From patchwork Wed May 4 15:25:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12838086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96BB5C433EF for ; Wed, 4 May 2022 15:26:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343761AbiEDP3l (ORCPT ); Wed, 4 May 2022 11:29:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352581AbiEDP3I (ORCPT ); Wed, 4 May 2022 11:29:08 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84F54DEC9 for ; Wed, 4 May 2022 08:25:24 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id n126-20020a1c2784000000b0038e8af3e788so1048444wmn.1 for ; Wed, 04 May 2022 08:25:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=v2bVxgiXyAJb5mJiV2jLmDxYyCHvoUPxWAqoNPpKzrk=; b=PDjNsLGwEeGiJ4QuVVgYGVssIcLv4sY0Fufo76eqKjEhnAKto9BVuabPY41r0litoj bmWkkVYeU+Uzzi62N3PB5ksCBA+i45+VwWlv+cWNjlT8CvGQ/fd3yyfLsB1S5SJQ1/od dY9wXzVclR58SEnOFa0fT6V5uw0YLbs1tr5XY4ha9N+Kr5sISSaI75sFH5uwFDX8wuuw RE4ylWvKYzFOLQfYhDx50wH3+asJrVyZPYGiLBWU5AjkC9PPjCJkAuDJQNxGepHoHjCk GsQ3SX/fSLyUgN2hmxD0gO5kA0DisnDgvNCD3dQbhT459Jecf/CqkSbmz+oPSA/RcNST 5LgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=v2bVxgiXyAJb5mJiV2jLmDxYyCHvoUPxWAqoNPpKzrk=; b=GupXNOE4AeW0d2KQMo7KaS3PZ8MHCi7qfVK6xFazxfXtQcz31rv5OHtsfWoZB+TZRA bp8FSqs+Lkn3JkQNOpyKulxv7vPslvsH7h8JLD/BQQ/OBOpsx2ZyPlmNH6cgs6xyfLkp dQPnJB3Nn3JE5IDnhTMSsL/2StB8LBu8kcxUTOMFTZ567trjep9W90fiLl3H+lW6NvXW Vg/4xgCIXbkF/yH7OmLbWxVlWI9sJ4fD+r96l73q9pZcvdV4Qwjk6q8X/MZKL5bFkKic yGbFEBXVr71eThM6mGWDMJ5OFoHCWX+E2aMJloZHVsRxconoVXaZBYdYIjLBG9gWqi2e 7arA== X-Gm-Message-State: AOAM532jtsO7aoyqYBjNpfeEjqacK3IV4fz9WrImZNBdySfLBTQtt2S/ gkP3YkSLcp5L0xRJriop74N+DyeroEk= X-Google-Smtp-Source: ABdhPJzksD2VdGzcWxaTI8+MqOoLfngaIeGYK6HRBfXtdABiJXHsJblKTYK6FP9dy+odXfomCQZO+A== X-Received: by 2002:a7b:c088:0:b0:394:451e:34a0 with SMTP id r8-20020a7bc088000000b00394451e34a0mr16397wmh.10.1651677922347; Wed, 04 May 2022 08:25:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m6-20020adfc586000000b0020c5253d908sm12636711wrg.84.2022.05.04.08.25.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:21 -0700 (PDT) Message-Id: <45662cf582ab7c8b1c32f55c9a34f4d73a28b71d.1651677919.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 04 May 2022 15:25:13 +0000 Subject: [PATCH v3 1/7] archive: optionally add "virtual" files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin With the `--add-file-with-content=:` option, `git archive` now supports use cases where relatively trivial files need to be added that do not exist on disk. This will allow us to generate `.zip` files with generated content, without having to add said content to the object database and without having to write it out to disk. Signed-off-by: Johannes Schindelin --- Documentation/git-archive.txt | 11 ++++++++ archive.c | 51 +++++++++++++++++++++++++++++------ t/t5003-archive-zip.sh | 12 +++++++++ 3 files changed, 66 insertions(+), 8 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index bc4e76a7834..a0edc9167b2 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -61,6 +61,17 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . +--add-file-with-content=::: + Add the specified contents to the archive. Can be repeated to add + multiple files. The path of the file in the archive is built + by concatenating the value for `--prefix` (if any) and the + basename of . ++ +The `` cannot contain any colon, the file mode is limited to +a regular file, and the option may be subject to platform-dependent +command-line limits. For non-trivial cases, write an untracked file +and use `--add-file` instead. + --worktree-attributes:: Look for attributes in .gitattributes files in the working tree as well (see <>). diff --git a/archive.c b/archive.c index a3bbb091256..d798624cd5f 100644 --- a/archive.c +++ b/archive.c @@ -263,6 +263,7 @@ static int queue_or_write_archive_entry(const struct object_id *oid, struct extra_file_info { char *base; struct stat stat; + void *content; }; int write_archive_entries(struct archiver_args *args, @@ -337,7 +338,13 @@ int write_archive_entries(struct archiver_args *args, strbuf_addstr(&path_in_archive, basename(path)); strbuf_reset(&content); - if (strbuf_read_file(&content, path, info->stat.st_size) < 0) + if (info->content) + err = write_entry(args, &fake_oid, path_in_archive.buf, + path_in_archive.len, + info->stat.st_mode, + info->content, info->stat.st_size); + else if (strbuf_read_file(&content, path, + info->stat.st_size) < 0) err = error_errno(_("could not read '%s'"), path); else err = write_entry(args, &fake_oid, path_in_archive.buf, @@ -493,6 +500,7 @@ static void extra_file_info_clear(void *util, const char *str) { struct extra_file_info *info = util; free(info->base); + free(info->content); free(info); } @@ -514,14 +522,38 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) if (!arg) return -1; - path = prefix_filename(args->prefix, arg); - item = string_list_append_nodup(&args->extra_files, path); - item->util = info = xmalloc(sizeof(*info)); + info = xmalloc(sizeof(*info)); info->base = xstrdup_or_null(base); - if (stat(path, &info->stat)) - die(_("File not found: %s"), path); - if (!S_ISREG(info->stat.st_mode)) - die(_("Not a regular file: %s"), path); + + if (!strcmp(opt->long_name, "add-file")) { + path = prefix_filename(args->prefix, arg); + if (stat(path, &info->stat)) + die(_("File not found: %s"), path); + if (!S_ISREG(info->stat.st_mode)) + die(_("Not a regular file: %s"), path); + info->content = NULL; /* read the file later */ + } else { + const char *colon = strchr(arg, ':'); + char *p; + + if (!colon) + die(_("missing colon: '%s'"), arg); + + p = xstrndup(arg, colon - arg); + if (!args->prefix) + path = p; + else { + path = prefix_filename(args->prefix, p); + free(p); + } + memset(&info->stat, 0, sizeof(info->stat)); + info->stat.st_mode = S_IFREG | 0644; + info->content = xstrdup(colon + 1); + info->stat.st_size = strlen(info->content); + } + item = string_list_append_nodup(&args->extra_files, path); + item->util = info; + return 0; } @@ -554,6 +586,9 @@ static int parse_archive_args(int argc, const char **argv, { OPTION_CALLBACK, 0, "add-file", args, N_("file"), N_("add untracked file to archive"), 0, add_file_cb, (intptr_t)&base }, + { OPTION_CALLBACK, 0, "add-file-with-content", args, + N_("path:content"), N_("add untracked file to archive"), 0, + add_file_cb, (intptr_t)&base }, OPT_STRING('o', "output", &output, N_("file"), N_("write the archive to this file")), OPT_BOOL(0, "worktree-attributes", &worktree_attributes, diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index 1e6d18b140e..8ff1257f1a0 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -206,6 +206,18 @@ test_expect_success 'git archive --format=zip --add-file' ' check_zip with_untracked check_added with_untracked untracked untracked +test_expect_success UNZIP 'git archive --format=zip --add-file-with-content' ' + git archive --format=zip >with_file_with_content.zip \ + --add-file-with-content=hello:world $EMPTY_TREE && + test_when_finished "rm -rf tmp-unpack" && + mkdir tmp-unpack && ( + cd tmp-unpack && + "$GIT_UNZIP" ../with_file_with_content.zip && + test_path_is_file hello && + test world = $(cat hello) + ) +' + test_expect_success 'git archive --format=zip --add-file twice' ' echo untracked >untracked && git archive --format=zip --prefix=one/ --add-file=untracked \ From patchwork Wed May 4 15:25:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12838085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC2A5C433FE for ; Wed, 4 May 2022 15:26:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352395AbiEDP3j (ORCPT ); Wed, 4 May 2022 11:29:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352587AbiEDP3I (ORCPT ); Wed, 4 May 2022 11:29:08 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 875137641 for ; Wed, 4 May 2022 08:25:25 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id u3so2524055wrg.3 for ; Wed, 04 May 2022 08:25:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=kk09cc6aH5JwrraG91yckuDsGytEkoSbAAZiDDO1h4A=; b=o2Frpi+eUZOatKZTUZ2ub7IDtc+3SR6kZMlR9gWBazaJK3ZnIstZS1kcWmsGAxO45N 6cXfZkqPIdXZbC2vX6EH+HlNZqjBvhFrpDVvx10Xv1h02kKJY0ktWDwjkJ/uV9B4TDkQ ullftKcG5qSMiGy/pjYZpBW/fo3TYRfpyhyXxSc8c5ALqt12FQ/sYsCAQ5aUHsbu4cr3 77m8VPJKW6qDKzMRtpiDGeZYJbSjPZ737H+IF+ffM2LKVZ/d0kXQfFQvnD2Hru+MFTli Ql1k7M+lEmUdC1up8wplPDjFPhtcniRCsCUfL06Xp7XZuRxwQN0xxSg0oCVG0jIH/xs9 JjdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=kk09cc6aH5JwrraG91yckuDsGytEkoSbAAZiDDO1h4A=; b=yegLdz0IwiEqRCKU0N5oKu/xFO5/dYODeU9jUEASYvUyByxNFot7yMn1yRa/G+zL0j U3qI8Yf1qUFsXcnAJvqXlZ2OpvAx8xfNR1fgozMwQIZ11hadChfqEFKezZemP4BQWaq3 rSRALlWJrWe2JwSNpj6jdlF3qxH98mf7svnj4tL5iTEa6CCyaBnEBJuabvUaFlrBXpjw SK9UrZQhCSh2/0jixDnsMpCerAdA0SF385C66a5QCsh7BHM5Sv8t7d9Nc5E6ijS7Ppps Iqp7Dkl+baoSqfjtQrI3R+ow6xnTKG+yHL8eCYbAKXnoM2jwwOMNoaPta9MEcGpT7vx8 L2nQ== X-Gm-Message-State: AOAM531AZDvbbxnGvntoqX6cfZd0cfJyylG+MnngnmnDRDa+PHF/qrkN WiOHOaItu74KSL9eYCP2gD9UpVONn1g= X-Google-Smtp-Source: ABdhPJw6sLov9y/dZy6b5/tnNSp64aQR+fyPm5pY3X2wEIZZCobgSWV5KjfnhYZliUXBI69d1UEU1A== X-Received: by 2002:a05:6000:1844:b0:20c:566b:ddab with SMTP id c4-20020a056000184400b0020c566bddabmr15408987wri.706.1651677923759; Wed, 04 May 2022 08:25:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k12-20020adfc70c000000b0020c5253d8fcsm14945276wrg.72.2022.05.04.08.25.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:23 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 04 May 2022 15:25:14 +0000 Subject: [PATCH v3 2/7] archive --add-file-with-contents: allow paths containing colons Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin By allowing the path to be enclosed in double-quotes, we can avoid the limitation that paths cannot contain colons. Signed-off-by: Johannes Schindelin --- Documentation/git-archive.txt | 13 +++++++++---- archive.c | 34 +++++++++++++++++++++++++++++----- t/t5003-archive-zip.sh | 8 ++++++++ 3 files changed, 46 insertions(+), 9 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index a0edc9167b2..1789ce4c232 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -67,10 +67,15 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . + -The `` cannot contain any colon, the file mode is limited to -a regular file, and the option may be subject to platform-dependent -command-line limits. For non-trivial cases, write an untracked file -and use `--add-file` instead. +The `` argument can start and end with a literal double-quote +character. In this case, the backslash is interpreted as escape +character. The path must be quoted if it contains a colon, to avoid +the colon from being misinterpreted as the separator between the +path and the contents. ++ +The file mode is limited to a regular file, and the option may be +subject to platform-dependent command-line limits. For non-trivial +cases, write an untracked file and use `--add-file` instead. --worktree-attributes:: Look for attributes in .gitattributes files in the working tree diff --git a/archive.c b/archive.c index d798624cd5f..3b751027143 100644 --- a/archive.c +++ b/archive.c @@ -533,13 +533,37 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) die(_("Not a regular file: %s"), path); info->content = NULL; /* read the file later */ } else { - const char *colon = strchr(arg, ':'); char *p; - if (!colon) - die(_("missing colon: '%s'"), arg); + if (*arg != '"') { + const char *colon = strchr(arg, ':'); + + if (!colon) + die(_("missing colon: '%s'"), arg); + p = xstrndup(arg, colon - arg); + arg = colon + 1; + } else { + struct strbuf buf = STRBUF_INIT; + const char *orig = arg; + + for (;;) { + if (!*(++arg)) + die(_("unclosed quote: '%s'"), orig); + if (*arg == '"') + break; + if (*arg == '\\' && *(++arg) == '\0') + die(_("trailing backslash: '%s"), orig); + else + strbuf_addch(&buf, *arg); + } + + if (*(++arg) != ':') + die(_("missing colon: '%s'"), orig); + + p = strbuf_detach(&buf, NULL); + arg++; + } - p = xstrndup(arg, colon - arg); if (!args->prefix) path = p; else { @@ -548,7 +572,7 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) } memset(&info->stat, 0, sizeof(info->stat)); info->stat.st_mode = S_IFREG | 0644; - info->content = xstrdup(colon + 1); + info->content = xstrdup(arg); info->stat.st_size = strlen(info->content); } item = string_list_append_nodup(&args->extra_files, path); diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index 8ff1257f1a0..5b8bbfc2692 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -207,13 +207,21 @@ check_zip with_untracked check_added with_untracked untracked untracked test_expect_success UNZIP 'git archive --format=zip --add-file-with-content' ' + if test_have_prereq FUNNYNAMES + then + QUOTED=quoted:colon + else + QUOTED=quoted + fi && git archive --format=zip >with_file_with_content.zip \ + --add-file-with-content=\"$QUOTED\": \ --add-file-with-content=hello:world $EMPTY_TREE && test_when_finished "rm -rf tmp-unpack" && mkdir tmp-unpack && ( cd tmp-unpack && "$GIT_UNZIP" ../with_file_with_content.zip && test_path_is_file hello && + test_path_is_file $QUOTED && test world = $(cat hello) ) ' From patchwork Wed May 4 15:25:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12838087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FC24C433EF for ; Wed, 4 May 2022 15:26:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352451AbiEDP3p (ORCPT ); Wed, 4 May 2022 11:29:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352594AbiEDP3I (ORCPT ); Wed, 4 May 2022 11:29:08 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7635165AF for ; Wed, 4 May 2022 08:25:26 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id i5so2488439wrc.13 for ; Wed, 04 May 2022 08:25:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=aGPDfIL7ucaH2apD8/TG7UTxME18NXFoiFTgp6N8cwnRny8xT2yEfUl/weztLE0O3f Yhg2tyckQDJHoKAqi7ByV/yEY7fbpTX9TdH9ea631I1+FnY6CR+RZjJ9Qu97p6ZcLxrK yyHkZ2sOicUdAbVr+7ztTSCZnqPa4loHExqvkN7jJm7A46ZuFe5/GXsj1KI637xTfeMv u0MaqpZh2ocW0/5UZPu7r90fs1ajm2zQS+Qc9qGq2ABkFKzuS2p5LgLe3CKaNULnOj82 xrY09bRL6qqKjYdEFLob8nDhVucOVB3ChrpLdSqs6Yt/uDMCkJ+lOXJ2g/1TwRULQaiK v/og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=UM0ODh5KbF3Y9W9svSOhxq5t3Q+0saxkHL4yenzydiNTyv/n7o1qskA+iHgvjlFqTv YTpyRLdGiPxFmaq+hzUwMhQwyvmir6Kg8ae0WZ3Ak3ucpzuoEqYEh/OtQVK9ik9nfOhL R/1RaESUdIEYzXrV3HN/vs5QUR6Dqy+wGP4rDLcD486kKc7oaBSXN3N++mlRvGcg5fc4 VcdlLllU0QHpUw19/G+R7QsdQHO33vxexts9p4l0iZcDr1qs/uLvcp0kdGqA+EnTjePb r5ki3T72pGaA23QM8UbHZH6YAzd14tss6bK6J0F6jznUbXSx5z76du9Srgtje3atsFUO +t1g== X-Gm-Message-State: AOAM530XgtEYce/iculpdTA4Z6j2rSyMlLeunsoEPziUD0/AYyinOpH3 gRkQeOu3mFSVXUI324KpOOxVwwMk/e4= X-Google-Smtp-Source: ABdhPJwFIsNWDtqSPBNC3NBaUGh+BkL89our2A4yv1VOZ5B/gCEIJaho9cS/JfBU0/9eKNyC63kBVA== X-Received: by 2002:a05:6000:170a:b0:20c:5e04:af27 with SMTP id n10-20020a056000170a00b0020c5e04af27mr12638601wrc.547.1651677925041; Wed, 04 May 2022 08:25:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t8-20020a7bc3c8000000b0038eba413181sm4057993wmj.1.2022.05.04.08.25.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:24 -0700 (PDT) Message-Id: <5a3eeb5540943279d1677c1338df86b58239abc1.1651677919.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 04 May 2022 15:25:15 +0000 Subject: [PATCH v3 3/7] scalar: validate the optional enlistment argument Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The `scalar` command needs a Scalar enlistment for many subcommands, and looks in the current directory for such an enlistment (traversing the parent directories until it finds one). These is subcommands can also be called with an optional argument specifying the enlistment. Here, too, we traverse parent directories as needed, until we find an enlistment. However, if the specified directory does not even exist, or is not a directory, we should stop right there, with an error message. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 6 ++++-- contrib/scalar/t/t9099-scalar.sh | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 1ce9c2b00e8..00dcd4b50ef 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -43,9 +43,11 @@ static void setup_enlistment_directory(int argc, const char **argv, usage_with_options(usagestr, options); /* find the worktree, determine its corresponding root */ - if (argc == 1) + if (argc == 1) { strbuf_add_absolute_path(&path, argv[0]); - else if (strbuf_getcwd(&path) < 0) + if (!is_directory(path.buf)) + die(_("'%s' does not exist"), path.buf); + } else if (strbuf_getcwd(&path) < 0) die(_("need a working directory")); strbuf_trim_trailing_dir_sep(&path); diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 2e1502ad45e..9d83fdf25e8 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -85,4 +85,9 @@ test_expect_success 'scalar delete with enlistment' ' test_path_is_missing cloned ' +test_expect_success '`scalar [...] ` errors out when dir is missing' ' + ! scalar run config cloned 2>err && + grep "cloned. does not exist" err +' + test_done From patchwork Wed May 4 15:25:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12838091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BD47C433F5 for ; Wed, 4 May 2022 15:26:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352486AbiEDP3x (ORCPT ); Wed, 4 May 2022 11:29:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352595AbiEDP3I (ORCPT ); Wed, 4 May 2022 11:29:08 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3820B13DD9 for ; Wed, 4 May 2022 08:25:28 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id i5so2488542wrc.13 for ; Wed, 04 May 2022 08:25:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jBMFPmkZenZY6DB8DFVMWpRAm6yiTvHdwZ3nPs9kwFg=; b=LRf4T4S1EGEgwbq5cy/QuEf25SSxS8HlVfnkGPlVcEXv13lnPZ7O9hvJ2m3hhPGfVP 5ijxuocfZ1r6Ri5XlHoL7PdqBoJmCyo2/4HKDTQ1R8QN7peMYpoc25G5ouUgMMqhm5J2 D/YyOunDk7nPuuB7gckUti7PC1aaDvxoCEB7tgOyCOzIu5c+05eGJm8P2/FOvPBh7aoy INL/qF2v3WgtbLmeqtcczBe2ZlYLZgGUXtTsnaflmggkKZZniUAT4k5gPghtvMSH/v9G nQh4lHHCohtUr/F8LIMgXSPEVj1czrB2MOuyOQ/hjl6vZOdSBWqsc64o0wsNUmNtUBfJ ohEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jBMFPmkZenZY6DB8DFVMWpRAm6yiTvHdwZ3nPs9kwFg=; b=jRdni2Z0PNw/0A5ksOVT2QAWyll5d3jUAnjA1vCwbo9SZXriA8MUy7z4vNICqx2c7v TZ0barpS9OHe6dU3FAWDJJRqIKKpgvKVeTBPP6sm/ZEcCAdNH0MAalrZtH0kHHTE+ZHD dNd3OK+dv/ExiqgEKfooITBqJoL8Ae6kbVcXUcuGlze0gj7esDfhY6NJSaVpnI/UcyQB Np+ZsZE4I3Na8sOR0bCu3yKkJqjTlC2KqQo5nrOUd9EXfeYzMPwwmqqVi/0ko82X7IyP fS2bELmU9OdPCPoBHsAlA4IjTbpFo6q5tcaS231pnAlM/fxwFrKE2/xHSxWEY/qbREiX POEQ== X-Gm-Message-State: AOAM5310xZcsUQBkCW8I0zLxv31vwzkHypmkpIZLRgYmh9Lt+9fKa9ly PnANhNe7+yEXRdo2ceTKu/bi7bjUZ4s= X-Google-Smtp-Source: ABdhPJwaLrIQivS2mpx2IIKklmTQU8piRmR+Pkl/7/Scm+w35VpclNcmnSX7Mmr7YeGBmeOf378vrg== X-Received: by 2002:adf:f4d0:0:b0:20a:ca54:1fcc with SMTP id h16-20020adff4d0000000b0020aca541fccmr17006988wrp.681.1651677926233; Wed, 04 May 2022 08:25:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n185-20020a1c27c2000000b003942a244ecdsm4275268wmn.18.2022.05.04.08.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:25 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 04 May 2022 15:25:16 +0000 Subject: [PATCH v3 4/7] Implement `scalar diagnose` Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin Over the course of Scalar's development, it became obvious that there is a need for a command that can gather all kinds of useful information that can help identify the most typical problems with large worktrees/repositories. The `diagnose` command is the culmination of this hard-won knowledge: it gathers the installed hooks, the config, a couple statistics describing the data shape, among other pieces of information, and then wraps everything up in a tidy, neat `.zip` archive. Note: originally, Scalar was implemented in C# using the .NET API, where we had the luxury of a comprehensive standard library that includes basic functionality such as writing a `.zip` file. In the C version, we lack such a commodity. Rather than introducing a dependency on, say, libzip, we slightly abuse Git's `archive` machinery: we write out a `.zip` of the empty try, augmented by a couple files that are added via the `--add-file*` options. We are careful trying not to modify the current repository in any way lest the very circumstances that required `scalar diagnose` to be run are changed by the `diagnose` run itself. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 141 +++++++++++++++++++++++++++++++ contrib/scalar/scalar.txt | 12 +++ contrib/scalar/t/t9099-scalar.sh | 14 +++ 3 files changed, 167 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 00dcd4b50ef..a290e52e1d2 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -11,6 +11,7 @@ #include "dir.h" #include "packfile.h" #include "help.h" +#include "archive.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -261,6 +262,44 @@ static int unregister_dir(void) return res; } +static int add_directory_to_archiver(struct strvec *archiver_args, + const char *path, int recurse) +{ + int at_root = !*path; + DIR *dir = opendir(at_root ? "." : path); + struct dirent *e; + struct strbuf buf = STRBUF_INIT; + size_t len; + int res = 0; + + if (!dir) + return error(_("could not open directory '%s'"), path); + + if (!at_root) + strbuf_addf(&buf, "%s/", path); + len = buf.len; + strvec_pushf(archiver_args, "--prefix=%s", buf.buf); + + while (!res && (e = readdir(dir))) { + if (!strcmp(".", e->d_name) || !strcmp("..", e->d_name)) + continue; + + strbuf_setlen(&buf, len); + strbuf_addstr(&buf, e->d_name); + + if (e->d_type == DT_REG) + strvec_pushf(archiver_args, "--add-file=%s", buf.buf); + else if (e->d_type != DT_DIR) + res = -1; + else if (recurse) + add_directory_to_archiver(archiver_args, buf.buf, recurse); + } + + closedir(dir); + strbuf_release(&buf); + return res; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -501,6 +540,107 @@ cleanup: return res; } +static int cmd_diagnose(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar diagnose []"), + NULL + }; + struct strbuf zip_path = STRBUF_INIT; + struct strvec archiver_args = STRVEC_INIT; + char **argv_copy = NULL; + int stdout_fd = -1, archiver_fd = -1; + time_t now = time(NULL); + struct tm tm; + struct strbuf path = STRBUF_INIT, buf = STRBUF_INIT; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, &zip_path); + + strbuf_addstr(&zip_path, "/.scalarDiagnostics/scalar_"); + strbuf_addftime(&zip_path, + "%Y%m%d_%H%M%S", localtime_r(&now, &tm), 0, 0); + strbuf_addstr(&zip_path, ".zip"); + switch (safe_create_leading_directories(zip_path.buf)) { + case SCLD_EXISTS: + case SCLD_OK: + break; + default: + error_errno(_("could not create directory for '%s'"), + zip_path.buf); + goto diagnose_cleanup; + } + stdout_fd = dup(1); + if (stdout_fd < 0) { + res = error_errno(_("could not duplicate stdout")); + goto diagnose_cleanup; + } + + archiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666); + if (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) { + res = error_errno(_("could not redirect output")); + goto diagnose_cleanup; + } + + init_zip_archiver(); + strvec_pushl(&archiver_args, "scalar-diagnose", "--format=zip", NULL); + + strbuf_reset(&buf); + strbuf_addstr(&buf, "Collecting diagnostic info\n\n"); + get_version_info(&buf, 1); + + strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + write_or_die(stdout_fd, buf.buf, buf.len); + strvec_pushf(&archiver_args, + "--add-file-with-content=diagnostics.log:%.*s", + (int)buf.len, buf.buf); + + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/logs", 1)) || + (res = add_directory_to_archiver(&archiver_args, ".git/objects/info", 0))) + goto diagnose_cleanup; + + strvec_pushl(&archiver_args, "--prefix=", + oid_to_hex(the_hash_algo->empty_tree), "--", NULL); + + /* `write_archive()` modifies the `argv` passed to it. Let it. */ + argv_copy = xmemdupz(archiver_args.v, + sizeof(char *) * archiver_args.nr); + res = write_archive(archiver_args.nr, (const char **)argv_copy, NULL, + the_repository, NULL, 0); + if (res) { + error(_("failed to write archive")); + goto diagnose_cleanup; + } + + if (!res) + fprintf(stderr, "\n" + "Diagnostics complete.\n" + "All of the gathered info is captured in '%s'\n", + zip_path.buf); + +diagnose_cleanup: + if (archiver_fd >= 0) { + close(1); + dup2(stdout_fd, 1); + } + free(argv_copy); + strvec_clear(&archiver_args); + strbuf_release(&zip_path); + strbuf_release(&path); + strbuf_release(&buf); + + return res; +} + static int cmd_list(int argc, const char **argv) { if (argc != 1) @@ -802,6 +942,7 @@ static struct { { "reconfigure", cmd_reconfigure }, { "delete", cmd_delete }, { "version", cmd_version }, + { "diagnose", cmd_diagnose }, { NULL, NULL}, }; diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt index f416d637289..22583fe046e 100644 --- a/contrib/scalar/scalar.txt +++ b/contrib/scalar/scalar.txt @@ -14,6 +14,7 @@ scalar register [] scalar unregister [] scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] scalar reconfigure [ --all | ] +scalar diagnose [] scalar delete DESCRIPTION @@ -129,6 +130,17 @@ reconfigure the enlistment. With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. Use this option after each Scalar upgrade. +Diagnose +~~~~~~~~ + +diagnose []:: + When reporting issues with Scalar, it is often helpful to provide the + information gathered by this command, including logs and certain + statistics describing the data shape of the current enlistment. ++ +The output of this command is a `.zip` file that is written into +a directory adjacent to the worktree in the `src` directory. + Delete ~~~~~~ diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 9d83fdf25e8..bbd07a44426 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -90,4 +90,18 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' grep "cloned. does not exist" err ' +SQ="'" +test_expect_success UNZIP 'scalar diagnose' ' + scalar clone "file://$(pwd)" cloned --single-branch && + scalar diagnose cloned >out && + sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && + zip_path=$(cat zip_path) && + test -n "$zip_path" && + unzip -v "$zip_path" && + folder=${zip_path%.zip} && + test_path_is_missing "$folder" && + unzip -p "$zip_path" diagnostics.log >out && + test_file_not_empty out +' + test_done From patchwork Wed May 4 15:25:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12838088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C7F2C433F5 for ; Wed, 4 May 2022 15:26:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352457AbiEDP3r (ORCPT ); Wed, 4 May 2022 11:29:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352616AbiEDP3J (ORCPT ); Wed, 4 May 2022 11:29:09 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5ED2ECB for ; Wed, 4 May 2022 08:25:32 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id x18so2552712wrc.0 for ; Wed, 04 May 2022 08:25:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=BOwgeAE/S9vapjUog7cLOLjx8iiqWE8Gkn3eKwDI6+k=; b=BaYqYe/t8L2ihULODbUD/6yJfbzxj8yq2BlwMBUmU3BzC+48poK96BihjuLxCfMAHK PV5tvrfogGgQPGdQJzsNAjgjonkkMu9gUQoSIBqg82RlQDwJEtQfyoH85tfD7qjp2WV7 06gJm3cjdYGDMjmDKA3UJPHB6+icCoAM21rCBdfNwT28LvA2Uc65Lz05Zcq8LMNYlhvw SQI5lpE9oDHHHOeAgVmrmPgz+REb/4Jzhqq5ErLkGyFxeXnBbPxdJnO4Z5QefXaZiLBt 3Kl5A2Oeaht/a6b5Ui/VqSSbaJxQog3Fhx2WoSn/jneiTF5L+diLNXuRboVfGIiKorBZ hMBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=BOwgeAE/S9vapjUog7cLOLjx8iiqWE8Gkn3eKwDI6+k=; b=LDKSi+D+1M4YHjaGsZqN17XBPJ7YFHr7pKv+J+Gm4NsCSc/tyB6qrorVj7GfnArbDs CQQHVqbrttHe1QnNOjMfZuFbzfH915PeRKw2w3U/ZVmtvP3vozjkLYmPREdSYfhGVB26 deAupS/25mZ+ZdSvVX3Y2l0+9XBfLdXOT3c2ihr/DiDIK0g++h8stTAFr71P2UI0JF6w 9sj7u6IOL0dfTOxAIayl9B336B9lVrig81KCRtu5ikcwz5+79Rum0RcNCdfSqcBV99Gd QYS0sRI95Xh4rgFGyHdlLYEhPgQEs7uY8ZVV9rVdtjZIdDzEnP/lS2W4bGbsedR+gq9x fGtg== X-Gm-Message-State: AOAM530i6ibN0Y6bCWyQlLpLURLSNr8DgwiXUV+ATPjSvN0dleSzOdxk w3WXloj5uMJzen0VQNxD81IBt30SUS0= X-Google-Smtp-Source: ABdhPJwHqhFx5cMBXIx+JcZGg5AR0smvVsJsM5mAqUvSPnhFydCopPnUz0M3ASPWshIi8dhHOiufOQ== X-Received: by 2002:adf:e241:0:b0:20a:d762:c4 with SMTP id bl1-20020adfe241000000b0020ad76200c4mr17599132wrb.371.1651677927681; Wed, 04 May 2022 08:25:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s6-20020a1cf206000000b003942a244f3fsm4032392wmc.24.2022.05.04.08.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:27 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 04 May 2022 15:25:17 +0000 Subject: [PATCH v3 5/7] scalar diagnose: include disk space information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When analyzing problems with large worktrees/repositories, it is useful to know how close to a "full disk" situation Scalar/Git operates. Let's include this information. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 53 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 1 + 2 files changed, 54 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index a290e52e1d2..df44902c909 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -300,6 +300,58 @@ static int add_directory_to_archiver(struct strvec *archiver_args, return res; } +#ifndef WIN32 +#include +#endif + +static int get_disk_info(struct strbuf *out) +{ +#ifdef WIN32 + struct strbuf buf = STRBUF_INIT; + char volume_name[MAX_PATH], fs_name[MAX_PATH]; + DWORD serial_number, component_length, flags; + ULARGE_INTEGER avail2caller, total, avail; + + strbuf_realpath(&buf, ".", 1); + if (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) { + error(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_setlen(&buf, offset_1st_component(buf.buf)); + if (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name), + &serial_number, &component_length, &flags, + fs_name, sizeof(fs_name))) { + error(_("could not get info for '%s'"), buf.buf); + strbuf_release(&buf); + return -1; + } + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, avail2caller.QuadPart); + strbuf_addch(out, '\n'); + strbuf_release(&buf); +#else + struct strbuf buf = STRBUF_INIT; + struct statvfs stat; + + strbuf_realpath(&buf, ".", 1); + if (statvfs(buf.buf, &stat) < 0) { + error_errno(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail)); + strbuf_addf(out, " (mount flags 0x%lx)\n", stat.f_flag); + strbuf_release(&buf); +#endif + return 0; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -596,6 +648,7 @@ static int cmd_diagnose(int argc, const char **argv) get_version_info(&buf, 1); strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + get_disk_info(&buf); write_or_die(stdout_fd, buf.buf, buf.len); strvec_pushf(&archiver_args, "--add-file-with-content=diagnostics.log:%.*s", diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index bbd07a44426..f3d037823c8 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -94,6 +94,7 @@ SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && scalar diagnose cloned >out && + grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && zip_path=$(cat zip_path) && test -n "$zip_path" && From patchwork Wed May 4 15:25:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12838090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 710E5C433EF for ; Wed, 4 May 2022 15:26:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352483AbiEDP3v (ORCPT ); Wed, 4 May 2022 11:29:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352612AbiEDP3I (ORCPT ); Wed, 4 May 2022 11:29:08 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1ABF193C6 for ; Wed, 4 May 2022 08:25:30 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id e24so2506426wrc.9 for ; Wed, 04 May 2022 08:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=VhRAkQGsOMMZYApuyWPvng0WDXvqAdGMjGolWIX9Q10=; b=JeOJwkRVr763YWikAA143DiKXhO4u7Fee4vi05SmMaJ3nJDM7JlxIQ+wWlofvHBMOo OC1yz9Q99/HXAGIOb0Jahb0rxrczoSlNNUCh/E1XQZaDGpYEpa1fcfU9+9dpmE08e080 qzZ/FgUCfky+VyhdeMmtB7pILF5Ex4nRdkt2/9LRrl1fZ4ehEBS+c1qkEjn+LLtPqPO6 Lgm3YcyHMFrkujj3vwt/MEChe93TRR7+mDJIORBB6v4ieoT0tV3ejKYZ7TuKbraVczwo wyg9uBtei2sxB1bgX+Golg64g9u8OBAhkO4869icHuOR159p6xzq4v+jA1kEl7dgujim +mOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=VhRAkQGsOMMZYApuyWPvng0WDXvqAdGMjGolWIX9Q10=; b=E4kWu7xV5StrEMbUNEc8XlPaKRi86nA+2kn9wGW+PaZbzjMcS5ND3timkc0OmyM2cS HygHRHCXXZXhI7jOXDMfufXC41xz6qT1np8QdmOLlPEXQsLc7naS2/2vQyh9iGbmjZC2 luP6VXyuGqojdGdNsyjkXJ4yNBO6XUSUR00J37nh76fTKBB5NqrOqrAmOfeDhoFTbLZI FuWAdWedQ4eCvwtsSeG8PX+J+/ZjCsI+SYdxBCKckCORhF98bhyiR3vy+o6eZJdFRmu8 mBxsShgBxzoIOwUQ8Ay9fIf5SbC32Wq9IKkYSUHy4yY83Zu45WsjcKidKKRlYlAZiOIF tYMA== X-Gm-Message-State: AOAM530ML9cHTWja725OFosJxpuEO1ixyAKUT9ldzu0pWMeg1F0hpYv/ ORMskBtlGLvy44PgP+c1wxW6EOZps1M= X-Google-Smtp-Source: ABdhPJx8E79gGep2tQiy7ZzPIkSvg2Cp32R5vWs07muoC7JhQO0aqrj+pIEn8bID20QgLBhFVrhJog== X-Received: by 2002:a05:6000:1861:b0:20c:8125:ee0a with SMTP id d1-20020a056000186100b0020c8125ee0amr2663674wri.713.1651677929105; Wed, 04 May 2022 08:25:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l1-20020a1ced01000000b003942a244f4fsm4010670wmh.40.2022.05.04.08.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:28 -0700 (PDT) Message-Id: <32aaad7cce1436ee1c6a7607cc8656b1275aac2f.1651677919.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 04 May 2022 15:25:18 +0000 Subject: [PATCH v3 6/7] scalar: teach `diagnose` to gather packfile info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham It's helpful to see if there are other crud files in the pack directory. Let's teach the `scalar diagnose` command to gather file size information about pack files. While at it, also enumerate the pack files in the alternate object directories, if any are registered. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 30 ++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 6 +++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index df44902c909..9adde8cf4b9 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -12,6 +12,7 @@ #include "packfile.h" #include "help.h" #include "archive.h" +#include "object-store.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -592,6 +593,29 @@ cleanup: return res; } +static void dir_file_stats_objects(const char *full_path, size_t full_path_len, + const char *file_name, void *data) +{ + struct strbuf *buf = data; + struct stat st; + + if (!stat(full_path, &st)) + strbuf_addf(buf, "%-70s %16" PRIuMAX "\n", file_name, + (uintmax_t)st.st_size); +} + +static int dir_file_stats(struct object_directory *object_dir, void *data) +{ + struct strbuf *buf = data; + + strbuf_addf(buf, "Contents of %s:\n", object_dir->path); + + for_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects, + data); + + return 0; +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -654,6 +678,12 @@ static int cmd_diagnose(int argc, const char **argv) "--add-file-with-content=diagnostics.log:%.*s", (int)buf.len, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-file-with-content=packs-local.txt:"); + dir_file_stats(the_repository->objects->odb, &buf); + foreach_alt_odb(dir_file_stats, &buf); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index f3d037823c8..e049221609d 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -93,6 +93,8 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && + git repack && + echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && scalar diagnose cloned >out && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -102,7 +104,9 @@ test_expect_success UNZIP 'scalar diagnose' ' folder=${zip_path%.zip} && test_path_is_missing "$folder" && unzip -p "$zip_path" diagnostics.log >out && - test_file_not_empty out + test_file_not_empty out && + unzip -p "$zip_path" packs-local.txt >out && + grep "$(pwd)/.git/objects" out ' test_done From patchwork Wed May 4 15:25:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12838089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EEAEC433FE for ; Wed, 4 May 2022 15:26:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352452AbiEDP3t (ORCPT ); Wed, 4 May 2022 11:29:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352613AbiEDP3I (ORCPT ); Wed, 4 May 2022 11:29:08 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14D5E2AC for ; Wed, 4 May 2022 08:25:32 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id l62-20020a1c2541000000b0038e4570af2fso1041713wml.5 for ; Wed, 04 May 2022 08:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cuVoRoGcLognUfUtAbrG/alAPfTx3uU2pG6PzEbPVe4=; b=WeTjerE77VuD7mjBEAX1NqLxE8Y+DxKS2LWQHeLQPhl84x/kEcP4kNr9dxedVhWePr QeINm9DIpx6lO6WYMQ6Re4HVm4A3kpZweUZjuMcUZzFlpJZ0FAHQCWKFctV/8B7fRtMe vWi6eMiugFvKDkHe7xo4d0Gi8lSzO+rYrF6QeNF5pP5/lk1Jpl55LlWse6RZzuaxyQ98 jd2epayhseafnNnlgHVGJ23FdI/sjdpiExVoBbrpTXfdiFEDQG9N4oO4Kv8paRZUjCn9 TgufBLW4+7cO1nmkKnE9iPbRWRx8sbZvAnqczWW8zL4E7ozXARqiSIPwYEtCMRF8XBOj oWTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cuVoRoGcLognUfUtAbrG/alAPfTx3uU2pG6PzEbPVe4=; b=Wbi75Cx4vOZjCAliYBDftZrFFibSi6gttGIUGWpPw3VM5y5hGSvTa9ZNy/yJ1Ka/k6 YW2dBc0lk6cspa9vJCUPQGw9WXyeN/cAkwLsbYsOldwqYPfBfKKnX44WF8CaKFlNh0QU 8K3NNUILlzp+Hh7U1pOMQnfrj0wO4qhWh9omRKZUehg2yNu4FxF5FdcRJUzFA+ZqJgqg d8buUpIoSLI5SpmXKTpqth5CfRRAVJMB2my0PkAKdoEAp1ux3OBfw8UUFki3WRoh1lKu 8VAn6DsYMrGBplPnVnBIPumw16PtPjKXUsW8oUqcYRO6qTOh46fVWAbFaotEOszWJGWt KX1g== X-Gm-Message-State: AOAM532RPhAs8eNO0/ByS+t9FGRi1tfkHPIpSCQMH/EXcDARki0XrmEI rx6AWr7aV1QFW3kqW1l6yoVG8r7Pl5I= X-Google-Smtp-Source: ABdhPJw4inmzqgvgFn58QTN/LO7SgbVHnV8jtVVovuKeRRwi4QpMQq2QSCDX3U+EMvj44UHfvzeYTw== X-Received: by 2002:a1c:f606:0:b0:394:1a21:f68c with SMTP id w6-20020a1cf606000000b003941a21f68cmr8249455wmc.131.1651677930232; Wed, 04 May 2022 08:25:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p26-20020adfa21a000000b0020c5253d8ecsm11766169wra.56.2022.05.04.08.25.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 May 2022 08:25:29 -0700 (PDT) Message-Id: <322932f0bb87db2e11fc7b8447c3fc0fe134ae9a.1651677919.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 04 May 2022 15:25:19 +0000 Subject: [PATCH v3 7/7] scalar: teach `diagnose` to gather loose objects information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham When operating at the scale that Scalar wants to support, certain data shapes are more likely to cause undesirable performance issues, such as large numbers of loose objects. By including statistics about this, `scalar diagnose` now makes it easier to identify such scenarios. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 59 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 5 ++- 2 files changed, 63 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 9adde8cf4b9..f2fe3858eca 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -616,6 +616,60 @@ static int dir_file_stats(struct object_directory *object_dir, void *data) return 0; } +static int count_files(char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count = 0; + + if (!dir) + return 0; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG) + count++; + + closedir(dir); + return count; +} + +static void loose_objs_stats(struct strbuf *buf, const char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count; + int total = 0; + unsigned char c; + struct strbuf count_path = STRBUF_INIT; + size_t base_path_len; + + if (!dir) + return; + + strbuf_addstr(buf, "Object directory stats for "); + strbuf_add_absolute_path(buf, path); + strbuf_addstr(buf, ":\n"); + + strbuf_add_absolute_path(&count_path, path); + strbuf_addch(&count_path, '/'); + base_path_len = count_path.len; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && + e->d_type == DT_DIR && strlen(e->d_name) == 2 && + !hex_to_bytes(&c, e->d_name, 1)) { + strbuf_setlen(&count_path, base_path_len); + strbuf_addstr(&count_path, e->d_name); + total += (count = count_files(count_path.buf)); + strbuf_addf(buf, "%s : %7d files\n", e->d_name, count); + } + + strbuf_addf(buf, "Total: %d loose objects", total); + + strbuf_release(&count_path); + closedir(dir); +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -684,6 +738,11 @@ static int cmd_diagnose(int argc, const char **argv) foreach_alt_odb(dir_file_stats, &buf); strvec_push(&archiver_args, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-file-with-content=objects-local.txt:"); + loose_objs_stats(&buf, ".git/objects"); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index e049221609d..9b4eedbb0aa 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -95,6 +95,7 @@ test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && + test_commit -C cloned/src loose && scalar diagnose cloned >out && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -106,7 +107,9 @@ test_expect_success UNZIP 'scalar diagnose' ' unzip -p "$zip_path" diagnostics.log >out && test_file_not_empty out && unzip -p "$zip_path" packs-local.txt >out && - grep "$(pwd)/.git/objects" out + grep "$(pwd)/.git/objects" out && + unzip -p "$zip_path" objects-local.txt >out && + grep "^Total: [1-9]" out ' test_done