From patchwork Wed Jan 26 08:41:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12724727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDA7CC2BA4C for ; Wed, 26 Jan 2022 08:41:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238545AbiAZIlx (ORCPT ); Wed, 26 Jan 2022 03:41:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230519AbiAZIlw (ORCPT ); Wed, 26 Jan 2022 03:41:52 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65855C06161C for ; Wed, 26 Jan 2022 00:41:52 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id c23so8704206wrb.5 for ; Wed, 26 Jan 2022 00:41:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=K0LgJSigfNyrn4WSwKovSk+APwo3VYPrw32hX33/Q40=; b=TyTWWhvV0zXsUyjzTnUELz8XqhtsfUgTlnzOOmb/fRNSe42YFxKKE/6xKD/WvGCryb 5ZmO9KrQRIWjYMAwtBFmECRLwZW6vGQ1bgrh/97gSAr9B5pUfb4pqWAn+ryD/GUfkMd2 tYMEp7NTv9sZErvLG4qF8xsz6rAP27YtEXovlc3SOoyjSCNxIf9dAA4iXRLGCpC/Bb6k 5I6qK1pepCrcsQMShUHv3cNv9qyrbKjHfpv81OlDET+jy6cFyva3fjabOxB1oDnGsON+ Xd+CO58rNSQ2c0S0h9TpZxbtcGRT85l4w53+F9LUH0j62cFKrPzy8+W1H8uq3j6t8PXF BRlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=K0LgJSigfNyrn4WSwKovSk+APwo3VYPrw32hX33/Q40=; b=6uY/7a42pOENCc1Z7lDb7bEwkY+/T/lHQE32VwKcoxN8vZPMUHcgEzclj2g2c7xECO DdB/qHINUv5mIiiaHizxa/dVj4I9eMj73ZqF2OaAAVX7EBuau/6x8V9uYQm1XXCKH5yW 3PmfGH+KQ0TIWdqwI+JU6fnFqau0gLIZc6r6kGPaMKB8aNgdNSw+A7Z1zGTVqjkkAgJU gaq7WtXMtTAoMg6Cj2rF5hFimOLCb9f8XND3HNEa5TyPcyHlDrkWOfjLRi2b3mpCHCi7 w8ilyXfTmVnk8iOzUKVDltEaRRbFWYLV1QcgaeNEOWQHbnUJy0oPSKOg7hEhIgaPyONT aqCg== X-Gm-Message-State: AOAM532PRw3Nc8/T1ByL1fIkHLIuFcgifBUaBJ8tM5vKTGgLGvkmQwLX CzdSLu3HTmdQCMSBY5M6la1IAilNUBc= X-Google-Smtp-Source: ABdhPJy58qr03mqCGQGBb7odEoceCGSpRmMIGrqCJbYVb0XFIFG2GaOOlJfZ/GL7ywl+Y71ZxdgDoQ== X-Received: by 2002:adf:b64f:: with SMTP id i15mr16780017wre.228.1643186510718; Wed, 26 Jan 2022 00:41:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t5sm19706564wrw.92.2022.01.26.00.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 00:41:50 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 26 Jan 2022 08:41:43 +0000 Subject: [PATCH 1/5] Implement `scalar diagnose` Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin Over the course of Scalar's development, it became obvious that there is a need for a command that can gather all kinds of useful information that can help identify the most typical problems with large worktrees/repositories. The `diagnose` command is the culmination of this hard-won knowledge: it gathers the installed hooks, the config, a couple statistics describing the data shape, among other pieces of information, and then wraps everything up in a tidy, neat `.zip` archive. Note: originally, Scalar was implemented in C# using the .NET API, where we had the luxury of a comprehensive standard library that includes basic functionality such as writing a `.zip` file. In the C version, we lack such a commodity. Rather than introducing a dependency on, say, libzip, we slightly abuse Git's `archive` command: Instead of writing the `.zip` file directly, we stage the file contents in a Git index of a temporary, bare repository, only to let `git archive` have at it, and finally removing the temporary repository. Also note: Due to the frequently-spawned `git hash-object` processes, this command is quite a bit slow on Windows. Should it turn out to be a big problem, the lack of a batch mode of the `hash-object` command could potentially be worked around via using `git fast-import` with a crafted `stdin`. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 170 +++++++++++++++++++++++++++++++ contrib/scalar/scalar.txt | 12 +++ contrib/scalar/t/t9099-scalar.sh | 13 +++ 3 files changed, 195 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 1ce9c2b00e8..13f2b0f4d5a 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -259,6 +259,108 @@ static int unregister_dir(void) return res; } +static int stage(const char *git_dir, struct strbuf *buf, const char *path) +{ + struct strbuf cacheinfo = STRBUF_INIT; + struct child_process cp = CHILD_PROCESS_INIT; + int res; + + strbuf_addstr(&cacheinfo, "100644,"); + + cp.git_cmd = 1; + strvec_pushl(&cp.args, "--git-dir", git_dir, + "hash-object", "-w", "--stdin", NULL); + res = pipe_command(&cp, buf->buf, buf->len, &cacheinfo, 256, NULL, 0); + if (!res) { + strbuf_rtrim(&cacheinfo); + strbuf_addch(&cacheinfo, ','); + /* We cannot stage `.git`, use `_git` instead. */ + if (starts_with(path, ".git/")) + strbuf_addf(&cacheinfo, "_%s", path + 1); + else + strbuf_addstr(&cacheinfo, path); + + child_process_init(&cp); + cp.git_cmd = 1; + strvec_pushl(&cp.args, "--git-dir", git_dir, + "update-index", "--add", "--cacheinfo", + cacheinfo.buf, NULL); + res = run_command(&cp); + } + + strbuf_release(&cacheinfo); + return res; +} + +static int stage_file(const char *git_dir, const char *path) +{ + struct strbuf buf = STRBUF_INIT; + int res; + + if (strbuf_read_file(&buf, path, 0) < 0) + return error(_("could not read '%s'"), path); + + res = stage(git_dir, &buf, path); + + strbuf_release(&buf); + return res; +} + +static int stage_directory(const char *git_dir, const char *path, int recurse) +{ + int at_root = !*path; + DIR *dir = opendir(at_root ? "." : path); + struct dirent *e; + struct strbuf buf = STRBUF_INIT; + size_t len; + int res = 0; + + if (!dir) + return error(_("could not open directory '%s'"), path); + + if (!at_root) + strbuf_addf(&buf, "%s/", path); + len = buf.len; + + while (!res && (e = readdir(dir))) { + if (!strcmp(".", e->d_name) || !strcmp("..", e->d_name)) + continue; + + strbuf_setlen(&buf, len); + strbuf_addstr(&buf, e->d_name); + + if ((e->d_type == DT_REG && stage_file(git_dir, buf.buf)) || + (e->d_type == DT_DIR && recurse && + stage_directory(git_dir, buf.buf, recurse))) + res = -1; + } + + closedir(dir); + strbuf_release(&buf); + return res; +} + +static int index_to_zip(const char *git_dir) +{ + struct child_process cp = CHILD_PROCESS_INIT; + struct strbuf oid = STRBUF_INIT; + + cp.git_cmd = 1; + strvec_pushl(&cp.args, "--git-dir", git_dir, "write-tree", NULL); + if (pipe_command(&cp, NULL, 0, &oid, the_hash_algo->hexsz + 1, + NULL, 0)) + return error(_("could not write temporary tree object")); + + strbuf_rtrim(&oid); + child_process_init(&cp); + cp.git_cmd = 1; + strvec_pushl(&cp.args, "--git-dir", git_dir, "archive", "-o", NULL); + strvec_pushf(&cp.args, "%s.zip", git_dir); + strvec_pushl(&cp.args, oid.buf, "--", NULL); + strbuf_release(&oid); + return run_command(&cp); +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -499,6 +601,73 @@ cleanup: return res; } +static int cmd_diagnose(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar diagnose []"), + NULL + }; + struct strbuf tmp_dir = STRBUF_INIT; + time_t now = time(NULL); + struct tm tm; + struct strbuf path = STRBUF_INIT, buf = STRBUF_INIT; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, &buf); + + strbuf_addstr(&buf, "/.scalarDiagnostics/scalar_"); + strbuf_addftime(&buf, "%Y%m%d_%H%M%S", localtime_r(&now, &tm), 0, 0); + if (run_git("init", "-q", "-b", "dummy", "--bare", buf.buf, NULL)) { + res = error(_("could not initialize temporary repository: %s"), + buf.buf); + goto diagnose_cleanup; + } + strbuf_realpath(&tmp_dir, buf.buf, 1); + + strbuf_reset(&buf); + strbuf_addf(&buf, "Collecting diagnostic info into temp folder %s\n\n", + tmp_dir.buf); + + get_version_info(&buf, 1); + + strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + fwrite(buf.buf, buf.len, 1, stdout); + + if ((res = stage(tmp_dir.buf, &buf, "diagnostics.log"))) + goto diagnose_cleanup; + + if ((res = stage_directory(tmp_dir.buf, ".git", 0)) || + (res = stage_directory(tmp_dir.buf, ".git/hooks", 0)) || + (res = stage_directory(tmp_dir.buf, ".git/info", 0)) || + (res = stage_directory(tmp_dir.buf, ".git/logs", 1)) || + (res = stage_directory(tmp_dir.buf, ".git/objects/info", 0))) + goto diagnose_cleanup; + + res = index_to_zip(tmp_dir.buf); + + if (!res) + res = remove_dir_recursively(&tmp_dir, 0); + + if (!res) + printf("\n" + "Diagnostics complete.\n" + "All of the gathered info is captured in '%s.zip'\n", + tmp_dir.buf); + +diagnose_cleanup: + strbuf_release(&tmp_dir); + strbuf_release(&path); + strbuf_release(&buf); + + return res; +} + static int cmd_list(int argc, const char **argv) { if (argc != 1) @@ -800,6 +969,7 @@ static struct { { "reconfigure", cmd_reconfigure }, { "delete", cmd_delete }, { "version", cmd_version }, + { "diagnose", cmd_diagnose }, { NULL, NULL}, }; diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt index f416d637289..22583fe046e 100644 --- a/contrib/scalar/scalar.txt +++ b/contrib/scalar/scalar.txt @@ -14,6 +14,7 @@ scalar register [] scalar unregister [] scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] scalar reconfigure [ --all | ] +scalar diagnose [] scalar delete DESCRIPTION @@ -129,6 +130,17 @@ reconfigure the enlistment. With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. Use this option after each Scalar upgrade. +Diagnose +~~~~~~~~ + +diagnose []:: + When reporting issues with Scalar, it is often helpful to provide the + information gathered by this command, including logs and certain + statistics describing the data shape of the current enlistment. ++ +The output of this command is a `.zip` file that is written into +a directory adjacent to the worktree in the `src` directory. + Delete ~~~~~~ diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 2e1502ad45e..ecd06e207c2 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -65,6 +65,19 @@ test_expect_success 'scalar clone' ' ) ' +SQ="'" +test_expect_success UNZIP 'scalar diagnose' ' + scalar diagnose cloned >out && + sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && + zip_path=$(cat zip_path) && + test -n "$zip_path" && + unzip -v "$zip_path" && + folder=${zip_path%.zip} && + test_path_is_missing "$folder" && + unzip -p "$zip_path" diagnostics.log >out && + test_file_not_empty out +' + test_expect_success 'scalar reconfigure' ' git init one/src && scalar register one && From patchwork Wed Jan 26 08:41:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12724728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17E9AC28CF5 for ; Wed, 26 Jan 2022 08:41:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230519AbiAZIl4 (ORCPT ); Wed, 26 Jan 2022 03:41:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238548AbiAZIlz (ORCPT ); Wed, 26 Jan 2022 03:41:55 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2852C061744 for ; Wed, 26 Jan 2022 00:41:54 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id e2so8990154wra.2 for ; Wed, 26 Jan 2022 00:41:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7ewypPI1Z9cMHCUUQ1HrSlBl1OW939EsBTHhSNw3s78=; b=D7TcTHWhUkUO57M7B0YrOTd4AgivXQcySDvDVCYewn5mms++TuqUxovv6p7Ou8sOOL Zo8tyvRcLEHD1+d0oiK849bGaL+NBfdo8EkiWChsm4gBMl519NRLH05LYEAc5/nOKOgp ObFejqQNejin/HD3siUwnpUNntt9A2ywnsU0BzUN3ScMD45Tv0hWcE2GbCOivjY7sH3u swOFT9+iZbrYGt2BZ5QZ0Wqz88jpMqltJIi7rqeoUDbDcITHhr221k94I4wI+6CNtzhF 1dve6sN7O4bacHpOiYj9cOVqf/hGqQo3Duq/PjcDyd4iOB/IweTSCDXcFcS4TsvAvYIf wHdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7ewypPI1Z9cMHCUUQ1HrSlBl1OW939EsBTHhSNw3s78=; b=f2mzOzY6Qjkyfe5WP7nxfMuivdYFaHzS0eXc5NcCxTTR0rxDKxzLxoeH2XgfaIUCzQ cdqHiwvJsEpk/vMP+youFXO6hZtfUCIgcpoVnbTdBktQYUIGL4cSZfCo69AejnxdkNpg aJw0lfSzm5Jal2gEDwn6USj7oXwtU3OdbAmOYB4FDHrnawy5c8HwaLGSiFouASWaxrwn KVJUmwnOGpdWrd9/WbBQiI4IF+by52b5ZiAuNY52q34XqbJdXBEOwoyXv0QVbhsO+ksA /2+EL2uw8C3JWKLYAPe6xkWZVsDnnseLsge2XHZu+Ym5T3y9jkbwLGTFV+bekBahQ7tn 1ZpA== X-Gm-Message-State: AOAM5328pytDCwpgK8ooywi9V9pFpZq5EXxp4d2aC3k5yypCS0Vmjwug SqbQG/co63BNk6KNBIhiwZm2T0fnqfY= X-Google-Smtp-Source: ABdhPJy0GubvuKvqfHMN1pVBqFFUtKydH4L5WK0UHiAqhAuGVgSeF7Nv4N0RJI82yj8rqhpkB6OgBA== X-Received: by 2002:a5d:5846:: with SMTP id i6mr13064418wrf.4.1643186513149; Wed, 26 Jan 2022 00:41:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f8sm3138831wmg.44.2022.01.26.00.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 00:41:51 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 26 Jan 2022 08:41:44 +0000 Subject: [PATCH 2/5] scalar diagnose: include disk space information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When analyzing problems with large worktrees/repositories, it is useful to know how close to a "full disk" situation Scalar/Git operates. Let's include this information. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 53 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 13f2b0f4d5a..e26fb2fc018 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -361,6 +361,58 @@ static int index_to_zip(const char *git_dir) return run_command(&cp); } +#ifndef WIN32 +#include +#endif + +static int get_disk_info(struct strbuf *out) +{ +#ifdef WIN32 + struct strbuf buf = STRBUF_INIT; + char volume_name[MAX_PATH], fs_name[MAX_PATH]; + DWORD serial_number, component_length, flags; + ULARGE_INTEGER avail2caller, total, avail; + + strbuf_realpath(&buf, ".", 1); + if (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) { + error(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_setlen(&buf, offset_1st_component(buf.buf)); + if (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name), + &serial_number, &component_length, &flags, + fs_name, sizeof(fs_name))) { + error(_("could not get info for '%s'"), buf.buf); + strbuf_release(&buf); + return -1; + } + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, avail2caller.QuadPart); + strbuf_addch(out, '\n'); + strbuf_release(&buf); +#else + struct strbuf buf = STRBUF_INIT; + struct statvfs stat; + + strbuf_realpath(&buf, ".", 1); + if (statvfs(buf.buf, &stat) < 0) { + error_errno(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail)); + strbuf_addf(out, " (mount flags 0x%lx)\n", stat.f_flag); + strbuf_release(&buf); +#endif + return 0; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -637,6 +689,7 @@ static int cmd_diagnose(int argc, const char **argv) get_version_info(&buf, 1); strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + get_disk_info(&buf); fwrite(buf.buf, buf.len, 1, stdout); if ((res = stage(tmp_dir.buf, &buf, "diagnostics.log"))) From patchwork Wed Jan 26 08:41:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12724729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D5AFC28CF5 for ; Wed, 26 Jan 2022 08:41:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238556AbiAZIl5 (ORCPT ); Wed, 26 Jan 2022 03:41:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238551AbiAZIl4 (ORCPT ); Wed, 26 Jan 2022 03:41:56 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15B23C061744 for ; Wed, 26 Jan 2022 00:41:56 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id s18so24356677wrv.7 for ; Wed, 26 Jan 2022 00:41:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3G1lUAZNqs+dHXlYXdtb9rjLEGLGv6eMZuLNYmd4lsA=; b=AIfbj6rGe+1/1xtoYRHC6KXAKbkTKChIRSk4TNJGrp/x+S05m4v64k7F2AnKaE/Eyn Y2OdT4ayX4eYmHL25cWHdmDpH+x4v1KbIzJvwzh8lGgE/fTuHbOOI1fb5MxpQHcKxuvu r6XxbA6WSAJqHi9Gg0UrvzlrF4Ev4AbIUlabR92aCqBNdGmdnCQBBRLVjzCDqlvf8po0 0ao6BbB62MVt5U8p780PiWE0O5iBoUKMvvIV5i7eAhkU9dhDoPQtxEFJdpVb7hhfGUyb +ynv0IQJNmRu9Q++w9cJcOUrlMJ6AoOHNB3NYGRvWQ/D49cGgUZksqTPTqKIOEeDuVyM urQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3G1lUAZNqs+dHXlYXdtb9rjLEGLGv6eMZuLNYmd4lsA=; b=2Wfk3JSwyDMcWfKSBDbNiEJ04H52/YPa2sgtiQCxQP4oe55Hol8eeX/RVHhoWG+Fui HiAKMmJ6AWoUUO4vVMhuQGCrmwNaUUVorteiEk7sMABeK0zWXutyXv1mgdZCQMv58Yua j6eEe54c+uQ51YB9ed8BDxYZ/yAKbM/i9WS07hk/pbiaYdOXj46oV+JD7QwdxXWdKgmD Pr4NKgBIMCFmMlldD+octDnhu8DjPy/MQtDlkJ50faqAT65++9KtFSlYfDcb6An+C2cf l4Tps/yUIJjhiN86KrMOCr+SpoPWdLuWTdm6eBDEu7oWA4J4Um7tyTFZKG/RhPXbx4fB 8JSw== X-Gm-Message-State: AOAM532dVgvsrPgsv0xg/QTs3gW2jxu7wk2xKZfJA6Uz+zOgbTG/eDOR 5H/PjhJZraBGbs6v1lbiPtP43JtzVaA= X-Google-Smtp-Source: ABdhPJwRDJHZ9zb2pQWOSUIsGqmjTNtZQd3rE3SIC768Uj+rcXlNULY9DIJ79RCimqJoVf+gNqAVbg== X-Received: by 2002:adf:f48a:: with SMTP id l10mr21358768wro.220.1643186514477; Wed, 26 Jan 2022 00:41:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x13sm19089293wru.28.2022.01.26.00.41.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 00:41:53 -0800 (PST) Message-Id: <330b36de799f82425c22bec50e6e42f0e495cab8.1643186507.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 26 Jan 2022 08:41:45 +0000 Subject: [PATCH 3/5] scalar: teach `diagnose` to gather packfile info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Teach the `scalar diagnose` command to gather file size information about pack files. Signed-off-by: Matthew John Cheetham --- contrib/scalar/scalar.c | 39 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 2 ++ 2 files changed, 41 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index e26fb2fc018..690933ffdf3 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -653,6 +653,39 @@ cleanup: return res; } +static void dir_file_stats(struct strbuf *buf, const char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + struct stat e_stat; + struct strbuf file_path = STRBUF_INIT; + size_t base_path_len; + + if (!dir) + return; + + strbuf_addstr(buf, "Contents of "); + strbuf_add_absolute_path(buf, path); + strbuf_addstr(buf, ":\n"); + + strbuf_add_absolute_path(&file_path, path); + strbuf_addch(&file_path, '/'); + base_path_len = file_path.len; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG) { + strbuf_setlen(&file_path, base_path_len); + strbuf_addstr(&file_path, e->d_name); + if (!stat(file_path.buf, &e_stat)) + strbuf_addf(buf, "%-70s %16"PRIuMAX"\n", + e->d_name, + (uintmax_t)e_stat.st_size); + } + + strbuf_release(&file_path); + closedir(dir); +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -695,6 +728,12 @@ static int cmd_diagnose(int argc, const char **argv) if ((res = stage(tmp_dir.buf, &buf, "diagnostics.log"))) goto diagnose_cleanup; + strbuf_reset(&buf); + dir_file_stats(&buf, ".git/objects/pack"); + + if ((res = stage(tmp_dir.buf, &buf, "packs-local.txt"))) + goto diagnose_cleanup; + if ((res = stage_directory(tmp_dir.buf, ".git", 0)) || (res = stage_directory(tmp_dir.buf, ".git/hooks", 0)) || (res = stage_directory(tmp_dir.buf, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index ecd06e207c2..b1745851e31 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -75,6 +75,8 @@ test_expect_success UNZIP 'scalar diagnose' ' folder=${zip_path%.zip} && test_path_is_missing "$folder" && unzip -p "$zip_path" diagnostics.log >out && + test_file_not_empty out && + unzip -p "$zip_path" packs-local.txt >out && test_file_not_empty out ' From patchwork Wed Jan 26 08:41:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12724730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57E77C28CF5 for ; Wed, 26 Jan 2022 08:42:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238551AbiAZImC (ORCPT ); Wed, 26 Jan 2022 03:42:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238554AbiAZIl5 (ORCPT ); Wed, 26 Jan 2022 03:41:57 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5BCBC061747 for ; Wed, 26 Jan 2022 00:41:56 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id d138-20020a1c1d90000000b0034e043aaac7so669321wmd.5 for ; Wed, 26 Jan 2022 00:41:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/GfXPlMur0t49zu+ymA1xmT5Pa6jK23dItzp/6/2Fnw=; b=khGshg4f3me+/c3TbNyDHrvENLndGZNd5NqA6hHqLA0+KmD+TdFtw4Mb3t97xiL9j4 8jwA2vbx6wiSMBjl1baqwlIhKrdFCVFmjtU2rUlyUs9w7xjieRXNma9XcQ17sdvmeAlT CvbsFkhP2kABzIPa12F/xckDuYqdPXbWXonqSn39XWfwCF0poRCBGbgi9GL0MpS2stj9 9hFbMoAEKCm2dpnxFEJP3n9Ku9t6b/Y3lJL4Fo3tHzxnS89+QuZqsa0wjb9+2BB7WL/B JO7riRaP8c5dUXnLGj8ETDvMAs4aoKb/5sY3jCwFjt/d/xtp8+dOdPPViIZpV6brRV6m w4Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/GfXPlMur0t49zu+ymA1xmT5Pa6jK23dItzp/6/2Fnw=; b=oHo/DvUFdWLfH/W1BLzY6VS3Ar8o2eMrTh+ol5sG6eOmbPHMVux/gu/jKtyt2sOHUo 4H0WBeBC63e1sx0qhbEZU/WG+sZ58joqxmYTrPSdbNbY0CKJnPtiL2n7MqHDE3O6uGbR KdeCejvOztLB6uOdW0s1aQa0zAW2Sh+FuCzv8zOSMMwfpGYj0Mzq3qvPfXiSE+4ZspyX h/JrSEFMNvz9gbJdx4DgxbyIzMvWFI3h3YtvZGvLa1eCHjXhM7d3cPASB4h0tUoGx6N7 aY5/Ug26vAKHRGRkqeLpA8vN4IdD6Gfg7u9QrA/bjdn+Rcqdr2RVcYGIDnt29NrIMm3V siYA== X-Gm-Message-State: AOAM533FTFv0IVjOXji9tWQlNBWwahGnCGBh70scPqdtF3kuCjA43Hbd 1muwN0DvN4E2f6/qgZPoAdVQVlAgmxk= X-Google-Smtp-Source: ABdhPJxztujxtP8CMLtS/Ws47ohfcid3B1MAWYZcErW044NbG4HE+CJNa2umeIkPPuqeb8BCeI5+Vw== X-Received: by 2002:a05:600c:2906:: with SMTP id i6mr6360531wmd.14.1643186515318; Wed, 26 Jan 2022 00:41:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h127sm2863000wmh.27.2022.01.26.00.41.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 00:41:54 -0800 (PST) Message-Id: <213f2c94b73f90fc758c2e3872804cf640cb2005.1643186507.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 26 Jan 2022 08:41:46 +0000 Subject: [PATCH 4/5] scalar: teach `diagnose` to gather loose objects information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham When operating at the scale that Scalar wants to support, certain data shapes are more likely to cause undesirable performance issues, such as large numbers or large sizes of loose objects. By including statistics about this, `scalar diagnose` now makes it easier to identify such scenarios. Signed-off-by: Matthew John Cheetham --- contrib/scalar/scalar.c | 60 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 2 ++ 2 files changed, 62 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 690933ffdf3..c0ad4948215 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -686,6 +686,60 @@ static void dir_file_stats(struct strbuf *buf, const char *path) closedir(dir); } +static int count_files(char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count = 0; + + if (!dir) + return 0; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG) + count++; + + closedir(dir); + return count; +} + +static void loose_objs_stats(struct strbuf *buf, const char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count; + int total = 0; + unsigned char c; + struct strbuf count_path = STRBUF_INIT; + size_t base_path_len; + + if (!dir) + return; + + strbuf_addstr(buf, "Object directory stats for "); + strbuf_add_absolute_path(buf, path); + strbuf_addstr(buf, ":\n"); + + strbuf_add_absolute_path(&count_path, path); + strbuf_addch(&count_path, '/'); + base_path_len = count_path.len; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && + e->d_type == DT_DIR && strlen(e->d_name) == 2 && + !hex_to_bytes(&c, e->d_name, 1)) { + strbuf_setlen(&count_path, base_path_len); + strbuf_addstr(&count_path, e->d_name); + total += (count = count_files(count_path.buf)); + strbuf_addf(buf, "%s : %7d files\n", e->d_name, count); + } + + strbuf_addf(buf, "Total: %d loose objects", total); + + strbuf_release(&count_path); + closedir(dir); +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -734,6 +788,12 @@ static int cmd_diagnose(int argc, const char **argv) if ((res = stage(tmp_dir.buf, &buf, "packs-local.txt"))) goto diagnose_cleanup; + strbuf_reset(&buf); + loose_objs_stats(&buf, ".git/objects"); + + if ((res = stage(tmp_dir.buf, &buf, "objects-local.txt"))) + goto diagnose_cleanup; + if ((res = stage_directory(tmp_dir.buf, ".git", 0)) || (res = stage_directory(tmp_dir.buf, ".git/hooks", 0)) || (res = stage_directory(tmp_dir.buf, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index b1745851e31..f2ec156d819 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -77,6 +77,8 @@ test_expect_success UNZIP 'scalar diagnose' ' unzip -p "$zip_path" diagnostics.log >out && test_file_not_empty out && unzip -p "$zip_path" packs-local.txt >out && + test_file_not_empty out && + unzip -p "$zip_path" objects-local.txt >out && test_file_not_empty out ' From patchwork Wed Jan 26 08:41:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12724731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5885FC2BA4C for ; Wed, 26 Jan 2022 08:42:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238558AbiAZImD (ORCPT ); Wed, 26 Jan 2022 03:42:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238548AbiAZIl6 (ORCPT ); Wed, 26 Jan 2022 03:41:58 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E29C5C06161C for ; Wed, 26 Jan 2022 00:41:57 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id s9so8560143wrb.6 for ; Wed, 26 Jan 2022 00:41:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8NExF/OXA41vxMib/2H60zyyvrhnSsyIIZN6k1MNhaw=; b=AWCRUpAg2hdrnQc5Bd7G6DQmg3H0zmJ95gmeKS0V4iBP5PdGzzqCK3M4fiDV5Ma/ba Gm2jFMvydr/NkFMlKzF2tyQFJHfylxEbSIB9UVRqUhD0uhGNXIkSYnLzABxeTXECLa/A PGl376I1fL8mYTJZgrEAKw7MabQKpL+vO1XVeCp0ov1UIlxS5PUeaY1lhBwmj7/HxxEG PK7Q6DjKntxvwfa88hnIr90BEp05P7l3Rl/xPwUHHGLg6kilvrwCn8BCYOwWe5XLcLoB qzDJrbYalw8aPuLrWtunBEcuo1qH7r6CvoiTwVlGlnJIfiprqGf5vYLWD/sJB5Aq8ftx Cj1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8NExF/OXA41vxMib/2H60zyyvrhnSsyIIZN6k1MNhaw=; b=xIHqoxoyA0g/e9JCr+8Gnq8wgz0dYK16ms6wGjINt1HlyHOtiIl91QOtlZ+wwesvhZ ldjMu4qMmQAF0V4DoSSlxdwg1NEzuELBOOm5rN1u12YiX3lv08OQRq26vrFww+ZL6tOV Sm30Otnvt3Hb73yZApP+4Pqx1Yqjb33XndXoKWt5d91qIkdH7snmoUrg7/Q3KqAA8rqW +tG605bOSoNRUXqnCFxkmu+X8/pGruZYA+raOly8g7opXpSrif5V01+imxCtX38RvfK1 C24jkkZH/aIl/NpVa9Jj2WIHDc11xCTPJKiuOTNJlhVXNYiBr0sq/xGaXHTC4SS1H+cT jc4A== X-Gm-Message-State: AOAM530ki7qK+IzSL1ZzlRfLqThi4UnIhOkKUr1wLgHfCKcRvfISFPwg tRT/0J1I6h3VIo+Hdt9xvY8LR65lNlU= X-Google-Smtp-Source: ABdhPJwxgAV3doLCl68Yy3OijlRpgaPk+KHTvn6fCnjqFHVPqRUZAaGvGTlgyUFXUJOZGzbzV25FtA== X-Received: by 2002:a05:6000:1569:: with SMTP id 9mr13451934wrz.265.1643186516072; Wed, 26 Jan 2022 00:41:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m2sm2729483wmq.35.2022.01.26.00.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 00:41:55 -0800 (PST) Message-Id: <3a2cdce554a1755210592f3b9bb056d936ca98b6.1643186507.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 26 Jan 2022 08:41:47 +0000 Subject: [PATCH 5/5] scalar diagnose: show a spinner while staging content Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin It can take a while to gather all the information that `scalar diagnose` wants to accumulate. Typically this happens when the user is in need of quick solutions and therefore their patience is tested already. By showing a little spinner that spins around, we hope to help the user muster just a tiny bit more patience until `scalar diagnose` is done. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index c0ad4948215..224329f38f5 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -259,12 +259,26 @@ static int unregister_dir(void) return res; } +static void spinner(void) +{ + static const char whee[] = "|\010/\010-\010\\\010", *next = whee; + + if (!next) + return; + if (write(2, next, 2) < 0) + next = NULL; + else + next = next[2] ? next + 2 : whee; +} + static int stage(const char *git_dir, struct strbuf *buf, const char *path) { struct strbuf cacheinfo = STRBUF_INIT; struct child_process cp = CHILD_PROCESS_INIT; int res; + spinner(); + strbuf_addstr(&cacheinfo, "100644,"); cp.git_cmd = 1;