From patchwork Wed Feb 16 15:02:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Cai X-Patchwork-Id: 12748680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C44D1C433EF for ; Wed, 16 Feb 2022 15:02:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235313AbiBPPCl (ORCPT ); Wed, 16 Feb 2022 10:02:41 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:50388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234084AbiBPPCh (ORCPT ); Wed, 16 Feb 2022 10:02:37 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A285B2982F for ; Wed, 16 Feb 2022 07:02:24 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id v12so3972140wrv.2 for ; Wed, 16 Feb 2022 07:02:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SNpIy/07GQZSBrZ5H2sk7/nd388dliKlQOi7EWsbFEw=; b=WlM9IuvK53NeYZg3rmRoNPRrkVUrrfi3R+llzFSTQJ7MtRCmaU0zT5S1yLppI/o3ro +X3ItFdggO5RfdwFb5Js2c/FmPKQN7Xz5tGjyp44FuFt/gkauyu+ytPVUTYKUSi9d9sK lwCYBOVpCitu7QxxK+2Sz+NzE+9s9DJwSFOfazmnCTJS8oZb3oul2SReDkWG21hEjGO7 hoa4mKqdVG6OqKOMPHUUj2aCkmZ2UINoW8SVp23joE+p9rk6yMsMRiyWqxrYwf0rxvfI +sOsnjwxzdYODTDfJfWG403XmgrwRc8h/J3gqqm3+LK77INabcbr/jJwK95yQCEM9YKJ X+TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SNpIy/07GQZSBrZ5H2sk7/nd388dliKlQOi7EWsbFEw=; b=794hPAGxUKV9QrWNyu/szhQpQsg/vRTaDfJ1ewwkpPWHGOE52h6lM3NQerheciUTYR pO6JYFyydiNH/sjD+ln4KMKc4OzH0KwsyAqz6Xxgcfz7AdGkbZtWFfJgWSkl+AnMZics 27N0Txyzh3nJxPsJmwrR5+yCp5rEcsHwNRZveSdoEHRgZOyiTIV+imv8pQEw8tXkSk6B R0AKsJFprwH2ThCp/KOYGtGthebSVHaGbb4G0BENXwwTEjft+CXs7H46y+JZv2d/qjXl c9/AmBjo43EJ9utG9N4tvOSqW9eVX4feJjEBjMBVmYw17C4yFIYAg505/lB2gKWXfkLu SC6A== X-Gm-Message-State: AOAM5316gsMnqMnz/H2LVNJOQEzOfDR6n9TVWYr3ZMXC6VDamNfKhjbb p+ebeHiahLr+C1zjtjO9Ly/+EjKTmAU= X-Google-Smtp-Source: ABdhPJxwHGUBusaYaNcY9xGCXVQkcgjrMDR5vi75ggxgaE3eDtlUsxwwXcVP6Rjyw8SMGXsLEFQIYw== X-Received: by 2002:adf:b645:0:b0:1e3:bab:7594 with SMTP id i5-20020adfb645000000b001e30bab7594mr2661767wre.346.1645023743045; Wed, 16 Feb 2022 07:02:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b10sm40354735wrd.8.2022.02.16.07.02.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 07:02:22 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 16 Feb 2022 15:02:17 +0000 Subject: [PATCH v8 1/4] cat-file: rename cmdmode to transform_mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, phillip.wood123@gmail.com, avarab@gmail.com, e@80x24.org, bagasdotme@gmail.com, gitster@pobox.com, Eric Sunshine , Jonathan Tan , Christian Couder , John Cai , John Cai Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: John Cai From: John Cai In the next patch, we will add an enum on the batch_options struct that indicates which type of batch operation will be used: --batch, --batch-check and the soon to be --batch-command that will read commands from stdin. --batch-command mode might get confused with the cmdmode flag. There is value in renaming cmdmode in any case. cmdmode refers to how the result output of the blob will be transformed, either according to --filter or --textconv. So transform_mode is a more descriptive name for the flag. Rename cmdmode to transform_mode in cat-file.c Signed-off-by: John Cai --- builtin/cat-file.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 7b3f42950ec..5f015e71096 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -24,7 +24,7 @@ struct batch_options { int buffer_output; int all_objects; int unordered; - int cmdmode; /* may be 'w' or 'c' for --filters or --textconv */ + int transform_mode; /* may be 'w' or 'c' for --filters or --textconv */ const char *format; }; @@ -302,19 +302,19 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d if (data->type == OBJ_BLOB) { if (opt->buffer_output) fflush(stdout); - if (opt->cmdmode) { + if (opt->transform_mode) { char *contents; unsigned long size; if (!data->rest) die("missing path for '%s'", oid_to_hex(oid)); - if (opt->cmdmode == 'w') { + if (opt->transform_mode == 'w') { if (filter_object(data->rest, 0100644, oid, &contents, &size)) die("could not convert '%s' %s", oid_to_hex(oid), data->rest); - } else if (opt->cmdmode == 'c') { + } else if (opt->transform_mode == 'c') { enum object_type type; if (!textconv_object(the_repository, data->rest, 0100644, oid, @@ -326,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d die("could not convert '%s' %s", oid_to_hex(oid), data->rest); } else - BUG("invalid cmdmode: %c", opt->cmdmode); + BUG("invalid transform_mode: %c", opt->transform_mode); batch_write(opt, contents, size); free(contents); } else { @@ -529,7 +529,7 @@ static int batch_objects(struct batch_options *opt) strbuf_expand(&output, opt->format, expand_format, &data); data.mark_query = 0; strbuf_release(&output); - if (opt->cmdmode) + if (opt->transform_mode) data.split_on_whitespace = 1; /* @@ -742,7 +742,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix) /* Return early if we're in batch mode? */ if (batch.enabled) { if (opt_cw) - batch.cmdmode = opt; + batch.transform_mode = opt; else if (opt && opt != 'b') usage_msg_optf(_("'-%c' is incompatible with batch mode"), usage, options, opt); From patchwork Wed Feb 16 15:02:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Cai X-Patchwork-Id: 12748679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAA91C433FE for ; Wed, 16 Feb 2022 15:02:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235310AbiBPPCi (ORCPT ); Wed, 16 Feb 2022 10:02:38 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:50456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235303AbiBPPCh (ORCPT ); Wed, 16 Feb 2022 10:02:37 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B932729811 for ; Wed, 16 Feb 2022 07:02:25 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id w11so3955537wra.4 for ; Wed, 16 Feb 2022 07:02:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ielkz8QHotBtE/FrvWE7acgciQk61gjydC4noInf6m0=; b=CVfjweJg6yMG3Wfiq8yvPIpYnGoNiZ0CasAcpqpZjJXxQrIsRFYMSUm2MPudAjMXmL Bg6Us0uV4cuAiZNalgblJ1A46g0lJnVfkoUX/RMbqoASW92a+PUZ6WplW0wX6f7khJ47 9J2RNVva+ukM+flBvOsmZyVUg82RwuXRvodX0ZfE/0eyFyByBxLsYDysTamBlGYub1y0 HZjsTL5aRMtMkE21eAkVwWUWz7NIi/gbMRaEHCqT1q/IW2JRLvVIfLejPzHquW1aD/x2 ulauZrfnaCcTXYqiYUHiawbOiKfwoUhdUAlhPfkyZEWe4bY2zqojImVAn/XBvGIFzGJK 4qhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ielkz8QHotBtE/FrvWE7acgciQk61gjydC4noInf6m0=; b=qqCZsN6z7nx3GmhkuLHx4MR8ssEg4DeP1JrdIsDMy5Z2p1e3ZpAK/ws9e/6NtOWjq7 OGOtsVFr6D4WkVm7hmi3bi/86EI8X9+XJ0VxcMyTj0Stjble0DEpQkucmrQ2DrkR18MN bHgoJouBRkZV25BiK5tDsfBTBT7SoYHIFRYs+qcfNptzrGwsnQud5DjALnneZ2fpxiOR 2te8YpS6jlqmw3HIEwvwf9qCvozmsmTcEB/15mI3ub9oBqCSAyIqJ8FP56zH2N77aT8I GfQSygJcEYKRukfGlszbp9Mw3qSrTFMRaXiP552JSqB9sH6300plNII8wmGvJ5aS7OVu e2kg== X-Gm-Message-State: AOAM533LS2XFaxkfoleDtpChnh1KO2WZNp2qn0DaEiC5G0tjmc2LOZYS 6aMGY/jCD1ln+lB9Mii0FxErjxfT1jY= X-Google-Smtp-Source: ABdhPJyg3U1WKgD9aub5usOmf2Kc3fS2f7fmvXLNKwQ1M4tgh3xGYhbThURwiRxf63BZXMizBWEbMw== X-Received: by 2002:a05:6000:25a:b0:1e3:2010:4d14 with SMTP id m26-20020a056000025a00b001e320104d14mr2663199wrz.455.1645023743982; Wed, 16 Feb 2022 07:02:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n5sm19061351wrm.3.2022.02.16.07.02.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 07:02:23 -0800 (PST) Message-Id: <1a038097bfc1e0e7dc7737e4fb664e8f79a2da50.1645023740.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Feb 2022 15:02:18 +0000 Subject: [PATCH v8 2/4] cat-file: introduce batch_mode enum to replace print_contents Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, phillip.wood123@gmail.com, avarab@gmail.com, e@80x24.org, bagasdotme@gmail.com, gitster@pobox.com, Eric Sunshine , Jonathan Tan , Christian Couder , John Cai , John Cai Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: John Cai From: John Cai A future patch introduces a new --batch-command flag. Including --batch and --batch-check, we will have a total of three batch modes. print_contents is the only boolean on the batch_options sturct used to distinguish between the different modes. This makes the code harder to read. To reduce potential confusion, replace print_contents with an enum to help readability and clarity. Helped-by: Junio C Hamano Signed-off-by: John Cai --- builtin/cat-file.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 5f015e71096..5e38af82af1 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -17,10 +17,15 @@ #include "object-store.h" #include "promisor-remote.h" +enum batch_mode { + BATCH_MODE_CONTENTS, + BATCH_MODE_INFO, +}; + struct batch_options { int enabled; int follow_symlinks; - int print_contents; + enum batch_mode batch_mode; int buffer_output; int all_objects; int unordered; @@ -386,7 +391,7 @@ static void batch_object_write(const char *obj_name, strbuf_addch(scratch, '\n'); batch_write(opt, scratch->buf, scratch->len); - if (opt->print_contents) { + if (opt->batch_mode == BATCH_MODE_CONTENTS) { print_object_or_die(opt, data); batch_write(opt, "\n", 1); } @@ -536,7 +541,7 @@ static int batch_objects(struct batch_options *opt) * If we are printing out the object, then always fill in the type, * since we will want to decide whether or not to stream. */ - if (opt->print_contents) + if (opt->batch_mode == BATCH_MODE_CONTENTS) data.info.typep = &data.type; if (opt->all_objects) { @@ -635,7 +640,14 @@ static int batch_option_callback(const struct option *opt, } bo->enabled = 1; - bo->print_contents = !strcmp(opt->long_name, "batch"); + + if (!strcmp(opt->long_name, "batch")) + bo->batch_mode = BATCH_MODE_CONTENTS; + else if (!strcmp(opt->long_name, "batch-check")) + bo->batch_mode = BATCH_MODE_INFO; + else + BUG("%s given to batch-option-callback", opt->long_name); + bo->format = arg; return 0; From patchwork Wed Feb 16 15:02:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Cai X-Patchwork-Id: 12748681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3783AC433EF for ; Wed, 16 Feb 2022 15:02:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235312AbiBPPCr (ORCPT ); Wed, 16 Feb 2022 10:02:47 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:50544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233199AbiBPPCj (ORCPT ); Wed, 16 Feb 2022 10:02:39 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5E192982F for ; Wed, 16 Feb 2022 07:02:26 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id f3so3771227wrh.7 for ; Wed, 16 Feb 2022 07:02:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UC9fF+KP+NxF+UpJJq9w730NXczpWlLCADgKQQkkAMM=; b=jdVeM4uARJGHRlcHCbJrGDRcY8fb0jnM9Ju/6nUG0tTvrpFoiodCwLVeA2oupwCH9q HDVy54o/xa8wK+IClcNjcguN66035y9c3QJf/mNegDJFjQsWSa9PHJM7tTTPFHZF3kn9 fbnsxM+GIaKndvOla3bLtIB2vMvph8Rxvt/V68tgXbfoka5BzteSvDEwYp4B4xYkF47F p+0fnYC6RDTQ2HzYnaZbXS3CUn15Ra+hZ8I2zpqTwa2PDfbv3I+fP+0j7jTEAvFl7K2Q kUtytjadCj1C3SS6KYb023foFTYtrPdyUO/UfgwiERGsCkDBHXE6A6M8lWoD9719dTYf eY8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UC9fF+KP+NxF+UpJJq9w730NXczpWlLCADgKQQkkAMM=; b=vsW2JvCn+bdPQ1ahLMV7p0xTq7nZ9ox8RX3eSTh05/WIjvv1fQtmnlqMSZ/OzpgapW fF9ShGz3t8MYGa7DrBDFfcZHyWLKQIRPZpbt5Qf0ZQ3bH+atZzLW+UR5Tciyn1JFJxff EqGlKuSDGA0sGQJB8I/vIKdZolnLI6eAX++tgcnlCd977x3S206zTYzSNQOfiYKVDZrX 5K12Ye47IAF+ITNmfU5/v5h9/JFz31S7Flclv5C6ijQjqdZmtPCF2vMOQmX6eNHlLC1m Xwi7rjcRPSvgprfr8nIpAdFScELXbV1VVvnLBgWDqn8kC/hHy81+fcdVR/3wSw73UvG8 ulHw== X-Gm-Message-State: AOAM532hJlyrMO7lhGR4VtjpwaEHAVUw2dQEAXAZUhzxWcFHubO+HYaZ X5QqZ4GrtELgqqaRx9WxS3nimAuQxWM= X-Google-Smtp-Source: ABdhPJwocTZ5+Z0GpT3DMEKiD0t6GwSCuJ9sheKOhqRY2jUZgs1/jY3J96EjDHwaG7iuivCVBOwITQ== X-Received: by 2002:a5d:6da1:0:b0:1e3:2bf5:13c with SMTP id u1-20020a5d6da1000000b001e32bf5013cmr2774921wrs.316.1645023745071; Wed, 16 Feb 2022 07:02:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l12sm17654756wmd.44.2022.02.16.07.02.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 07:02:24 -0800 (PST) Message-Id: <486ee84781679d587f137b9c24fb5377e4b138d8.1645023740.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Feb 2022 15:02:19 +0000 Subject: [PATCH v8 3/4] cat-file: add remove_timestamp helper Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, phillip.wood123@gmail.com, avarab@gmail.com, e@80x24.org, bagasdotme@gmail.com, gitster@pobox.com, Eric Sunshine , Jonathan Tan , Christian Couder , John Cai , John Cai Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: John Cai From: John Cai maybe_remove_timestamp() takes arguments, but it would be useful to have a function that reads from stdin and strips the timestamp. This would allow tests to pipe data into a function to remove timestamps, and wouldn't have to always assign a variable. This is especially helpful when the data is multiple lines. Keep maybe_remove_timestamp() the same, but add a remove_timestamp helper that reads from stdin. The tests in the next patch will make use of this. Signed-off-by: John Cai --- t/t1006-cat-file.sh | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh index 145eee11df9..2d52851dadc 100755 --- a/t/t1006-cat-file.sh +++ b/t/t1006-cat-file.sh @@ -105,13 +105,18 @@ strlen () { } maybe_remove_timestamp () { - if test -z "$2"; then - echo_without_newline "$1" - else - echo_without_newline "$(printf '%s\n' "$1" | sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//')" - fi + if test -z "$2"; then + echo_without_newline "$1" + else + echo_without_newline "$(printf '%s\n' "$1" | remove_timestamp)" + fi } +remove_timestamp () { + sed -e 's/ [0-9][0-9]* [-+][0-9][0-9][0-9][0-9]$//' +} + + run_tests () { type=$1 sha1=$2 From patchwork Wed Feb 16 15:02:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Cai X-Patchwork-Id: 12748682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A7FAC433EF for ; Wed, 16 Feb 2022 15:02:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235337AbiBPPCt (ORCPT ); Wed, 16 Feb 2022 10:02:49 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:50792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235316AbiBPPCm (ORCPT ); Wed, 16 Feb 2022 10:02:42 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F6422AE27 for ; Wed, 16 Feb 2022 07:02:28 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id i19so1376528wmq.5 for ; Wed, 16 Feb 2022 07:02:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=xv4t2aZxqMGyWIvNovy4YtHVx+f8VAoqZNjMMU9DOpc=; b=ilLGyUcTrhLllgLfYmJgQDPQBKWvnnYbt9Hcno5hwmBZlEu3Bx2PqNGeu4vgBwK0GA DRl0guDGgQ6uAM7u0mX8w33o+P9pnPD2G9Ui7SKMjr9GzDFtCQNh3aazudmTaQhtTM1l R+hbtZTvaVOolgRshkkm3jC2EqQK8HANKfTmHDYENdb9V86eoLSygshnhd3dwGgck2pv xcJc9baiIl+ctDOSvGyQTgJis/cHMhfB1/Zk5Rm9pS1BCAw+hxdQVqNznTtE/eipuq8I 4O+emsEBHKD0Qlk91NC+LwnxnX0A7nM9p2NAeioIhqRWPidCAlfuZsst8ugxZrBLMqx5 FLIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=xv4t2aZxqMGyWIvNovy4YtHVx+f8VAoqZNjMMU9DOpc=; b=KecpahwoCKd2yhmQIGga06b7Xyx4366zsECNryCdUSyVf4NrjuZixDGiQFcYJdYio5 wCM4TLSEXaPdsGc/nw3WDWm1wKfOrYqMDVHgWRNhsyX/UEIodoPvbMs9k0S3Igq26IfB dh89SyU9YSbgCQUpao84MHygQyq+b7bhXoOZ2qgazO3qJX7SD7dWz6yJ1K8Q5WTlW4gW V4+m2WjeADdIVe9uWbsFTWld1zX5GOBUN4sCyUQspnQipK/FRYeJ2uhNvtUH0p5kfNeA MPIPDPhigYBTYHwB3npLdOjBg3ROHs05VQxu5ACept6xnebLtRXHxQYz8V1vSyy1pXU7 9njA== X-Gm-Message-State: AOAM5321zGe+rjh4qnxwhfhBwiCqDIrZoMV2Ldmtm/HQHsZhUQiR9txZ 36I1OB6zK9pMHyiZEc2GZemtPNFnr3I= X-Google-Smtp-Source: ABdhPJzXZhT17FL6g4P708l3VJody7a7Bby3rRwcV0IehL4M/1ZL716+QfnvA9AdkTUdmwScLC+70Q== X-Received: by 2002:a7b:c0d2:0:b0:37b:e9e6:75d6 with SMTP id s18-20020a7bc0d2000000b0037be9e675d6mr1950975wmh.39.1645023746186; Wed, 16 Feb 2022 07:02:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o20sm18086223wmq.21.2022.02.16.07.02.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 07:02:25 -0800 (PST) Message-Id: <8edf80574b8dd72ca8489df6e480cf6820b98070.1645023740.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Feb 2022 15:02:20 +0000 Subject: [PATCH v8 4/4] cat-file: add --batch-command mode MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: me@ttaylorr.com, phillip.wood123@gmail.com, avarab@gmail.com, e@80x24.org, bagasdotme@gmail.com, gitster@pobox.com, Eric Sunshine , Jonathan Tan , Christian Couder , John Cai , John Cai Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: John Cai From: John Cai Add a new flag --batch-command that accepts commands and arguments from stdin, similar to git-update-ref --stdin. At GitLab, we use a pair of long running cat-file processes when accessing object content. One for iterating over object metadata with --batch-check, and the other to grab object contents with --batch. However, if we had --batch-command, we wouldn't need to keep both processes around, and instead just have one --batch-command process where we can flip between getting object info, and getting object contents. Since we have a pair of cat-file processes per repository, this means we can get rid of roughly half of long lived git cat-file processes. Given there are many repositories being accessed at any given time, this can lead to huge savings. git cat-file --batch-command will enter an interactive command mode whereby the user can enter in commands and their arguments that get queued in memory: [arg1] [arg2] LF [arg1] [arg2] LF When --buffer mode is used, commands will be queued in memory until a flush command is issued that execute them: flush LF The reason for a flush command is that when a consumer process (A) talks to a git cat-file process (B) and interactively writes to and reads from it in --buffer mode, (A) needs to be able to control when the buffer is flushed to stdout. Currently, from (A)'s perspective, the only way is to either 1. kill (B)'s process 2. send an invalid object to stdin. 1. is not ideal from a performance perspective as it will require spawning a new cat-file process each time, and 2. is hacky and not a good long term solution. With this mechanism of queueing up commands and letting (A) issue a flush command, process (A) can control when the buffer is flushed and can guarantee it will receive all of the output when in --buffer mode. --batch-command also will not allow (B) to flush to stdout until a flush is received. This patch adds the basic structure for adding command which can be extended in the future to add more commands. It also adds the following two commands (on top of the flush command): contents LF info LF The contents command takes an argument and prints out the object contents. The info command takes an argument and prints out the object metadata. These can be used in the following way with --buffer: info LF contents LF contents LF info LF flush LF info LF flush LF When used without --buffer: info LF contents LF contents LF info LF info LF Helped-by: Ævar Arnfjörð Bjarmason Signed-off-by: John Cai --- Documentation/git-cat-file.txt | 42 +++++++++- builtin/cat-file.c | 140 ++++++++++++++++++++++++++++++++- t/README | 3 + t/t1006-cat-file.sh | 107 ++++++++++++++++++++++++- 4 files changed, 286 insertions(+), 6 deletions(-) diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt index bef76f4dd06..70c5b4f12d1 100644 --- a/Documentation/git-cat-file.txt +++ b/Documentation/git-cat-file.txt @@ -96,6 +96,33 @@ OPTIONS need to specify the path, separated by whitespace. See the section `BATCH OUTPUT` below for details. +--batch-command:: +--batch-command=:: + Enter a command mode that reads commands and arguments from stdin. May + only be combined with `--buffer`, `--textconv` or `--filters`. In the + case of `--textconv` or `--filters`, the input lines also need to specify + the path, separated by whitespace. See the section `BATCH OUTPUT` below + for details. ++ +`--batch-command` recognizes the following commands: ++ +-- +contents :: + Print object contents for object reference ``. This corresponds to + the output of `--batch`. + +info :: + Print object info for object reference ``. This corresponds to the + output of `--batch-check`. + +flush:: + Used with `--buffer` to execute all preceding commands that were issued + since the beginning or since the last flush was issued. When `--buffer` + is used, no output will come until a `flush` is issued. When `--buffer` + is not used, commands are flushed each time without issuing `flush`. +-- ++ + --batch-all-objects:: Instead of reading a list of objects on stdin, perform the requested batch operation on all objects in the repository and @@ -110,7 +137,7 @@ OPTIONS that a process can interactively read and write from `cat-file`. With this option, the output uses normal stdio buffering; this is much more efficient when invoking - `--batch-check` on a large number of objects. + `--batch-check` or `--batch-command` on a large number of objects. --unordered:: When `--batch-all-objects` is in use, visit objects in an @@ -202,6 +229,13 @@ from stdin, one per line, and print information about them. By default, the whole line is considered as an object, as if it were fed to linkgit:git-rev-parse[1]. +When `--batch-command` is given, `cat-file` will read commands from stdin, +one per line, and print information based on the command given. With +`--batch-command`, the `info` command followed by an object will print +information about the object the same way `--batch-check` would, and the +`contents` command followed by an object prints contents in the same way +`--batch` would. + You can specify the information shown for each object by using a custom ``. The `` is copied literally to stdout for each object, with placeholders of the form `%(atom)` expanded, followed by a @@ -237,9 +271,9 @@ newline. The available atoms are: If no format is specified, the default format is `%(objectname) %(objecttype) %(objectsize)`. -If `--batch` is specified, the object information is followed by the -object contents (consisting of `%(objectsize)` bytes), followed by a -newline. +If `--batch` is specified, or if `--batch-command` is used with the `contents` +command, the object information is followed by the object contents (consisting +of `%(objectsize)` bytes), followed by a newline. For example, `--batch` without a custom format would produce: diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 5e38af82af1..39d486dd737 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -20,6 +20,7 @@ enum batch_mode { BATCH_MODE_CONTENTS, BATCH_MODE_INFO, + BATCH_MODE_QUEUE_AND_DISPATCH, }; struct batch_options { @@ -513,6 +514,132 @@ static int batch_unordered_packed(const struct object_id *oid, data); } +typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *, + struct strbuf *, struct expand_data *); + +struct queued_cmd { + parse_cmd_fn_t fn; + char *line; +}; + +static void parse_cmd_contents(struct batch_options *opt, + const char *line, + struct strbuf *output, + struct expand_data *data) +{ + opt->batch_mode = BATCH_MODE_CONTENTS; + batch_one_object(line, output, opt, data); +} + +static void parse_cmd_info(struct batch_options *opt, + const char *line, + struct strbuf *output, + struct expand_data *data) +{ + opt->batch_mode = BATCH_MODE_INFO; + batch_one_object(line, output, opt, data); +} + +/* Loop through each queued_cmd, dispatch the function, free the + * memory associated with line so the cmd array can be reused. + * Callers must set nr back to 0 in order to reuse the cmd array. + */ +static void dispatch_calls(struct batch_options *opt, + struct strbuf *output, + struct expand_data *data, + struct queued_cmd *cmd, + int nr) +{ + int i; + + if (!opt->buffer_output) + die(_("flush is only for --buffer mode")); + + for (i = 0; i < nr; i++) { + cmd[i].fn(opt, cmd[i].line, output, data); + FREE_AND_NULL(cmd[i].line); + } + + fflush(stdout); +} + +static const struct parse_cmd { + const char *name; + parse_cmd_fn_t fn; + unsigned takes_args; +} commands[] = { + { "contents", parse_cmd_contents, 1}, + { "info", parse_cmd_info, 1}, + { "flush", NULL, 0}, +}; + +static void batch_objects_command(struct batch_options *opt, + struct strbuf *output, + struct expand_data *data) +{ + struct strbuf input = STRBUF_INIT; + struct queued_cmd *queued_cmd = NULL; + size_t alloc = 0, nr = 0; + + while (!strbuf_getline(&input, stdin)) { + int i; + const struct parse_cmd *cmd = NULL; + const char *p = NULL, *cmd_end; + struct queued_cmd call = {0}; + + if (!input.len) + die(_("empty command in input")); + if (isspace(*input.buf)) + die(_("whitespace before command: '%s'"), input.buf); + + for (i = 0; i < ARRAY_SIZE(commands); i++) { + if (!skip_prefix(input.buf, commands[i].name, &cmd_end)) + continue; + + cmd = &commands[i]; + if (cmd->takes_args) { + if (*cmd_end != ' ') + die(_("%s requires arguments"), + commands[i].name); + + p = cmd_end + 1; + } else if (*cmd_end) { + die(_("%s takes no arguments"), + commands[i].name); + } + + break; + } + + if (!cmd) + die(_("unknown command: '%s'"), input.buf); + + if (!strcmp(cmd->name, "flush")) { + dispatch_calls(opt, output, data, queued_cmd, nr); + nr = 0; + continue; + } + + if (!opt->buffer_output) { + cmd->fn(opt, p, output, data); + continue; + } + + ALLOC_GROW(queued_cmd, nr + 1, alloc); + call.fn = cmd->fn; + call.line = xstrdup_or_null(p); + queued_cmd[nr++] = call; + } + + if (opt->buffer_output && + nr && + !git_env_bool("GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT", 0)) + dispatch_calls(opt, output, data, queued_cmd, nr); + + free(queued_cmd); + strbuf_release(&input); +} + static int batch_objects(struct batch_options *opt) { struct strbuf input = STRBUF_INIT; @@ -595,6 +722,10 @@ static int batch_objects(struct batch_options *opt) save_warning = warn_on_object_refname_ambiguity; warn_on_object_refname_ambiguity = 0; + if (opt->batch_mode == BATCH_MODE_QUEUE_AND_DISPATCH) { + batch_objects_command(opt, &output, &data); + goto cleanup; + } while (strbuf_getline(&input, stdin) != EOF) { if (data.split_on_whitespace) { /* @@ -613,6 +744,7 @@ static int batch_objects(struct batch_options *opt) batch_one_object(input.buf, &output, opt, &data); } + cleanup: strbuf_release(&input); strbuf_release(&output); warn_on_object_refname_ambiguity = save_warning; @@ -645,6 +777,8 @@ static int batch_option_callback(const struct option *opt, bo->batch_mode = BATCH_MODE_CONTENTS; else if (!strcmp(opt->long_name, "batch-check")) bo->batch_mode = BATCH_MODE_INFO; + else if (!strcmp(opt->long_name, "batch-command")) + bo->batch_mode = BATCH_MODE_QUEUE_AND_DISPATCH; else BUG("%s given to batch-option-callback", opt->long_name); @@ -666,7 +800,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix) N_("git cat-file "), N_("git cat-file (-e | -p) "), N_("git cat-file (-t | -s) [--allow-unknown-type] "), - N_("git cat-file (--batch | --batch-check) [--batch-all-objects]\n" + N_("git cat-file (--batch | --batch-check | --batch-command) [--batch-all-objects]\n" " [--buffer] [--follow-symlinks] [--unordered]\n" " [--textconv | --filters]"), N_("git cat-file (--textconv | --filters)\n" @@ -695,6 +829,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix) N_("like --batch, but don't emit "), PARSE_OPT_OPTARG | PARSE_OPT_NONEG, batch_option_callback), + OPT_CALLBACK_F(0, "batch-command", &batch, N_("format"), + N_("read commands from stdin"), + PARSE_OPT_OPTARG | PARSE_OPT_NONEG, + batch_option_callback), OPT_CMDMODE(0, "batch-all-objects", &opt, N_("with --batch[-check]: ignores stdin, batches all known objects"), 'b'), /* Batch-specific options */ diff --git a/t/README b/t/README index f48e0542cdc..bcd813b0c59 100644 --- a/t/README +++ b/t/README @@ -472,6 +472,9 @@ a test and then fails then the whole test run will abort. This can help to make sure the expected tests are executed and not silently skipped when their dependency breaks or is simply not present in a new environment. +GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=, when true will prevent cat-file +--batch-command from flushing to output on exit. + Naming Tests ------------ diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh index 2d52851dadc..74f0e36b69e 100755 --- a/t/t1006-cat-file.sh +++ b/t/t1006-cat-file.sh @@ -182,6 +182,24 @@ $content" test_cmp expect actual ' + for opt in --buffer --no-buffer + do + test -z "$content" || + test_expect_success "--batch-command $opt output of $type content is correct" ' + maybe_remove_timestamp "$batch_output" $no_ts >expect && + maybe_remove_timestamp "$(test_write_lines "contents $sha1" \ + | git cat-file --batch-command $opt)" $no_ts >actual && + test_cmp expect actual + ' + + test_expect_success "--batch-command $opt output of $type info is correct" ' + echo "$sha1 $type $size" >expect && + test_write_lines "info $sha1" \ + | git cat-file --batch-command $opt >actual && + test_cmp expect actual + ' + done + test_expect_success "custom --batch-check format" ' echo "$type $sha1" >expect && echo $sha1 | git cat-file --batch-check="%(objecttype) %(objectname)" >actual && @@ -229,6 +247,22 @@ test_expect_success "setup" ' run_tests 'blob' $hello_sha1 $hello_size "$hello_content" "$hello_content" +test_expect_success '--batch-command --buffer with flush for blob info' ' + echo "$hello_sha1 blob $hello_size" >expect && + test_write_lines "info $hello_sha1" "flush" | \ + GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \ + git cat-file --batch-command --buffer >actual && + test_cmp expect actual +' + +test_expect_success '--batch-command --buffer without flush for blob info' ' + touch output && + test_write_lines "info $hello_sha1" | \ + GIT_TEST_CAT_FILE_NO_FLUSH_ON_EXIT=1 \ + git cat-file --batch-command --buffer >>output && + test_must_be_empty output +' + test_expect_success '--batch-check without %(rest) considers whole line' ' echo "$hello_sha1 blob $hello_size" >expect && git update-index --add --cacheinfo 100644 $hello_sha1 "white space" && @@ -272,7 +306,7 @@ test_expect_success \ "Reach a blob from a tag pointing to it" \ "test '$hello_content' = \"\$(git cat-file blob $tag_sha1)\"" -for batch in batch batch-check +for batch in batch batch-check batch-command do for opt in t s e p do @@ -378,6 +412,42 @@ test_expect_success "--batch-check with multiple sha1s gives correct format" ' "$(echo_without_newline "$batch_check_input" | git cat-file --batch-check)" ' +test_expect_success '--batch-command with multiple info calls gives correct format' ' + cat >expect <<-EOF && + $hello_sha1 blob $hello_size + $tree_sha1 tree $tree_size + $commit_sha1 commit $commit_size + $tag_sha1 tag $tag_size + deadbeef missing + EOF + + test_write_lines "info $hello_sha1"\ + "info $tree_sha1"\ + "info $commit_sha1"\ + "info $tag_sha1"\ + "info deadbeef" | git cat-file --batch-command --buffer >actual && + test_cmp expect actual +' + +test_expect_success '--batch-command with multiple command calls gives correct format' ' + remove_timestamp >expect <<-EOF && + $hello_sha1 blob $hello_size + $hello_content + $commit_sha1 commit $commit_size + $commit_content + $tag_sha1 tag $tag_size + $tag_content + deadbeef missing + EOF + + test_write_lines "contents $hello_sha1"\ + "contents $commit_sha1"\ + "contents $tag_sha1"\ + "contents deadbeef"\ + "flush" | git cat-file --batch-command --buffer | remove_timestamp >actual && + test_cmp expect actual +' + test_expect_success 'setup blobs which are likely to delta' ' test-tool genrandom foo 10240 >foo && { cat foo && echo plus; } >foo-plus && @@ -968,5 +1038,40 @@ test_expect_success 'cat-file --batch-all-objects --batch-check ignores replace' echo "$orig commit $orig_size" >expect && test_cmp expect actual ' +test_expect_success 'batch-command empty command' ' + echo "" >cmd && + test_expect_code 128 git cat-file --batch-command err && + grep "^fatal:.*empty command in input.*" err +' + +test_expect_success 'batch-command whitespace before command' ' + echo " info deadbeef" >cmd && + test_expect_code 128 git cat-file --batch-command err && + grep "^fatal:.*whitespace before command.*" err +' + +test_expect_success 'batch-command unknown command' ' + echo unknown_command >cmd && + test_expect_code 128 git cat-file --batch-command err && + grep "^fatal:.*unknown command.*" err +' + +test_expect_success 'batch-command missing arguments' ' + echo "info" >cmd && + test_expect_code 128 git cat-file --batch-command err && + grep "^fatal:.*info requires arguments.*" err +' + +test_expect_success 'batch-command flush with arguments' ' + echo "flush arg" >cmd && + test_expect_code 128 git cat-file --batch-command --buffer err && + grep "^fatal:.*flush takes no arguments.*" err +' + +test_expect_success 'batch-command flush without --buffer' ' + echo "flush" >cmd && + test_expect_code 128 git cat-file --batch-command err && + grep "^fatal:.*flush is only for --buffer mode.*" err +' test_done