From patchwork Sat May 21 22:18:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B43AC433F5 for ; Sat, 21 May 2022 22:19:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347058AbiEUWTE (ORCPT ); Sat, 21 May 2022 18:19:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245400AbiEUWTC (ORCPT ); Sat, 21 May 2022 18:19:02 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B25514EF60 for ; Sat, 21 May 2022 15:19:00 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id w4so15875590wrg.12 for ; Sat, 21 May 2022 15:19:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ychXGwGi9MKGlFm4LBv/h3HDn8TPIu/o+OfIz7tey24=; b=MjX3yuMselYclC1hS8A8e5FPRj5MBq5HMIIIFy0aZ7dk/khPHh9tY0gwABOpmR/Z+Q ulytuVZ/JAuJ+G9EapwwyehhgZlAMzb9vGyrJcuD5CCFQOQ29xKmdjYZ1eNtRxXcyMys UNa5TdH0M0blJSWEokXOvao9EkAg0ijuRXdP8JpJ/Krxo2n6a0scSSx8UX7ztsRFfxIb avArEbzJu0By8/zxr33LPBAJotqKdqYkNmUD1eAZrnoaRZIzNBqXw/Ql+TEGO8JVUG9P IZAAol33/fYNt3PsDh4QFpxwUSuIlkBsNbqK+02ZU4D8WfS2w5Jl5McoJsB18T1P4Y4r bwWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ychXGwGi9MKGlFm4LBv/h3HDn8TPIu/o+OfIz7tey24=; b=La8UeG0PqN/S2KXxKUDfPDY2QYgWMTB0R5p3vn7DIlJBfDa1vprfWiNLKLM3+3HJtx VlZSHyJcx/5Rwk2qWlCn7nMxlT8Pc1/L7wcgxLMbr5SbLzCIDcD4xpAiBPAS1A68BvCB lALQciVnQ3EPG8A5ShHony30JRev9X0aAZWPyw7aAmb+LFcLatGexNJw1Sc9BcJZuxBp IPBwIE7Ze3wI7VmwWRVy+ZTfsS3lq36X//U/b5YeO4B7KxtLCiHVEsGlc0wDPuYxS93q +kPgeg035M3bK379gK+0GUozzbdEi7L3L1poHSjJUS3D/KngbV2INgpLzAynWtBq0DR4 0MLg== X-Gm-Message-State: AOAM532162t6lrT4M6scxalNjwwyGU8+IK/Ihe+V46kKGFS1E0bOhHpl AkEVy2+V27sg3ruHOJ8mf/39jznkOsw= X-Google-Smtp-Source: ABdhPJxVZPV2zLOqzZ/lDUpnwNRWRlLcWUb0AzqGzoERG8cK+wOrX06aUVOiLj4NK/UiRLnCx3lpNA== X-Received: by 2002:adf:cd06:0:b0:20d:a8a:c096 with SMTP id w6-20020adfcd06000000b0020d0a8ac096mr13576986wrm.637.1653171538936; Sat, 21 May 2022 15:18:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m40-20020a05600c3b2800b00397381a7ae8sm5167410wms.30.2022.05.21.15.18.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:18:58 -0700 (PDT) Message-Id: <68793fcec62dce83a972a055fee46f48cedd115a.1653171536.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 22:18:45 +0000 Subject: [PATCH v3 01/12] ci: fix code style Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin In b92cb86ea14 (travis-ci: check that all build artifacts are .gitignore-d, 2017-12-31), a function was introduced with a code style that is different from the surrounding code: it added the opening curly brace on its own line, when all the existing functions in the same file cuddle that brace on the same line as the function name. Let's make the code style consistent again. Signed-off-by: Johannes Schindelin --- ci/lib.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 86e37da9bc5..d718f4e386d 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -69,8 +69,7 @@ skip_good_tree () { exit 0 } -check_unignored_build_artifacts () -{ +check_unignored_build_artifacts () { ! git ls-files --other --exclude-standard --error-unmatch \ -- ':/*' 2>/dev/null || { From patchwork Sat May 21 22:18:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 878E1C433F5 for ; Sat, 21 May 2022 22:19:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347144AbiEUWTI (ORCPT ); Sat, 21 May 2022 18:19:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347037AbiEUWTD (ORCPT ); Sat, 21 May 2022 18:19:03 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD3094EF69 for ; Sat, 21 May 2022 15:19:01 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id k30so15902607wrd.5 for ; Sat, 21 May 2022 15:19:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=le2cv+HomVUYLKGp+TzkR9V/sqB0i/ZnSks/92czT8U=; b=HDgkzuXiHjP8E5zaxYcWdxmjSK/kDj4GRU4UTTP6tVe14yMNOjue39y6lj1/z8KHxu 4z08HS7yW4zQz5F+h4kbUtz1YlzIWiEsqYsH6/JlOaOqD6bUViIVVpzKrnY+d37rnBGU gM0K4a+6yUXY/Jb/Bw279ScTejXYtQwq0zEdgyIhYy0bD3CEe5v44/aXVYgV15ES1Urz D1Ui6t8JUGleLKPuKGSAcesZCRvUqB1T1b1cJ0ngS3iJs/iyIBNwaIn+oZn5K5jfxEnr i86nGMckt/6+bKvV5wK2qrywo9Xws0Ldea4SYB3I0ZRKniQvrHBlqrUbdpD25iSCL41v 357A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=le2cv+HomVUYLKGp+TzkR9V/sqB0i/ZnSks/92czT8U=; b=Cfx/hp9YVvFKkj4pujtuXJ77F0Wy6C0AYrS0npOV8XsPg7BNFLgTmXQW8ScRqc77a2 iUHmNCXPP4ne2T0ERoPYNU0KVLudSSbXpWc4eQnS5ef3OgDzUpCLY5C5J3Xz7Oj5Tyl/ PkRWIKyN78S6Jr+jeagfym0gLylLbS74vcYnBgu2rZvOTesIKsLcRofvbvOJDEur1prW b6GuQLzXTDNIWay7fjE/jKNGcl89+l4sl2rUSKA7b9J1EeiayVUZg/iOPNEAPzBBV5ag W1LDlNgXGk/g55OmRUsKlGJ1ZZIcEDQ642UFozMX9q5CM4Jx1wOjaRcKb9+ot/BjAi5H CLMQ== X-Gm-Message-State: AOAM531c2v+QDIFKfeZNY5Fiaj5RoHKOhNR6sjO9NRwZR5DtngyA17EO VBzJNsm8N1VdIV+3jHdsIJHpDGiFrHY= X-Google-Smtp-Source: ABdhPJx6RT5Y5sYaHPB7x2jiYgn5o9aooEQwVUAVHjNQyCMJwtRtYDEgCzj1e2zzl4ivjjjW3pxHlQ== X-Received: by 2002:adf:eb4c:0:b0:20e:64b8:1ce8 with SMTP id u12-20020adfeb4c000000b0020e64b81ce8mr13425669wrn.609.1653171540022; Sat, 21 May 2022 15:19:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m40-20020a05600c3b2800b00397381a7ae8sm5167433wms.30.2022.05.21.15.18.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:18:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 21 May 2022 22:18:46 +0000 Subject: [PATCH v3 02/12] tests: refactor --write-junit-xml code Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The code writing JUnit XML is interspersed directly with all the code in `t/test-lib.sh`, and it is therefore not only ill-separated, but introducing yet another output format would make the situation even worse. Let's introduce an abstraction layer by hiding the JUnit XML code behind four new functions that are supposed to be called before and after each test and test case. This is not just an academic exercise, refactoring for refactoring's sake. We _actually_ want to introduce such a new output format, to make it substantially easier to diagnose test failures in our GitHub workflow, therefore we do need this refactoring. This commit is best viewed with `git show --color-moved --color-moved-ws=allow-indentation-change `. Signed-off-by: Johannes Schindelin --- t/test-lib-junit.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++ t/test-lib.sh | 124 ++++++------------------------------------- 2 files changed, 142 insertions(+), 108 deletions(-) create mode 100644 t/test-lib-junit.sh diff --git a/t/test-lib-junit.sh b/t/test-lib-junit.sh new file mode 100644 index 00000000000..9d55d74d764 --- /dev/null +++ b/t/test-lib-junit.sh @@ -0,0 +1,126 @@ +# Library of functions to format test scripts' output in JUnit XML +# format, to support Git's test suite result to be presented in an +# easily digestible way on Azure Pipelines. +# +# Copyright (c) 2022 Johannes Schindelin +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/ . +# +# The idea is for `test-lib.sh` to source this file when the user asks +# for JUnit XML; these functions will then override (empty) functions +# that are are called at the appropriate times during the test runs. + +start_test_output () { + junit_xml_dir="$TEST_OUTPUT_DIRECTORY/out" + mkdir -p "$junit_xml_dir" + junit_xml_base=${1##*/} + junit_xml_path="$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml" + junit_attrs="name=\"${junit_xml_base%.sh}\"" + junit_attrs="$junit_attrs timestamp=\"$(TZ=UTC \ + date +%Y-%m-%dT%H:%M:%S)\"" + write_junit_xml --truncate "" " " + junit_suite_start=$(test-tool date getnanos) + if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + GIT_TEST_TEE_OFFSET=0 + fi +} + +start_test_case_output () { + junit_start=$(test-tool date getnanos) +} + +finalize_test_case_output () { + test_case_result=$1 + shift + case "$test_case_result" in + ok) + set "$*" + ;; + failure) + junit_insert="" + junit_insert="$junit_insert $(xml_attr_encode \ + "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + test-tool path-utils skip-n-bytes \ + "$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET + else + printf '%s\n' "$@" | sed 1d + fi)")" + junit_insert="$junit_insert" + if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + junit_insert="$junit_insert$(xml_attr_encode \ + "$(cat "$GIT_TEST_TEE_OUTPUT_FILE")")" + fi + set "$1" " $junit_insert" + ;; + fixed) + set "$* (breakage fixed)" + ;; + broken) + set "$* (known breakage)" + ;; + skip) + message="$(xml_attr_encode "$skipped_reason")" + set "$1" " " + ;; + esac + + junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" + shift + junit_attrs="$junit_attrs classname=\"$this_test\"" + junit_attrs="$junit_attrs time=\"$(test-tool \ + date getnanos $junit_start)\"" + write_junit_xml "$(printf '%s\n' \ + " " "$@" " ")" + junit_have_testcase=t +} + +finalize_test_output () { + if test -n "$junit_xml_path" + then + test -n "$junit_have_testcase" || { + junit_start=$(test-tool date getnanos) + write_junit_xml_testcase "all tests skipped" + } + + # adjust the overall time + junit_time=$(test-tool date getnanos $junit_suite_start) + sed -e "s/\(]*/& time=\"$junit_time\"/" \ + -e '/^ *<\/testsuite/d' \ + <"$junit_xml_path" >"$junit_xml_path.new" + mv "$junit_xml_path.new" "$junit_xml_path" + + write_junit_xml " " "" + write_junit_xml= + fi +} + +write_junit_xml () { + case "$1" in + --truncate) + >"$junit_xml_path" + junit_have_testcase= + shift + ;; + esac + printf '%s\n' "$@" >>"$junit_xml_path" +} + +xml_attr_encode () { + printf '%s\n' "$@" | test-tool xml-encode +} diff --git a/t/test-lib.sh b/t/test-lib.sh index f09e8f3efce..bdb11e28eea 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -137,6 +137,12 @@ mark_option_requires_arg () { store_arg_to=$2 } +# These functions can be overridden e.g. to output JUnit XML +start_test_output () { :; } +start_test_case_output () { :; } +finalize_test_case_output () { :; } +finalize_test_output () { :; } + parse_option () { local opt="$1" @@ -196,7 +202,7 @@ parse_option () { tee=t ;; --write-junit-xml) - write_junit_xml=t + . "$TEST_DIRECTORY/test-lib-junit.sh" ;; --stress) stress=t ;; @@ -664,7 +670,7 @@ exec 6<&0 exec 7>&2 _error_exit () { - finalize_junit_xml + finalize_test_output GIT_EXIT_OK=t exit 1 } @@ -774,35 +780,13 @@ trap '{ code=$?; set +x; } 2>/dev/null; exit $code' INT TERM HUP # the test_expect_* functions instead. test_ok_ () { - if test -n "$write_junit_xml" - then - write_junit_xml_testcase "$*" - fi + finalize_test_case_output ok "$@" test_success=$(($test_success + 1)) say_color "" "ok $test_count - $@" } test_failure_ () { - if test -n "$write_junit_xml" - then - junit_insert="" - junit_insert="$junit_insert $(xml_attr_encode \ - "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" - then - test-tool path-utils skip-n-bytes \ - "$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET - else - printf '%s\n' "$@" | sed 1d - fi)")" - junit_insert="$junit_insert" - if test -n "$GIT_TEST_TEE_OUTPUT_FILE" - then - junit_insert="$junit_insert$(xml_attr_encode \ - "$(cat "$GIT_TEST_TEE_OUTPUT_FILE")")" - fi - write_junit_xml_testcase "$1" " $junit_insert" - fi + finalize_test_case_output failure "$@" test_failure=$(($test_failure + 1)) say_color error "not ok $test_count - $1" shift @@ -815,19 +799,13 @@ test_failure_ () { } test_known_broken_ok_ () { - if test -n "$write_junit_xml" - then - write_junit_xml_testcase "$* (breakage fixed)" - fi + finalize_test_case_output fixed "$@" test_fixed=$(($test_fixed+1)) say_color error "ok $test_count - $@ # TODO known breakage vanished" } test_known_broken_failure_ () { - if test -n "$write_junit_xml" - then - write_junit_xml_testcase "$* (known breakage)" - fi + finalize_test_case_output broken "$@" test_broken=$(($test_broken+1)) say_color warn "not ok $test_count - $@ # TODO known breakage" } @@ -1104,10 +1082,7 @@ test_start_ () { test_count=$(($test_count+1)) maybe_setup_verbose maybe_setup_valgrind - if test -n "$write_junit_xml" - then - junit_start=$(test-tool date getnanos) - fi + start_test_case_output } test_finish_ () { @@ -1158,12 +1133,7 @@ test_skip () { case "$to_skip" in t) - if test -n "$write_junit_xml" - then - message="$(xml_attr_encode "$skipped_reason")" - write_junit_xml_testcase "$1" \ - " " - fi + finalize_test_case_output skip "$@" say_color skip "ok $test_count # skip $1 ($skipped_reason)" : true @@ -1179,53 +1149,6 @@ test_at_end_hook_ () { : } -write_junit_xml () { - case "$1" in - --truncate) - >"$junit_xml_path" - junit_have_testcase= - shift - ;; - esac - printf '%s\n' "$@" >>"$junit_xml_path" -} - -xml_attr_encode () { - printf '%s\n' "$@" | test-tool xml-encode -} - -write_junit_xml_testcase () { - junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" - shift - junit_attrs="$junit_attrs classname=\"$this_test\"" - junit_attrs="$junit_attrs time=\"$(test-tool \ - date getnanos $junit_start)\"" - write_junit_xml "$(printf '%s\n' \ - " " "$@" " ")" - junit_have_testcase=t -} - -finalize_junit_xml () { - if test -n "$write_junit_xml" && test -n "$junit_xml_path" - then - test -n "$junit_have_testcase" || { - junit_start=$(test-tool date getnanos) - write_junit_xml_testcase "all tests skipped" - } - - # adjust the overall time - junit_time=$(test-tool date getnanos $junit_suite_start) - sed -e "s/\(]*/& time=\"$junit_time\"/" \ - -e '/^ *<\/testsuite/d' \ - <"$junit_xml_path" >"$junit_xml_path.new" - mv "$junit_xml_path.new" "$junit_xml_path" - - write_junit_xml " " "" - write_junit_xml= - fi -} - test_atexit_cleanup=: test_atexit_handler () { # In a succeeding test script 'test_atexit_handler' is invoked @@ -1248,7 +1171,7 @@ test_done () { # removed, so the commands can access pidfiles and socket files. test_atexit_handler - finalize_junit_xml + finalize_test_output if test -z "$HARNESS_ACTIVE" then @@ -1539,22 +1462,7 @@ fi # in subprocesses like git equals our $PWD (for pathname comparisons). cd -P "$TRASH_DIRECTORY" || exit 1 -if test -n "$write_junit_xml" -then - junit_xml_dir="$TEST_OUTPUT_DIRECTORY/out" - mkdir -p "$junit_xml_dir" - junit_xml_base=${0##*/} - junit_xml_path="$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml" - junit_attrs="name=\"${junit_xml_base%.sh}\"" - junit_attrs="$junit_attrs timestamp=\"$(TZ=UTC \ - date +%Y-%m-%dT%H:%M:%S)\"" - write_junit_xml --truncate "" " " - junit_suite_start=$(test-tool date getnanos) - if test -n "$GIT_TEST_TEE_OUTPUT_FILE" - then - GIT_TEST_TEE_OFFSET=0 - fi -fi +start_test_output "$0" # Convenience # A regexp to match 5 and 35 hexdigits From patchwork Sat May 21 22:18:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BF7FC433F5 for ; Sat, 21 May 2022 22:19:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347182AbiEUWTM (ORCPT ); Sat, 21 May 2022 18:19:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346344AbiEUWTD (ORCPT ); Sat, 21 May 2022 18:19:03 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E58AA4EF7A for ; Sat, 21 May 2022 15:19:02 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id e28so15307994wra.10 for ; Sat, 21 May 2022 15:19:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iGAE3SZGJU7eojvkhG0qydiDez1X+uPg8UYp38uGbF0=; b=pnpgYtzE8eCl+OV5Z9Qsot3aHuK9I6XMOW/tXVdFGMfKAkllTarp/scudzRP9H0ued VPwdEEC6MD1MlkEpNyA0A+t/YNo+pAwNujSTbb9vwsI31hZz95aY8eJXEbR5ZCPR2LjR 8oDwzNeebh2Bv4DCjRy8BQ39RwtHidFQOYsfWjbk3d5Mu7GmLlqAZqyv6jXDebh3pnGY LY+SnkNJe4odJypUtd+kRPYKpnCXb1it5zga0neekjyQyINb96uIKNKWmH6q9QavvHwI kik1cx6gAyt3HxR7amy4fIBINZcrm62J+wHTzL23dCbrYgomVPYdCYHqrDzYComfWQ+C /fow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iGAE3SZGJU7eojvkhG0qydiDez1X+uPg8UYp38uGbF0=; b=shP1R27ZZPo3u7ZJlFyR6e3e0HIfxK3DXbZ22/rqvwk/rXzS+6R+I4c96lm4ivXMcI d8/Utanlij5sPIfDaH5OjNUG+9yb4j5Vh/T4ajtYOTsBo6G24dZDUUX+Incfr4I72D8X lz+7r3eqq7amfVZ60PpDq2U+YYaPqkueldYYEHPIGCgTgQ0mLmSzh4pF/rMY84rWWG6w HxvQf2t9TV/IoSBtjd5osPB6dANhPl61TCexHKaFHZWCLhKo2NHaG+gISLxRE0Nho/rP phLKb6y3lrkCj3r64gEJmOmGfs8mmx4TRVxS348upObolu1j+R8BO7bIObdmgehGYGjb PVxw== X-Gm-Message-State: AOAM533zafciX/cGwFmI30QrowMSCfXPQHUYELBHHL097frxPhJGzlQD kMMaxY5qbYCwogmheBAwAzxwFjJFfGk= X-Google-Smtp-Source: ABdhPJxTDJ4ii3jI9F4cZ4CuUf6SrbDt2lWOtgtamp6IuAzYvc29b/mKFDukgB48zi2tcpTXt9eQVA== X-Received: by 2002:a05:6000:1841:b0:20c:788b:9306 with SMTP id c1-20020a056000184100b0020c788b9306mr13129820wri.369.1653171541147; Sat, 21 May 2022 15:19:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w9-20020adf8bc9000000b0020d07958bb3sm6119835wra.3.2022.05.21.15.19.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:19:00 -0700 (PDT) Message-Id: <30ccd602108fea74d64a5b81e6665fcc7efb6f0a.1653171536.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 22:18:47 +0000 Subject: [PATCH v3 03/12] test(junit): avoid line feeds in XML attributes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin In the test case's output, we do want newline characters, but in the XML attributes we do not want them. However, the `xml_attr_encode` function always adds a Line Feed at the end (which are then encoded as ` `, even for XML attributes. This seems not to faze Azure Pipelines' XML parser, but it still is incorrect, so let's fix it. Signed-off-by: Johannes Schindelin --- t/test-lib-junit.sh | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/t/test-lib-junit.sh b/t/test-lib-junit.sh index 9d55d74d764..c959183c7e2 100644 --- a/t/test-lib-junit.sh +++ b/t/test-lib-junit.sh @@ -50,7 +50,7 @@ finalize_test_case_output () { ;; failure) junit_insert="" + junit_insert="$junit_insert $(xml_attr_encode --no-lf "$1")\">" junit_insert="$junit_insert $(xml_attr_encode \ "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" then @@ -74,12 +74,12 @@ finalize_test_case_output () { set "$* (known breakage)" ;; skip) - message="$(xml_attr_encode "$skipped_reason")" + message="$(xml_attr_encode --no-lf "$skipped_reason")" set "$1" " " ;; esac - junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" + junit_attrs="name=\"$(xml_attr_encode --no-lf "$this_test.$test_count $1")\"" shift junit_attrs="$junit_attrs classname=\"$this_test\"" junit_attrs="$junit_attrs time=\"$(test-tool \ @@ -122,5 +122,11 @@ write_junit_xml () { } xml_attr_encode () { - printf '%s\n' "$@" | test-tool xml-encode + if test "x$1" = "x--no-lf" + then + shift + printf '%s' "$*" | test-tool xml-encode + else + printf '%s\n' "$@" | test-tool xml-encode + fi } From patchwork Sat May 21 22:18:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28D29C433EF for ; Sat, 21 May 2022 22:19:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347184AbiEUWTQ (ORCPT ); Sat, 21 May 2022 18:19:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347077AbiEUWTF (ORCPT ); Sat, 21 May 2022 18:19:05 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 061164EF53 for ; Sat, 21 May 2022 15:19:04 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id w4so15875671wrg.12 for ; Sat, 21 May 2022 15:19:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IALL0QMC0v8ysobvDm4KtZMwa6z5A/re01nuKPjfC2M=; b=SmCjqgVfXftHeCK7Ug1T6doqz7axxkZFem0AiCG7JV2ZTI/ts2JJzlWJ5e9uBJNaJR CQ+3XsR+VmK/SynE9qZ7ieu6NLZPgKyA4kZjH8U9dmt/Hi0lw7KdSGOhwyHEXfn0DiSq xYuOqCsLh/+wh83mA7GR5hUTF8E/1iHoc6nlwJLtJ0sMrjG7BXjeV5PalawVwpXJJUAb cBIrbtd+wJ0z1xDojqAfUpf8MSo/MVT6HhPXelb3pX1a9ySDl+pnxY/MUNiO4fdNzsv6 ePwsWbngvwrqDA1ONfj+FU4UHhTwfz18G4lC7n6v/bAJUiupf3tL7zLMJ8v192+HZnD1 RoJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IALL0QMC0v8ysobvDm4KtZMwa6z5A/re01nuKPjfC2M=; b=irZRBr4yt+kX2WP0Uy6kLlSHe9S/fhfJ10rqpspHB0MYwh68nnyOXEcqpgJ4MUfAY7 pmkJIV6JkL/7Gfg4RlqSILv4TVGu+tsMHsJhMvm3AQOAADSkJMtTKRryu27Raq3CXooq 8eT/uh30JJX28cEbAbAzsKV5o39Vx7e1X0Mpe7YZ4nVTg0VEWyIiJAKydmzCQl3KcFX+ hQUJe0Z8PgUb5p1lV2JyEUJ9sVU5mgCPT1DaLQl1NnJo7wBtrYaIjAGK9uhZVZ/Jpk6+ t+8wkoGqJpdMxwAXYIMlryguOhDhQSSF3xmHo4sYdTwNRBvSQ+OEs/d35S8UQzuiGK+d fa/Q== X-Gm-Message-State: AOAM5320M7gBQzN58B3EV0I2h3Fq04QyyUpONylmc2gjZ9cFud+Puat0 3uGaiCE1fukGg7mmv4UZ6P2XEWiXj3E= X-Google-Smtp-Source: ABdhPJwIVZd4ORzpGg/bdbaENaxOQFONEmrTNBZBNpmiB1po9DBKrS+Dysn5GD7rgsMvLgXIcBpJmA== X-Received: by 2002:adf:f611:0:b0:20d:1317:5bae with SMTP id t17-20020adff611000000b0020d13175baemr13056841wrp.359.1653171542144; Sat, 21 May 2022 15:19:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g10-20020adfa58a000000b0020e60baffd1sm2702432wrc.52.2022.05.21.15.19.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:19:01 -0700 (PDT) Message-Id: <8f5b112bd08dcaf63ea764be71a17942c184de0e.1653171536.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 22:18:48 +0000 Subject: [PATCH v3 04/12] ci/run-build-and-tests: take a more high-level view Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin In the web UI of GitHub workflows, failed runs are presented with the job step that failed auto-expanded. In the current setup, this is not helpful at all because that shows only the output of `prove`, which says which test failed, but not in what way. What would help understand the reader what went wrong is the verbose test output of the failed test. The logs of the failed runs do contain that verbose test output, but it is shown in the _next_ step (which is marked as succeeding, and is therefore _not_ auto-expanded). Anyone not intimately familiar with this would completely miss the verbose test output, being left mostly puzzled with the test failures. We are about to show the failed test cases' output in the _same_ step, so that the user has a much easier time to figure out what was going wrong. But first, we must partially revert the change that tried to improve the CI runs by combining the `Makefile` targets to build into a single `make` invocation. That might have sounded like a good idea at the time, but it does make it rather impossible for the CI script to determine whether the _build_ failed, or the _tests_. If the tests were run at all, that is. So let's go back to calling `make` for the build, and call `make test` separately so that we can easily detect that _that_ invocation failed, and react appropriately. Signed-off-by: Johannes Schindelin --- ci/run-build-and-tests.sh | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 280dda7d285..2818b3046ae 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -10,7 +10,7 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; *) ln -s "$cache_dir/.prove" t/.prove;; esac -export MAKE_TARGETS="all test" +run_tests=t case "$jobname" in linux-gcc) @@ -41,14 +41,15 @@ pedantic) # Don't run the tests; we only care about whether Git can be # built. export DEVOPTS=pedantic - export MAKE_TARGETS=all + run_tests= ;; esac -# Any new "test" targets should not go after this "make", but should -# adjust $MAKE_TARGETS. Otherwise compilation-only targets above will -# start running tests. -make $MAKE_TARGETS +make +if test -n "$run_tests" +then + make test +fi check_unignored_build_artifacts save_good_tree From patchwork Sat May 21 22:18:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB1EFC433F5 for ; Sat, 21 May 2022 22:19:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347194AbiEUWTS (ORCPT ); Sat, 21 May 2022 18:19:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347080AbiEUWTG (ORCPT ); Sat, 21 May 2022 18:19:06 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C7214EF60 for ; Sat, 21 May 2022 15:19:05 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id s28so15780943wrb.7 for ; Sat, 21 May 2022 15:19:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ganxqkS9kK8VPMrpMbSNTO8ET3r2m3OId4BMtNcMrsg=; b=YmhQYkfkVFq5YIbWw3S8n8kkavBojkXZUQ1jdGtwQVfo/ILrKg8MM3XyToL0gz7zdb rDA+o+0i31v3Uz9GN0EiCox6np+tPbdIo9fLoVXkiHxwGKot5t/TuzeQvZ29t4MYvw0M TW+uBwtAFdU83bXrRO2SVgTIWABjDerptr/XKjNxCXl8IycD0Rfb2fRgT/PE4tdojTdR bnlMDmYt6jUhtFKMDP/t0pzSfpseu/inqxgeQtMYyucx/x1+pXLDvLOHrY8QLbRWzEbg yqw442de1eBUXqD4pheufnGb7izF9Gu9j87Xi4fPM1AvtAi0vygyacLKF+Xud4VoeGqd amrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ganxqkS9kK8VPMrpMbSNTO8ET3r2m3OId4BMtNcMrsg=; b=mnCdqRo8zaYe0nC8CKpNU40OgFGljm4oXnKtpNC/WutVp35sMI78Y42Ra8++Fb9LzK YB6COVMCdJPcy82wlqbKLMsG4dg7OqrepFfRzj/xvdMvplXf9Icib5jFp8ZNmlrzUfpB 8U07p0dzt/ItMr7GFcAo5RknriUM14sLC3fCnaIQLY2Mewwn/MHioGzvCOUPpW6swYVb Qpg2BDxHfiG/s67zQKCMfpPGzhkx28CpBgJbp6G7MTadcqBNnhrt+URxNXR8qqOmApQu MD78GJ8iAuOYmZT/hpNAQI+NxMfJNrCQ6P20gFN+8pSsYmlqzYggwq0CHf/WRS0erUWs EZmg== X-Gm-Message-State: AOAM530g1pukY0DeG4i+CoOkyuPtR7Myc8rsu5cZtxnhXxHPFjmh7NLA Aj2ulJYdWOqW49F0yNL1pwYDoINJxxM= X-Google-Smtp-Source: ABdhPJxXtz4cUakv5VWLIJ34Pqc4UM69e1QQOLDyS8vKTrGIW8csogcwpxxmkXcuk9IlY7CHgRrCrA== X-Received: by 2002:adf:f90f:0:b0:20e:5fd4:5d06 with SMTP id b15-20020adff90f000000b0020e5fd45d06mr13826147wrr.371.1653171543350; Sat, 21 May 2022 15:19:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u6-20020adfc646000000b0020d0c48d135sm6243854wrg.15.2022.05.21.15.19.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:19:02 -0700 (PDT) Message-Id: <417f702a2454316f0bdc5b66eb16751272ff2989.1653171536.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 22:18:49 +0000 Subject: [PATCH v3 05/12] ci: make it easier to find failed tests' logs in the GitHub workflow Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When investigating a test failure, the time that matters most is the time it takes from getting aware of the failure to displaying the output of the failing test case. You currently have to know a lot of implementation details when investigating test failures in the CI runs. The first step is easy: the failed job is marked quite clearly, but when opening it, the failed step is expanded, which in our case is the one running `ci/run-build-and-tests.sh`. This step, most notably, only offers a high-level view of what went wrong: it prints the output of `prove` which merely tells the reader which test script failed. The actually interesting part is in the detailed log of said failed test script. But that log is shown in the CI run's step that runs `ci/print-test-failures.sh`. And that step is _not_ expanded in the web UI by default. It is even marked as "successful", which makes it very easy to miss that there is useful information hidden in there. Let's help the reader by showing the failed tests' detailed logs in the step that is expanded automatically, i.e. directly after the test suite failed. This also helps the situation where the _build_ failed and the `print-test-failures` step was executed under the assumption that the _test suite_ failed, and consequently failed to find any failed tests. An alternative way to implement this patch would be to source `ci/print-test-failures.sh` in the `handle_test_failures` function to show these logs. However, over the course of the next few commits, we want to introduce some grouping which would be harder to achieve that way (for example, we do want a leaner, and colored, preamble for each failed test script, and it would be trickier to accommodate the lack of nested groupings in GitHub workflows' output). Signed-off-by: Johannes Schindelin --- .github/workflows/main.yml | 12 ------------ ci/lib.sh | 23 +++++++++++++++++++++++ ci/run-build-and-tests.sh | 3 ++- ci/run-test-slice.sh | 3 ++- 4 files changed, 27 insertions(+), 14 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c35200defb9..3fa88b78b6d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -119,10 +119,6 @@ jobs: - name: test shell: bash run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 - - name: ci/print-test-failures.sh - if: failure() - shell: bash - run: ci/print-test-failures.sh - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v2 @@ -204,10 +200,6 @@ jobs: env: NO_SVN_TESTS: 1 run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 - - name: ci/print-test-failures.sh - if: failure() - shell: bash - run: ci/print-test-failures.sh - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v2 @@ -261,8 +253,6 @@ jobs: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - run: ci/run-build-and-tests.sh - - run: ci/print-test-failures.sh - if: failure() - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v2 @@ -292,8 +282,6 @@ jobs: - uses: actions/checkout@v1 - run: ci/install-docker-dependencies.sh - run: ci/run-build-and-tests.sh - - run: ci/print-test-failures.sh - if: failure() - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v1 diff --git a/ci/lib.sh b/ci/lib.sh index d718f4e386d..65f5188a550 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -78,6 +78,10 @@ check_unignored_build_artifacts () { } } +handle_failed_tests () { + return 1 +} + # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} @@ -123,6 +127,25 @@ then CI_JOB_ID="$GITHUB_RUN_ID" CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t + handle_failed_tests () { + mkdir -p t/failed-test-artifacts + echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV + + for test_exit in t/test-results/*.exit + do + test 0 != "$(cat "$test_exit")" || continue + + test_name="${test_exit%.exit}" + test_name="${test_name##*/}" + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" + cat "t/test-results/$test_name.out" + + trash_dir="t/trash directory.$test_name" + cp "t/test-results/$test_name.out" t/failed-test-artifacts/ + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + done + return 1 + } cache_dir="$HOME/none" diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 2818b3046ae..1ede75e5556 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -48,7 +48,8 @@ esac make if test -n "$run_tests" then - make test + make test || + handle_failed_tests fi check_unignored_build_artifacts diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index f8c2c3106a2..63358c23e11 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -12,6 +12,7 @@ esac make --quiet -C t T="$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | - tr '\n' ' ')" + tr '\n' ' ')" || +handle_failed_tests check_unignored_build_artifacts From patchwork Sat May 21 22:18:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6FC8C433EF for ; Sat, 21 May 2022 22:19:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347445AbiEUWTa (ORCPT ); Sat, 21 May 2022 18:19:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347111AbiEUWTH (ORCPT ); Sat, 21 May 2022 18:19:07 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5340A4EF69 for ; Sat, 21 May 2022 15:19:06 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id m20so4776127wrb.13 for ; Sat, 21 May 2022 15:19:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3L0yUdu/rjFTtYj9ucbGZSQMuykrsevUUqWmK3QrWro=; b=UHvRGsd9/Ie5hVfDzHx/HCxa4Wr8ZwklUyklj+wO/2jFIwxgL5SwkhXj4XF8dumffL Fki4EKDRsnmj3i+1THitCnRCMEb4r2/4DqGD+MNtQxX+sPudHl6ghG2gb373xlWiuWu9 R/dvyPCUgO7eMypGthcYaxCO/3Ri/+LIUX25EJuCgqrji2jtvm8tHtAZw50IBlYRFpGn CEKO08CmZXtK+GvMakl+1SB78kTIXP9Mzcjh4XuHTEtmfKN0JUWHlLzQ64ZRULCHKUgu VIDRXBqgw1EmCOPd8aigPcIHbePhjEBs07Z8CtustiV9j0SSCqpz81ofSQVlNskRnPuw k3Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3L0yUdu/rjFTtYj9ucbGZSQMuykrsevUUqWmK3QrWro=; b=HXT4Cx1XGr93Yvkd4hdxE9Sf8XeQHRnLjmLZu+JZbvql+DzlndyKRm/NvzW/e4fMpq OUK2OQimJcv1YKPIDPZ2DnU+hzdHWOnH/ycd+5bIJDk1/jv+8h0NquNOfL9nNAgPRF/b B6PCLUIUJ78UmYKi+pRAggGZzZSbeH1f/NDME5C4LfnCaJuL9raR4JKDbyB9W96ure9Y M/CUK+kqN6ui1/4Tig/Q3LSNtTtOfj3dxfgkLEQmL+EwLrpHNYtwj414RtNBPZ5eKHYO b5fxNEeQysh2dmlepoTdZ0pqrA9U08HZUk4AZF1WSqXU1RxEPU9XuVhOz0bIo6oKVcpl cnJg== X-Gm-Message-State: AOAM533IWXRa/qvmvWP7hfHFDK77B1VSJfOtNfbSiMTgLZ2WIOcc0OTa y5cEYxPasdz1NnQennzQgaXW1vK7q6k= X-Google-Smtp-Source: ABdhPJzErT/SG/9JqnY1trK1rAWqSTEfeJwDsK1oHdxemCWL4N3bD7ErpDPLVU4s16AeaidDu887jQ== X-Received: by 2002:a5d:6d8f:0:b0:20c:7fb7:d59c with SMTP id l15-20020a5d6d8f000000b0020c7fb7d59cmr14225921wrs.77.1653171544484; Sat, 21 May 2022 15:19:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u9-20020adfb209000000b0020c7ec0fdf4sm6983049wra.117.2022.05.21.15.19.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:19:04 -0700 (PDT) Message-Id: <7d2284314efc0071ad9cc9b8f2559e07310e35b0.1653171536.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 22:18:50 +0000 Subject: [PATCH v3 06/12] ci/run-build-and-tests: add some structure to the GitHub workflow output Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The current output of Git's GitHub workflow can be quite confusing, especially for contributors new to the project. To make it more helpful, let's introduce some collapsible grouping. Initially, readers will see the high-level view of what actually happened (did the build fail, or the test suite?). To drill down, the respective group can be expanded. Note: sadly, workflow output currently cannot contain any nested groups (see https://github.com/actions/runner/issues/802 for details), therefore we take pains to ensure to end any previous group before starting a new one. Signed-off-by: Johannes Schindelin --- ci/lib.sh | 56 ++++++++++++++++++++++++++++++++++----- ci/run-build-and-tests.sh | 4 +-- ci/run-test-slice.sh | 2 +- 3 files changed, 52 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 65f5188a550..f8cb79e44f0 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,5 +1,50 @@ # Library of functions shared by all CI scripts +if test true != "$GITHUB_ACTIONS" +then + begin_group () { :; } + end_group () { :; } + + group () { + shift + "$@" + } + set -x +else + begin_group () { + need_to_end_group=t + echo "::group::$1" >&2 + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + echo '::endgroup::' >&2 + } + trap end_group EXIT + + group () { + set +x + begin_group "$1" + shift + "$@" + res=$? + end_group + return $res + } + + begin_group "CI setup" +fi + +# Set 'exit on error' for all CI scripts to let the caller know that +# something went wrong. +# +# We already enabled tracing executed commands earlier. This helps by showing +# how # environment variables are set and and dependencies are installed. +set -e + skip_branch_tip_with_tag () { # Sometimes, a branch is pushed at the same time the tag that points # at the same commit as the tip of the branch is pushed, and building @@ -88,12 +133,6 @@ export TERM=${TERM:-dumb} # Clear MAKEFLAGS that may come from the outside world. export MAKEFLAGS= -# Set 'exit on error' for all CI scripts to let the caller know that -# something went wrong. -# Set tracing executed commands, primarily setting environment variables -# and installing dependencies. -set -ex - if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines @@ -138,7 +177,7 @@ then test_name="${test_exit%.exit}" test_name="${test_name##*/}" printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" - cat "t/test-results/$test_name.out" + group "Failed test: $test_name" cat "t/test-results/$test_name.out" trash_dir="t/trash directory.$test_name" cp "t/test-results/$test_name.out" t/failed-test-artifacts/ @@ -233,3 +272,6 @@ linux-leaks) esac MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" + +end_group +set -x diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 1ede75e5556..7abfa00adc0 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -45,10 +45,10 @@ pedantic) ;; esac -make +group Build make if test -n "$run_tests" then - make test || + group "Run tests" make test || handle_failed_tests fi check_unignored_build_artifacts diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index 63358c23e11..a3c67956a8d 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -10,7 +10,7 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; *) ln -s "$cache_dir/.prove" t/.prove;; esac -make --quiet -C t T="$(cd t && +group "Run tests" make --quiet -C t T="$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | tr '\n' ' ')" || handle_failed_tests From patchwork Sat May 21 22:18:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12858055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1FCBC433F5 for ; Sat, 21 May 2022 22:19:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347497AbiEUWTb (ORCPT ); Sat, 21 May 2022 18:19:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347118AbiEUWTI (ORCPT ); Sat, 21 May 2022 18:19:08 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CB9A4F462 for ; Sat, 21 May 2022 15:19:07 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id f2so15949280wrc.0 for ; Sat, 21 May 2022 15:19:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IcC8TjxkAxolQXyBFCqHk/x+zhXALadYy+cGgGUwAt4=; b=qtwofsR1WSXI6OcrMGPsQO/UWeoFPLrZfEMaDW530Kazjas06l5gw/A0ZBI30JulPK GX7o2xwQBG39dXrJZnzJ4RCbj0mkFkNtkty2BV7EgHIiTqF4ePbopuvOvErh9WW22+8p wvWtzVNmOq6lY8ZwQgoofhErahXQutKKVXdJRpTjr/5bC16pxCHqpYhvtSQateXUbG6f xKYxauE06OTE8Kz/DgFtG5A43Zef5I25mR9WMOBd3XS+Q4tR8nc1XikYC71Wqm+NSwo8 RGYVUYw8CYb5ky2in68Du75xD/7Tb9RKCBZOYU1HVf2fXbqHT4ecrlPRSdPsPCNl7vhN 60GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IcC8TjxkAxolQXyBFCqHk/x+zhXALadYy+cGgGUwAt4=; b=yyXprmWZ9N1u6ewro+aciw7y/pm9U+AmkiP1HGIP11ZO6+CRPiKTUjR6pzcSKVt6h3 tB1+m2ZQejcIzQ8LdE2Iz6q41jq8tXz5I/MSOeEAYVsBQGufO+PbvaifIqW6LdG2oZga YbFHkwmYHBB29VXFrVCaA3NX6VLv2J4FUzZoqJmILQKLch+8JQnLa4boJMqzbJf804uM xQNKw3khB1HZ1DziBrbiDLsgxbm3EFSWZVTiZDh68UIrKXeWDCBHON5TjNXC3yRGZTkx hDMpU8xgijKLj9DcT8C1YvXHuL7DSeI2RixVO4TBjPrpx1fPU89tj6AgX5by82DskV8K IL2w== X-Gm-Message-State: AOAM531mNsNSH8WplCehwvdHpISE+g4DMjDa2az/XzyK797Mx/evlIFZ YY69FsEYpKhbp/yPl/II88Ukm6RGi6A= X-Google-Smtp-Source: ABdhPJwQtCC62Fp//uzyBdT82eCRh0FysO6GHItCZknB4V9fHeMAEg7U0iLbYLO8axZdvJZUFDoJYA== X-Received: by 2002:a5d:598a:0:b0:20e:7211:9681 with SMTP id n10-20020a5d598a000000b0020e72119681mr10949729wri.572.1653171545633; Sat, 21 May 2022 15:19:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k5-20020adfb345000000b0020d09f0b766sm6056937wrd.71.2022.05.21.15.19.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 15:19:05 -0700 (PDT) Message-Id: <98059b94a881485f763efc54346e8b19103ee64b.1653171536.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 22:18:51 +0000 Subject: [PATCH v3 07/12] ci: optionally mark up output in the GitHub workflow Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Phillip Wood , Victoria Dye , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin A couple of commands exist to spruce up the output in GitHub workflows: https://docs.github.com/en/actions/learn-github-actions/workflow-commands-for-github-actions In addition to the `::group::