From patchwork Wed Nov 1 13:02:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442810 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0B53125A4 for ; Wed, 1 Nov 2023 13:02:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="mWzW3I/f"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Dd/3JS7T" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 749C710C for ; Wed, 1 Nov 2023 06:02:49 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 0F2815C0429; Wed, 1 Nov 2023 09:02:47 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 01 Nov 2023 09:02:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843767; x=1698930167; bh=Nn BHNOi9hucY5UlSonC3hhtnCN1xzlDiHCccJM2fuWE=; b=mWzW3I/f9QA22qWdn6 bNxH2JgTGOM/WA0rf3PZfNAk+HClww+F64a90C+H8FOfVgFfs7PVuX1B2z2mibv5 d/loSn6KTci6zSYxsUldHs0rQ7wo3MjmLm/cOPiODfHfAmgfm+mzmtE94s3Tra2E RUj6E8e7BhJUONsnMR1HLt3IscO+gcVgy3YhYdHjqjTKO+s7sgE11kBNGGQ/Ygb7 UUJaqjDZyJeDgll+hAY+K2xh4L+KGoGQJLFJLOJwcnM6mKg1C+SuXhQvR9CfWyxi 92hoeeUA/5ZWkG6LSGealm40ywSN4x2MD7ADRKZP03w8P3/X1DlvMEiJlkaPJxpE HzWQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843767; x=1698930167; bh=NnBHNOi9hucY5 UlSonC3hhtnCN1xzlDiHCccJM2fuWE=; b=Dd/3JS7TGOTG9kN3kFx4DFpeBDY9R ttpd69Is6TLNqixDhwG2aD4ycWVNC+ShaOWFH2u/4izaCJBg7VHbRdN3rHs6YErm rn/8A9jD+Qys2zdrHJvRSPIR9bR8Dj6He6hvVbGhpngCuCvq2sNkBCButqO+UmM5 fyKTpA5Nc0aUfy4vwTTz4Gk2TbGF/jp3YSreyXAFIsCAHlt4a5m9LLV8VRpJHABu Sa9IVuXf57ohScUpHN35yKVyRnmkwF8nx0+ZGLEAJYzPUXImEhxuELbqJRP0wOTU 6AMnUJAxv5230jql97Btplp8hHLRc3XekUK09MDCvpn4l32Lg9/scIIlA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:02:45 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 1e736688 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:34 +0000 (UTC) Date: Wed, 1 Nov 2023 14:02:43 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 1/8] ci: reorder definitions for grouping functions Message-ID: <0ba396f2a331ce1e9f7e7122e13dc683e3ecc752.1698843660.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We define a set of grouping functions that are used to group together output in our CI, where these groups then end up as collapsible sections in the respective pipeline platform. The way these functions are defined is not easily extensible though as we have an up front check for the CI _not_ being GitHub Actions, where we define the non-stub logic in the else branch. Reorder the conditional branches such that we explicitly handle GitHub Actions. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 6fbb5bade12..eb384f4e952 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,16 +1,7 @@ # Library of functions shared by all CI scripts -if test true != "$GITHUB_ACTIONS" +if test true = "$GITHUB_ACTIONS" then - begin_group () { :; } - end_group () { :; } - - group () { - shift - "$@" - } - set -x -else begin_group () { need_to_end_group=t echo "::group::$1" >&2 @@ -42,6 +33,15 @@ else } begin_group "CI setup" +else + begin_group () { :; } + end_group () { :; } + + group () { + shift + "$@" + } + set -x fi # Set 'exit on error' for all CI scripts to let the caller know that From patchwork Wed Nov 1 13:02:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442811 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7DD1125D1 for ; Wed, 1 Nov 2023 13:02:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="OqqEWQ1E"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="D1SYxj+t" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B52E0F4 for ; Wed, 1 Nov 2023 06:02:51 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 2EA425C04A7; Wed, 1 Nov 2023 09:02:51 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Wed, 01 Nov 2023 09:02:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843771; x=1698930171; bh=DK cHsghbITCg5CVzN6KHVu0FC1otSQM+4z7ZaXo5Wug=; b=OqqEWQ1EiJ04BSxbBw Da+YBY6eBwqXSJZ+qX+E9MKXQd8u6eRUwYB3iYFPV7CTKeFAXIX5keEI6k1sWlXi GvJVebfJnmbhz35n/0LIq1JFwu4eRE7OjYqCgr61GvqP5mzXurGdG/uysVobBdcI kcqyHPDAwiZE8R+M5w/DG9b5B8YpQ8Od2hepXGjwU0HnLb+uKmgafyv7r9hPROjn ROhbBsQUj6cRBjtibefPNLuTNyutGMXYUz8wd5vPyUgEde+ldvcev3BrZRPyVwgN qQAC+EL/2C1FIH08pkQo/RYokMbBArNEeIJP9nM9tQj8a+829U+dltg8XxBTLnq3 8xmw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843771; x=1698930171; bh=DKcHsghbITCg5 CVzN6KHVu0FC1otSQM+4z7ZaXo5Wug=; b=D1SYxj+t3oE5gvdVCwk8ypYVyFIKm kg8xg2q4s4LIXohhMEmksItFVL+zLEbz5aP8zV5HKfoN9RjgkPh8W2o4TV1hkwY1 hvJjpS0veCqUFT7/J192ZI1eTNROe2aLccFbSLhRz4PlvNgFt+ldxnDVhJVkzANg Pce+uQ4YvwBPPi+el4pnBBWwYwSVmNWEHItF4lN9GEb1i6rhIkf+pegCaTvC7n6X RxNmwLcPCpUwpcO7TQVRcA6nQl/3Lq6vvspY/Ts4pEmnTS/+2SoDVqbYnG2npyD4 TORY2HKsjEI7lsRivhH4COwDvDc5TLHaW18AX7LFcHGouJB/93MWWyTyw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:02:49 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 3321c172 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:39 +0000 (UTC) Date: Wed, 1 Nov 2023 14:02:47 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 2/8] ci: make grouping setup more generic Message-ID: <821cfcd61254ace6981288a9ad23999502892873.1698843660.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Make the grouping setup more generic by always calling `begin_group ()` and `end_group ()` regardless of whether we have stubbed those functions or not. This ensures we can more readily add support for additional CI platforms. Furthermore, the `group ()` function is made generic so that it is the same for both GitHub Actions and for other platforms. There is a semantic conflict here though: GitHub Actions used to call `set +x` in `group ()` whereas the non-GitHub case unconditionally uses `set -x`. The latter would get overriden if we kept the `set +x` in the generic version of `group ()`. To resolve this conflict, we simply drop the `set +x` in the generic variant of this function. As `begin_group ()` calls `set -x` anyway this is not much of a change though, as the only commands that aren't printed anymore now are the ones between the beginning of `group ()` and the end of `begin_group ()`. Last, this commit changes `end_group ()` to also accept a parameter that indicates _which_ group should end. This will be required by a later commit that introduces support for GitLab CI. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 46 ++++++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index eb384f4e952..b3411afae8e 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,36 +14,34 @@ then need_to_end_group= echo '::endgroup::' >&2 } - trap end_group EXIT - - group () { - set +x - begin_group "$1" - shift - # work around `dash` not supporting `set -o pipefail` - ( - "$@" 2>&1 - echo $? >exit.status - ) | - sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' - res=$(cat exit.status) - rm exit.status - end_group - return $res - } - - begin_group "CI setup" else begin_group () { :; } end_group () { :; } - group () { - shift - "$@" - } set -x fi +group () { + group="$1" + shift + begin_group "$group" + + # work around `dash` not supporting `set -o pipefail` + ( + "$@" 2>&1 + echo $? >exit.status + ) | + sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' + res=$(cat exit.status) + rm exit.status + + end_group "$group" + return $res +} + +begin_group "CI setup" +trap "end_group 'CI setup'" EXIT + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # @@ -287,5 +285,5 @@ esac MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" -end_group +end_group "CI setup" set -x From patchwork Wed Nov 1 13:02:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442813 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 775D912B9F for ; Wed, 1 Nov 2023 13:03:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="z0eHxd5E"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="WIcfiv1j" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 708F5B7 for ; Wed, 1 Nov 2023 06:02:56 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id DC6105C026B; Wed, 1 Nov 2023 09:02:55 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 01 Nov 2023 09:02:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843775; x=1698930175; bh=35 5BVwRQsgtIKi62U4dRA0FroUzxmbsaxHR7YF2U0wo=; b=z0eHxd5EG1V7Tq9qMi uehhDfSpciU8XMySXsAEVtNQoe6K94vMmB+AgZal94IJ1NsDzrbxWBKykVw/aXWg wWWE4vl+IRWz7ycJodYHB5Z+of+4STV64TCG3a0pkCobm+MAd6k4op9SkLNAWT7N LHDYw0s2AafyZ12D/dEWwtmtUENj7F5bMYNLSmA7dJJMu+4pZh7cwoGSb0JN8HZ3 qi3oewwQdIfU3ExpHFoHEan54pkeRgvd5DtylRnXC3z1uC9tjiZ+kzzSBr5/PdHI 9/upj/6mP9Dgrx8jWtebPrCVQE+O3CK+oYLZogmvVH5NaFakPgNL1Q2Qst5CnRZV rf9A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843775; x=1698930175; bh=355BVwRQsgtIK i62U4dRA0FroUzxmbsaxHR7YF2U0wo=; b=WIcfiv1j4aDxgDBfP0OpbvR1SEC1E UmSxLTSOXdwJOV+Xuy4NCHvtyXSb9nwKXS+q1q29+GrQVHnPKEhrg1PVjR4pCU02 w+3SkwZ8dKeDmNMjx2YeXcfE4Ish8jMOcXkdabv29qLa956J/dLigLZ2xxTSjPHN uh1NlnJQ5QVzW3zsuX49jKHe267g+TZfoHxDtnj3BoN/hyOj1hakAqoZ23W7Ep/Y cNL7igNNFCzL/umxG2ZPc0dXGpTVsD6DUedeIKBDurz96p0r9LXrgzhwtxRuQidR opLXOh2hZR2+VIa40hOMinfxap+PZCAiuNL6R2loaywHQkbvO8m8Pa1QQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:02:54 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 8d69b092 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:43 +0000 (UTC) Date: Wed, 1 Nov 2023 14:02:52 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 3/8] ci: group installation of Docker dependencies Message-ID: <6e5bcf143c86650f153de63b0438c9e09c9dc005.1698843660.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The output of CI jobs tends to be quite long-winded and hard to digest. To help with this, many CI systems provide the ability to group output into collapsible sections, and we're also doing this in some of our scripts. One notable omission is the script to install Docker dependencies. Address it to bring more structure to the output for Docker-based jobs. Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 78b7e326da6..d0bc19d3bb3 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -3,6 +3,10 @@ # Install dependencies required to build and test Git inside container # +. ${0%/*}/lib.sh + +begin_group "Install dependencies" + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -20,3 +24,5 @@ pedantic) dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ;; esac + +end_group "Install dependencies" From patchwork Wed Nov 1 13:02:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442814 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA02C134A0 for ; Wed, 1 Nov 2023 13:03:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="DCtDHzwk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="kQvLYYSd" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49F7FDC for ; Wed, 1 Nov 2023 06:03:00 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id A79195C0269; Wed, 1 Nov 2023 09:02:59 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 01 Nov 2023 09:02:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843779; x=1698930179; bh=TN gHE3eW3QgP29f/nRrshY84o6/TYHcWEg//ZXA83nc=; b=DCtDHzwkj6Z30MtDgg uYumjiykYxcpevdGoaLUW1LDPAiPWsVndTSs9mW8PdIzG9uD8sGWujJR5Kk6xopE 3xCNcy+blXO4OeyTsjNCyxOA2dfNgJcwrvNK263a+/U2wJUuj9Q3/W97nE2wMngx ypgn6ewj+J3+SbB01Udhg+9+tjeZFyA7Tqf2CqmCFq1I0QT/6rP85Z9Y2/rw1E7r 9OyzibJtnefc8Vcp/LlOlkd61q4eP7NQnL1155Kr3TVo0XfcnkQwykfFWJXymN+0 lVujqi6JPtOeAIGxeullDtcNnZTorm1NMgCUuKc+wefN3AeROg01ZXegScz5aeVf /Bog== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843779; x=1698930179; bh=TNgHE3eW3QgP2 9f/nRrshY84o6/TYHcWEg//ZXA83nc=; b=kQvLYYSdTc0ir7CacVNTV0rBIOIOq DwHpp75w4+F9qZYileTGfgj9nPnj1DWuUUUsgemfkzUwI9GzBtECWdz5GF1LaB2w NLOjNK5LXLkkBHpvjhA10LrfFbIpdMnrE4IPfXxPKt+ewqAQlT0WkBxdfy4pjVRu WoB3sy5vnHI0/fRanNzv2UZNjyle86QpHTNCWsHntDgoE4rsXSTW6bXbIjxZ+NV1 g43/4PLAuRuGknp5WAAOx6xjqpPUu7fzVFBqf3Iggl7ZjYI+hIVDrkkoD0WPkQAI yUAlWRFo2hVzChNxQqJYDHzPjKnYwv9/4Su438kl5GKz3PNltDLo6fZCw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:02:58 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 1987f110 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:47 +0000 (UTC) Date: Wed, 1 Nov 2023 14:02:56 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 4/8] ci: split out logic to set up failed test artifacts Message-ID: <2182acf5bfce5abf00287b035d321a9cb5b18941.1698843660.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have some logic in place to create a directory with the output from failed tests, which will then subsequently be uploaded as CI artifacts. We're about to add support for GitLab CI, which will want to reuse the logic. Split the logic into a separate function so that it is reusable. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index b3411afae8e..9ffdf743903 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -131,6 +131,27 @@ handle_failed_tests () { return 1 } +create_failed_test_artifacts () { + mkdir -p t/failed-test-artifacts + + for test_exit in t/test-results/*.exit + do + test 0 != "$(cat "$test_exit")" || continue + + test_name="${test_exit%.exit}" + test_name="${test_name##*/}" + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" + echo "The full logs are in the 'print test failures' step below." + echo "See also the 'failed-tests-*' artifacts attached to this run." + cat "t/test-results/$test_name.markup" + + trash_dir="t/trash directory.$test_name" + cp "t/test-results/$test_name.out" t/failed-test-artifacts/ + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + done + return 1 +} + # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} @@ -171,25 +192,8 @@ then CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t handle_failed_tests () { - mkdir -p t/failed-test-artifacts echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV - - for test_exit in t/test-results/*.exit - do - test 0 != "$(cat "$test_exit")" || continue - - test_name="${test_exit%.exit}" - test_name="${test_name##*/}" - printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" - echo "The full logs are in the 'print test failures' step below." - echo "See also the 'failed-tests-*' artifacts attached to this run." - cat "t/test-results/$test_name.markup" - - trash_dir="t/trash directory.$test_name" - cp "t/test-results/$test_name.out" t/failed-test-artifacts/ - tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" - done - return 1 + create_failed_test_artifacts } cache_dir="$HOME/none" From patchwork Wed Nov 1 13:03:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442815 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65EF412E44 for ; Wed, 1 Nov 2023 13:03:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="vZpsxjdM"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="bKCak9gF" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D74A2103 for ; Wed, 1 Nov 2023 06:03:04 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 52E265C0429; Wed, 1 Nov 2023 09:03:04 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Wed, 01 Nov 2023 09:03:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843784; x=1698930184; bh=38 oBy1IBjhcoP6a36VWXeV9rcI0uvMGQM9wk0+CFs8Q=; b=vZpsxjdMCurDyiKsiW pAiWFKHE67VIvDEuQTcSIXN2ZEMKwvSauPKHyLdIIE28wnADFL9LdQffNyHQfsfJ RXJTwwONZg2oOHJC+VoNRKAETfmgArNQqYkUFeDEefU7rPXD3Hm0jPHETCFrd9B+ rSRDSEhZpd5DZfVeyinUrigauOLdVQY/QYQhjJ5Vp0Lvsc/33n9yIr+JOn877WTg dn9iHMprPEGWzRfA/VOgQb/8QPFkgjI9WuP2k7r8hN6F9qYOWkfFoxO9eBZebCat X2nhYl6AzDBnhlvI3zqK17CgvyvWqJsIV2LD/UKQHo1iBCEmK7wcQcG36NB84soT Hpvw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843784; x=1698930184; bh=38oBy1IBjhcoP 6a36VWXeV9rcI0uvMGQM9wk0+CFs8Q=; b=bKCak9gFLr9vqC2wYbq1LWNlssLcU XSTdYVk1wi5qCuhbgJ1uuWL27/LyJkxK2sY9gjoHWV9kek9X67R50XbxCjFCwsBw Zer3pfttqx55rcpXD7TixsBKorWmVuRSdibpK6+mhWaxEDHllmNIU3jv9/oezWTW MiISXe7m/Wgswa9jw/0XKDQk3pcJh7W59jSpho+iWsx3huMAaUyF+74eMHJ2hipl Y5B2LJxpK4By/6Z9TIhaaDdMSuyYhVDIY7pcH+1G57ncIGrPhoVTvdauPwxtVYVM fYoQ87jo6W78aIfJHIJpGxgI74FcLayPxR3xItTrAzTV7EDP9yRwgxy+Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:03:02 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a50feb2d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:51 +0000 (UTC) Date: Wed, 1 Nov 2023 14:03:00 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 5/8] ci: unify setup of some environment variables Message-ID: <6078aea246d874ba18bbd9ce2c0a7fb5c03e1564.1698843660.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Both GitHub Actions and Azure Pipelines set up the environment variables GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are actually the same, the setup is completely duplicate. With the upcoming support for GitLab CI this duplication would only extend even further. Unify the setup of those environment variables so that only the uncommon parts are separated. While at it, we also perform some additional small improvements: - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS. It doesn't hurt on platforms where we don't persist the state, so this further reduces boilerplate. - When running on Windows systems we set `--no-chain-lint` and `--no-bin-wrappers`. Interestingly though, we did so _after_ already having exported the respective environment variables. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 9ffdf743903..0b35da3cfdb 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -175,11 +175,8 @@ then # among *all* phases) cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" - export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" - export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows_nt != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--write-junit-xml" + JOBS=10 elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions @@ -198,17 +195,27 @@ then cache_dir="$HOME/none" - export GIT_PROVE_OPTS="--timer --jobs 10" - export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--github-workflow-markup" + JOBS=10 else echo "Could not identify CI type" >&2 env >&2 exit 1 fi +MAKEFLAGS="$MAKEFLAGS --jobs=$JOBS" +GIT_PROVE_OPTS="--timer --jobs $JOBS --state=failed,slow,save" + +GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" +case "$CI_OS_NAME" in +windows|windows_nt) + GIT_TEST_OPTS="$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers" + ;; +esac + +export GIT_TEST_OPTS +export GIT_PROVE_OPTS + good_trees_file="$cache_dir/good-trees" mkdir -p "$cache_dir" From patchwork Wed Nov 1 13:03:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442816 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11AD1134A3 for ; Wed, 1 Nov 2023 13:03:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ci/2Pd6r"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Iszsms5V" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F5CA101 for ; Wed, 1 Nov 2023 06:03:08 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id EF2A95C0489; Wed, 1 Nov 2023 09:03:07 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 01 Nov 2023 09:03:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843787; x=1698930187; bh=AB G0fF0MuK5hQGlyyEvPA1vscbhIytN+V44Um6WKHw0=; b=ci/2Pd6r3kPMmCxbDF PXrPW4BV7oHrUV8GDxJX2C1hK6HP0xKXu4UKj7EByQIs92ExZOaj1fyk4cPsfhRc 4cQj7COAq6Rsjy0f5sZstFOEroDYr+8Xr5d/KzbwfvQHMT71eexioS51ozlRgrq+ 9q/TNjFwxbyUsNerNZReXm2nDCKTEpxAsCa89ZweKdDymkjuKkMEWifGiOM6bmfT fJKn+4Jct1DjCQz4j60OCDBn2iaUbT1m4fKcxhZGdK50OQfdkpsCWEewspuVXb06 jOS7WZ4zIjmGVv84q4Y17jHvO0pfcWLC6dOJbHLmR0i9qB8jPCJLuuIqALLNtLmb VGoA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843787; x=1698930187; bh=ABG0fF0MuK5hQ GlyyEvPA1vscbhIytN+V44Um6WKHw0=; b=Iszsms5V02E++t9v6U2UqG8dws0xV dpKN/3LV1ZUguzxXVvyjCrskAO6eoCjCdQ/bHMCqod2PAZv/j5g6PXPFYdVJZpix Q7HARU9QFp2hSypg+EClBOYcnEelnyt39SZRICEyTMSwjjsEa9/APoCf479PY3Jr zlmhnT1CaHImJhAJqTrwgSQ1wf2tfhTfBvWuOMYw2+CBFvmV3H7hTpRo1ijvhAKi Gfqdzu51+TWCzDZfvBuEKeTTUMPD7g7E1dmcw+D8+ZZmABXEzhWnaAUxNzE32yrl wWti40i4tLFvWOb81LF0As4y8l9kGiPOJU7EWXr+Ufm+R7BnBFTfnuVCQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:03:06 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 18a295b2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:56 +0000 (UTC) Date: Wed, 1 Nov 2023 14:03:04 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 6/8] ci: squelch warnings when testing with unusable Git repo Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Our CI jobs that run on Docker also use mostly the same architecture to build and test Git via the "ci/run-build-and-tests.sh" script. These scripts also provide some functionality to massage the Git repository we're supposedly operating in. In our Docker-based infrastructure we may not even have a Git repository available though, which leads to warnings when those functions execute. Make the helpers exit gracefully in case either there is no Git in our PATH, or when not running in a Git repository. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/ci/lib.sh b/ci/lib.sh index 0b35da3cfdb..f0a2f80f094 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -69,10 +69,32 @@ skip_branch_tip_with_tag () { fi } +# Check whether we can use the path passed via the first argument as Git +# repository. +is_usable_git_repository () { + # We require Git in our PATH, otherwise we cannot access repositories + # at all. + if ! command -v git >/dev/null + then + return 1 + fi + + # And the target directory needs to be a proper Git repository. + if ! git -C "$1" rev-parse 2>/dev/null + then + return 1 + fi +} + # Save some info about the current commit's tree, so we can skip the build # job if we encounter the same tree again and can provide a useful info # message. save_good_tree () { + if ! is_usable_git_repository . + then + return + fi + echo "$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID" >>"$good_trees_file" # limit the file size tail -1000 "$good_trees_file" >"$good_trees_file".tmp @@ -88,6 +110,11 @@ skip_good_tree () { return fi + if ! is_usable_git_repository . + then + return + fi + if ! good_tree_info="$(grep "^$(git rev-parse $CI_COMMIT^{tree}) " "$good_trees_file")" then # Haven't seen this tree yet, or no cached good trees file yet. @@ -119,6 +146,11 @@ skip_good_tree () { } check_unignored_build_artifacts () { + if ! is_usable_git_repository . + then + return + fi + ! git ls-files --other --exclude-standard --error-unmatch \ -- ':/*' 2>/dev/null || { From patchwork Wed Nov 1 13:03:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442817 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EADF4134B4 for ; Wed, 1 Nov 2023 13:03:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="uyYlzwUn"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="vJxRei8N" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57696102 for ; Wed, 1 Nov 2023 06:03:13 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id C48FE5C01B9; Wed, 1 Nov 2023 09:03:12 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 01 Nov 2023 09:03:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843792; x=1698930192; bh=DM DBDZgDDCqa6rBpvIOR21fsjffbC/YoZpfGHuO8kCY=; b=uyYlzwUnvIaqGt3dD8 WZMq3nCFYLTQRBP14+tlgRAvwqf6fzmGcuS6Sc0T7dtkLxIwFwKzrpufx3/5y4oV d5FBMp1P3JbUqUaTRFtwdsUf2JqK9ucwaLhkEH7B+qqmglUaJAbaikUnmkCNi/5t TVyMDAX56dXL6X/0fFVlHr9XeNJi1w1InEVB8TgDmqXvYwwl9COvLAOJRX9zlId+ mTBvLNKhCagKwcrUO8a5dtRjZ9lh6jNzRQ1vm3purSf9fsNTneVbvBKkToxPRK0g wU4UnKZXvMOkzfF6cMvto7GSlsrh+kLctRo4Vd4mgCJ16OdZUVZvv8fXUsEpmH58 YeDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843792; x=1698930192; bh=DMDBDZgDDCqa6 rBpvIOR21fsjffbC/YoZpfGHuO8kCY=; b=vJxRei8NE+kOdNb603zDzVJ/bv4tf 4U5aTk7HnZBzzb8bZ8627Iikv4++3IoZ2mjJXoP2uEaCdMv63cQTTgAIf0aNggrl SAH97s3fEDA7q2nvca4AsUOPdWaXK8YLEqy50poWl6g6Zdg4iPhTaXjo3ZB/FjnY JgCAezdrfh1dD/a+IwXvz0vtEXtWeaN8rKjydh5kXV55/8h7Lo3NwVGQUfTFi9KS q69BJSk3xioweeIcxJB5ZjBb41i3ppPK65eLmuPTAvJs5S9gEptA6GB8gthHQUXI 3fjfZwR2EW4wgfS/MxGhjsIVxhFwqCRv1rB/tHmIL5R7ZcFV5dUFH5HsA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepjeetfeelieettdelfeeuudfhgefhhfevudevgeduueduvdetvdetgefhtdfhgeev necuffhomhgrihhnpegrlhhpihhnvghlihhnuhigrdhorhhgpdhhthhtphgurdhshhenuc evlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhk shdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:03:11 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 093af6dd (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:03:00 +0000 (UTC) Date: Wed, 1 Nov 2023 14:03:09 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 7/8] ci: install test dependencies for linux-musl Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The linux-musl CI job executes tests on Alpine Linux, which is based on musl libc instead of glibc. We're missing some test dependencies though, which causes us to skip a subset of tests. Install these test dependencies to increase our test coverage on this platform. There are still some missing test dependecies, but these do not have a corresponding package in the Alpine repositories: - p4 and p4d, both parts of the Perforce version control system. - cvsps, which generates patch sets for CVS. - Subversion and the SVN::Core Perl library, the latter of which is not available in the Alpine repositories. While the tool itself is available, all Subversion-related tests are skipped without the SVN::Core Perl library anyway. The Apache2-based tests require a bit more care though. For one, the module path is different on Alpine Linux, which requires us to add it to the list of known module paths to detect it. But second, the WebDAV module on Alpine Linux is broken because it does not bundle the default database backend [1]. We thus need to skip the WebDAV-based tests on Alpine Linux for now. [1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112 Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 4 +++- t/lib-httpd.sh | 17 ++++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index d0bc19d3bb3..6e845283680 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -17,7 +17,9 @@ linux32) ;; linux-musl) apk add --update build-base curl-dev openssl-dev expat-dev gettext \ - pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null + pcre2-dev python3 musl-libintl perl-utils ncurses \ + apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ + bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; pedantic) dnf -yq update >/dev/null && diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh index 2fb1b2ae561..9ea74927c40 100644 --- a/t/lib-httpd.sh +++ b/t/lib-httpd.sh @@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \ '/usr/lib/apache2/modules' \ '/usr/lib64/httpd/modules' \ '/usr/lib/httpd/modules' \ - '/usr/libexec/httpd' + '/usr/libexec/httpd' \ + '/usr/lib/apache2' do if test -d "$DEFAULT_HTTPD_MODULE_PATH" then @@ -127,6 +128,20 @@ else "Could not identify web server at '$LIB_HTTPD_PATH'" fi +if test -n "$LIB_HTTPD_DAV" && test -f /etc/os-release +then + case "$(grep "^ID=" /etc/os-release | cut -d= -f2-)" in + alpine) + # The WebDAV module in Alpine Linux is broken at least up to + # Alpine v3.16 as the default DBM driver is missing. + # + # https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112 + test_skip_or_die GIT_TEST_HTTPD \ + "Apache WebDAV module does not have default DBM backend driver" + ;; + esac +fi + install_script () { write_script "$HTTPD_ROOT_PATH/$1" <"$TEST_PATH/$1" } From patchwork Wed Nov 1 13:03:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442818 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A786B12B90 for ; Wed, 1 Nov 2023 13:03:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="bUdsu7+T"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="LLgrnniB" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5609EA6 for ; Wed, 1 Nov 2023 06:03:18 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id BFA8A5C01B9; Wed, 1 Nov 2023 09:03:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Wed, 01 Nov 2023 09:03:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843797; x=1698930197; bh=Jw W21xk7NABA7qUL+dA6c4kuCd7iKQsRfmVBpLSfZC0=; b=bUdsu7+T1/m7aO2QNo dSDWkAirikgvHYPPWMhOsd3B7mXovaOqQzGlcw99gIdMG2nur3lz/3+eHAtZOW7D gtpp2tqK4GHydYlXjZ1e5RffRg/DdzgvFzo91u82CjOQTPEXYDyx14lu8gc2sTZr MRsaOGN3bIZP0I8W+E/E5H3wcEwrlmSU6Nlcv7PRA2UrK71++C6BQMpxLmgHq9Vr i7Z2oLpbI568QIZBLxuSwUxsJgaEsAdllZWQc5eMM5TYZJIlxCSn7eGd1+V5zon1 AN7hujpwIF80meCiK/rBFE0NE/CmOxDJ3Sdj4R0iVvpUhYKfS3Y6lXl+Wgt96QfG KiYg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843797; x=1698930197; bh=JwW21xk7NABA7 qUL+dA6c4kuCd7iKQsRfmVBpLSfZC0=; b=LLgrnniBBwKeuoUl92hQp3HOoKTGj hXRngwv0OVgvk2iQKBzs1B6gBCEe10A6dt+z+SWtqg36B2EA1ga66OE5bV2riVW3 W4woUJ3zPXvXNQwPqUdfA+uUegX2gPfW0/FYlsVXR0c3XVfImiyWbuFkprahYXZm WMvVqlX6xwUDzJ4ibKZWX5Fyc6nz4B1DR+0dR56kwn1Dshb4HgF0bDd/XbY41k8G Ptg5ei5h5rTxB4sVhrTH17SlnN1vSJRZCf2ycTglxMaTIsMjZfp6ojcsff9Z7ftV RRENT8Eg/zNt94wB8Fi5G1gU6cbVN6X/32s7l3brtRLRawF9nJZxMZS7w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnheplefgiedtffekvdetffegudeuvdfftdffheetfeelffevvdffhedvffegleeggeei necuffhomhgrihhnpehgihhtlhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:03:15 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id b19a96e1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:03:05 +0000 (UTC) Date: Wed, 1 Nov 2023 14:03:13 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 8/8] ci: add support for GitLab CI Message-ID: <5784d03a6f102cac88f7a857e57987968e82a809.1698843660.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer. Results of these pipeline runs will be made available (at least) in GitLab's mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt --- .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++ ci/install-docker-dependencies.sh | 13 +++++++- ci/lib.sh | 44 +++++++++++++++++++++++++ ci/print-test-failures.sh | 6 ++++ 4 files changed, 115 insertions(+), 1 deletion(-) create mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 00000000000..cd98bcb18aa --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,53 @@ +default: + timeout: 2h + +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_REF_PROTECTED == "true" + +test: + image: $image + before_script: + - ./ci/install-docker-dependencies.sh + script: + - useradd builder --create-home + - chown -R builder "${CI_PROJECT_DIR}" + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh + after_script: + - | + if test "$CI_JOB_STATUS" != 'success' + then + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh + fi + parallel: + matrix: + - jobname: linux-sha256 + image: ubuntu:latest + CC: clang + - jobname: linux-gcc + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-TEST-vars + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-gcc-default + image: ubuntu:latest + CC: gcc + - jobname: linux-leaks + image: ubuntu:latest + CC: gcc + - jobname: linux-asan-ubsan + image: ubuntu:latest + CC: clang + - jobname: pedantic + image: fedora:latest + - jobname: linux-musl + image: alpine:latest + artifacts: + paths: + - t/failed-test-artifacts + when: on_failure diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 6e845283680..48c43f0f907 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -16,11 +16,22 @@ linux32) ' ;; linux-musl) - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ + apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ pcre2-dev python3 musl-libintl perl-utils ncurses \ apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; +linux-*) + # Required so that apt doesn't wait for user input on certain packages. + export DEBIAN_FRONTEND=noninteractive + + apt update -q && + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ + perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \ + libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \ + apache2 cvs cvsps gnupg libcgi-pm-perl subversion + ;; pedantic) dnf -yq update >/dev/null && dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null diff --git a/ci/lib.sh b/ci/lib.sh index f0a2f80f094..2718ce8776c 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,6 +14,22 @@ then need_to_end_group= echo '::endgroup::' >&2 } +elif test true = "$GITLAB_CI" +then + begin_group () { + need_to_end_group=t + printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n" + trap "end_group '$1'" EXIT + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n" + trap - EXIT + } else begin_group () { :; } end_group () { :; } @@ -229,6 +245,34 @@ then GIT_TEST_OPTS="--github-workflow-markup" JOBS=10 +elif test true = "$GITLAB_CI" +then + CI_TYPE=gitlab-ci + CI_BRANCH="$CI_COMMIT_REF_NAME" + CI_COMMIT="$CI_COMMIT_SHA" + case "$CI_JOB_IMAGE" in + macos-*) + CI_OS_NAME=osx;; + alpine:*|fedora:*|ubuntu:*) + CI_OS_NAME=linux;; + *) + echo "Could not identify OS image" >&2 + env >&2 + exit 1 + ;; + esac + CI_REPO_SLUG="$CI_PROJECT_PATH" + CI_JOB_ID="$CI_JOB_ID" + CC="${CC_PACKAGE:-${CC:-gcc}}" + DONT_SKIP_TAGS=t + handle_failed_tests () { + create_failed_test_artifacts + } + + cache_dir="$HOME/none" + + runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -) + JOBS=$(nproc) else echo "Could not identify CI type" >&2 env >&2 diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 57277eefcd0..c33ad4e3a22 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -51,6 +51,12 @@ do tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" continue ;; + gitlab-ci) + mkdir -p failed-test-artifacts + cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ + tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + continue + ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1