From patchwork Wed Nov 1 13:02:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13442812 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BC2C12B93 for ; Wed, 1 Nov 2023 13:02:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="xbpeTej6"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="BOHlTbAp" Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF05DFC for ; Wed, 1 Nov 2023 06:02:53 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 42A525C0269; Wed, 1 Nov 2023 09:02:43 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 01 Nov 2023 09:02:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698843763; x=1698930163; bh=sM l7EUUPZ4Jc8BLwXaFM2/NCdSPhA4clsrRIt/962ws=; b=xbpeTej63h7XEo5QNc W0pHK3MdAB8fNEFyLvJtyMVAGTd8zdRE/1CwHEgSJ3UQWWyykuazxmSAA5zU/pT8 kV79kcgCSSbFmgRu9n8QeRsK0c+tDUwnnKb5WkeR6nm3fITU56AqcKHq7ylzjjEq /1pO64GAEIVTn9+ALoaFR3qqBaAHU16fo88UjxqP8HquKQzQ4ihlcW8W7RUJT5ir DDn9iY09h2TP8sGdnKOnE9sM2dnNuglTuyo2LnDTjoYVXHGADz8I+HMgeHH1L7aT CmNJlx9BkSnlFX07GRGWwRL5STKpKbgV7VwnniP+a4+7PaB4mszoXZjFcIDbHfFB +k5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698843763; x=1698930163; bh=sMl7EUUPZ4Jc8 BLwXaFM2/NCdSPhA4clsrRIt/962ws=; b=BOHlTbAp+CNcgrYlU5j9FEOlDC5T8 EBZE2gXArDJ2SOXej/EFhI0YuwKuzB+DPenquwfn3eLPwVwGV/5YoFhS8BUf4/nN RbgC2oIdwHEOqj7KRZVtGLSpV3c10eKiYRo82SVtLlrPKzGS5gMZz76kIxIpsFpG 4d6dJ12Esc7UNSG0yLOShc/CPjBTC4P73icgdtlsCcFBJyJxRXw6Dm1HOItPXZC1 8uSpGJcxqBiOp/iT7uKagAYXN4YD8K6SB/G80Am+3P+8kIGixCDcFVZ1ZhFG1Ugv cwLWW7N7Z50W5zbAXG79PGOsIjnKGLG4t0rxH8iKWE95NQO7W9FehvugQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtgedggeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepveehueevvdffvdegkeeiffeuieevjeeliefffeeigedvudehudelgedujeefiedu necuffhomhgrihhnpehgihhtlhgrsgdrtghomhdphhhtthhpugdrshhhnecuvehluhhsth gvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 09:02:41 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 4c520733 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Nov 2023 13:02:30 +0000 (UTC) Date: Wed, 1 Nov 2023 14:02:38 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen , Victoria Dye Subject: [PATCH v5 0/8] ci: add support for GitLab CI Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Hi, this is the fifth version of my patch series that introduces support for GitLab CI. There are only minor changes compared to v4: - Patch 5: Fixed a typo in the commit message. Furthermore, I've dropped the note about `export VAR=value` being a Bashism. - Patch 5: Fixed the accidentally-dropped "windows_nt" case. - Patch 5: Introduce the JOBS variable right away so that there is less churn in the patch that introduces GitLab CI. - Patch 8: Moved `DEBIAN_FRONTEND` into the "linux-*" case to make clear that it shouldn't impact anything else. - Patch 8: Dropped the paragraph about GitLab CI not being a first class citizen based on Junio's feedback. This setup is going to be actively used (at least) by us at GitLab, and thus we're also going to maintain it. After both Junio's and Victoria's feedback I've kept `.gitlab-ci.yml` in its canonical location. The pipeline for this version can be found at [1]. Thanks for all the discussion and feedback so far! Patrick [1]: https://gitlab.com/gitlab-org/git/-/pipelines/1057566736 Patrick Steinhardt (8): ci: reorder definitions for grouping functions ci: make grouping setup more generic ci: group installation of Docker dependencies ci: split out logic to set up failed test artifacts ci: unify setup of some environment variables ci: squelch warnings when testing with unusable Git repo ci: install test dependencies for linux-musl ci: add support for GitLab CI .gitlab-ci.yml | 53 +++++++++ ci/install-docker-dependencies.sh | 23 +++- ci/lib.sh | 191 +++++++++++++++++++++--------- ci/print-test-failures.sh | 6 + t/lib-httpd.sh | 17 ++- 5 files changed, 234 insertions(+), 56 deletions(-) create mode 100644 .gitlab-ci.yml Range-diff against v4: 1: 8595fe5016a = 1: 0ba396f2a33 ci: reorder definitions for grouping functions 2: 7358a943392 = 2: 821cfcd6125 ci: make grouping setup more generic 3: 6d842592c6f = 3: 6e5bcf143c8 ci: group installation of Docker dependencies 4: e15651b3f5d = 4: 2182acf5bfc ci: split out logic to set up failed test artifacts 5: a64799b6e25 ! 5: 6078aea246d ci: unify setup of some environment variables @@ Metadata ## Commit message ## ci: unify setup of some environment variables - Both GitHub Actions and Azue Pipelines set up the environment variables + Both GitHub Actions and Azure Pipelines set up the environment variables GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are actually the same, the setup is completely duplicate. With the upcoming support for GitLab CI this duplication would only extend even further. @@ Commit message `--no-bin-wrappers`. Interestingly though, we did so _after_ already having exported the respective environment variables. - - We stop using `export VAR=value` syntax, which is a Bashism. It's - not quite worth it as we still use this syntax all over the place, - but it doesn't hurt readability either. - Signed-off-by: Patrick Steinhardt ## ci/lib.sh ## @@ ci/lib.sh: then - test windows_nt != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--write-junit-xml" ++ JOBS=10 elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions @@ ci/lib.sh: then - test windows != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--github-workflow-markup" ++ JOBS=10 else echo "Could not identify CI type" >&2 env >&2 exit 1 fi -+MAKEFLAGS="$MAKEFLAGS --jobs=10" -+GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" ++MAKEFLAGS="$MAKEFLAGS --jobs=$JOBS" ++GIT_PROVE_OPTS="--timer --jobs $JOBS --state=failed,slow,save" + +GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" -+if test windows = "$CI_OS_NAME" -+then ++case "$CI_OS_NAME" in ++windows|windows_nt) + GIT_TEST_OPTS="$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers" -+fi ++ ;; ++esac + +export GIT_TEST_OPTS +export GIT_PROVE_OPTS 6: f7d2a8666fe = 6: d69bde92f2f ci: squelch warnings when testing with unusable Git repo 7: 9b43b0d90e3 = 7: b911c005bae ci: install test dependencies for linux-musl 8: f3f2c98a0dc ! 8: 5784d03a6f1 ci: add support for GitLab CI @@ Commit message that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for - the maintainer. - - The integration does not necessarily have to be a first-class citizen, - which would in practice only add to the fallout that pipeline failures - have for the maintainer. That being said, we are happy to maintain this - alternative CI setup for the Git project and will make test results - available as part of our own mirror of the Git project at [1]. + the maintainer. Results of these pipeline runs will be made available + (at least) in GitLab's mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI @@ .gitlab-ci.yml (new) + when: on_failure ## ci/install-docker-dependencies.sh ## -@@ - - begin_group "Install dependencies" - -+# Required so that apt doesn't wait for user input on certain packages. -+export DEBIAN_FRONTEND=noninteractive -+ - case "$jobname" in - linux32) - linux32 --32bit i386 sh -c ' @@ ci/install-docker-dependencies.sh: linux32) ' ;; @@ ci/install-docker-dependencies.sh: linux32) bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; +linux-*) ++ # Required so that apt doesn't wait for user input on certain packages. ++ export DEBIAN_FRONTEND=noninteractive ++ + apt update -q && + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ @@ ci/lib.sh: then begin_group () { :; } end_group () { :; } @@ ci/lib.sh: then - cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" - - GIT_TEST_OPTS="--write-junit-xml" -+ JOBS=10 - elif test true = "$GITHUB_ACTIONS" - then - CI_TYPE=github-actions -@@ ci/lib.sh: then - cache_dir="$HOME/none" GIT_TEST_OPTS="--github-workflow-markup" -+ JOBS=10 + JOBS=10 +elif test true = "$GITLAB_CI" +then + CI_TYPE=gitlab-ci @@ ci/lib.sh: then else echo "Could not identify CI type" >&2 env >&2 - exit 1 - fi - --MAKEFLAGS="$MAKEFLAGS --jobs=10" --GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" -+MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}" -+GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save" - - GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" - if test windows = "$CI_OS_NAME" ## ci/print-test-failures.sh ## @@ ci/print-test-failures.sh: do base-commit: 692be87cbba55e8488f805d236f2ad50483bd7d5