From patchwork Wed Oct 14 05:21:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 11836793 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 326C6921 for ; Wed, 14 Oct 2020 05:25:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DB480214D8 for ; Wed, 14 Oct 2020 05:24:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SqiMFFFE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB480214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50686 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kSZHW-0008Iy-Vp for patchwork-qemu-devel@patchwork.kernel.org; Wed, 14 Oct 2020 01:24:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58582) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kSZEa-0004Qq-FP for qemu-devel@nongnu.org; Wed, 14 Oct 2020 01:21:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:37417) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kSZEW-0003Bp-Ew for qemu-devel@nongnu.org; Wed, 14 Oct 2020 01:21:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1602652911; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dldSYyrF6HCag5OpMym1QmT2Bt6q7MvSxhhXerFWxjE=; b=SqiMFFFEg1nAs2ryKs5LR5MUdLT2uPhykJCIThPk9RHGl+lIffqW5ADK5AMpHyb1Jf7QKp 9EQqx53mRK1lMdOTatDV2IS7BlUKO0u6gq1kPQKoICj/ZHa5ygUeRj1Dw/dlxHLfvgZbhe CHqYQ0IMwYKaBeTb9Z2Z1i4009XB4uM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-261-q_2SCzY1OaqeP-Qvs32xLQ-1; Wed, 14 Oct 2020 01:21:47 -0400 X-MC-Unique: q_2SCzY1OaqeP-Qvs32xLQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4F23E802B74; Wed, 14 Oct 2020 05:21:46 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-119-55.rdu2.redhat.com [10.10.119.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D6E4E5D9CD; Wed, 14 Oct 2020 05:21:41 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org, =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell Subject: [PATCH v3 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI) Date: Wed, 14 Oct 2020 01:21:36 -0400 Message-Id: <20201014052140.1146924-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=crosa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=crosa@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/14 01:12:43 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Eduardo Habkost , Erik Skultety , Stefan Hajnoczi , Andrea Bolognani , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Beraldo Leal Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" TL;DR: this should allow the QEMU maintainer to push to the staging branch, and have custom jobs running on the project's aarch64 and s390x machines. Simple usage looks like: git push remote staging ./scripts/ci/gitlab-pipeline-status --verbose --wait Long version: The idea about a public facing Gating CI for QEMU was summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first. At this point, there are two specific runners (an aarch64 and an s390) registered with GitLab, at https://gitlab.com/qemu-project, currently setup to the "qemu" repository. Changes from v2: - The overall idea of "Gating CI" has been re-worded "custom runners", given that the other jobs running on shared runners are also considered gating (Daniel) - Fixed wording and typos on the documentation, including: * update -> up to date (Erik) * a different set of CI jobs -> different CI jobs (Erik) * Pull requests will only be merged -> code will only be merged (Stefan) * Setup -> set up (Stefan) * them -> they (Stefan) * the -> where the (Stefan) * dropped "in the near future" (Stefan) - Changed comment on "build-environment.yml" regarding the origin of the package list (Stefan) - Removed inclusion of "vars.yml" from "build-environment.yml", given that no external variable is used there - Updated package list in "build-environment.yml" from current dockerfiles - Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30), and noted that it's possible to use it on those distros - Moved CI documentation from "testing.rst" to its own file (Phillipe) - Split "GitLab Gating CI: initial set of jobs, documentation and scripts" into (Phillipe): 1) Basic documentation and configuration (gitlab-ci.yml) placeholder 2) Playbooks for setting up a build environment 3) Playbooks for setting up gitlab-runner 4) Actual GitLab CI jobs configuration - Set custom jobs to be on the "build" stage, given that they combine build and test. - Set custom jobs to not depend on any other job, so they can start right away. - Set rules for starting jobs so that all pushing to any branch that start with name "staging". This allows the project maintainer to use the "push to staging" workflow, while also allowing others to generate similar jobs. If this project has configured custom runners, the jobs will run, if not, the pipeline will be marked as "stuck". - Changed "scripts" on custom jobs to follow the now common pattern (on other jobs) of creating a "build" directory. Changes from v1: - Added jobs that require specific GitLab runners already available (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x) - Removed jobs that require specific GitLab runners not yet available (Fedora 30, FreeBSD 12.1) - Updated documentation - Added copyright and license to new scripts - Moved script to from "contrib" to "scripts/ci/" - Moved setup playbooks form "contrib" to "scripts/ci/setup" - Moved "gating.yml" to ".gitlab-ci.d" directory - Removed "staging" only branch restriction on jobs defined in ".gitlab-ci.yml", assumes that the additional jobs on the staging branch running on the freely available gitlab shared runner are positive - Dropped patches 1-3 (already merged) - Simplified amount of version specifity on Ubuntu, from 18.04.3 to simply 18.04 (assumes no diverse minor levels will be used or specific runners) Changes from the RFC patches[2] accompanying the RFC document: - Moved gating job definitions to .gitlab-ci-gating.yml - Added info on "--disable-libssh" build option requirement (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs - Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test) - Added w32 and w64 cross builds based on Fedora 30 - Added a FreeBSD based job that builds all targets and runs `make check` - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs - Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included - New patch with script to check or wait on a pipeline execution [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html Cleber Rosa (4): Jobs based on custom runners: documentation and configuration placeholder Jobs based on custom runners: build environment docs and playbook Jobs based on custom runners: docs and gitlab-runner setup playbook Jobs based on custom runners: add job definitions for QEMU's machines .gitlab-ci.d/custom-runners.yml | 206 +++++++++++++++++++++++ .gitlab-ci.yml | 1 + docs/devel/ci.rst | 149 +++++++++++++++++ docs/devel/index.rst | 1 + scripts/ci/setup/.gitignore | 1 + scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++ scripts/ci/setup/gitlab-runner.yml | 72 ++++++++ scripts/ci/setup/inventory | 2 + scripts/ci/setup/vars.yml.template | 13 ++ 9 files changed, 665 insertions(+) create mode 100644 .gitlab-ci.d/custom-runners.yml create mode 100644 docs/devel/ci.rst create mode 100644 scripts/ci/setup/.gitignore create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/inventory create mode 100644 scripts/ci/setup/vars.yml.template