From patchwork Thu Mar 12 19:36:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 11435289 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E9CAE1392 for ; Thu, 12 Mar 2020 19:37:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C055F206E2 for ; Thu, 12 Mar 2020 19:37:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EYhCvq4I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C055F206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:48510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTe6-0000dY-1h for patchwork-qemu-devel@patchwork.kernel.org; Thu, 12 Mar 2020 15:37:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44135) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTdP-0005zw-5j for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCTdO-0007mk-5j for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:47 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:35037 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jCTdO-0007mX-2B for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584041805; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2qBNted8CFpYRl2d21kOkpTYOKH+hXz5AA/g+4M5bAw=; b=EYhCvq4IkGBMewq5lDGFZnP0pWCgP1rHoFjW7URBtaic8fqCQf3f+EgCOGHpwEi0VloRjs r0Dl0e5bUu5EePHafE57i6+zHzOjrTRbvP9Mi6uyHZM6unOaxlWZVg8lN8QiVPQ92knY9x 04Cwy4gAILYKQRLd1OcwPHE3lROs6VI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-418-eZVfrXAzMs6WquwmRMVJvQ-1; Thu, 12 Mar 2020 15:36:41 -0400 X-MC-Unique: eZVfrXAzMs6WquwmRMVJvQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CD24A14E6; Thu, 12 Mar 2020 19:36:40 +0000 (UTC) Received: from dhcp-17-173.bos.redhat.com (dhcp-17-173.bos.redhat.com [10.18.17.173]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AF76563769; Thu, 12 Mar 2020 19:36:31 +0000 (UTC) From: Cleber Rosa To: =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell , qemu-devel@nongnu.org Subject: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile Date: Thu, 12 Mar 2020 15:36:12 -0400 Message-Id: <20200312193616.438922-2-crosa@redhat.com> In-Reply-To: <20200312193616.438922-1-crosa@redhat.com> References: <20200312193616.438922-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Thomas Huth , Eduardo Habkost , Erik Skultety , Wainer Moschetta , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Beraldo Leal Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Which is currenly missing, and will be referenced later in the contributed CI playbooks. Signed-off-by: Cleber Rosa Reviewed-by: Alex Bennée --- tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 tests/docker/dockerfiles/centos8.docker diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker new file mode 100644 index 0000000000..bfa0d33c9c --- /dev/null +++ b/tests/docker/dockerfiles/centos8.docker @@ -0,0 +1,32 @@ +FROM centos:8.1.1911 + +RUN dnf -y update +ENV PACKAGES \ + SDL-devel \ + bison \ + bzip2 \ + bzip2-devel \ + dbus-daemon \ + flex \ + gcc \ + gcc-c++ \ + gettext \ + git \ + glib2-devel \ + libaio-devel \ + libepoxy-devel \ + lzo-devel \ + make \ + mesa-libEGL-devel \ + nettle-devel \ + perl-Test-Harness \ + pixman-devel \ + python36 \ + rdma-core-devel \ + spice-glib-devel \ + spice-server \ + tar \ + zlib-devel + +RUN dnf install -y $PACKAGES +RUN rpm -q $PACKAGES | sort > /packages.txt From patchwork Thu Mar 12 19:36:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 11435293 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66CAB1392 for ; Thu, 12 Mar 2020 19:37:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3DD4E206E2 for ; Thu, 12 Mar 2020 19:37:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="P9s1q55y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3DD4E206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:48512 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTeG-0001GJ-FX for patchwork-qemu-devel@patchwork.kernel.org; Thu, 12 Mar 2020 15:37:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44194) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTda-0006ml-AZ for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCTdZ-0007r6-Au for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:58 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:25889 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jCTdZ-0007qp-7P for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584041816; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pL/4tsiBJijnv/MS3/MnnZB7fz4iUpoEpqQeJ2YCZog=; b=P9s1q55ytdTEXVNMGjyTHX6wOi8fWbSe4BfyTrpihGRPcnmB3kmaZvmpiWDmW/UQkGiaEp b3UrdjLPNA3DK5zZgvEVqxO9nD9x44sARlENJLxWSo6ZqQoJBR4vhZQg+vfHkFTtWWi74i urR2aXI+vWp2tpOB1n9HQuHCEIsG/rY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-349-UbPbwy9VMLussHF6CuapzA-1; Thu, 12 Mar 2020 15:36:53 -0400 X-MC-Unique: UbPbwy9VMLussHF6CuapzA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D8BCC6BE62; Thu, 12 Mar 2020 19:36:51 +0000 (UTC) Received: from dhcp-17-173.bos.redhat.com (dhcp-17-173.bos.redhat.com [10.18.17.173]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E49615C1D8; Thu, 12 Mar 2020 19:36:40 +0000 (UTC) From: Cleber Rosa To: =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell , qemu-devel@nongnu.org Subject: [PATCH 2/5] tests/docker: make "buildah bud" output similar to "docker build" Date: Thu, 12 Mar 2020 15:36:13 -0400 Message-Id: <20200312193616.438922-3-crosa@redhat.com> In-Reply-To: <20200312193616.438922-1-crosa@redhat.com> References: <20200312193616.438922-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Thomas Huth , Eduardo Habkost , Erik Skultety , Wainer Moschetta , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Beraldo Leal Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Podman users will most often be using buildah to build containers. Among the differences between "buildah bud|build-using-dockerfile" and a traditional "docker build" is that buildah does not run a container during build. To the best of my knowledge and experiments, this means that runtime variables, such as ENV from one base image will not propagate into another. The end result is that the location for the cross compiler binaries, defined in the base "qemu/debian9-mxe" image, are not passed through this image. Consequently, the cross compilers are not on PATH and the build fails. Signed-off-by: Cleber Rosa Acked-by: Alex Bennée --- tests/docker/dockerfiles/debian-win32-cross.docker | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker index 9d7053e59d..d16d6431bc 100644 --- a/tests/docker/dockerfiles/debian-win32-cross.docker +++ b/tests/docker/dockerfiles/debian-win32-cross.docker @@ -9,7 +9,7 @@ MAINTAINER Philippe Mathieu-Daudé ENV TARGET i686 -ENV PATH $PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin +ENV PATH $PATH:/usr/lib/mxe/usr/bin:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin ENV PKG_CONFIG_PATH \ $PKG_CONFIG_PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/lib/pkgconfig From patchwork Thu Mar 12 19:36:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 11435297 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B9681392 for ; Thu, 12 Mar 2020 19:38:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 123CD206E2 for ; Thu, 12 Mar 2020 19:38:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Z5RQ/S2y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 123CD206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:48524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTfL-0004Lh-8Q for patchwork-qemu-devel@patchwork.kernel.org; Thu, 12 Mar 2020 15:38:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44212) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTdc-0006zo-My for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCTdb-0007s1-I3 for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:00 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:57254 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jCTdb-0007ro-EN for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584041819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FVtqhaDz31hPkY62S3ERAd5gLcsSXZwSZmNt1nLa3ww=; b=Z5RQ/S2ydCwA9JaaZhKayx0hhIxRuSIGtpEQncLVIW/ZpuRvrNxAHqXQkQWxjIK/o5nzI5 TgAZDo91FnCMjN3oYviVbc8TMZjK2vJcQLeLr/qL//aWjFB4cZ5a6Awswy3CPhVWN5QaGv yBFHIEkOXtfW0I/xCgmB41SzH1bO+OA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-146-pFxr24iPMYCpeOHrYcnXPw-1; Thu, 12 Mar 2020 15:36:55 -0400 X-MC-Unique: pFxr24iPMYCpeOHrYcnXPw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CA943801F88; Thu, 12 Mar 2020 19:36:53 +0000 (UTC) Received: from dhcp-17-173.bos.redhat.com (dhcp-17-173.bos.redhat.com [10.18.17.173]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F34065C1B5; Thu, 12 Mar 2020 19:36:51 +0000 (UTC) From: Cleber Rosa To: =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell , qemu-devel@nongnu.org Subject: [PATCH 3/5] GitLab CI: avoid calling before_scripts on unintended jobs Date: Thu, 12 Mar 2020 15:36:14 -0400 Message-Id: <20200312193616.438922-4-crosa@redhat.com> In-Reply-To: <20200312193616.438922-1-crosa@redhat.com> References: <20200312193616.438922-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Thomas Huth , Eduardo Habkost , Erik Skultety , Wainer Moschetta , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Beraldo Leal Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" At this point it seems that all jobs depend on those steps, with maybe the EDK2 jobs as exceptions. The jobs that will be added will not want those scripts to be run, so let's move these steps to the appropriate jobs, while still trying to avoid repetition. Signed-off-by: Cleber Rosa Reviewed-by: Wainer dos Santos Moschetta Acked-by: Thomas Huth --- .gitlab-ci.yml | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 72f8b8aa51..3ef1551273 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,11 +1,10 @@ include: - local: '/.gitlab-ci-edk2.yml' -before_script: +build-system1: + before_script: &before_scr_apt - apt-get update -qq - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage - -build-system1: script: - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev @@ -18,6 +17,8 @@ build-system1: - make -j2 check build-system2: + before_script: + *before_scr_apt script: - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev @@ -31,6 +32,8 @@ build-system2: - make -j2 check build-disabled: + before_script: + *before_scr_apt script: - mkdir build - cd build @@ -45,6 +48,8 @@ build-disabled: - make -j2 check-qtest SPEED=slow build-tcg-disabled: + before_script: + *before_scr_apt script: - apt-get install -y -qq clang libgtk-3-dev libusb-dev - mkdir build @@ -63,6 +68,8 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: + before_script: + *before_scr_apt script: - mkdir build - cd build @@ -72,6 +79,8 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: + before_script: + *before_scr_apt script: - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev @@ -84,6 +93,8 @@ build-clang: - make -j2 check build-tci: + before_script: + *before_scr_apt script: - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" - mkdir build From patchwork Thu Mar 12 19:36:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 11435301 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5141690 for ; Thu, 12 Mar 2020 19:39:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 282F0206E2 for ; Thu, 12 Mar 2020 19:39:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KWZunL/m" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 282F0206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:48526 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTfa-0004yh-CE for patchwork-qemu-devel@patchwork.kernel.org; Thu, 12 Mar 2020 15:39:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44222) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTdd-00074e-HK for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCTdc-0007sH-0k for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:01 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:29622 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jCTdb-0007s9-TE for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:36:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584041819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZPizTgORafpKHefddcD0j2V6GIQK6ROg5iFW9mvYRmY=; b=KWZunL/mgmAdT4QLGEYaSq91VGy1VIGKhlh1k0WhuN35EGfWVL9RNo5xocRyQ2c8PGnsnM HhZHPPvZgZpYTHTI1EMALTUNmExOTVhyesjlfWbYVi4C+A7PVWjc8RWYpHhsQiS3f8sK46 Rwc9R5NhTHuyvz7H492HQgjBY8XpbdQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-198-mwYTm12SOric2lx0YNd8zA-1; Thu, 12 Mar 2020 15:36:56 -0400 X-MC-Unique: mwYTm12SOric2lx0YNd8zA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3C31E19251A9; Thu, 12 Mar 2020 19:36:55 +0000 (UTC) Received: from dhcp-17-173.bos.redhat.com (dhcp-17-173.bos.redhat.com [10.18.17.173]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E5D075C1B5; Thu, 12 Mar 2020 19:36:53 +0000 (UTC) From: Cleber Rosa To: =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell , qemu-devel@nongnu.org Subject: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script Date: Thu, 12 Mar 2020 15:36:15 -0400 Message-Id: <20200312193616.438922-5-crosa@redhat.com> In-Reply-To: <20200312193616.438922-1-crosa@redhat.com> References: <20200312193616.438922-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Thomas Huth , Eduardo Habkost , Erik Skultety , Wainer Moschetta , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Beraldo Leal Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This script is intended to be used right after a push to a branch. By default, it will look for the pipeline associated with the commit that is the HEAD of the *local* staging branch. It can be used as a one time check, or with the `--wait` option to wait until the pipeline completes. If the pipeline is successful, then a merge of the staging branch into the master branch should be the next step. Signed-off-by: Cleber Rosa --- contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100755 contrib/ci/scripts/gitlab-pipeline-status diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status new file mode 100755 index 0000000000..83d412daec --- /dev/null +++ b/contrib/ci/scripts/gitlab-pipeline-status @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 + +""" +Checks the GitLab pipeline status for a given commit commit +""" + +# pylint: disable=C0103 + +import argparse +import http.client +import json +import os +import subprocess +import time +import sys + + +def get_local_staging_branch_commit(): + """ + Returns the commit sha1 for the *local* branch named "staging" + """ + result = subprocess.run(['git', 'rev-parse', 'staging'], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + cwd=os.path.dirname(__file__), + universal_newlines=True).stdout.strip() + if result == 'staging': + raise ValueError("There's no local staging branch") + if len(result) != 40: + raise ValueError("Branch staging HEAD doesn't look like a sha1") + return result + + +def get_pipeline_status(project_id, commit_sha1): + """ + Returns the JSON content of the pipeline status API response + """ + url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id, + commit_sha1) + connection = http.client.HTTPSConnection('gitlab.com') + connection.request('GET', url=url) + response = connection.getresponse() + if response.code != http.HTTPStatus.OK: + raise ValueError("Failed to receive a successful response") + json_response = json.loads(response.read()) + # afaict, there should one one pipeline for the same project + commit + # if this assumption is false, we can add further filters to the + # url, such as username, and order_by. + if not json_response: + raise ValueError("No pipeline found") + return json_response[0] + + +def wait_on_pipeline_success(timeout, interval, + project_id, commit_sha): + """ + Waits for the pipeline to end up to the timeout given + """ + start = time.time() + while True: + if time.time() >= (start + timeout): + print("Waiting on the pipeline success timed out") + return False + + status = get_pipeline_status(project_id, commit_sha) + if status['status'] == 'running': + time.sleep(interval) + print('running...') + continue + + if status['status'] == 'success': + return True + + msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url'] + print(msg) + return False + + +def main(): + """ + Script entry point + """ + parser = argparse.ArgumentParser( + prog='pipeline-status', + description='check or wait on a pipeline status') + + parser.add_argument('-t', '--timeout', type=int, default=7200, + help=('Amount of time (in seconds) to wait for the ' + 'pipeline to complete. Defaults to ' + '%(default)s')) + parser.add_argument('-i', '--interval', type=int, default=60, + help=('Amount of time (in seconds) to wait between ' + 'checks of the pipeline status. Defaults ' + 'to %(default)s')) + parser.add_argument('-w', '--wait', action='store_true', default=False, + help=('Wether to wait, instead of checking only once ' + 'the status of a pipeline')) + parser.add_argument('-p', '--project-id', type=int, default=11167699, + help=('The GitLab project ID. Defaults to the project ' + 'for https://gitlab.com/qemu-project/qemu, that ' + 'is, "%(default)s"')) + try: + default_commit = get_local_staging_branch_commit() + commit_required = False + except ValueError: + default_commit = '' + commit_required = True + parser.add_argument('-c', '--commit', required=commit_required, + default=default_commit, + help=('Look for a pipeline associated with the given ' + 'commit. If one is not explicitly given, the ' + 'commit associated with the local branch named ' + '"staging" is used. Default: %(default)s')) + parser.add_argument('--verbose', action='store_true', default=False, + help=('A minimal verbosity level that prints the ' + 'overall result of the check/wait')) + + args = parser.parse_args() + + try: + if args.wait: + success = wait_on_pipeline_success( + args.timeout, + args.interval, + args.project_id, + args.commit) + else: + status = get_pipeline_status(args.project_id, + args.commit) + success = status['status'] == 'success' + except Exception as error: # pylint: disable=W0703 + success = False + if args.verbose: + print("ERROR: %s" % error.args[0]) + + if success: + if args.verbose: + print('success') + sys.exit(0) + else: + if args.verbose: + print('failure') + sys.exit(1) + + +if __name__ == '__main__': + main() From patchwork Thu Mar 12 19:36:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 11435307 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7829390 for ; Thu, 12 Mar 2020 19:40:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3DB52206E2 for ; Thu, 12 Mar 2020 19:40:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="JH1rP0O7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3DB52206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:48540 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTgd-0007Xx-Er for patchwork-qemu-devel@patchwork.kernel.org; Thu, 12 Mar 2020 15:40:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44240) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCTdj-0007Vl-8j for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCTdf-0007tO-8K for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:06 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:59095 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jCTdf-0007tE-2c for qemu-devel@nongnu.org; Thu, 12 Mar 2020 15:37:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584041822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q6A6a2hxXlqqiYzbdM/gqVm92XQIxhN3NPrL/ps8Ryk=; b=JH1rP0O7qhC+hEZZU/kMBShw82ScZHm4bIElqdqIRZ1Nj1GI399kd92WuKVA5/wMHq1k0v qWEtXEs7W30S9VdiMnrmoYgorgqL89UNsYqH+S4T37s65xEetcVemnacJtg9E6daY9Zvly zwk4KxwjQcicdrrqiFEuo6in08O5rUU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-490-DhRmXr5oPpm3AkOyk-losw-1; Thu, 12 Mar 2020 15:36:57 -0400 X-MC-Unique: DhRmXr5oPpm3AkOyk-losw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8D1C81013CF8; Thu, 12 Mar 2020 19:36:56 +0000 (UTC) Received: from dhcp-17-173.bos.redhat.com (dhcp-17-173.bos.redhat.com [10.18.17.173]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 558D65C1B5; Thu, 12 Mar 2020 19:36:55 +0000 (UTC) From: Cleber Rosa To: =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell , qemu-devel@nongnu.org Subject: [PATCH 5/5] GitLab Gating CI: initial set of jobs, documentation and scripts Date: Thu, 12 Mar 2020 15:36:16 -0400 Message-Id: <20200312193616.438922-6-crosa@redhat.com> In-Reply-To: <20200312193616.438922-1-crosa@redhat.com> References: <20200312193616.438922-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Thomas Huth , Eduardo Habkost , Erik Skultety , Wainer Moschetta , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Beraldo Leal Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This is a mapping of Peter's "remake-merge-builds" and "pull-buildtest" scripts, gone through some updates, adding some build option and removing others. To add to some of the diversity of platforms that QEMU covers, this initial set of jobs require three machines: - Ubuntu 18.04 - Fedora 30 - FreeBSD 12.1 All jobs have been marked as x86_64, but others can be introduced with different architectures. During the development of this set of jobs, the GitLab CI was tested with many other architectures, including ppc64, s390x and aarch64. More information can be found in the documentation itself. Signed-off-by: Cleber Rosa --- .gitlab-ci-gating.yml | 111 +++++++++++ .gitlab-ci.yml | 15 ++ contrib/ci/orgs/qemu/build-environment.yml | 208 +++++++++++++++++++++ contrib/ci/orgs/qemu/gitlab-runner.yml | 65 +++++++ contrib/ci/orgs/qemu/inventory | 2 + contrib/ci/orgs/qemu/vars.yml | 13 ++ docs/devel/testing.rst | 142 ++++++++++++++ 7 files changed, 556 insertions(+) create mode 100644 .gitlab-ci-gating.yml create mode 100644 contrib/ci/orgs/qemu/build-environment.yml create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml create mode 100644 contrib/ci/orgs/qemu/inventory create mode 100644 contrib/ci/orgs/qemu/vars.yml diff --git a/.gitlab-ci-gating.yml b/.gitlab-ci-gating.yml new file mode 100644 index 0000000000..4c6010bcc4 --- /dev/null +++ b/.gitlab-ci-gating.yml @@ -0,0 +1,111 @@ +variables: + GIT_SUBMODULE_STRATEGY: recursive + +# All ubuntu-18.04.3 jobs should run successfully in an environment +# setup by the contrib/ci/orgs/qemu/build-environment.yml task +# "Install basic packages to build QEMU on Ubuntu 18.04" +ubuntu-18.04.3-x86_64-all-linux-static: + tags: + - ubuntu_18.04.3 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763 + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages + - ./configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + - make --output-sync -j`nproc` check-tcg V=1 + +ubuntu-18.04.3-x86_64-all: + tags: + - ubuntu_18.04.3 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04.3-x86_64-alldbg: + tags: + - ubuntu_18.04.3 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --enable-debug --disable-libssh + - make clean + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04.3-x86_64-clang: + tags: + - ubuntu_18.04.3 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04.3-x86_64-tci: + tags: + - ubuntu_18.04.3 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --enable-tcg-interpreter + - make --output-sync -j`nproc` + +ubuntu-18.04.3-x86_64-notcg: + tags: + - ubuntu_18.04.3 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --disable-libssh --disable-tcg + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +# All fedora-30 jobs should run successfully in an environment +# setup by the contrib/ci/orgs/qemu/build-environment.yml task +# "Install basic packages to build QEMU on Fedora 30" +fedora-30-x86_64-cross-w32: + tags: + - fedora_30 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --cross-prefix=i686-w64-mingw32- + - make --output-sync -j`nproc` + +fedora-30-x86_64-cross-w64: + tags: + - fedora_30 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure --cross-prefix=x86_64-w64-mingw32- + - make --output-sync -j`nproc` + +# All freebsd_12.1 jobs should run successfully in an environment +# setup by the contrib/ci/orgs/qemu/build-environment.yml task +# "Install basic packages to build QEMU on FreeBSD 12.x" +freebsd-12.1-x86_64-all: + tags: + - freebsd_12.1 + - x86_64 + rules: + - if: '$CI_COMMIT_REF_NAME == "staging"' + script: + - ./configure + - gmake --output-sync -j`sysctl -n hw.ncpu` + - gmake --output-sync -j`sysctl -n hw.ncpu` check V=1 diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 3ef1551273..8c51672dc1 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,7 +1,10 @@ include: - local: '/.gitlab-ci-edk2.yml' + - local: '/.gitlab-ci-gating.yml' build-system1: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: &before_scr_apt - apt-get update -qq - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage @@ -17,6 +20,8 @@ build-system1: - make -j2 check build-system2: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: *before_scr_apt script: @@ -32,6 +37,8 @@ build-system2: - make -j2 check build-disabled: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: *before_scr_apt script: @@ -48,6 +55,8 @@ build-disabled: - make -j2 check-qtest SPEED=slow build-tcg-disabled: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: *before_scr_apt script: @@ -68,6 +77,8 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: *before_scr_apt script: @@ -79,6 +90,8 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: *before_scr_apt script: @@ -93,6 +106,8 @@ build-clang: - make -j2 check build-tci: + rules: + - if: '$CI_COMMIT_REF_NAME != "staging"' before_script: *before_scr_apt script: diff --git a/contrib/ci/orgs/qemu/build-environment.yml b/contrib/ci/orgs/qemu/build-environment.yml new file mode 100644 index 0000000000..ea839ded4c --- /dev/null +++ b/contrib/ci/orgs/qemu/build-environment.yml @@ -0,0 +1,208 @@ +--- +- name: Installation of basic packages to build QEMU + hosts: all + vars_files: + - vars.yml + tasks: + - name: Install basic packages to build QEMU on Ubuntu 18.04 + apt: + update_cache: yes + # This matches the packages on tests/docker/Dockerfiles/ubuntu1804.docker + pkg: + - flex + - bison + - ccache + - clang + - gcc + - gettext + - git + - glusterfs-common + - libaio-dev + - libattr1-dev + - libbrlapi-dev + - libbz2-dev + - libcacard-dev + - libcap-ng-dev + - libcurl4-gnutls-dev + - libdrm-dev + - libepoxy-dev + - libfdt-dev + - libgbm-dev + - libgtk-3-dev + - libibverbs-dev + - libiscsi-dev + - libjemalloc-dev + - libjpeg-turbo8-dev + - liblzo2-dev + - libncurses5-dev + - libncursesw5-dev + - libnfs-dev + - libnss3-dev + - libnuma-dev + - libpixman-1-dev + - librados-dev + - librbd-dev + - librdmacm-dev + - libsasl2-dev + - libsdl2-dev + - libseccomp-dev + - libsnappy-dev + - libspice-protocol-dev + - libspice-server-dev + - libssh-dev + - libusb-1.0-0-dev + - libusbredirhost-dev + - libvdeplug-dev + - libvte-2.91-dev + - libxen-dev + - libzstd-dev + - make + - python3-yaml + - python3-sphinx + - sparse + - texinfo + - xfslibs-dev + state: present + when: "ansible_facts['distribution'] == 'Ubuntu'" + + - name: Install basic packages to build QEMU on FreeBSD 12.x + pkgng: + # This matches the packages on .cirrus.yml under the freebsd_12_task + name: bash,bison,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,perl5,pixman,pkgconf,png,usbredir + state: present + when: "ansible_facts['os_family'] == 'FreeBSD'" + + - name: Install basic packages to build QEMU on Fedora 30 + dnf: + # This matches the packages on tests/docker/Dockerfiles/fedora.docker + name: + - bc + - bison + - brlapi-devel + - bzip2 + - bzip2-devel + - ccache + - clang + - cyrus-sasl-devel + - dbus-daemon + - device-mapper-multipath-devel + - findutils + - flex + - gcc + - gcc-c++ + - gettext + - git + - glib2-devel + - glusterfs-api-devel + - gnutls-devel + - gtk3-devel + - hostname + - libaio-devel + - libasan + - libattr-devel + - libblockdev-mpath-devel + - libcap-ng-devel + - libcurl-devel + - libfdt-devel + - libiscsi-devel + - libjpeg-devel + - libpmem-devel + - libpng-devel + - librbd-devel + - libseccomp-devel + - libssh-devel + - libubsan + - libusbx-devel + - libxml2-devel + - llvm + - lzo-devel + - make + - mingw32-bzip2 + - mingw32-curl + - mingw32-glib2 + - mingw32-gmp + - mingw32-gnutls + - mingw32-gtk3 + - mingw32-libjpeg-turbo + - mingw32-libpng + - mingw32-libtasn1 + - mingw32-nettle + - mingw32-nsis + - mingw32-pixman + - mingw32-pkg-config + - mingw32-SDL2 + - mingw64-bzip2 + - mingw64-curl + - mingw64-glib2 + - mingw64-gmp + - mingw64-gnutls + - mingw64-gtk3 + - mingw64-libjpeg-turbo + - mingw64-libpng + - mingw64-libtasn1 + - mingw64-nettle + - mingw64-pixman + - mingw64-pkg-config + - mingw64-SDL2 + - ncurses-devel + - nettle-devel + - nss-devel + - numactl-devel + - perl + - perl-Test-Harness + - pixman-devel + - python3 + - python3-sphinx + - PyYAML + - rdma-core-devel + - SDL2-devel + - snappy-devel + - sparse + - spice-server-devel + - systemd-devel + - systemtap-sdt-devel + - tar + - texinfo + - usbredir-devel + - virglrenderer-devel + - vte291-devel + - which + - xen-devel + - zlib-devel + - libzstd-devel + state: present + when: "ansible_facts['distribution'] == 'Fedora'" + + - name: Install basic packages to build QEMU on CentOS 8 + dnf: + # This matches the packages on tests/docker/Dockerfiles/centos8.docker + name: + - SDL-devel + - bison + - bzip2 + - bzip2-devel + - dbus-daemon + - flex + - gcc + - gcc-c++ + - gettext + - git + - glib2-devel + - libaio-devel + - libepoxy-devel + - lzo-devel + - make + - mesa-libEGL-devel + - nettle-devel + - perl-Test-Harness + - pixman-devel + - python36 + - rdma-core-devel + - spice-glib-devel + - spice-server + - tar + - zlib-devel + state: present + when: + - "ansible_facts['distribution'] == 'CentOS'" + - "ansible_facts['distribution_major_version'] == '8'" diff --git a/contrib/ci/orgs/qemu/gitlab-runner.yml b/contrib/ci/orgs/qemu/gitlab-runner.yml new file mode 100644 index 0000000000..19dc31a299 --- /dev/null +++ b/contrib/ci/orgs/qemu/gitlab-runner.yml @@ -0,0 +1,65 @@ +--- +- name: Installation of gitlab-runner + hosts: all + vars_files: + - vars.yml + tasks: + - debug: + msg: 'Checking for a valid GitLab registration token' + failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'" + + - name: Checks the availability of official gitlab-runner builds in the archive + uri: + url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-linux-386 + method: HEAD + status_code: + - 200 + - 403 + register: gitlab_runner_available_archive + + - name: Update base url + set_fact: + gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner- + when: gitlab_runner_available_archive.status == 200 + - debug: + msg: Base gitlab-runner url is {{ gitlab_runner_base_url }} + + - name: Set OS name (FreeBSD) + set_fact: + gitlab_runner_os: freebsd + when: "ansible_facts['system'] == 'FreeBSD'" + + - name: Create a group for the gitlab-runner service + group: + name: gitlab-runner + - name: Create a user for the gitlab-runner service + user: + user: gitlab-runner + group: gitlab-runner + comment: GitLab Runner + home: /home/gitlab-runner + shell: /bin/bash + + - name: Downloads the matching gitlab-runner + get_url: + dest: /usr/local/bin/gitlab-runner + url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" + owner: gitlab-runner + group: gitlab-runner + mode: u=rwx,g=rwx,o=rx + + - name: Register the gitlab-runner + command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" + + - name: Install the gitlab-runner service using its own functionality + command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner + register: gitlab_runner_install_service_result + failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" + when: "ansible_facts['os_family'] != 'FreeBSD'" + + - name: Enable the gitlab-runner service + service: + name: gitlab-runner + state: started + enabled: yes + when: "ansible_facts['os_family'] != 'FreeBSD'" diff --git a/contrib/ci/orgs/qemu/inventory b/contrib/ci/orgs/qemu/inventory new file mode 100644 index 0000000000..8bb7ba6b33 --- /dev/null +++ b/contrib/ci/orgs/qemu/inventory @@ -0,0 +1,2 @@ +[local] +localhost diff --git a/contrib/ci/orgs/qemu/vars.yml b/contrib/ci/orgs/qemu/vars.yml new file mode 100644 index 0000000000..0095851172 --- /dev/null +++ b/contrib/ci/orgs/qemu/vars.yml @@ -0,0 +1,13 @@ +# The version of the gitlab-runner to use +gitlab_runner_version: 12.8.0 +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner- +# The URL of the gitlab server to use, usually https://gitlab.com unless you're +# using a private GitLab instance +gitlab_runner_server_url: https://gitlab.com +# Defaults to linux, checks can be used to change this +gitlab_runner_os: linux +# Defaults to amd64 (x86_64), checks can be used to change this +gitlab_runner_arch: amd64 +# A unique token made available by GitLab to your project for registering runners +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 770a987ea4..6f8e04e079 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -896,3 +896,145 @@ exercise as many corner cases as possible. It is a useful test suite to run to exercise QEMU's linux-user code:: https://linux-test-project.github.io/ + +CI +== + +QEMU has configurations enabled for a number of different CI services. +The most update information about them and their status can be found +at:: + + https://wiki.qemu.org/Testing/CI + +Gating CI +---------- + +A Pull Requests will only to be merged if they successfully go through +a different set of CI jobs. GitLab's CI is the service/framework used +for executing the gating jobs. + +The architecture of GitLab's CI service allows different machines to be +setup with GitLab's "agent", called gitlab-runner, which will take care +of running jobs created by events such as a push to a branch. + +Even though gitlab-runner can execute jobs on environments such as +containers, this initial implementation assumes the shell executor is +used, effectively running jobs on the same machine (be them physical +or virtual) the gitlab-runner agent is running. This means those +machines must be setup in advance, with the requirements matching the +jobs expected to be executed there. + +Machine configuration for gating jobs +------------------------------------- + +The GitLab's CI architecture allows different parties to provide +different machines that will run different jobs. At this point, QEMU +will deploy a limited set of machines and jobs. Documentation and/or +scripts to setup those machines is located under:: + + contrib/ci/orgs/qemu + +Ansible playbooks have been provided to perform two different tasks +related to setting gitlab-runner and the build environment. + +Other organizations involved in QEMU development may, in the near +future, contribute their own setup documentation/scripts under +``contrib/ci/orgs/$ORG_NAME`` directory. + +The GitLab CI jobs definition for the gating jobs are located under:: + + .gitlab-ci-gating.yml + +It's expected that these will be moved to a ``.gitlab-ci.d/`` +directory shortly. Once the gating CI has proved mature with this set +of initial jobs, other members in the community may provide their own +machine configuration documentation/scripts, and accompanying job +definitions. Those contributed jobs should run as non-gating, until +their reliability is verified. + +Machine Setup Howto +------------------- + +For all Linux based systems, the setup can be mostly automated by the +execution of two Ansible playbooks. Start by adding your machines to +the ``inventory`` file under ``contrib/ci/orgs/qemu``, such as this:: + + [local] + fully.qualified.domain + other.machine.hostname + +You may need to set some variables in the inventory file itself. One +very common need is to tell Ansible to use a Python 3 interpreter on +those hosts. This would look like:: + + [local] + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 + +Build environment +~~~~~~~~~~~~~~~~~ + +The ``contrib/ci/orgs/qemu/build-environment.yml`` Ansible playbook +will setup machines with the environment needed to perform builds of +QEMU and runs of the tests defined in the current gating jobs. It +covers a number of different Linux distributions and FreeBSD. + +To run the playbook, execute:: + + cd contrib/ci/orgs/qemu + ansible-playbook -i inventory build-environment.yml + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that +will run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * Under "Set up a specific Runner manually", look for the value under + "Use the following registration token during setup" + +Edit the ``contrib/ci/orgs/qemu/vars.yml`` file, setting the +``gitlab_runner_registration_token`` variable to the value obtained +earlier. + +To run the playbook, execute:: + + cd contrib/ci/orgs/qemu + ansible-playbook -i inventory gitlab-runner.yml + +.. note:: there are currently limitations to gitlab-runner itself when + setting up a service under FreeBSD systems. You will need to + perform steps 4 to 10 manually, as described at + https://docs.gitlab.com/runner/install/freebsd.html + +Following the registration, it's necessary to configure the runner tags, +and optionally other configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * "Runners activated for this project", then + * Click on the "Edit" icon (next to the "Lock" Icon) + +Under tags, add values matching the jobs a runner should run. For a +FreeBSD 12.1 x86_64 system, the tags should be set as:: + + freebsd12.1,x86_64 + +Because the job definition at ``.gitlab-ci-gating.yml`` contains:: + + freebsd-12.1-x86_64-all: + tags: + - freebsd_12.1 + - x86_64 + +It's also recommended to: + + * increase the "Maximum job timeout" to something like ``2h`` + * uncheck the "Run untagged jobs" check box + * give it a better Description