From patchwork Fri Feb 25 21:01:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 12760960 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E979EC4321E for ; Fri, 25 Feb 2022 21:24:04 +0000 (UTC) Received: from localhost ([::1]:52168 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nNi4K-0000EZ-2I for qemu-devel@archiver.kernel.org; Fri, 25 Feb 2022 16:24:04 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60044) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nNhk7-0006i4-Ri for qemu-devel@nongnu.org; Fri, 25 Feb 2022 16:03:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:40060) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nNhk4-0005O3-PP for qemu-devel@nongnu.org; Fri, 25 Feb 2022 16:03:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645822988; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+2Dt1lT9Scd1l2FLzZxR50BVG4rXXdKWj0GOBfUSf6Y=; b=CGUVhXuDnWFHYf0RSvBtl0CWAMBLM1RQHrqkRBee5rJSaHRFngPVzc7eQOhVSXi5nydMo/ UeuqzoRv+f0GgX/Xq3mQ+4DRzVm1RtOgRcPRAb+YO/r1cUmk/hHHzlryVTFM6VXTrkQuJS 0MV1NXCBWXiqzqH00pWJ7xYTG7QhCSE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-ov4IEeldP0yszeT4MjFVsA-1; Fri, 25 Feb 2022 16:03:00 -0500 X-MC-Unique: ov4IEeldP0yszeT4MjFVsA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 740D11006AA5; Fri, 25 Feb 2022 21:02:55 +0000 (UTC) Received: from p50.localhost.localdomain.com (unknown [10.22.16.50]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CB4B32ED79; Fri, 25 Feb 2022 21:02:01 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org, =?utf-8?q?Alex_Benn=C3=A9e?= , =?utf-8?q?Daniel_P_?= =?utf-8?q?=2E_Berrang=C3=A9?= , Peter Maydell Subject: [PATCH 0/9] Avocado tests: filter out tests using TCG booting full blown distros Date: Fri, 25 Feb 2022 16:01:47 -0500 Message-Id: <20220225210156.2032055-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=crosa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=crosa@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Huacai Chen , Mark Cave-Ayland , Eric Farman , KONRAD Frederic , "Edgar E. Iglesias" , Alexandre Iooss , Yoshinori Sato , Daniel Henrique Barboza , Wainer dos Santos Moschetta , Halil Pasic , =?utf-8?q?Herv=C3=A9_Poussineau?= , Antony Pavlov , Christian Borntraeger , Thomas Huth , Greg Kurz , Fabien Chouteau , Eric Auger , qemu-s390x@nongnu.org, qemu-arm@nongnu.org, Michael Rolnik , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Pavel Dovgalyuk , Cleber Rosa , David Gibson , Beraldo Leal , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , qemu-ppc@nongnu.org, Paolo Bonzini , Mahmoud Mandour , Aurelien Jarno Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" It was previously reported[1] and discussed that tests booting full blown distros and relying on TCG would take too much time to run, especially in the environments given by GitLab CI's shared runners. This is an implementation of a proposal to exclude those tests from being run by default on `make check-avocado` invocations. To make it extra clear, all tests are still available, but those that are tagged with "accel:tcg" and "boots:distro", are filtered out by default on `make check-avocado`. This is the situation of the Avocado GitLab CI jobs with and without the changes in this PS: +------------------+--- ----------------+ | Now[2] | Before[3] | +------------------------+------------------+--------------------+ | Job | Length | Tests | Length | Tests | | Name | (mm:ss) | Run | (mm:ss) | Run | +------------------------+------------------+-------------------+ |avocado-system-alpine | 06:33 16 | 20:30 18 | |avocado-system-debian | 12:06 24 | 13:05 24 | |avocado-system-centos | 09:58 41 | 24:15 44 | |avocado-system-fedora | 08:50 35 | 08:59 35 | |avocado-system-opensuse | 08:09 38 | 27:21 42 | |avocado-system-ubuntu | 06:52 16 | 18:52 18 | |avocado-cfi-x86_64 | 05:43 27 | 15:07 29 | +------------------------+------------------+--------------------+ |TOTALS | 58:11 197 | 2:08:09 210 | +------------------------+------------------+--------------------+ Assuming the jobs run in parallel, the overall wait time for all the Avocado jobs to complete is now ~12 minutes. [1] https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg07271.html [2] https://gitlab.com/cleber.gnu/qemu/-/pipelines/479720240 [3] https://gitlab.com/qemu-project/qemu/-/pipelines/478580581 Cleber Rosa (9): Avocado GitLab CI jobs: don't reset TARGETS and simplify commands Avocado tests: use logging namespace that is preserved in test logs Avocado migration test: adapt to "utils.network" API namespace change Avocado: bump to version 95.0 tests/avocado/linux_ssh_mips_malta.py: add missing accel (tcg) tag tests/avocado/virtiofs_submounts.py: shared_dir may not exist Avocado tests: improve documentation on tag filtering Avocado tests: classify tests based on what it's booted Avocado tests: don't run tests with TCG that boot full blown distros .gitlab-ci.d/buildtest-template.yml | 3 ++ .gitlab-ci.d/buildtest.yml | 9 ---- docs/devel/testing.rst | 22 +++++++++ tests/Makefile.include | 6 ++- tests/avocado/avocado_qemu/__init__.py | 10 ++--- tests/avocado/boot_linux.py | 4 ++ tests/avocado/boot_linux_console.py | 54 +++++++++++++++++++++++ tests/avocado/boot_xen.py | 3 ++ tests/avocado/hotplug_cpu.py | 1 + tests/avocado/intel_iommu.py | 1 + tests/avocado/linux_initrd.py | 5 ++- tests/avocado/linux_ssh_mips_malta.py | 5 +++ tests/avocado/machine_arm_canona1100.py | 1 + tests/avocado/machine_arm_integratorcp.py | 7 ++- tests/avocado/machine_arm_n8x0.py | 2 + tests/avocado/machine_avr6.py | 1 + tests/avocado/machine_m68k_nextcube.py | 1 + tests/avocado/machine_microblaze.py | 1 + tests/avocado/machine_mips_fuloong2e.py | 1 + tests/avocado/machine_mips_loongson3v.py | 1 + tests/avocado/machine_mips_malta.py | 6 ++- tests/avocado/machine_rx_gdbsim.py | 2 + tests/avocado/machine_s390_ccw_virtio.py | 4 ++ tests/avocado/machine_sparc64_sun4u.py | 1 + tests/avocado/machine_sparc_leon3.py | 1 + tests/avocado/migration.py | 4 +- tests/avocado/multiprocess.py | 4 ++ tests/avocado/ppc_405.py | 2 + tests/avocado/ppc_bamboo.py | 2 + tests/avocado/ppc_mpc8544ds.py | 1 + tests/avocado/ppc_prep_40p.py | 1 + tests/avocado/ppc_pseries.py | 1 + tests/avocado/ppc_virtex_ml507.py | 1 + tests/avocado/replay_kernel.py | 33 ++++++++++++-- tests/avocado/replay_linux.py | 6 +-- tests/avocado/reverse_debugging.py | 6 +-- tests/avocado/smmu.py | 1 + tests/avocado/tcg_plugins.py | 3 ++ tests/avocado/tesseract_utils.py | 6 +-- tests/avocado/virtio-gpu.py | 2 + tests/avocado/virtio_check_params.py | 3 +- tests/avocado/virtiofs_submounts.py | 8 ++-- tests/requirements.txt | 2 +- 43 files changed, 197 insertions(+), 41 deletions(-)