From patchwork Fri Sep 24 18:54:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 12516641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DFBDC433EF for ; Fri, 24 Sep 2021 19:05:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2CDEE610FD for ; Fri, 24 Sep 2021 19:05:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2CDEE610FD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:33264 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mTqVK-0004Mq-Bb for qemu-devel@archiver.kernel.org; Fri, 24 Sep 2021 15:05:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39214) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTqO9-0003K8-TO for qemu-devel@nongnu.org; Fri, 24 Sep 2021 14:57:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:37438) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTqO8-0005Cm-3F for qemu-devel@nongnu.org; Fri, 24 Sep 2021 14:57:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632509855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=nL7iz+RrEoo9yAMNBG9KYSlsAKM1sZClVcQmCtA5nGE=; b=AOUNsuGHsatgKWw7W/TT5NtmqJgtj/XX9diAE4R4twh/fDV3NCIh6fV96LJS8tCJNQaQJt kVq1odIrskeMdRhehoxA3oZFgGTqYPs1VeOxeVb0reTrAs0K+04d2qpEE7TuLafOmbQ1kO Tp7+X547ZCtLfnc92PnXHDnuzkuoIRM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-534-vVsrc7b1MniNXiqodQvqxw-1; Fri, 24 Sep 2021 14:56:26 -0400 X-MC-Unique: vVsrc7b1MniNXiqodQvqxw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1F649100C664; Fri, 24 Sep 2021 18:56:24 +0000 (UTC) Received: from p50.net (unknown [10.22.33.186]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 711815FC13; Fri, 24 Sep 2021 18:55:12 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Subject: [PATCH 00/16] Acceptance Tests: use Avocado 91.0 features and other improvements Date: Fri, 24 Sep 2021 14:54:50 -0400 Message-Id: <20210924185506.2542588-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=crosa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=crosa@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.473, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-ppc@nongnu.org, Thomas Huth , Beraldo Leal , Yoshinori Sato , Pavel Dovgalyuk , =?utf-8?q?Alex_Benn=C3=A9e?= , =?utf-8?q?Herv=C3=A9?= =?utf-8?q?_Poussineau?= , Greg Kurz , Wainer dos Santos Moschetta , Auger Eric , Kamil Rytarowski , Paolo Bonzini , Reinoud Zandijk , David Gibson , Willian Rampazzo , Cleber Rosa , Alexandre Iooss , Mahmoud Mandour , Ryo ONODERA , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is a collection of patches for the Acceptance Tests to leverage some of the features of Avocado 91.0. With the Avocado version bump by itself, there would be a change in the default "test runner" implementation that Avocado uses, from the one simply known as "runner" to the new one called "nrunner". Among the changes from one implementation to the other, is the fact that "nrunner" will run tests in parallel by default. This is *not yet* enabled by default on "make check-acceptance", but users can choose to use simply by setting the "AVOCADO_RUNNER" variable, that is: make AVOCADO_RUNNER=nrunner check-acceptance If you are curious about the architectural differences of the nrunner, please refer to: https://avocado-framework.readthedocs.io/en/91.0/guides/contributor/chapters/runners.html One other noteworthy proposal is a convention to tag tests that either have known issues, or that touch on QEMU features that have known issues. By tagging those tests accordingly, they will be automatically excluded from the regular execution with "make check-acceptance". Finally, some updates to assets locations and some tests refactors and cleanups. Cleber Rosa (16): Acceptance Tests: bump Avocado requirement to 91.0 Acceptance Tests: improve check-acceptance description Acceptance Tests: add mechanism for listing tests Acceptance Tests: keep track and disable tests with known issues Acceptance Tests: add standard clean up at test tearDown() Acceptance Tests: use extract from package from avocado.utils Acceptance Tests: workaround expired mipsdistros.mips.com HTTPS cert acceptance/tests/vnc.py: use explicit syntax for enabling passwords tests/acceptance/boot_xen.py: merge base classes tests/acceptance/boot_xen.py: unify tags tests/acceptance/boot_xen.py: fetch kernel during test setUp() tests/acceptance/boot_xen.py: removed unused import tests/acceptance/boot_xen.py: use class attribute tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update tests/acceptance/ppc_prep_40p.py: clean up unused import tests/acceptance/ppc_prep_40p.py: unify tags docs/devel/testing.rst | 40 ++++++++++ tests/Makefile.include | 15 +++- tests/acceptance/avocado_qemu/__init__.py | 1 + tests/acceptance/boot_linux_console.py | 93 +++++++++-------------- tests/acceptance/boot_xen.py | 54 ++++--------- tests/acceptance/machine_rx_gdbsim.py | 3 + tests/acceptance/ppc_prep_40p.py | 17 ++--- tests/acceptance/replay_kernel.py | 18 ++--- tests/acceptance/tcg_plugins.py | 2 +- tests/acceptance/vnc.py | 2 +- tests/requirements.txt | 2 +- 11 files changed, 128 insertions(+), 119 deletions(-)