From patchwork Thu Sep 20 15:19:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 10607957 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27C09112B for ; Thu, 20 Sep 2018 15:22:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 190142DC92 for ; Thu, 20 Sep 2018 15:22:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1739E2DCB2; Thu, 20 Sep 2018 15:22:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 818DF2DC92 for ; Thu, 20 Sep 2018 15:22:24 +0000 (UTC) Received: from localhost ([::1]:51369 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g30md-0000rX-OP for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Sep 2018 11:22:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50378) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g30lD-0007mf-9m for qemu-devel@nongnu.org; Thu, 20 Sep 2018 11:20:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g30l9-00085m-D6 for qemu-devel@nongnu.org; Thu, 20 Sep 2018 11:20:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36250) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g30l9-000843-7C for qemu-devel@nongnu.org; Thu, 20 Sep 2018 11:20:51 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 27D713091D65 for ; Thu, 20 Sep 2018 15:20:50 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-125-88.rdu2.redhat.com [10.10.125.88]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 41AA87D65E; Thu, 20 Sep 2018 15:20:49 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Thu, 20 Sep 2018 11:19:55 -0400 Message-Id: <20180920151957.28803-2-crosa@redhat.com> In-Reply-To: <20180920151957.28803-1-crosa@redhat.com> References: <20180920151957.28803-1-crosa@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Thu, 20 Sep 2018 15:20:50 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/2] Bootstrap Python venv for tests X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Fam Zheng , Eduardo Habkost , Stefan Hajnoczi , Cleber Rosa Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP A number of QEMU tests are written in Python, and may benefit from an untainted Python venv. By using make rules, tests that depend on specific Python libs can set that rule as a requiment, along with rules that require the presence or installation of specific libraries. The tests/venv-requirements.txt is supposed to contain the Python requirements that should be added to the venv created by check-venv. Signed-off-by: Cleber Rosa --- tests/Makefile.include | 19 +++++++++++++++++++ tests/venv-requirements.txt | 3 +++ 2 files changed, 22 insertions(+) create mode 100644 tests/venv-requirements.txt diff --git a/tests/Makefile.include b/tests/Makefile.include index 87c81d1dcc..9bb90a83d4 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -12,6 +12,7 @@ check-help: @echo " $(MAKE) check-block Run block tests" @echo " $(MAKE) check-tcg Run TCG tests" @echo " $(MAKE) check-report.html Generates an HTML test report" + @echo " $(MAKE) check-venv Creates a Python venv for tests" @echo " $(MAKE) check-clean Clean the tests" @echo @echo "Please note that HTML reports do not regenerate if the unit tests" @@ -999,6 +1000,23 @@ check-decodetree: ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \ TEST, decodetree.py) +# Python venv for running tests + +.PHONY: check-venv + +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv +TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt + +$(TESTS_VENV_DIR): + $(call quiet-command, \ + $(PYTHON) -m venv $@, \ + VENV, $@) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \ + PIP, $(TESTS_VENV_REQ)) + +check-venv: $(TESTS_VENV_DIR) + # Consolidated targets .PHONY: check-qapi-schema check-qtest check-unit check check-clean @@ -1012,6 +1030,7 @@ check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y)) rm -f tests/test-qapi-gen-timestamp + rm -rf $(TESTS_VENV_DIR) clean: check-clean diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt new file mode 100644 index 0000000000..d39f9d1576 --- /dev/null +++ b/tests/venv-requirements.txt @@ -0,0 +1,3 @@ +# Add Python module requirements, one per line, to be installed +# in the tests/venv Python virtual environment. For more info, +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1 From patchwork Thu Sep 20 15:19:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 10607955 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C16C6174F for ; Thu, 20 Sep 2018 15:22:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B112F2DC88 for ; Thu, 20 Sep 2018 15:22:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AF2282DC90; Thu, 20 Sep 2018 15:22:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 34F802DC92 for ; Thu, 20 Sep 2018 15:22:23 +0000 (UTC) Received: from localhost ([::1]:51368 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g30mc-0000oO-Go for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Sep 2018 11:22:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50381) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g30lD-0007mh-Ah for qemu-devel@nongnu.org; Thu, 20 Sep 2018 11:20:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g30lA-00086M-62 for qemu-devel@nongnu.org; Thu, 20 Sep 2018 11:20:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37334) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g30l9-00085q-T1 for qemu-devel@nongnu.org; Thu, 20 Sep 2018 11:20:52 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 41E82550D6 for ; Thu, 20 Sep 2018 15:20:51 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-125-88.rdu2.redhat.com [10.10.125.88]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5C0EA7A4DA; Thu, 20 Sep 2018 15:20:50 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Thu, 20 Sep 2018 11:19:56 -0400 Message-Id: <20180920151957.28803-3-crosa@redhat.com> In-Reply-To: <20180920151957.28803-1-crosa@redhat.com> References: <20180920151957.28803-1-crosa@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 20 Sep 2018 15:20:51 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Fam Zheng , Eduardo Habkost , Stefan Hajnoczi , Cleber Rosa Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP The acceptance (aka functional, aka Avocado-based) tests are Python files located in "tests/acceptance" that need to be run with the Avocado libs and test runner. Let's provide a convenient way for QEMU developers to run them, by making use of the tests-venv with the required setup. Also, while the Avocado test runner will take care of creating a location to save test results to, it was understood that it's better if the results are kept within the build tree. Signed-off-by: Cleber Rosa --- docs/devel/testing.rst | 28 +++++++++++++++++++++++----- tests/Makefile.include | 17 +++++++++++++++-- tests/venv-requirements.txt | 1 + 3 files changed, 39 insertions(+), 7 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 727c4019b5..0fbf0d0aac 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -545,10 +545,24 @@ Tests based on ``avocado_qemu.Test`` can easily: - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html -Installation ------------- +Running tests +------------- -To install Avocado and its dependencies, run: +You can run the acceptance tests simply by executing: + +.. code:: + + make check-acceptance + +This involves the automatic creation of Python virtual environment +within the build tree (at ``tests/venv``) which will have all the +right dependencies, and will save tests results also within the +build tree (at ``tests/results``). + +Manual Installation +------------------- + +To manually install Avocado and its dependencies, run: .. code:: @@ -689,11 +703,15 @@ The exact QEMU binary to be used on QEMUMachine. Uninstalling Avocado -------------------- -If you've followed the installation instructions above, you can easily -uninstall Avocado. Start by listing the packages you have installed:: +If you've followed the manual installation instructions above, you can +easily uninstall Avocado. Start by listing the packages you have +installed:: pip list --user And remove any package you want with:: pip uninstall + +If you've used ``make check-acceptance``, the Python virtual environment where +Avocado is installed will be cleaned up as part of ``make check-clean``. diff --git a/tests/Makefile.include b/tests/Makefile.include index 9bb90a83d4..8cef694954 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -11,6 +11,7 @@ check-help: @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-block Run block tests" @echo " $(MAKE) check-tcg Run TCG tests" + @echo " $(MAKE) check-acceptance Run all acceptance (functional) tests" @echo " $(MAKE) check-report.html Generates an HTML test report" @echo " $(MAKE) check-venv Creates a Python venv for tests" @echo " $(MAKE) check-clean Clean the tests" @@ -1002,10 +1003,11 @@ check-decodetree: # Python venv for running tests -.PHONY: check-venv +.PHONY: check-venv check-acceptance TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results $(TESTS_VENV_DIR): $(call quiet-command, \ @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR): $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \ PIP, $(TESTS_VENV_REQ)) +$(TESTS_RESULTS_DIR): + $(call quiet-command, mkdir -p $@, \ + MKDIR, $@) + check-venv: $(TESTS_VENV_DIR) +check-acceptance: check-venv $(TESTS_RESULTS_DIR) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/avocado \ + --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) --failfast=on \ + $(SRC_PATH)/tests/acceptance, \ + "AVOCADO", "tests/acceptance") + # Consolidated targets .PHONY: check-qapi-schema check-qtest check-unit check check-clean @@ -1030,7 +1043,7 @@ check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y)) rm -f tests/test-qapi-gen-timestamp - rm -rf $(TESTS_VENV_DIR) + rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) clean: check-clean diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt index d39f9d1576..1734d0ce27 100644 --- a/tests/venv-requirements.txt +++ b/tests/venv-requirements.txt @@ -1,3 +1,4 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 +avocado-framework==64.0