| Message ID | 20220627072856.1529357-10-ani@anisinha.ca (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Introduce new acpi/smbios qtests using biosbits | expand |
On 27/06/2022 09.28, Ani Sinha wrote: > This change adds python based qtest framework that can be used to run > qtests from within a virtual environment. A bash script creates the virtual > environment and then runs the python based tests from within that environment. > All dependent python packages are installed in the virtual environment using > pip module. QEMU python test modules are also available in the environment for > spawning the QEMU based VMs. > > It also introduces QEMU acpi/smbios biosbits python test script which is run > from within the python virtual environment. > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/qtest/acpi-bits/acpi-bits-test-venv.sh | 59 ++++ > tests/qtest/acpi-bits/acpi-bits-test.py | 327 +++++++++++++++++++ > tests/qtest/acpi-bits/meson.build | 39 +++ > tests/qtest/acpi-bits/requirements.txt | 1 + > 4 files changed, 426 insertions(+) > create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh > create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py > create mode 100644 tests/qtest/acpi-bits/meson.build > create mode 100644 tests/qtest/acpi-bits/requirements.txt > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh > new file mode 100644 > index 0000000000..124e03ce18 > --- /dev/null > +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh > @@ -0,0 +1,59 @@ > +#!/usr/bin/env bash > +# Generates a python virtual environment for the test to run. > +# Then runs python test scripts from within that virtual environment. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Author: Ani Sinha <ani@anisinha.ca> > + > +set -e > + > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > +MYDIR=$(dirname $MYPATH) > + > +if [ -z "$QTEST_SOURCE_ROOT" ]; then > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > + echo " to the root of the qemu source tree." > + echo -n "This is required so that the test can find the " > + echo "python modules that it needs for execution." > + exit 1 > +fi > +SRCDIR=$QTEST_SOURCE_ROOT > +TESTSCRIPTS=("acpi-bits-test.py") > +PIPCMD="-m pip -q --disable-pip-version-check" > +# we need to save the old value of PWD before we do a change-dir later > +QTEST_PWD=$PWD > + > +TESTS_PYTHON=/usr/bin/python3 > +TESTS_VENV_REQ=requirements.txt > + > +# sadly for pip -e and -t options do not work together. > +# please see https://github.com/pypa/pip/issues/562 > +cd $MYDIR > + > +$TESTS_PYTHON -m venv . > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > +[ -f $TESTS_VENV_REQ ] && \ > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ > + > +# venv is activated at this point. > + > +# run the test > +for testscript in ${TESTSCRIPTS[@]} ; do > + export QTEST_PWD; python3 $testscript > +done > + > +cd $QTEST_PWD > + > +exit 0 > diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py > new file mode 100644 > index 0000000000..673567bf8e > --- /dev/null > +++ b/tests/qtest/acpi-bits/acpi-bits-test.py > @@ -0,0 +1,327 @@ > +#!/usr/bin/env python3 > +# group: rw quick > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > +# https://biosbits.org/ > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Some parts are slightly taken from qtest.py and iotests.py > +# > +# Authors: > +# Ani Sinha <ani@anisinha.ca> > + > +# pylint: disable=invalid-name > + > +""" > +QEMU bios tests using biosbits available at > +https://biosbits.org/. > +""" > + > +import logging > +import os > +import re > +import shutil > +import subprocess > +import sys > +import tarfile > +import tempfile > +import time > +import unittest > +import zipfile > +from typing import ( > + List, > + Optional, > + Sequence, > +) > +from tap import TAPTestRunner > +from qemu.machine import QEMUMachine > + > +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY') > +QTEST_PWD = os.getenv('QTEST_PWD') > + > +def get_arch(): > + """finds the arch from the qemu binary name""" > + match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG) > + if match: > + return match.group(1) > + return 'x86_64' > + > +ARCH = get_arch() > + > +class QEMUBitsMachine(QEMUMachine): > + """ > + A QEMU VM, with isa-debugcon enabled and bits iso passed > + using -cdrom to QEMU commandline. > + """ > + def __init__(self, > + binary: str, > + args: Sequence[str] = (), > + wrapper: Sequence[str] = (), > + name: Optional[str] = None, > + base_temp_dir: str = "/var/tmp", > + debugcon_log: str = "debugcon-log.txt", > + debugcon_addr: str = "0x403", > + sock_dir: Optional[str] = None, > + qmp_timer: Optional[float] = None): > + # pylint: disable=too-many-arguments > + > + if name is None: > + name = "qemu-bits-%d" % os.getpid() > + if sock_dir is None: > + sock_dir = base_temp_dir > + super().__init__(binary, args, wrapper=wrapper, name=name, > + base_temp_dir=base_temp_dir, > + sock_dir=sock_dir, qmp_timer=qmp_timer) > + self.debugcon_log = debugcon_log > + self.debugcon_addr = debugcon_addr > + self.base_temp_dir = base_temp_dir > + > + @property > + def _base_args(self) -> List[str]: > + args = super()._base_args > + args.extend([ > + '-chardev', > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > + self.debugcon_log), > + '-device', > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > + ]) > + return args So is this patch series *really* related to qtests? qtests are using a special "accelerator" mode of QEMU where it gets launched with the "-qtest" parameter. I can't see that you're using "-qtest" here or anywhere else, so this rather looks like another framework to me to run python-based QEMU tests (similar to the avocado tests). Thus if this is really not related to qtests, may I suggest to move this into another folder instead? Maybe tests/pytests/acpi-bits or something similar? Thomas
On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote: > > On 27/06/2022 09.28, Ani Sinha wrote: > > This change adds python based qtest framework that can be used to run > > qtests from within a virtual environment. A bash script creates the virtual > > environment and then runs the python based tests from within that environment. > > All dependent python packages are installed in the virtual environment using > > pip module. QEMU python test modules are also available in the environment for > > spawning the QEMU based VMs. > > > > It also introduces QEMU acpi/smbios biosbits python test script which is run > > from within the python virtual environment. > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/qtest/acpi-bits/acpi-bits-test-venv.sh | 59 ++++ > > tests/qtest/acpi-bits/acpi-bits-test.py | 327 +++++++++++++++++++ > > tests/qtest/acpi-bits/meson.build | 39 +++ > > tests/qtest/acpi-bits/requirements.txt | 1 + > > 4 files changed, 426 insertions(+) > > create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh > > create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py > > create mode 100644 tests/qtest/acpi-bits/meson.build > > create mode 100644 tests/qtest/acpi-bits/requirements.txt > > > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh > > new file mode 100644 > > index 0000000000..124e03ce18 > > --- /dev/null > > +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh > > @@ -0,0 +1,59 @@ > > +#!/usr/bin/env bash > > +# Generates a python virtual environment for the test to run. > > +# Then runs python test scripts from within that virtual environment. > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Author: Ani Sinha <ani@anisinha.ca> > > + > > +set -e > > + > > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > > +MYDIR=$(dirname $MYPATH) > > + > > +if [ -z "$QTEST_SOURCE_ROOT" ]; then > > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > > + echo " to the root of the qemu source tree." > > + echo -n "This is required so that the test can find the " > > + echo "python modules that it needs for execution." > > + exit 1 > > +fi > > +SRCDIR=$QTEST_SOURCE_ROOT > > +TESTSCRIPTS=("acpi-bits-test.py") > > +PIPCMD="-m pip -q --disable-pip-version-check" > > +# we need to save the old value of PWD before we do a change-dir later > > +QTEST_PWD=$PWD > > + > > +TESTS_PYTHON=/usr/bin/python3 > > +TESTS_VENV_REQ=requirements.txt > > + > > +# sadly for pip -e and -t options do not work together. > > +# please see https://github.com/pypa/pip/issues/562 > > +cd $MYDIR > > + > > +$TESTS_PYTHON -m venv . > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > > +[ -f $TESTS_VENV_REQ ] && \ > > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ > > + > > +# venv is activated at this point. > > + > > +# run the test > > +for testscript in ${TESTSCRIPTS[@]} ; do > > + export QTEST_PWD; python3 $testscript > > +done > > + > > +cd $QTEST_PWD > > + > > +exit 0 > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py > > new file mode 100644 > > index 0000000000..673567bf8e > > --- /dev/null > > +++ b/tests/qtest/acpi-bits/acpi-bits-test.py > > @@ -0,0 +1,327 @@ > > +#!/usr/bin/env python3 > > +# group: rw quick > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > > +# https://biosbits.org/ > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Some parts are slightly taken from qtest.py and iotests.py > > +# > > +# Authors: > > +# Ani Sinha <ani@anisinha.ca> > > + > > +# pylint: disable=invalid-name > > + > > +""" > > +QEMU bios tests using biosbits available at > > +https://biosbits.org/. > > +""" > > + > > +import logging > > +import os > > +import re > > +import shutil > > +import subprocess > > +import sys > > +import tarfile > > +import tempfile > > +import time > > +import unittest > > +import zipfile > > +from typing import ( > > + List, > > + Optional, > > + Sequence, > > +) > > +from tap import TAPTestRunner > > +from qemu.machine import QEMUMachine > > + > > +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY') > > +QTEST_PWD = os.getenv('QTEST_PWD') > > + > > +def get_arch(): > > + """finds the arch from the qemu binary name""" > > + match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG) > > + if match: > > + return match.group(1) > > + return 'x86_64' > > + > > +ARCH = get_arch() > > + > > +class QEMUBitsMachine(QEMUMachine): > > + """ > > + A QEMU VM, with isa-debugcon enabled and bits iso passed > > + using -cdrom to QEMU commandline. > > + """ > > + def __init__(self, > > + binary: str, > > + args: Sequence[str] = (), > > + wrapper: Sequence[str] = (), > > + name: Optional[str] = None, > > + base_temp_dir: str = "/var/tmp", > > + debugcon_log: str = "debugcon-log.txt", > > + debugcon_addr: str = "0x403", > > + sock_dir: Optional[str] = None, > > + qmp_timer: Optional[float] = None): > > + # pylint: disable=too-many-arguments > > + > > + if name is None: > > + name = "qemu-bits-%d" % os.getpid() > > + if sock_dir is None: > > + sock_dir = base_temp_dir > > + super().__init__(binary, args, wrapper=wrapper, name=name, > > + base_temp_dir=base_temp_dir, > > + sock_dir=sock_dir, qmp_timer=qmp_timer) > > + self.debugcon_log = debugcon_log > > + self.debugcon_addr = debugcon_addr > > + self.base_temp_dir = base_temp_dir > > + > > + @property > > + def _base_args(self) -> List[str]: > > + args = super()._base_args > > + args.extend([ > > + '-chardev', > > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > > + self.debugcon_log), > > + '-device', > > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > > + ]) > > + return args > > So is this patch series *really* related to qtests? qtests are using a > special "accelerator" mode of QEMU where it gets launched with the "-qtest" > parameter. I can't see that you're using "-qtest" here or anywhere else, so > this rather looks like another framework to me to run python-based QEMU > tests (similar to the avocado tests). yes you are right. This does not use or need the qtest accelerator because we are not inspecting the guest memory for anything. > > Thus if this is really not related to qtests, may I suggest to move this > into another folder instead? Maybe tests/pytests/acpi-bits or something similar? The problem I faced with this test is that it does not quite fall into the qtest category. Nor does it fall into the integration test category. I asked Igor and he suggested I use the qtest framework. Should we invent a new class of tests then? How many such tests are we going to have in the future? Lets see what others think. > > Thomas >
On 28/06/2022 09.26, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote: >> >> On 27/06/2022 09.28, Ani Sinha wrote: >>> This change adds python based qtest framework that can be used to run >>> qtests from within a virtual environment. A bash script creates the virtual >>> environment and then runs the python based tests from within that environment. >>> All dependent python packages are installed in the virtual environment using >>> pip module. QEMU python test modules are also available in the environment for >>> spawning the QEMU based VMs. >>> >>> It also introduces QEMU acpi/smbios biosbits python test script which is run >>> from within the python virtual environment. >>> >>> Signed-off-by: Ani Sinha <ani@anisinha.ca> >>> --- >>> tests/qtest/acpi-bits/acpi-bits-test-venv.sh | 59 ++++ >>> tests/qtest/acpi-bits/acpi-bits-test.py | 327 +++++++++++++++++++ >>> tests/qtest/acpi-bits/meson.build | 39 +++ >>> tests/qtest/acpi-bits/requirements.txt | 1 + >>> 4 files changed, 426 insertions(+) >>> create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh >>> create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py >>> create mode 100644 tests/qtest/acpi-bits/meson.build >>> create mode 100644 tests/qtest/acpi-bits/requirements.txt >>> >>> diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh >>> new file mode 100644 >>> index 0000000000..124e03ce18 >>> --- /dev/null >>> +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh >>> @@ -0,0 +1,59 @@ >>> +#!/usr/bin/env bash >>> +# Generates a python virtual environment for the test to run. >>> +# Then runs python test scripts from within that virtual environment. >>> +# >>> +# This program is free software; you can redistribute it and/or modify >>> +# it under the terms of the GNU General Public License as published by >>> +# the Free Software Foundation; either version 2 of the License, or >>> +# (at your option) any later version. >>> +# >>> +# This program is distributed in the hope that it will be useful, >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >>> +# GNU General Public License for more details. >>> +# >>> +# You should have received a copy of the GNU General Public License >>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. >>> +# >>> +# Author: Ani Sinha <ani@anisinha.ca> >>> + >>> +set -e >>> + >>> +MYPATH=$(realpath ${BASH_SOURCE:-$0}) >>> +MYDIR=$(dirname $MYPATH) >>> + >>> +if [ -z "$QTEST_SOURCE_ROOT" ]; then >>> + echo -n "Please set QTEST_SOURCE_ROOT env pointing" >>> + echo " to the root of the qemu source tree." >>> + echo -n "This is required so that the test can find the " >>> + echo "python modules that it needs for execution." >>> + exit 1 >>> +fi >>> +SRCDIR=$QTEST_SOURCE_ROOT >>> +TESTSCRIPTS=("acpi-bits-test.py") >>> +PIPCMD="-m pip -q --disable-pip-version-check" >>> +# we need to save the old value of PWD before we do a change-dir later >>> +QTEST_PWD=$PWD >>> + >>> +TESTS_PYTHON=/usr/bin/python3 >>> +TESTS_VENV_REQ=requirements.txt >>> + >>> +# sadly for pip -e and -t options do not work together. >>> +# please see https://github.com/pypa/pip/issues/562 >>> +cd $MYDIR >>> + >>> +$TESTS_PYTHON -m venv . >>> +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ >>> +[ -f $TESTS_VENV_REQ ] && \ >>> + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ >>> + >>> +# venv is activated at this point. >>> + >>> +# run the test >>> +for testscript in ${TESTSCRIPTS[@]} ; do >>> + export QTEST_PWD; python3 $testscript >>> +done >>> + >>> +cd $QTEST_PWD >>> + >>> +exit 0 >>> diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py >>> new file mode 100644 >>> index 0000000000..673567bf8e >>> --- /dev/null >>> +++ b/tests/qtest/acpi-bits/acpi-bits-test.py >>> @@ -0,0 +1,327 @@ >>> +#!/usr/bin/env python3 >>> +# group: rw quick >>> +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, >>> +# https://biosbits.org/ >>> +# >>> +# This program is free software; you can redistribute it and/or modify >>> +# it under the terms of the GNU General Public License as published by >>> +# the Free Software Foundation; either version 2 of the License, or >>> +# (at your option) any later version. >>> +# >>> +# This program is distributed in the hope that it will be useful, >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >>> +# GNU General Public License for more details. >>> +# >>> +# You should have received a copy of the GNU General Public License >>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. >>> +# >>> +# Some parts are slightly taken from qtest.py and iotests.py >>> +# >>> +# Authors: >>> +# Ani Sinha <ani@anisinha.ca> >>> + >>> +# pylint: disable=invalid-name >>> + >>> +""" >>> +QEMU bios tests using biosbits available at >>> +https://biosbits.org/. >>> +""" >>> + >>> +import logging >>> +import os >>> +import re >>> +import shutil >>> +import subprocess >>> +import sys >>> +import tarfile >>> +import tempfile >>> +import time >>> +import unittest >>> +import zipfile >>> +from typing import ( >>> + List, >>> + Optional, >>> + Sequence, >>> +) >>> +from tap import TAPTestRunner >>> +from qemu.machine import QEMUMachine >>> + >>> +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY') >>> +QTEST_PWD = os.getenv('QTEST_PWD') >>> + >>> +def get_arch(): >>> + """finds the arch from the qemu binary name""" >>> + match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG) >>> + if match: >>> + return match.group(1) >>> + return 'x86_64' >>> + >>> +ARCH = get_arch() >>> + >>> +class QEMUBitsMachine(QEMUMachine): >>> + """ >>> + A QEMU VM, with isa-debugcon enabled and bits iso passed >>> + using -cdrom to QEMU commandline. >>> + """ >>> + def __init__(self, >>> + binary: str, >>> + args: Sequence[str] = (), >>> + wrapper: Sequence[str] = (), >>> + name: Optional[str] = None, >>> + base_temp_dir: str = "/var/tmp", >>> + debugcon_log: str = "debugcon-log.txt", >>> + debugcon_addr: str = "0x403", >>> + sock_dir: Optional[str] = None, >>> + qmp_timer: Optional[float] = None): >>> + # pylint: disable=too-many-arguments >>> + >>> + if name is None: >>> + name = "qemu-bits-%d" % os.getpid() >>> + if sock_dir is None: >>> + sock_dir = base_temp_dir >>> + super().__init__(binary, args, wrapper=wrapper, name=name, >>> + base_temp_dir=base_temp_dir, >>> + sock_dir=sock_dir, qmp_timer=qmp_timer) >>> + self.debugcon_log = debugcon_log >>> + self.debugcon_addr = debugcon_addr >>> + self.base_temp_dir = base_temp_dir >>> + >>> + @property >>> + def _base_args(self) -> List[str]: >>> + args = super()._base_args >>> + args.extend([ >>> + '-chardev', >>> + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, >>> + self.debugcon_log), >>> + '-device', >>> + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, >>> + ]) >>> + return args >> >> So is this patch series *really* related to qtests? qtests are using a >> special "accelerator" mode of QEMU where it gets launched with the "-qtest" >> parameter. I can't see that you're using "-qtest" here or anywhere else, so >> this rather looks like another framework to me to run python-based QEMU >> tests (similar to the avocado tests). > > yes you are right. This does not use or need the qtest accelerator > because we are not inspecting the guest memory for anything. > >> >> Thus if this is really not related to qtests, may I suggest to move this >> into another folder instead? Maybe tests/pytests/acpi-bits or something similar? > > The problem I faced with this test is that it does not quite fall into > the qtest category. Nor does it fall into the integration test > category. I asked Igor and he suggested I use the qtest framework. My python-foo really lacks, so putting python stuff under tests/qtest where I'm the maintainer is a really bad idea ;-) > Should we invent a new class of tests then? How many such tests are we > going to have in the future? Lets see what others think. I think we want to have another folder under tests/ for python-based tests that do not use the avocado framework. John is working on better python bindings, and since we switched almost all other tests to the meson test runner now, the avocado-based tests (with their own test runner) rather feel like an oddball in the qemu test suite now, so I asume we will have more python-based tests in the future which will work without avocado. The difficult question is - as always - to come up with a good name... - tests/pytest/ - tests/integration/ - tests/pyqmygoose/ - ... ... I don't have a real good clue, as you can see ;-) Thomas
On Tue, Jun 28, 2022 at 12:56:52PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote: > > > > On 27/06/2022 09.28, Ani Sinha wrote: > > > This change adds python based qtest framework that can be used to run > > > qtests from within a virtual environment. A bash script creates the virtual > > > environment and then runs the python based tests from within that environment. > > > All dependent python packages are installed in the virtual environment using > > > pip module. QEMU python test modules are also available in the environment for > > > spawning the QEMU based VMs. > > > > > > It also introduces QEMU acpi/smbios biosbits python test script which is run > > > from within the python virtual environment. > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > --- > > > tests/qtest/acpi-bits/acpi-bits-test-venv.sh | 59 ++++ > > > tests/qtest/acpi-bits/acpi-bits-test.py | 327 +++++++++++++++++++ > > > tests/qtest/acpi-bits/meson.build | 39 +++ > > > tests/qtest/acpi-bits/requirements.txt | 1 + > > > 4 files changed, 426 insertions(+) > > > create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh > > > create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py > > > create mode 100644 tests/qtest/acpi-bits/meson.build > > > create mode 100644 tests/qtest/acpi-bits/requirements.txt > > > > > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh > > > new file mode 100644 > > > index 0000000000..124e03ce18 > > > --- /dev/null > > > +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh > > > @@ -0,0 +1,59 @@ > > > +#!/usr/bin/env bash > > > +# Generates a python virtual environment for the test to run. > > > +# Then runs python test scripts from within that virtual environment. > > > +# > > > +# This program is free software; you can redistribute it and/or modify > > > +# it under the terms of the GNU General Public License as published by > > > +# the Free Software Foundation; either version 2 of the License, or > > > +# (at your option) any later version. > > > +# > > > +# This program is distributed in the hope that it will be useful, > > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > +# GNU General Public License for more details. > > > +# > > > +# You should have received a copy of the GNU General Public License > > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > > +# > > > +# Author: Ani Sinha <ani@anisinha.ca> > > > + > > > +set -e > > > + > > > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > > > +MYDIR=$(dirname $MYPATH) > > > + > > > +if [ -z "$QTEST_SOURCE_ROOT" ]; then > > > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > > > + echo " to the root of the qemu source tree." > > > + echo -n "This is required so that the test can find the " > > > + echo "python modules that it needs for execution." > > > + exit 1 > > > +fi > > > +SRCDIR=$QTEST_SOURCE_ROOT > > > +TESTSCRIPTS=("acpi-bits-test.py") > > > +PIPCMD="-m pip -q --disable-pip-version-check" > > > +# we need to save the old value of PWD before we do a change-dir later > > > +QTEST_PWD=$PWD > > > + > > > +TESTS_PYTHON=/usr/bin/python3 > > > +TESTS_VENV_REQ=requirements.txt > > > + > > > +# sadly for pip -e and -t options do not work together. > > > +# please see https://github.com/pypa/pip/issues/562 > > > +cd $MYDIR > > > + > > > +$TESTS_PYTHON -m venv . > > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > > > +[ -f $TESTS_VENV_REQ ] && \ > > > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ > > > + > > > +# venv is activated at this point. > > > + > > > +# run the test > > > +for testscript in ${TESTSCRIPTS[@]} ; do > > > + export QTEST_PWD; python3 $testscript > > > +done > > > + > > > +cd $QTEST_PWD > > > + > > > +exit 0 > > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py > > > new file mode 100644 > > > index 0000000000..673567bf8e > > > --- /dev/null > > > +++ b/tests/qtest/acpi-bits/acpi-bits-test.py > > > @@ -0,0 +1,327 @@ > > > +#!/usr/bin/env python3 > > > +# group: rw quick > > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > > > +# https://biosbits.org/ > > > +# > > > +# This program is free software; you can redistribute it and/or modify > > > +# it under the terms of the GNU General Public License as published by > > > +# the Free Software Foundation; either version 2 of the License, or > > > +# (at your option) any later version. > > > +# > > > +# This program is distributed in the hope that it will be useful, > > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > +# GNU General Public License for more details. > > > +# > > > +# You should have received a copy of the GNU General Public License > > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > > +# > > > +# Some parts are slightly taken from qtest.py and iotests.py > > > +# > > > +# Authors: > > > +# Ani Sinha <ani@anisinha.ca> > > > + > > > +# pylint: disable=invalid-name > > > + > > > +""" > > > +QEMU bios tests using biosbits available at > > > +https://biosbits.org/. > > > +""" > > > + > > > +import logging > > > +import os > > > +import re > > > +import shutil > > > +import subprocess > > > +import sys > > > +import tarfile > > > +import tempfile > > > +import time > > > +import unittest > > > +import zipfile > > > +from typing import ( > > > + List, > > > + Optional, > > > + Sequence, > > > +) > > > +from tap import TAPTestRunner > > > +from qemu.machine import QEMUMachine > > > + > > > +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY') > > > +QTEST_PWD = os.getenv('QTEST_PWD') > > > + > > > +def get_arch(): > > > + """finds the arch from the qemu binary name""" > > > + match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG) > > > + if match: > > > + return match.group(1) > > > + return 'x86_64' > > > + > > > +ARCH = get_arch() > > > + > > > +class QEMUBitsMachine(QEMUMachine): > > > + """ > > > + A QEMU VM, with isa-debugcon enabled and bits iso passed > > > + using -cdrom to QEMU commandline. > > > + """ > > > + def __init__(self, > > > + binary: str, > > > + args: Sequence[str] = (), > > > + wrapper: Sequence[str] = (), > > > + name: Optional[str] = None, > > > + base_temp_dir: str = "/var/tmp", > > > + debugcon_log: str = "debugcon-log.txt", > > > + debugcon_addr: str = "0x403", > > > + sock_dir: Optional[str] = None, > > > + qmp_timer: Optional[float] = None): > > > + # pylint: disable=too-many-arguments > > > + > > > + if name is None: > > > + name = "qemu-bits-%d" % os.getpid() > > > + if sock_dir is None: > > > + sock_dir = base_temp_dir > > > + super().__init__(binary, args, wrapper=wrapper, name=name, > > > + base_temp_dir=base_temp_dir, > > > + sock_dir=sock_dir, qmp_timer=qmp_timer) > > > + self.debugcon_log = debugcon_log > > > + self.debugcon_addr = debugcon_addr > > > + self.base_temp_dir = base_temp_dir > > > + > > > + @property > > > + def _base_args(self) -> List[str]: > > > + args = super()._base_args > > > + args.extend([ > > > + '-chardev', > > > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > > > + self.debugcon_log), > > > + '-device', > > > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > > > + ]) > > > + return args > > > > So is this patch series *really* related to qtests? qtests are using a > > special "accelerator" mode of QEMU where it gets launched with the "-qtest" > > parameter. I can't see that you're using "-qtest" here or anywhere else, so > > this rather looks like another framework to me to run python-based QEMU > > tests (similar to the avocado tests). > > yes you are right. This does not use or need the qtest accelerator > because we are not inspecting the guest memory for anything. Same is true for e.g. ./tests/qtest/bios-tables-test.c ... > > > > Thus if this is really not related to qtests, may I suggest to move this > > into another folder instead? Maybe tests/pytests/acpi-bits or something similar? > > The problem I faced with this test is that it does not quite fall into > the qtest category. Nor does it fall into the integration test > category. I asked Igor and he suggested I use the qtest framework. > Should we invent a new class of tests then? How many such tests are we > going to have in the future? Lets see what others think. > > > > > Thomas > >
On 28/06/2022 11.55, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 12:56:52PM +0530, Ani Sinha wrote: >> On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote: >>> >>> On 27/06/2022 09.28, Ani Sinha wrote: >>>> This change adds python based qtest framework that can be used to run >>>> qtests from within a virtual environment. A bash script creates the virtual >>>> environment and then runs the python based tests from within that environment. >>>> All dependent python packages are installed in the virtual environment using >>>> pip module. QEMU python test modules are also available in the environment for >>>> spawning the QEMU based VMs. >>>> >>>> It also introduces QEMU acpi/smbios biosbits python test script which is run >>>> from within the python virtual environment. >>>> >>>> Signed-off-by: Ani Sinha <ani@anisinha.ca> >>>> --- [...] >>>> +class QEMUBitsMachine(QEMUMachine): >>>> + """ >>>> + A QEMU VM, with isa-debugcon enabled and bits iso passed >>>> + using -cdrom to QEMU commandline. >>>> + """ >>>> + def __init__(self, >>>> + binary: str, >>>> + args: Sequence[str] = (), >>>> + wrapper: Sequence[str] = (), >>>> + name: Optional[str] = None, >>>> + base_temp_dir: str = "/var/tmp", >>>> + debugcon_log: str = "debugcon-log.txt", >>>> + debugcon_addr: str = "0x403", >>>> + sock_dir: Optional[str] = None, >>>> + qmp_timer: Optional[float] = None): >>>> + # pylint: disable=too-many-arguments >>>> + >>>> + if name is None: >>>> + name = "qemu-bits-%d" % os.getpid() >>>> + if sock_dir is None: >>>> + sock_dir = base_temp_dir >>>> + super().__init__(binary, args, wrapper=wrapper, name=name, >>>> + base_temp_dir=base_temp_dir, >>>> + sock_dir=sock_dir, qmp_timer=qmp_timer) >>>> + self.debugcon_log = debugcon_log >>>> + self.debugcon_addr = debugcon_addr >>>> + self.base_temp_dir = base_temp_dir >>>> + >>>> + @property >>>> + def _base_args(self) -> List[str]: >>>> + args = super()._base_args >>>> + args.extend([ >>>> + '-chardev', >>>> + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, >>>> + self.debugcon_log), >>>> + '-device', >>>> + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, >>>> + ]) >>>> + return args >>> >>> So is this patch series *really* related to qtests? qtests are using a >>> special "accelerator" mode of QEMU where it gets launched with the "-qtest" >>> parameter. I can't see that you're using "-qtest" here or anywhere else, so >>> this rather looks like another framework to me to run python-based QEMU >>> tests (similar to the avocado tests). >> >> yes you are right. This does not use or need the qtest accelerator >> because we are not inspecting the guest memory for anything. > > Same is true for e.g. ./tests/qtest/bios-tables-test.c ... No, bios-tables-test still uses functions like qtest_memread() and qtest_readb(), so it still uses the qtest framework, even if it's actually running code via TCG. Thomas
diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh new file mode 100644 index 0000000000..124e03ce18 --- /dev/null +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Generates a python virtual environment for the test to run. +# Then runs python test scripts from within that virtual environment. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Author: Ani Sinha <ani@anisinha.ca> + +set -e + +MYPATH=$(realpath ${BASH_SOURCE:-$0}) +MYDIR=$(dirname $MYPATH) + +if [ -z "$QTEST_SOURCE_ROOT" ]; then + echo -n "Please set QTEST_SOURCE_ROOT env pointing" + echo " to the root of the qemu source tree." + echo -n "This is required so that the test can find the " + echo "python modules that it needs for execution." + exit 1 +fi +SRCDIR=$QTEST_SOURCE_ROOT +TESTSCRIPTS=("acpi-bits-test.py") +PIPCMD="-m pip -q --disable-pip-version-check" +# we need to save the old value of PWD before we do a change-dir later +QTEST_PWD=$PWD + +TESTS_PYTHON=/usr/bin/python3 +TESTS_VENV_REQ=requirements.txt + +# sadly for pip -e and -t options do not work together. +# please see https://github.com/pypa/pip/issues/562 +cd $MYDIR + +$TESTS_PYTHON -m venv . +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ +[ -f $TESTS_VENV_REQ ] && \ + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ + +# venv is activated at this point. + +# run the test +for testscript in ${TESTSCRIPTS[@]} ; do + export QTEST_PWD; python3 $testscript +done + +cd $QTEST_PWD + +exit 0 diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py new file mode 100644 index 0000000000..673567bf8e --- /dev/null +++ b/tests/qtest/acpi-bits/acpi-bits-test.py @@ -0,0 +1,327 @@ +#!/usr/bin/env python3 +# group: rw quick +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, +# https://biosbits.org/ +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Some parts are slightly taken from qtest.py and iotests.py +# +# Authors: +# Ani Sinha <ani@anisinha.ca> + +# pylint: disable=invalid-name + +""" +QEMU bios tests using biosbits available at +https://biosbits.org/. +""" + +import logging +import os +import re +import shutil +import subprocess +import sys +import tarfile +import tempfile +import time +import unittest +import zipfile +from typing import ( + List, + Optional, + Sequence, +) +from tap import TAPTestRunner +from qemu.machine import QEMUMachine + +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY') +QTEST_PWD = os.getenv('QTEST_PWD') + +def get_arch(): + """finds the arch from the qemu binary name""" + match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG) + if match: + return match.group(1) + return 'x86_64' + +ARCH = get_arch() + +class QEMUBitsMachine(QEMUMachine): + """ + A QEMU VM, with isa-debugcon enabled and bits iso passed + using -cdrom to QEMU commandline. + """ + def __init__(self, + binary: str, + args: Sequence[str] = (), + wrapper: Sequence[str] = (), + name: Optional[str] = None, + base_temp_dir: str = "/var/tmp", + debugcon_log: str = "debugcon-log.txt", + debugcon_addr: str = "0x403", + sock_dir: Optional[str] = None, + qmp_timer: Optional[float] = None): + # pylint: disable=too-many-arguments + + if name is None: + name = "qemu-bits-%d" % os.getpid() + if sock_dir is None: + sock_dir = base_temp_dir + super().__init__(binary, args, wrapper=wrapper, name=name, + base_temp_dir=base_temp_dir, + sock_dir=sock_dir, qmp_timer=qmp_timer) + self.debugcon_log = debugcon_log + self.debugcon_addr = debugcon_addr + self.base_temp_dir = base_temp_dir + + @property + def _base_args(self) -> List[str]: + args = super()._base_args + args.extend([ + '-chardev', + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, + self.debugcon_log), + '-device', + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, + ]) + return args + + def base_args(self): + """return the base argument to QEMU binary""" + return self._base_args + +class AcpiBitsTest(unittest.TestCase): + """ACPI and SMBIOS tests using biosbits.""" + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self._vm = None + self._workdir = None + self._bitsver = 2095 + self._debugcon_addr = '0x403' + self._debugcon_log = 'debugcon-log.txt' + logging.basicConfig(level=logging.INFO) + + def copy_bits_config(self): + """ copies the bios bits config file into bits. + """ + config_file = 'bits-cfg.txt' + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') + target_config_dir = os.path.join(self._workdir, + 'bits-%d' %self._bitsver, 'boot') + self.assertTrue(os.path.exists(qemu_bits_config_dir)) + self.assertTrue(os.path.exists(target_config_dir)) + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, + config_file), os.R_OK)) + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), + target_config_dir) + logging.info('copied config file %s to %s', + config_file, target_config_dir) + + def copy_test_scripts(self): + """copies the python test scripts into bits. """ + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') + target_test_dir = os.path.join(self._workdir, 'bits-%d' %self._bitsver, + 'boot', 'python') + + self.assertTrue(os.path.exists(qemu_test_dir)) + self.assertTrue(os.path.exists(target_test_dir)) + + for filename in os.listdir(qemu_test_dir): + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ + filename.endswith('.py'): + shutil.copy2(os.path.join(qemu_test_dir, filename), + target_test_dir) + logging.info('copied test file %s to %s', + filename, target_test_dir) + + # now remove the pyc test file if it exists, otherwise the + # changes in the python test script won't be executed. + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' + if os.access(os.path.join(target_test_dir, testfile_pyc), + os.R_OK): + os.remove(os.path.join(target_test_dir, testfile_pyc)) + logging.info('removed compiled file %s', + os.path.join(target_test_dir, testfile_pyc)) + + def fix_mkrescue(self, mkrescue): + """ grub-mkrescue is a bash script with two variables, 'prefix' and + 'libdir'. They must be pointed to the right location so that the + iso can be generated appropriately. We point the two variables to + the directory where we have extracted our pre-built bits grub + tarball. + """ + grub_x86_64_mods = os.path.join(self._workdir, 'bits-grub', + 'grub-inst-x86_64-efi') + grub_i386_mods = os.path.join(self._workdir, 'bits-grub', + 'grub-inst') + + self.assertTrue(os.path.exists(grub_x86_64_mods)) + self.assertTrue(os.path.exists(grub_i386_mods)) + + new_script = "" + with open(mkrescue, 'r') as filehandle: + orig_script = filehandle.read() + new_script = re.sub('(^prefix=)(.*)', + r'\1"%s"' %grub_x86_64_mods, + orig_script, flags=re.M) + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, + new_script, flags=re.M) + + with open(mkrescue, 'w') as filehandle: + filehandle.write(new_script) + + def generate_bits_iso(self): + """ Uses grub-mkrescue to generate a fresh bits iso with the python + test scripts + """ + bits_dir = os.path.join(self._workdir, 'bits-%d' %self._bitsver) + iso_file = os.path.join(self._workdir, 'bits-%d.iso' %self._bitsver) + mkrescue_script = os.path.join(self._workdir, + 'bits-grub', 'grub-inst-x86_64-efi', + 'bin', 'grub-mkrescue') + + self.assertTrue(os.access(mkrescue_script, os.R_OK | os.W_OK)) + + self.fix_mkrescue(mkrescue_script) + + logging.info('calling grub-mkrescue to generate the biosbits iso ...') + + if os.getenv('V'): + subprocess.check_call([mkrescue_script, '-o', iso_file, bits_dir], + stdout=subprocess.DEVNULL) + else: + subprocess.check_call([mkrescue_script, '-o', iso_file, bits_dir], + stderr=subprocess.DEVNULL, + stdout=subprocess.DEVNULL) + + self.assertTrue(os.access(iso_file, os.R_OK)) + + logging.info('iso file %s successfully generated.', iso_file) + + def setUp(self): + self._workdir = tempfile.mkdtemp(prefix='acpi-bits-', + suffix='.tmp') + + logging.info('generated working dir: %s', self._workdir) + + # extract the bits software in the temp working directory + bits_zip_file = os.path.join(os.getcwd(), 'prebuilt', 'bits-%d.zip' + %self._bitsver) + grub_tar_file = os.path.join(os.getcwd(), 'prebuilt', + 'bits-%d-grub.tar.gz' %self._bitsver) + + with zipfile.ZipFile(bits_zip_file, 'r') as zref: + zref.extractall(self._workdir) + + with tarfile.open(grub_tar_file, 'r') as tarball: + tarball.extractall(self._workdir) + + self.copy_test_scripts() + self.copy_bits_config() + self.generate_bits_iso() + + def parse_log(self): + """parse the log generated by running bits tests and + check for failures. + """ + debugconf = os.path.join(self._workdir, self._debugcon_log) + log = "" + with open(debugconf, 'r') as filehandle: + log = filehandle.read() + + if os.getenv('V'): + print('\nlogs from biosbits follows:') + print('==========================================\n\n') + print(log) + print('\n==========================================\n') + + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*', + log) + for match in matchiter: + # verify that no test cases failed. + self.assertEqual(match.group(3).split()[0], '0', + 'Some bits tests seems to have failed. ' \ + 'Set V=1 in the environment to get the entire ' \ + 'log from bits.') + + def tearDown(self): + if self._vm: + self.assertFalse(not self._vm.is_running) + logging.info('removing the work directory %s', self._workdir) + shutil.rmtree(self._workdir) + + def test_acpi_smbios_bits(self): + """The main test case implementaion.""" + + qemu_bin = QTESTQEMUPROG + iso_file = os.path.join(self._workdir, 'bits-%d.iso' %self._bitsver) + + # QTESTQEMUPROG could be relative to the current directory + if not os.access(QTESTQEMUPROG, os.X_OK): + qemu_bin = os.path.join(QTEST_PWD, QTESTQEMUPROG) + + logging.info('QEMU binary used: %s', qemu_bin) + + self.assertTrue(os.access(qemu_bin, os.X_OK)) + self.assertTrue(os.access(iso_file, os.R_OK)) + + self._vm = QEMUBitsMachine(binary=qemu_bin, + base_temp_dir=self._workdir, + debugcon_log=self._debugcon_log, + debugcon_addr=self._debugcon_addr) + + self._vm.add_args('-cdrom', '%s' %iso_file) + + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ + " " + " ".join(str(arg) for arg in self._vm.args) + + logging.info("launching QEMU vm with the following arguments: %s", + args) + + self._vm.launch() + # biosbits has been configured to run all the specified test suites + # in batch mode and then automatically initiate a vm shutdown. + # sleep for maximum of one minute + max_sleep_time = time.monotonic() + 60 + while self._vm.is_running() and time.monotonic() < max_sleep_time: + time.sleep(1) + + self.assertFalse(time.monotonic() > max_sleep_time, + 'The VM seems to have failed to shutdown in time') + + self.parse_log() + +def execute_unittest(argv: List[str], debug: bool = False, + runner: TAPTestRunner = None) -> None: + """Executes unittests within the calling module.""" + + unittest.main(argv=argv, + testRunner=runner, + verbosity=2 if debug else 1, + warnings=None if sys.warnoptions else 'ignore') + +def main(): + """ The main function where execution begins. """ + + assert QTESTQEMUPROG is not None, \ + "Environment variable QTEST_QEMU_BINARY required." + + runner = TAPTestRunner() + runner.set_stream(True) + runner.set_format("%s/acpi-bits-test" %ARCH) + execute_unittest(sys.argv, False, runner) + +main() diff --git a/tests/qtest/acpi-bits/meson.build b/tests/qtest/acpi-bits/meson.build new file mode 100644 index 0000000000..95c096914f --- /dev/null +++ b/tests/qtest/acpi-bits/meson.build @@ -0,0 +1,39 @@ +xorriso = find_program('xorriso', required: true) +if not xorriso.found() + message('xorriso not found ... disabled bits acpi tests.') + subdir_done() +endif + +# biosbits qtests are currenly only supported on x86_64 platforms. +qtests_x86_64 += ['acpi-bits-test'] +qtests += {'acpi-bits-test' : files('acpi-bits-test.py'),} +slow_qtests += {'acpi-bits-test': 120, } + +subdir('prebuilt') +subdir('bits-tests') +subdir('bits-config') + +test_files = ['acpi-bits-test.py'] +requirements = 'requirements.txt' + +copytestfiles = custom_target('copy test files', + input : test_files, + output : test_files, + command : ['cp', '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) + +requirementsfiles = custom_target('copy py req files', + input : requirements, + output : requirements, + command : ['cp', '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) + +other_deps += [copytestfiles,requirementsfiles] + +qtest_executables += { + 'acpi-bits-test': configure_file(copy:true, + input:'acpi-bits-test-venv.sh', + output:'acpi-bits-test') +} diff --git a/tests/qtest/acpi-bits/requirements.txt b/tests/qtest/acpi-bits/requirements.txt new file mode 100644 index 0000000000..00cdad09ef --- /dev/null +++ b/tests/qtest/acpi-bits/requirements.txt @@ -0,0 +1 @@ +tap.py
This change adds python based qtest framework that can be used to run qtests from within a virtual environment. A bash script creates the virtual environment and then runs the python based tests from within that environment. All dependent python packages are installed in the virtual environment using pip module. QEMU python test modules are also available in the environment for spawning the QEMU based VMs. It also introduces QEMU acpi/smbios biosbits python test script which is run from within the python virtual environment. Signed-off-by: Ani Sinha <ani@anisinha.ca> --- tests/qtest/acpi-bits/acpi-bits-test-venv.sh | 59 ++++ tests/qtest/acpi-bits/acpi-bits-test.py | 327 +++++++++++++++++++ tests/qtest/acpi-bits/meson.build | 39 +++ tests/qtest/acpi-bits/requirements.txt | 1 + 4 files changed, 426 insertions(+) create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py create mode 100644 tests/qtest/acpi-bits/meson.build create mode 100644 tests/qtest/acpi-bits/requirements.txt