| Message ID | 20220710170014.1673480-8-ani@anisinha.ca (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Introduce new acpi/smbios python tests using biosbits | expand |
On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > This change adds python based test environment that can be used to run pytest > from within a virtual environment. A bash script sets up a virtual environment > and then runs the python based tests from within that environment. > All dependent python packages are installed in the virtual environment using > pip python module. QEMU python test modules are also available in the environment > for spawning the QEMU based VMs. > > It also introduces QEMU acpi/smbios biosbits python test script which is run > from within the python virtual environment. When the bios bits tests are run, > bios bits binaries are downloaded from an external repo/location. > Currently, the test points to an external private github repo where the bits > archives are checked in. > Oh, I see -- requirements are handled here in this patch. Is this test designed to run the host/build system? I'm a little confused about the environment here. Is this test going to be run "by default" or will users have to opt into running it using a special invocation? --js > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > tests/pytest/acpi-bits/meson.build | 33 ++ > tests/pytest/acpi-bits/requirements.txt | 1 + > 4 files changed, 475 insertions(+) > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > create mode 100644 tests/pytest/acpi-bits/meson.build > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > new file mode 100644 > index 0000000000..186395473b > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > @@ -0,0 +1,59 @@ > +#!/usr/bin/env bash > +# Generates a python virtual environment for the test to run. > +# Then runs python test scripts from within that virtual environment. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Author: Ani Sinha <ani@anisinha.ca> > + > +set -e > + > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > +MYDIR=$(dirname $MYPATH) > + > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > + echo " to the root of the qemu source tree." > + echo -n "This is required so that the test can find the " > + echo "python modules that it needs for execution." > + exit 1 > +fi > +SRCDIR=$PYTEST_SOURCE_ROOT > +TESTSCRIPTS=("acpi-bits-test.py") > +PIPCMD="-m pip -q --disable-pip-version-check" > +# we need to save the old value of PWD before we do a change-dir later > +PYTEST_PWD=$PWD > + > +TESTS_PYTHON=/usr/bin/python3 > +TESTS_VENV_REQ=requirements.txt > + > +# sadly for pip -e and -t options do not work together. > +# please see https://github.com/pypa/pip/issues/562 > +cd $MYDIR > + > +$TESTS_PYTHON -m venv . > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > +[ -f $TESTS_VENV_REQ ] && \ > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > + > +# venv is activated at this point. > + > +# run the test > +for testscript in ${TESTSCRIPTS[@]} ; do > + export PYTEST_PWD; python3 $testscript > +done > + > +cd $PYTEST_PWD > + > +exit 0 > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py > new file mode 100644 > index 0000000000..97e61eb709 > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > @@ -0,0 +1,382 @@ > +#!/usr/bin/env python3 > +# group: rw quick > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > +# https://biosbits.org/ > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Some parts are slightly taken from qtest.py and iotests.py > +# > +# Authors: > +# Ani Sinha <ani@anisinha.ca> > + > +# pylint: disable=invalid-name > + > +""" > +QEMU bios tests using biosbits available at > +https://biosbits.org/. > +""" > + > +import logging > +import os > +import re > +import shutil > +import subprocess > +import sys > +import tarfile > +import tempfile > +import time > +import unittest > +from urllib import request > +import zipfile > +from typing import ( > + List, > + Optional, > + Sequence, > +) > +from tap import TAPTestRunner > +from qemu.machine import QEMUMachine > + > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > +PYTEST_PWD = os.getenv('PYTEST_PWD') > + > +def get_arch(): > + """finds the arch from the qemu binary name""" > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > + if match: > + return match.group(1) > + return 'x86_64' > + > +ARCH = get_arch() > + > +class QEMUBitsMachine(QEMUMachine): > + """ > + A QEMU VM, with isa-debugcon enabled and bits iso passed > + using -cdrom to QEMU commandline. > + """ > + def __init__(self, > + binary: str, > + args: Sequence[str] = (), > + wrapper: Sequence[str] = (), > + name: Optional[str] = None, > + base_temp_dir: str = "/var/tmp", > + debugcon_log: str = "debugcon-log.txt", > + debugcon_addr: str = "0x403", > + sock_dir: Optional[str] = None, > + qmp_timer: Optional[float] = None): > + # pylint: disable=too-many-arguments > + > + if name is None: > + name = "qemu-bits-%d" % os.getpid() > + if sock_dir is None: > + sock_dir = base_temp_dir > + super().__init__(binary, args, wrapper=wrapper, name=name, > + base_temp_dir=base_temp_dir, > + sock_dir=sock_dir, qmp_timer=qmp_timer) > + self.debugcon_log = debugcon_log > + self.debugcon_addr = debugcon_addr > + self.base_temp_dir = base_temp_dir > + > + @property > + def _base_args(self) -> List[str]: > + args = super()._base_args > + args.extend([ > + '-chardev', > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > + self.debugcon_log), > + '-device', > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > + ]) > + return args > + > + def base_args(self): > + """return the base argument to QEMU binary""" > + return self._base_args > + > +class AcpiBitsTest(unittest.TestCase): > + """ACPI and SMBIOS tests using biosbits.""" > + def __init__(self, *args, **kwargs): > + super().__init__(*args, **kwargs) > + self._vm = None > + self._workDir = None > + self._bitsVer = 2100 > + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" > + self._debugcon_addr = '0x403' > + self._debugcon_log = 'debugcon-log.txt' > + logging.basicConfig(level=logging.INFO) > + > + def copy_bits_config(self): > + """ copies the bios bits config file into bits. > + """ > + config_file = 'bits-cfg.txt' > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > + target_config_dir = os.path.join(self._workDir, > + 'bits-%d' %self._bitsVer, 'boot') > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > + self.assertTrue(os.path.exists(target_config_dir)) > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > + config_file), os.R_OK)) > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > + target_config_dir) > + logging.info('copied config file %s to %s', > + config_file, target_config_dir) > + > + def copy_test_scripts(self): > + """copies the python test scripts into bits. """ > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, > + 'boot', 'python') > + > + self.assertTrue(os.path.exists(qemu_test_dir)) > + self.assertTrue(os.path.exists(target_test_dir)) > + > + for filename in os.listdir(qemu_test_dir): > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > + filename.endswith('.py'): > + shutil.copy2(os.path.join(qemu_test_dir, filename), > + target_test_dir) > + logging.info('copied test file %s to %s', > + filename, target_test_dir) > + > + # now remove the pyc test file if it exists, otherwise the > + # changes in the python test script won't be executed. > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > + if os.access(os.path.join(target_test_dir, testfile_pyc), > + os.F_OK): > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > + logging.info('removed compiled file %s', > + os.path.join(target_test_dir, testfile_pyc)) > + > + def fix_mkrescue(self, mkrescue): > + """ grub-mkrescue is a bash script with two variables, 'prefix' and > + 'libdir'. They must be pointed to the right location so that the > + iso can be generated appropriately. We point the two variables to > + the directory where we have extracted our pre-built bits grub > + tarball. > + """ > + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > + > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > + self.assertTrue(os.path.exists(grub_i386_mods)) > + > + new_script = "" > + with open(mkrescue, 'r') as filehandle: > + orig_script = filehandle.read() > + new_script = re.sub('(^prefix=)(.*)', > + r'\1"%s"' %grub_x86_64_mods, > + orig_script, flags=re.M) > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, > + new_script, flags=re.M) > + > + with open(mkrescue, 'w') as filehandle: > + filehandle.write(new_script) > + > + def generate_bits_iso(self): > + """ Uses grub-mkrescue to generate a fresh bits iso with the python > + test scripts > + """ > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > + mkrescue_script = os.path.join(self._workDir, > + 'grub-inst-x86_64-efi', 'bin', > + 'grub-mkrescue') > + > + self.assertTrue(os.access(mkrescue_script, > + os.R_OK | os.W_OK | os.X_OK)) > + > + self.fix_mkrescue(mkrescue_script) > + > + logging.info('calling grub-mkrescue to generate the biosbits iso ...') > + > + try: > + if os.getenv('V'): > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stdout=subprocess.DEVNULL) > + else: > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stderr=subprocess.DEVNULL, > + stdout=subprocess.DEVNULL) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("Error while generating the bits iso. " > + "Pass V=1 in the environment to get more details. " > + + str(e)) > + > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + logging.info('iso file %s successfully generated.', iso_file) > + > + def setUp(self): > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > + if BITS_LOC: > + prefix = BITS_LOC > + else: > + prefix = os.path.join(os.getcwd(), 'prebuilt') > + if not os.path.isdir(prefix): > + os.mkdir(prefix, mode=0o775) > + > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > + %self._bitsVer) > + grub_tar_file = os.path.join(prefix, > + 'bits-%d-grub.tar.gz' %self._bitsVer) > + # if the location of the bits binaries has been specified by the user > + # and they are not found in that location, skip the test. > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location %s." \ > + %BITS_LOC) > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location %s." \ > + %BITS_LOC) > + > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > + suffix='.tmp') > + logging.info('working dir: %s', self._workDir) > + > + localArchive = "bits-%d.zip" % self._bitsVer > + if not os.access(bits_zip_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) > + > + localArchive = "bits-%d-grub.tar.gz" % self._bitsVer > + if not os.access(grub_tar_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) > + > + # extract the bits software in the temp working directory > + with zipfile.ZipFile(bits_zip_file, 'r') as zref: > + zref.extractall(self._workDir) > + > + with tarfile.open(grub_tar_file, 'r') as tarball: > + tarball.extractall(self._workDir) > + > + self.copy_test_scripts() > + self.copy_bits_config() > + self.generate_bits_iso() > + > + def parse_log(self): > + """parse the log generated by running bits tests and > + check for failures. > + """ > + debugconf = os.path.join(self._workDir, self._debugcon_log) > + log = "" > + with open(debugconf, 'r') as filehandle: > + log = filehandle.read() > + > + if os.getenv('V'): > + print('\nlogs from biosbits follows:') > + print('==========================================\n') > + print(log) > + print('==========================================\n') > + > + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*', > + log) > + for match in matchiter: > + # verify that no test cases failed. > + self.assertEqual(match.group(3).split()[0], '0', > + 'Some bits tests seems to have failed. ' \ > + 'Set V=1 in the environment to get the entire ' \ > + 'log from bits.') > + > + def tearDown(self): > + if self._vm: > + self.assertFalse(not self._vm.is_running) > + logging.info('removing the work directory %s', self._workDir) > + shutil.rmtree(self._workDir) > + > + def test_acpi_smbios_bits(self): > + """The main test case implementaion.""" > + > + qemu_bin = PYTESTQEMUBIN > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > + > + # PYTESTQEMUBIN could be relative to the current directory > + if not os.access(PYTESTQEMUBIN, os.X_OK) and PYTEST_PWD: > + qemu_bin = os.path.join(PYTEST_PWD, PYTESTQEMUBIN) > + > + logging.info('QEMU binary used: %s', qemu_bin) > + > + self.assertTrue(os.access(qemu_bin, os.X_OK)) > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + self._vm = QEMUBitsMachine(binary=qemu_bin, > + base_temp_dir=self._workDir, > + debugcon_log=self._debugcon_log, > + debugcon_addr=self._debugcon_addr) > + > + self._vm.add_args('-cdrom', '%s' %iso_file) > + > + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ > + " " + " ".join(str(arg) for arg in self._vm.args) > + > + logging.info("launching QEMU vm with the following arguments: %s", > + args) > + > + self._vm.launch() > + # biosbits has been configured to run all the specified test suites > + # in batch mode and then automatically initiate a vm shutdown. > + # sleep for maximum of one minute > + max_sleep_time = time.monotonic() + 60 > + while self._vm.is_running() and time.monotonic() < max_sleep_time: > + time.sleep(1) > + > + self.assertFalse(time.monotonic() > max_sleep_time, > + 'The VM seems to have failed to shutdown in time') > + > + self.parse_log() > + > +def execute_unittest(argv: List[str], debug: bool = False, > + runner: TAPTestRunner = None) -> None: > + """Executes unittests within the calling module.""" > + > + unittest.main(argv=argv, > + testRunner=runner, > + verbosity=2 if debug else 1, > + warnings=None if sys.warnoptions else 'ignore') > + > +def main(): > + """ The main function where execution begins. """ > + > + assert PYTESTQEMUBIN is not None, \ > + "Environment variable PYTEST_QEMU_BINARY required." > + > + runner = TAPTestRunner() > + runner.set_stream(True) > + runner.set_format("%s/acpi-bits-test" %ARCH) > + execute_unittest(sys.argv, False, runner) > + > +main() > diff --git a/tests/pytest/acpi-bits/meson.build b/tests/pytest/acpi-bits/meson.build > new file mode 100644 > index 0000000000..099c191d57 > --- /dev/null > +++ b/tests/pytest/acpi-bits/meson.build > @@ -0,0 +1,33 @@ > +xorriso = find_program('xorriso', required: true) > +if not xorriso.found() > + message('xorriso not found ... disabled bits acpi tests.') > + subdir_done() > +endif > + > +subdir('bits-tests') > +subdir('bits-config') > + > +test_files = ['acpi-bits-test.py'] > +requirements = 'requirements.txt' > + > +copytestfiles = custom_target('copy test files', > + input : test_files, > + output : test_files, > + command : ['cp', '@INPUT@', '@OUTPUT@'], > + install : false, > + build_by_default : true) > + > +requirementsfiles = custom_target('copy py req files', > + input : requirements, > + output : requirements, > + command : ['cp', '@INPUT@', '@OUTPUT@'], > + install : false, > + build_by_default : true) > + > +other_deps += [copytestfiles,requirementsfiles] > + > +pytest_executables += { > + 'acpi-bits-test': configure_file(copy:true, > + input:'acpi-bits-test-venv.sh', > + output:'acpi-bits-test') > +} > diff --git a/tests/pytest/acpi-bits/requirements.txt b/tests/pytest/acpi-bits/requirements.txt > new file mode 100644 > index 0000000000..00cdad09ef > --- /dev/null > +++ b/tests/pytest/acpi-bits/requirements.txt > @@ -0,0 +1 @@ > +tap.py > -- > 2.25.1 >
On Mon, 11 Jul 2022, John Snow wrote: > On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > This change adds python based test environment that can be used to run pytest > > from within a virtual environment. A bash script sets up a virtual environment > > and then runs the python based tests from within that environment. > > All dependent python packages are installed in the virtual environment using > > pip python module. QEMU python test modules are also available in the environment > > for spawning the QEMU based VMs. > > > > It also introduces QEMU acpi/smbios biosbits python test script which is run > > from within the python virtual environment. When the bios bits tests are run, > > bios bits binaries are downloaded from an external repo/location. > > Currently, the test points to an external private github repo where the bits > > archives are checked in. > > > > Oh, I see -- requirements are handled here in this patch. > > Is this test designed to run the host/build system? I'm a little > confused about the environment here. yes it is supposed to run on the build system where "make check" is executed. > > Is this test going to be run "by default" or will users have to opt > into running it using a special invocation? run by default through "make check" or "make check-pytest" or "make check-pytest-x86_64". Users can also run this test manually by invoking the command directly just like qtests. If the bits artifacts are not found locally and cannot be downloaded, the test is skipped with a (hopefully) helpful message. $ make check-pytest GIT ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc slirp [1/9] Generating qemu-version.h with a custom command (wrapped by meson to capture output) /usr/bin/python3 -B /home/anisinha/workspace/qemu/meson/meson.py test --no-rebuild -t 0 --num-processes 1 --print-errorlogs --suite pytest 1/1 qemu:pytest+pytest-x86_64 / pytest-x86_64/acpi-bits-test SKIP 3.96s 0 subtests passed Ok: 0 Expected Fail: 0 Fail: 0 Unexpected Pass: 0 Skipped: 1 Timeout: 0 Full log written to /home/anisinha/workspace/qemu/build/meson-logs/testlog.txt ----------------------------------- output ----------------------------------- stdout: # TAP results for AcpiBitsTest ok 1 x86_64/acpi-bits-test # SKIP test skipped since biosbits binaries could not be obtained.<urlopen error [Errno -2] Name or service not known> 1..1 stderr: INFO:root:working dir: /tmp/acpi-bits-nbjjjvkt.tmp INFO:root:archive bits-2100.zip not found in /home/anisinha/workspace/qemu/build/tests/pytest/acpi-bits/prebuilt/bits-2100.zip, downloading ... ------------------------------------------------------------------------------ > > --js > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > > tests/pytest/acpi-bits/meson.build | 33 ++ > > tests/pytest/acpi-bits/requirements.txt | 1 + > > 4 files changed, 475 insertions(+) > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > > create mode 100644 tests/pytest/acpi-bits/meson.build > > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > new file mode 100644 > > index 0000000000..186395473b > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > @@ -0,0 +1,59 @@ > > +#!/usr/bin/env bash > > +# Generates a python virtual environment for the test to run. > > +# Then runs python test scripts from within that virtual environment. > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Author: Ani Sinha <ani@anisinha.ca> > > + > > +set -e > > + > > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > > +MYDIR=$(dirname $MYPATH) > > + > > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > > + echo " to the root of the qemu source tree." > > + echo -n "This is required so that the test can find the " > > + echo "python modules that it needs for execution." > > + exit 1 > > +fi > > +SRCDIR=$PYTEST_SOURCE_ROOT > > +TESTSCRIPTS=("acpi-bits-test.py") > > +PIPCMD="-m pip -q --disable-pip-version-check" > > +# we need to save the old value of PWD before we do a change-dir later > > +PYTEST_PWD=$PWD > > + > > +TESTS_PYTHON=/usr/bin/python3 > > +TESTS_VENV_REQ=requirements.txt > > + > > +# sadly for pip -e and -t options do not work together. > > +# please see https://github.com/pypa/pip/issues/562 > > +cd $MYDIR > > + > > +$TESTS_PYTHON -m venv . > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > > +[ -f $TESTS_VENV_REQ ] && \ > > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > > + > > +# venv is activated at this point. > > + > > +# run the test > > +for testscript in ${TESTSCRIPTS[@]} ; do > > + export PYTEST_PWD; python3 $testscript > > +done > > + > > +cd $PYTEST_PWD > > + > > +exit 0 > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py > > new file mode 100644 > > index 0000000000..97e61eb709 > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > > @@ -0,0 +1,382 @@ > > +#!/usr/bin/env python3 > > +# group: rw quick > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > > +# https://biosbits.org/ > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Some parts are slightly taken from qtest.py and iotests.py > > +# > > +# Authors: > > +# Ani Sinha <ani@anisinha.ca> > > + > > +# pylint: disable=invalid-name > > + > > +""" > > +QEMU bios tests using biosbits available at > > +https://biosbits.org/. > > +""" > > + > > +import logging > > +import os > > +import re > > +import shutil > > +import subprocess > > +import sys > > +import tarfile > > +import tempfile > > +import time > > +import unittest > > +from urllib import request > > +import zipfile > > +from typing import ( > > + List, > > + Optional, > > + Sequence, > > +) > > +from tap import TAPTestRunner > > +from qemu.machine import QEMUMachine > > + > > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > > +PYTEST_PWD = os.getenv('PYTEST_PWD') > > + > > +def get_arch(): > > + """finds the arch from the qemu binary name""" > > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > > + if match: > > + return match.group(1) > > + return 'x86_64' > > + > > +ARCH = get_arch() > > + > > +class QEMUBitsMachine(QEMUMachine): > > + """ > > + A QEMU VM, with isa-debugcon enabled and bits iso passed > > + using -cdrom to QEMU commandline. > > + """ > > + def __init__(self, > > + binary: str, > > + args: Sequence[str] = (), > > + wrapper: Sequence[str] = (), > > + name: Optional[str] = None, > > + base_temp_dir: str = "/var/tmp", > > + debugcon_log: str = "debugcon-log.txt", > > + debugcon_addr: str = "0x403", > > + sock_dir: Optional[str] = None, > > + qmp_timer: Optional[float] = None): > > + # pylint: disable=too-many-arguments > > + > > + if name is None: > > + name = "qemu-bits-%d" % os.getpid() > > + if sock_dir is None: > > + sock_dir = base_temp_dir > > + super().__init__(binary, args, wrapper=wrapper, name=name, > > + base_temp_dir=base_temp_dir, > > + sock_dir=sock_dir, qmp_timer=qmp_timer) > > + self.debugcon_log = debugcon_log > > + self.debugcon_addr = debugcon_addr > > + self.base_temp_dir = base_temp_dir > > + > > + @property > > + def _base_args(self) -> List[str]: > > + args = super()._base_args > > + args.extend([ > > + '-chardev', > > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > > + self.debugcon_log), > > + '-device', > > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > > + ]) > > + return args > > + > > + def base_args(self): > > + """return the base argument to QEMU binary""" > > + return self._base_args > > + > > +class AcpiBitsTest(unittest.TestCase): > > + """ACPI and SMBIOS tests using biosbits.""" > > + def __init__(self, *args, **kwargs): > > + super().__init__(*args, **kwargs) > > + self._vm = None > > + self._workDir = None > > + self._bitsVer = 2100 > > + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" > > + self._debugcon_addr = '0x403' > > + self._debugcon_log = 'debugcon-log.txt' > > + logging.basicConfig(level=logging.INFO) > > + > > + def copy_bits_config(self): > > + """ copies the bios bits config file into bits. > > + """ > > + config_file = 'bits-cfg.txt' > > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > > + target_config_dir = os.path.join(self._workDir, > > + 'bits-%d' %self._bitsVer, 'boot') > > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > > + self.assertTrue(os.path.exists(target_config_dir)) > > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > > + config_file), os.R_OK)) > > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > > + target_config_dir) > > + logging.info('copied config file %s to %s', > > + config_file, target_config_dir) > > + > > + def copy_test_scripts(self): > > + """copies the python test scripts into bits. """ > > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > > + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, > > + 'boot', 'python') > > + > > + self.assertTrue(os.path.exists(qemu_test_dir)) > > + self.assertTrue(os.path.exists(target_test_dir)) > > + > > + for filename in os.listdir(qemu_test_dir): > > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > > + filename.endswith('.py'): > > + shutil.copy2(os.path.join(qemu_test_dir, filename), > > + target_test_dir) > > + logging.info('copied test file %s to %s', > > + filename, target_test_dir) > > + > > + # now remove the pyc test file if it exists, otherwise the > > + # changes in the python test script won't be executed. > > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > > + if os.access(os.path.join(target_test_dir, testfile_pyc), > > + os.F_OK): > > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > > + logging.info('removed compiled file %s', > > + os.path.join(target_test_dir, testfile_pyc)) > > + > > + def fix_mkrescue(self, mkrescue): > > + """ grub-mkrescue is a bash script with two variables, 'prefix' and > > + 'libdir'. They must be pointed to the right location so that the > > + iso can be generated appropriately. We point the two variables to > > + the directory where we have extracted our pre-built bits grub > > + tarball. > > + """ > > + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') > > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > > + > > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > > + self.assertTrue(os.path.exists(grub_i386_mods)) > > + > > + new_script = "" > > + with open(mkrescue, 'r') as filehandle: > > + orig_script = filehandle.read() > > + new_script = re.sub('(^prefix=)(.*)', > > + r'\1"%s"' %grub_x86_64_mods, > > + orig_script, flags=re.M) > > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, > > + new_script, flags=re.M) > > + > > + with open(mkrescue, 'w') as filehandle: > > + filehandle.write(new_script) > > + > > + def generate_bits_iso(self): > > + """ Uses grub-mkrescue to generate a fresh bits iso with the python > > + test scripts > > + """ > > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > > + mkrescue_script = os.path.join(self._workDir, > > + 'grub-inst-x86_64-efi', 'bin', > > + 'grub-mkrescue') > > + > > + self.assertTrue(os.access(mkrescue_script, > > + os.R_OK | os.W_OK | os.X_OK)) > > + > > + self.fix_mkrescue(mkrescue_script) > > + > > + logging.info('calling grub-mkrescue to generate the biosbits iso ...') > > + > > + try: > > + if os.getenv('V'): > > + subprocess.check_call([mkrescue_script, '-o', > > + iso_file, bits_dir], > > + stdout=subprocess.DEVNULL) > > + else: > > + subprocess.check_call([mkrescue_script, '-o', > > + iso_file, bits_dir], > > + stderr=subprocess.DEVNULL, > > + stdout=subprocess.DEVNULL) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("Error while generating the bits iso. " > > + "Pass V=1 in the environment to get more details. " > > + + str(e)) > > + > > + self.assertTrue(os.access(iso_file, os.R_OK)) > > + > > + logging.info('iso file %s successfully generated.', iso_file) > > + > > + def setUp(self): > > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > > + if BITS_LOC: > > + prefix = BITS_LOC > > + else: > > + prefix = os.path.join(os.getcwd(), 'prebuilt') > > + if not os.path.isdir(prefix): > > + os.mkdir(prefix, mode=0o775) > > + > > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > > + %self._bitsVer) > > + grub_tar_file = os.path.join(prefix, > > + 'bits-%d-grub.tar.gz' %self._bitsVer) > > + # if the location of the bits binaries has been specified by the user > > + # and they are not found in that location, skip the test. > > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be found in the specified location %s." \ > > + %BITS_LOC) > > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be found in the specified location %s." \ > > + %BITS_LOC) > > + > > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > > + suffix='.tmp') > > + logging.info('working dir: %s', self._workDir) > > + > > + localArchive = "bits-%d.zip" % self._bitsVer > > + if not os.access(bits_zip_file, os.F_OK): > > + logging.info("archive %s not found in %s, downloading ...", > > + localArchive, bits_zip_file) > > + try: > > + req = request.urlopen(self._bitsLoc + localArchive) > > + with open(os.path.join(prefix, localArchive), > > + 'wb') as archivef: > > + archivef.write(req.read()) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be obtained." + str(e)) > > + else: > > + logging.info('using locally found %s', localArchive) > > + > > + localArchive = "bits-%d-grub.tar.gz" % self._bitsVer > > + if not os.access(grub_tar_file, os.F_OK): > > + logging.info("archive %s not found in %s, downloading ...", > > + localArchive, bits_zip_file) > > + try: > > + req = request.urlopen(self._bitsLoc + localArchive) > > + with open(os.path.join(prefix, localArchive), > > + 'wb') as archivef: > > + archivef.write(req.read()) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be obtained." + str(e)) > > + else: > > + logging.info('using locally found %s', localArchive) > > + > > + # extract the bits software in the temp working directory > > + with zipfile.ZipFile(bits_zip_file, 'r') as zref: > > + zref.extractall(self._workDir) > > + > > + with tarfile.open(grub_tar_file, 'r') as tarball: > > + tarball.extractall(self._workDir) > > + > > + self.copy_test_scripts() > > + self.copy_bits_config() > > + self.generate_bits_iso() > > + > > + def parse_log(self): > > + """parse the log generated by running bits tests and > > + check for failures. > > + """ > > + debugconf = os.path.join(self._workDir, self._debugcon_log) > > + log = "" > > + with open(debugconf, 'r') as filehandle: > > + log = filehandle.read() > > + > > + if os.getenv('V'): > > + print('\nlogs from biosbits follows:') > > + print('==========================================\n') > > + print(log) > > + print('==========================================\n') > > + > > + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*', > > + log) > > + for match in matchiter: > > + # verify that no test cases failed. > > + self.assertEqual(match.group(3).split()[0], '0', > > + 'Some bits tests seems to have failed. ' \ > > + 'Set V=1 in the environment to get the entire ' \ > > + 'log from bits.') > > + > > + def tearDown(self): > > + if self._vm: > > + self.assertFalse(not self._vm.is_running) > > + logging.info('removing the work directory %s', self._workDir) > > + shutil.rmtree(self._workDir) > > + > > + def test_acpi_smbios_bits(self): > > + """The main test case implementaion.""" > > + > > + qemu_bin = PYTESTQEMUBIN > > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > > + > > + # PYTESTQEMUBIN could be relative to the current directory > > + if not os.access(PYTESTQEMUBIN, os.X_OK) and PYTEST_PWD: > > + qemu_bin = os.path.join(PYTEST_PWD, PYTESTQEMUBIN) > > + > > + logging.info('QEMU binary used: %s', qemu_bin) > > + > > + self.assertTrue(os.access(qemu_bin, os.X_OK)) > > + self.assertTrue(os.access(iso_file, os.R_OK)) > > + > > + self._vm = QEMUBitsMachine(binary=qemu_bin, > > + base_temp_dir=self._workDir, > > + debugcon_log=self._debugcon_log, > > + debugcon_addr=self._debugcon_addr) > > + > > + self._vm.add_args('-cdrom', '%s' %iso_file) > > + > > + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ > > + " " + " ".join(str(arg) for arg in self._vm.args) > > + > > + logging.info("launching QEMU vm with the following arguments: %s", > > + args) > > + > > + self._vm.launch() > > + # biosbits has been configured to run all the specified test suites > > + # in batch mode and then automatically initiate a vm shutdown. > > + # sleep for maximum of one minute > > + max_sleep_time = time.monotonic() + 60 > > + while self._vm.is_running() and time.monotonic() < max_sleep_time: > > + time.sleep(1) > > + > > + self.assertFalse(time.monotonic() > max_sleep_time, > > + 'The VM seems to have failed to shutdown in time') > > + > > + self.parse_log() > > + > > +def execute_unittest(argv: List[str], debug: bool = False, > > + runner: TAPTestRunner = None) -> None: > > + """Executes unittests within the calling module.""" > > + > > + unittest.main(argv=argv, > > + testRunner=runner, > > + verbosity=2 if debug else 1, > > + warnings=None if sys.warnoptions else 'ignore') > > + > > +def main(): > > + """ The main function where execution begins. """ > > + > > + assert PYTESTQEMUBIN is not None, \ > > + "Environment variable PYTEST_QEMU_BINARY required." > > + > > + runner = TAPTestRunner() > > + runner.set_stream(True) > > + runner.set_format("%s/acpi-bits-test" %ARCH) > > + execute_unittest(sys.argv, False, runner) > > + > > +main() > > diff --git a/tests/pytest/acpi-bits/meson.build b/tests/pytest/acpi-bits/meson.build > > new file mode 100644 > > index 0000000000..099c191d57 > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/meson.build > > @@ -0,0 +1,33 @@ > > +xorriso = find_program('xorriso', required: true) > > +if not xorriso.found() > > + message('xorriso not found ... disabled bits acpi tests.') > > + subdir_done() > > +endif > > + > > +subdir('bits-tests') > > +subdir('bits-config') > > + > > +test_files = ['acpi-bits-test.py'] > > +requirements = 'requirements.txt' > > + > > +copytestfiles = custom_target('copy test files', > > + input : test_files, > > + output : test_files, > > + command : ['cp', '@INPUT@', '@OUTPUT@'], > > + install : false, > > + build_by_default : true) > > + > > +requirementsfiles = custom_target('copy py req files', > > + input : requirements, > > + output : requirements, > > + command : ['cp', '@INPUT@', '@OUTPUT@'], > > + install : false, > > + build_by_default : true) > > + > > +other_deps += [copytestfiles,requirementsfiles] > > + > > +pytest_executables += { > > + 'acpi-bits-test': configure_file(copy:true, > > + input:'acpi-bits-test-venv.sh', > > + output:'acpi-bits-test') > > +} > > diff --git a/tests/pytest/acpi-bits/requirements.txt b/tests/pytest/acpi-bits/requirements.txt > > new file mode 100644 > > index 0000000000..00cdad09ef > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/requirements.txt > > @@ -0,0 +1 @@ > > +tap.py > > -- > > 2.25.1 > > > >
On Sun, Jul 10, 2022 at 10:30:10PM +0530, Ani Sinha wrote: > This change adds python based test environment that can be used to run pytest > from within a virtual environment. A bash script sets up a virtual environment > and then runs the python based tests from within that environment. > All dependent python packages are installed in the virtual environment using > pip python module. QEMU python test modules are also available in the environment > for spawning the QEMU based VMs. > > It also introduces QEMU acpi/smbios biosbits python test script which is run > from within the python virtual environment. When the bios bits tests are run, > bios bits binaries are downloaded from an external repo/location. > Currently, the test points to an external private github repo where the bits > archives are checked in. > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > tests/pytest/acpi-bits/meson.build | 33 ++ > tests/pytest/acpi-bits/requirements.txt | 1 + > 4 files changed, 475 insertions(+) > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > create mode 100644 tests/pytest/acpi-bits/meson.build > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > new file mode 100644 > index 0000000000..186395473b > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > @@ -0,0 +1,59 @@ > +#!/usr/bin/env bash > +# Generates a python virtual environment for the test to run. > +# Then runs python test scripts from within that virtual environment. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Author: Ani Sinha <ani@anisinha.ca> > + > +set -e > + > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > +MYDIR=$(dirname $MYPATH) > + > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > + echo " to the root of the qemu source tree." > + echo -n "This is required so that the test can find the " > + echo "python modules that it needs for execution." > + exit 1 > +fi > +SRCDIR=$PYTEST_SOURCE_ROOT > +TESTSCRIPTS=("acpi-bits-test.py") > +PIPCMD="-m pip -q --disable-pip-version-check" > +# we need to save the old value of PWD before we do a change-dir later > +PYTEST_PWD=$PWD > + > +TESTS_PYTHON=/usr/bin/python3 > +TESTS_VENV_REQ=requirements.txt > + > +# sadly for pip -e and -t options do not work together. > +# please see https://github.com/pypa/pip/issues/562 > +cd $MYDIR > + > +$TESTS_PYTHON -m venv . > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > +[ -f $TESTS_VENV_REQ ] && \ > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > + > +# venv is activated at this point. > + > +# run the test > +for testscript in ${TESTSCRIPTS[@]} ; do > + export PYTEST_PWD; python3 $testscript > +done > + > +cd $PYTEST_PWD > + > +exit 0 > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py > new file mode 100644 > index 0000000000..97e61eb709 > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > @@ -0,0 +1,382 @@ > +#!/usr/bin/env python3 > +# group: rw quick > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > +# https://biosbits.org/ > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Some parts are slightly taken from qtest.py and iotests.py > +# > +# Authors: > +# Ani Sinha <ani@anisinha.ca> > + > +# pylint: disable=invalid-name > + > +""" > +QEMU bios tests using biosbits available at > +https://biosbits.org/. > +""" > + > +import logging > +import os > +import re > +import shutil > +import subprocess > +import sys > +import tarfile > +import tempfile > +import time > +import unittest > +from urllib import request > +import zipfile > +from typing import ( > + List, > + Optional, > + Sequence, > +) > +from tap import TAPTestRunner > +from qemu.machine import QEMUMachine > + > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > +PYTEST_PWD = os.getenv('PYTEST_PWD') > + > +def get_arch(): > + """finds the arch from the qemu binary name""" > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > + if match: > + return match.group(1) > + return 'x86_64' > + > +ARCH = get_arch() > + > +class QEMUBitsMachine(QEMUMachine): > + """ > + A QEMU VM, with isa-debugcon enabled and bits iso passed > + using -cdrom to QEMU commandline. > + """ > + def __init__(self, > + binary: str, > + args: Sequence[str] = (), > + wrapper: Sequence[str] = (), > + name: Optional[str] = None, > + base_temp_dir: str = "/var/tmp", > + debugcon_log: str = "debugcon-log.txt", > + debugcon_addr: str = "0x403", > + sock_dir: Optional[str] = None, > + qmp_timer: Optional[float] = None): > + # pylint: disable=too-many-arguments > + > + if name is None: > + name = "qemu-bits-%d" % os.getpid() > + if sock_dir is None: > + sock_dir = base_temp_dir > + super().__init__(binary, args, wrapper=wrapper, name=name, > + base_temp_dir=base_temp_dir, > + sock_dir=sock_dir, qmp_timer=qmp_timer) > + self.debugcon_log = debugcon_log > + self.debugcon_addr = debugcon_addr > + self.base_temp_dir = base_temp_dir > + > + @property > + def _base_args(self) -> List[str]: > + args = super()._base_args > + args.extend([ > + '-chardev', > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > + self.debugcon_log), > + '-device', > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > + ]) > + return args > + > + def base_args(self): > + """return the base argument to QEMU binary""" > + return self._base_args > + > +class AcpiBitsTest(unittest.TestCase): > + """ACPI and SMBIOS tests using biosbits.""" > + def __init__(self, *args, **kwargs): > + super().__init__(*args, **kwargs) > + self._vm = None > + self._workDir = None > + self._bitsVer = 2100 > + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" This URL location gives a 404 - was it supposed to be poiinting to the bits-builds branch ? eg to form a URL like: https://github.com/ani-sinha/bits/blob/bits-builds/bits-2100.zip?raw=true > + self._debugcon_addr = '0x403' > + self._debugcon_log = 'debugcon-log.txt' > + logging.basicConfig(level=logging.INFO) > + > + def copy_bits_config(self): > + """ copies the bios bits config file into bits. > + """ > + config_file = 'bits-cfg.txt' > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > + target_config_dir = os.path.join(self._workDir, > + 'bits-%d' %self._bitsVer, 'boot') > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > + self.assertTrue(os.path.exists(target_config_dir)) > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > + config_file), os.R_OK)) > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > + target_config_dir) > + logging.info('copied config file %s to %s', > + config_file, target_config_dir) > + > + def copy_test_scripts(self): > + """copies the python test scripts into bits. """ > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, > + 'boot', 'python') > + > + self.assertTrue(os.path.exists(qemu_test_dir)) > + self.assertTrue(os.path.exists(target_test_dir)) > + > + for filename in os.listdir(qemu_test_dir): > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > + filename.endswith('.py'): > + shutil.copy2(os.path.join(qemu_test_dir, filename), > + target_test_dir) > + logging.info('copied test file %s to %s', > + filename, target_test_dir) > + > + # now remove the pyc test file if it exists, otherwise the > + # changes in the python test script won't be executed. > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > + if os.access(os.path.join(target_test_dir, testfile_pyc), > + os.F_OK): > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > + logging.info('removed compiled file %s', > + os.path.join(target_test_dir, testfile_pyc)) > + > + def fix_mkrescue(self, mkrescue): > + """ grub-mkrescue is a bash script with two variables, 'prefix' and > + 'libdir'. They must be pointed to the right location so that the > + iso can be generated appropriately. We point the two variables to > + the directory where we have extracted our pre-built bits grub > + tarball. > + """ > + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > + > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > + self.assertTrue(os.path.exists(grub_i386_mods)) > + > + new_script = "" > + with open(mkrescue, 'r') as filehandle: > + orig_script = filehandle.read() > + new_script = re.sub('(^prefix=)(.*)', > + r'\1"%s"' %grub_x86_64_mods, > + orig_script, flags=re.M) > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, > + new_script, flags=re.M) > + > + with open(mkrescue, 'w') as filehandle: > + filehandle.write(new_script) > + > + def generate_bits_iso(self): > + """ Uses grub-mkrescue to generate a fresh bits iso with the python > + test scripts > + """ > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > + mkrescue_script = os.path.join(self._workDir, > + 'grub-inst-x86_64-efi', 'bin', > + 'grub-mkrescue') > + > + self.assertTrue(os.access(mkrescue_script, > + os.R_OK | os.W_OK | os.X_OK)) > + > + self.fix_mkrescue(mkrescue_script) > + > + logging.info('calling grub-mkrescue to generate the biosbits iso ...') > + > + try: > + if os.getenv('V'): > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stdout=subprocess.DEVNULL) > + else: > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stderr=subprocess.DEVNULL, > + stdout=subprocess.DEVNULL) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("Error while generating the bits iso. " > + "Pass V=1 in the environment to get more details. " > + + str(e)) > + > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + logging.info('iso file %s successfully generated.', iso_file) > + > + def setUp(self): > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > + if BITS_LOC: > + prefix = BITS_LOC > + else: > + prefix = os.path.join(os.getcwd(), 'prebuilt') > + if not os.path.isdir(prefix): > + os.mkdir(prefix, mode=0o775) > + > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > + %self._bitsVer) > + grub_tar_file = os.path.join(prefix, > + 'bits-%d-grub.tar.gz' %self._bitsVer) > + # if the location of the bits binaries has been specified by the user > + # and they are not found in that location, skip the test. > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location %s." \ > + %BITS_LOC) > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location %s." \ > + %BITS_LOC) > + > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > + suffix='.tmp') > + logging.info('working dir: %s', self._workDir) > + > + localArchive = "bits-%d.zip" % self._bitsVer > + if not os.access(bits_zip_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) so you skip downlaod if it already exists locally. IIUC it is looking in the CWD, which is presumably the directory the QEMU build is performed in ? So if dev cleans their build tree, the cache is lost ? Avocado has a more persistent cache outside the build tree IIUC. With regards, Daniel
On Thu, 14 Jul 2022, Daniel P. Berrangé wrote: > On Sun, Jul 10, 2022 at 10:30:10PM +0530, Ani Sinha wrote: > > This change adds python based test environment that can be used to run pytest > > from within a virtual environment. A bash script sets up a virtual environment > > and then runs the python based tests from within that environment. > > All dependent python packages are installed in the virtual environment using > > pip python module. QEMU python test modules are also available in the environment > > for spawning the QEMU based VMs. > > > > It also introduces QEMU acpi/smbios biosbits python test script which is run > > from within the python virtual environment. When the bios bits tests are run, > > bios bits binaries are downloaded from an external repo/location. > > Currently, the test points to an external private github repo where the bits > > archives are checked in. > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > > tests/pytest/acpi-bits/meson.build | 33 ++ > > tests/pytest/acpi-bits/requirements.txt | 1 + > > 4 files changed, 475 insertions(+) > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > > create mode 100644 tests/pytest/acpi-bits/meson.build > > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > new file mode 100644 > > index 0000000000..186395473b > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > @@ -0,0 +1,59 @@ > > +#!/usr/bin/env bash > > +# Generates a python virtual environment for the test to run. > > +# Then runs python test scripts from within that virtual environment. > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Author: Ani Sinha <ani@anisinha.ca> > > + > > +set -e > > + > > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > > +MYDIR=$(dirname $MYPATH) > > + > > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > > + echo " to the root of the qemu source tree." > > + echo -n "This is required so that the test can find the " > > + echo "python modules that it needs for execution." > > + exit 1 > > +fi > > +SRCDIR=$PYTEST_SOURCE_ROOT > > +TESTSCRIPTS=("acpi-bits-test.py") > > +PIPCMD="-m pip -q --disable-pip-version-check" > > +# we need to save the old value of PWD before we do a change-dir later > > +PYTEST_PWD=$PWD > > + > > +TESTS_PYTHON=/usr/bin/python3 > > +TESTS_VENV_REQ=requirements.txt > > + > > +# sadly for pip -e and -t options do not work together. > > +# please see https://github.com/pypa/pip/issues/562 > > +cd $MYDIR > > + > > +$TESTS_PYTHON -m venv . > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > > +[ -f $TESTS_VENV_REQ ] && \ > > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > > + > > +# venv is activated at this point. > > + > > +# run the test > > +for testscript in ${TESTSCRIPTS[@]} ; do > > + export PYTEST_PWD; python3 $testscript > > +done > > + > > +cd $PYTEST_PWD > > + > > +exit 0 > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py > > new file mode 100644 > > index 0000000000..97e61eb709 > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > > @@ -0,0 +1,382 @@ > > +#!/usr/bin/env python3 > > +# group: rw quick > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > > +# https://biosbits.org/ > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Some parts are slightly taken from qtest.py and iotests.py > > +# > > +# Authors: > > +# Ani Sinha <ani@anisinha.ca> > > + > > +# pylint: disable=invalid-name > > + > > +""" > > +QEMU bios tests using biosbits available at > > +https://biosbits.org/. > > +""" > > + > > +import logging > > +import os > > +import re > > +import shutil > > +import subprocess > > +import sys > > +import tarfile > > +import tempfile > > +import time > > +import unittest > > +from urllib import request > > +import zipfile > > +from typing import ( > > + List, > > + Optional, > > + Sequence, > > +) > > +from tap import TAPTestRunner > > +from qemu.machine import QEMUMachine > > + > > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > > +PYTEST_PWD = os.getenv('PYTEST_PWD') > > + > > +def get_arch(): > > + """finds the arch from the qemu binary name""" > > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > > + if match: > > + return match.group(1) > > + return 'x86_64' > > + > > +ARCH = get_arch() > > + > > +class QEMUBitsMachine(QEMUMachine): > > + """ > > + A QEMU VM, with isa-debugcon enabled and bits iso passed > > + using -cdrom to QEMU commandline. > > + """ > > + def __init__(self, > > + binary: str, > > + args: Sequence[str] = (), > > + wrapper: Sequence[str] = (), > > + name: Optional[str] = None, > > + base_temp_dir: str = "/var/tmp", > > + debugcon_log: str = "debugcon-log.txt", > > + debugcon_addr: str = "0x403", > > + sock_dir: Optional[str] = None, > > + qmp_timer: Optional[float] = None): > > + # pylint: disable=too-many-arguments > > + > > + if name is None: > > + name = "qemu-bits-%d" % os.getpid() > > + if sock_dir is None: > > + sock_dir = base_temp_dir > > + super().__init__(binary, args, wrapper=wrapper, name=name, > > + base_temp_dir=base_temp_dir, > > + sock_dir=sock_dir, qmp_timer=qmp_timer) > > + self.debugcon_log = debugcon_log > > + self.debugcon_addr = debugcon_addr > > + self.base_temp_dir = base_temp_dir > > + > > + @property > > + def _base_args(self) -> List[str]: > > + args = super()._base_args > > + args.extend([ > > + '-chardev', > > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > > + self.debugcon_log), > > + '-device', > > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > > + ]) > > + return args > > + > > + def base_args(self): > > + """return the base argument to QEMU binary""" > > + return self._base_args > > + > > +class AcpiBitsTest(unittest.TestCase): > > + """ACPI and SMBIOS tests using biosbits.""" > > + def __init__(self, *args, **kwargs): > > + super().__init__(*args, **kwargs) > > + self._vm = None > > + self._workDir = None > > + self._bitsVer = 2100 > > + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" > > This URL location gives a 404 - was it supposed to be poiinting to the > bits-builds branch ? eg to form a URL like: > You are looking at a partial URL. The code completes the URL based on the version we are using. Append to this URL the archive name, like bits-2100.zip or something. eg, https://github.com/ani-sinha/bits/raw/bits-builds/bits-2100.zip > https://github.com/ani-sinha/bits/blob/bits-builds/bits-2100.zip?raw=true > > > + self._debugcon_addr = '0x403' > > + self._debugcon_log = 'debugcon-log.txt' > > + logging.basicConfig(level=logging.INFO) > > + > > + def copy_bits_config(self): > > + """ copies the bios bits config file into bits. > > + """ > > + config_file = 'bits-cfg.txt' > > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > > + target_config_dir = os.path.join(self._workDir, > > + 'bits-%d' %self._bitsVer, 'boot') > > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > > + self.assertTrue(os.path.exists(target_config_dir)) > > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > > + config_file), os.R_OK)) > > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > > + target_config_dir) > > + logging.info('copied config file %s to %s', > > + config_file, target_config_dir) > > + > > + def copy_test_scripts(self): > > + """copies the python test scripts into bits. """ > > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > > + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, > > + 'boot', 'python') > > + > > + self.assertTrue(os.path.exists(qemu_test_dir)) > > + self.assertTrue(os.path.exists(target_test_dir)) > > + > > + for filename in os.listdir(qemu_test_dir): > > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > > + filename.endswith('.py'): > > + shutil.copy2(os.path.join(qemu_test_dir, filename), > > + target_test_dir) > > + logging.info('copied test file %s to %s', > > + filename, target_test_dir) > > + > > + # now remove the pyc test file if it exists, otherwise the > > + # changes in the python test script won't be executed. > > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > > + if os.access(os.path.join(target_test_dir, testfile_pyc), > > + os.F_OK): > > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > > + logging.info('removed compiled file %s', > > + os.path.join(target_test_dir, testfile_pyc)) > > + > > + def fix_mkrescue(self, mkrescue): > > + """ grub-mkrescue is a bash script with two variables, 'prefix' and > > + 'libdir'. They must be pointed to the right location so that the > > + iso can be generated appropriately. We point the two variables to > > + the directory where we have extracted our pre-built bits grub > > + tarball. > > + """ > > + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') > > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > > + > > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > > + self.assertTrue(os.path.exists(grub_i386_mods)) > > + > > + new_script = "" > > + with open(mkrescue, 'r') as filehandle: > > + orig_script = filehandle.read() > > + new_script = re.sub('(^prefix=)(.*)', > > + r'\1"%s"' %grub_x86_64_mods, > > + orig_script, flags=re.M) > > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, > > + new_script, flags=re.M) > > + > > + with open(mkrescue, 'w') as filehandle: > > + filehandle.write(new_script) > > + > > + def generate_bits_iso(self): > > + """ Uses grub-mkrescue to generate a fresh bits iso with the python > > + test scripts > > + """ > > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > > + mkrescue_script = os.path.join(self._workDir, > > + 'grub-inst-x86_64-efi', 'bin', > > + 'grub-mkrescue') > > + > > + self.assertTrue(os.access(mkrescue_script, > > + os.R_OK | os.W_OK | os.X_OK)) > > + > > + self.fix_mkrescue(mkrescue_script) > > + > > + logging.info('calling grub-mkrescue to generate the biosbits iso ...') > > + > > + try: > > + if os.getenv('V'): > > + subprocess.check_call([mkrescue_script, '-o', > > + iso_file, bits_dir], > > + stdout=subprocess.DEVNULL) > > + else: > > + subprocess.check_call([mkrescue_script, '-o', > > + iso_file, bits_dir], > > + stderr=subprocess.DEVNULL, > > + stdout=subprocess.DEVNULL) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("Error while generating the bits iso. " > > + "Pass V=1 in the environment to get more details. " > > + + str(e)) > > + > > + self.assertTrue(os.access(iso_file, os.R_OK)) > > + > > + logging.info('iso file %s successfully generated.', iso_file) > > + > > + def setUp(self): > > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > > + if BITS_LOC: > > + prefix = BITS_LOC > > + else: > > + prefix = os.path.join(os.getcwd(), 'prebuilt') > > + if not os.path.isdir(prefix): > > + os.mkdir(prefix, mode=0o775) > > + > > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > > + %self._bitsVer) > > + grub_tar_file = os.path.join(prefix, > > + 'bits-%d-grub.tar.gz' %self._bitsVer) > > + # if the location of the bits binaries has been specified by the user > > + # and they are not found in that location, skip the test. > > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be found in the specified location %s." \ > > + %BITS_LOC) > > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be found in the specified location %s." \ > > + %BITS_LOC) > > + > > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > > + suffix='.tmp') > > + logging.info('working dir: %s', self._workDir) > > + > > + localArchive = "bits-%d.zip" % self._bitsVer > > + if not os.access(bits_zip_file, os.F_OK): > > + logging.info("archive %s not found in %s, downloading ...", > > + localArchive, bits_zip_file) > > + try: > > + req = request.urlopen(self._bitsLoc + localArchive) > > + with open(os.path.join(prefix, localArchive), > > + 'wb') as archivef: > > + archivef.write(req.read()) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be obtained." + str(e)) > > + else: > > + logging.info('using locally found %s', localArchive) > > so you skip downlaod if it already exists locally. IIUC it is looking > in the CWD, which is presumably the directory the QEMU build is > performed in ? Yes, build/test/pytest/bits-test > So if dev cleans their build tree, the cache is lost ? Yes. > > Avocado has a more persistent cache outside the build tree IIUC. > So that is why I have the environment variable mechanism for passing to the test that will point to the location of the archives. The developer can download the files there and point to the test. Or I can change the scriprt accordingly if we know where we were downloading. We were discussing submodules and it was fiercely hated. So mst suggested another repo where to keep the binaries. My script that check out that repo somewhere outside the build directory and point the test to that location. I have kept several options open. We just need to make some decisions. > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Thu, Jul 14, 2022 at 19:49 Ani Sinha <ani@anisinha.ca> wrote: > > > On Thu, 14 Jul 2022, Daniel P. Berrangé wrote: > > > On Sun, Jul 10, 2022 at 10:30:10PM +0530, Ani Sinha wrote: > > > This change adds python based test environment that can be used to run > pytest > > > from within a virtual environment. A bash script sets up a virtual > environment > > > and then runs the python based tests from within that environment. > > > All dependent python packages are installed in the virtual environment > using > > > pip python module. QEMU python test modules are also available in the > environment > > > for spawning the QEMU based VMs. > > > > > > It also introduces QEMU acpi/smbios biosbits python test script which > is run > > > from within the python virtual environment. When the bios bits tests > are run, > > > bios bits binaries are downloaded from an external repo/location. > > > Currently, the test points to an external private github repo where > the bits > > > archives are checked in. > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > --- > > > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > > > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > > > tests/pytest/acpi-bits/meson.build | 33 ++ > > > tests/pytest/acpi-bits/requirements.txt | 1 + > > > 4 files changed, 475 insertions(+) > > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > > > create mode 100644 tests/pytest/acpi-bits/meson.build > > > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > > > > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > > new file mode 100644 > > > index 0000000000..186395473b > > > --- /dev/null > > > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > > @@ -0,0 +1,59 @@ > > > +#!/usr/bin/env bash > > > +# Generates a python virtual environment for the test to run. > > > +# Then runs python test scripts from within that virtual environment. > > > +# > > > +# This program is free software; you can redistribute it and/or modify > > > +# it under the terms of the GNU General Public License as published by > > > +# the Free Software Foundation; either version 2 of the License, or > > > +# (at your option) any later version. > > > +# > > > +# This program is distributed in the hope that it will be useful, > > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > +# GNU General Public License for more details. > > > +# > > > +# You should have received a copy of the GNU General Public License > > > +# along with this program. If not, see <http://www.gnu.org/licenses/ > >. > > > +# > > > +# Author: Ani Sinha <ani@anisinha.ca> > > > + > > > +set -e > > > + > > > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > > > +MYDIR=$(dirname $MYPATH) > > > + > > > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > > > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > > > + echo " to the root of the qemu source tree." > > > + echo -n "This is required so that the test can find the " > > > + echo "python modules that it needs for execution." > > > + exit 1 > > > +fi > > > +SRCDIR=$PYTEST_SOURCE_ROOT > > > +TESTSCRIPTS=("acpi-bits-test.py") > > > +PIPCMD="-m pip -q --disable-pip-version-check" > > > +# we need to save the old value of PWD before we do a change-dir later > > > +PYTEST_PWD=$PWD > > > + > > > +TESTS_PYTHON=/usr/bin/python3 > > > +TESTS_VENV_REQ=requirements.txt > > > + > > > +# sadly for pip -e and -t options do not work together. > > > +# please see https://github.com/pypa/pip/issues/562 > > > +cd $MYDIR > > > + > > > +$TESTS_PYTHON -m venv . > > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > > > +[ -f $TESTS_VENV_REQ ] && \ > > > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > > > + > > > +# venv is activated at this point. > > > + > > > +# run the test > > > +for testscript in ${TESTSCRIPTS[@]} ; do > > > + export PYTEST_PWD; python3 $testscript > > > +done > > > + > > > +cd $PYTEST_PWD > > > + > > > +exit 0 > > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py > b/tests/pytest/acpi-bits/acpi-bits-test.py > > > new file mode 100644 > > > index 0000000000..97e61eb709 > > > --- /dev/null > > > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > > > @@ -0,0 +1,382 @@ > > > +#!/usr/bin/env python3 > > > +# group: rw quick > > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > > > +# https://biosbits.org/ > > > +# > > > +# This program is free software; you can redistribute it and/or modify > > > +# it under the terms of the GNU General Public License as published by > > > +# the Free Software Foundation; either version 2 of the License, or > > > +# (at your option) any later version. > > > +# > > > +# This program is distributed in the hope that it will be useful, > > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > +# GNU General Public License for more details. > > > +# > > > +# You should have received a copy of the GNU General Public License > > > +# along with this program. If not, see <http://www.gnu.org/licenses/ > >. > > > +# > > > +# Some parts are slightly taken from qtest.py and iotests.py > > > +# > > > +# Authors: > > > +# Ani Sinha <ani@anisinha.ca> > > > + > > > +# pylint: disable=invalid-name > > > + > > > +""" > > > +QEMU bios tests using biosbits available at > > > +https://biosbits.org/. > > > +""" > > > + > > > +import logging > > > +import os > > > +import re > > > +import shutil > > > +import subprocess > > > +import sys > > > +import tarfile > > > +import tempfile > > > +import time > > > +import unittest > > > +from urllib import request > > > +import zipfile > > > +from typing import ( > > > + List, > > > + Optional, > > > + Sequence, > > > +) > > > +from tap import TAPTestRunner > > > +from qemu.machine import QEMUMachine > > > + > > > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > > > +PYTEST_PWD = os.getenv('PYTEST_PWD') > > > + > > > +def get_arch(): > > > + """finds the arch from the qemu binary name""" > > > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > > > + if match: > > > + return match.group(1) > > > + return 'x86_64' > > > + > > > +ARCH = get_arch() > > > + > > > +class QEMUBitsMachine(QEMUMachine): > > > + """ > > > + A QEMU VM, with isa-debugcon enabled and bits iso passed > > > + using -cdrom to QEMU commandline. > > > + """ > > > + def __init__(self, > > > + binary: str, > > > + args: Sequence[str] = (), > > > + wrapper: Sequence[str] = (), > > > + name: Optional[str] = None, > > > + base_temp_dir: str = "/var/tmp", > > > + debugcon_log: str = "debugcon-log.txt", > > > + debugcon_addr: str = "0x403", > > > + sock_dir: Optional[str] = None, > > > + qmp_timer: Optional[float] = None): > > > + # pylint: disable=too-many-arguments > > > + > > > + if name is None: > > > + name = "qemu-bits-%d" % os.getpid() > > > + if sock_dir is None: > > > + sock_dir = base_temp_dir > > > + super().__init__(binary, args, wrapper=wrapper, name=name, > > > + base_temp_dir=base_temp_dir, > > > + sock_dir=sock_dir, qmp_timer=qmp_timer) > > > + self.debugcon_log = debugcon_log > > > + self.debugcon_addr = debugcon_addr > > > + self.base_temp_dir = base_temp_dir > > > + > > > + @property > > > + def _base_args(self) -> List[str]: > > > + args = super()._base_args > > > + args.extend([ > > > + '-chardev', > > > + 'file,path=%s,id=debugcon' > %os.path.join(self.base_temp_dir, > > > + > self.debugcon_log), > > > + '-device', > > > + 'isa-debugcon,iobase=%s,chardev=debugcon' > %self.debugcon_addr, > > > + ]) > > > + return args > > > + > > > + def base_args(self): > > > + """return the base argument to QEMU binary""" > > > + return self._base_args > > > + > > > +class AcpiBitsTest(unittest.TestCase): > > > + """ACPI and SMBIOS tests using biosbits.""" > > > + def __init__(self, *args, **kwargs): > > > + super().__init__(*args, **kwargs) > > > + self._vm = None > > > + self._workDir = None > > > + self._bitsVer = 2100 > > > + self._bitsLoc = " > https://github.com/ani-sinha/bits/raw/bits-builds/" > > > > This URL location gives a 404 - was it supposed to be poiinting to the > > bits-builds branch ? eg to form a URL like: > > > > You are looking at a partial URL. The code completes the URL based on the > version we are using. Append to this URL the archive name, like > bits-2100.zip or something. > > eg, https://github.com/ani-sinha/bits/raw/bits-builds/bits-2100.zip > > > > https://github.com/ani-sinha/bits/blob/bits-builds/bits-2100.zip?raw=true > > > > > + self._debugcon_addr = '0x403' > > > + self._debugcon_log = 'debugcon-log.txt' > > > + logging.basicConfig(level=logging.INFO) > > > + > > > + def copy_bits_config(self): > > > + """ copies the bios bits config file into bits. > > > + """ > > > + config_file = 'bits-cfg.txt' > > > + qemu_bits_config_dir = os.path.join(os.getcwd(), > 'bits-config') > > > + target_config_dir = os.path.join(self._workDir, > > > + 'bits-%d' %self._bitsVer, > 'boot') > > > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > > > + self.assertTrue(os.path.exists(target_config_dir)) > > > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > > > + config_file), os.R_OK)) > > > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > > > + target_config_dir) > > > + logging.info('copied config file %s to %s', > > > + config_file, target_config_dir) > > > + > > > + def copy_test_scripts(self): > > > + """copies the python test scripts into bits. """ > > > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > > > + target_test_dir = os.path.join(self._workDir, 'bits-%d' > %self._bitsVer, > > > + 'boot', 'python') > > > + > > > + self.assertTrue(os.path.exists(qemu_test_dir)) > > > + self.assertTrue(os.path.exists(target_test_dir)) > > > + > > > + for filename in os.listdir(qemu_test_dir): > > > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) > and \ > > > + filename.endswith('.py'): > > > + shutil.copy2(os.path.join(qemu_test_dir, filename), > > > + target_test_dir) > > > + logging.info('copied test file %s to %s', > > > + filename, target_test_dir) > > > + > > > + # now remove the pyc test file if it exists, > otherwise the > > > + # changes in the python test script won't be executed. > > > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > > > + if os.access(os.path.join(target_test_dir, > testfile_pyc), > > > + os.F_OK): > > > + os.remove(os.path.join(target_test_dir, > testfile_pyc)) > > > + logging.info('removed compiled file %s', > > > + os.path.join(target_test_dir, > testfile_pyc)) > > > + > > > + def fix_mkrescue(self, mkrescue): > > > + """ grub-mkrescue is a bash script with two variables, > 'prefix' and > > > + 'libdir'. They must be pointed to the right location so > that the > > > + iso can be generated appropriately. We point the two > variables to > > > + the directory where we have extracted our pre-built bits > grub > > > + tarball. > > > + """ > > > + grub_x86_64_mods = os.path.join(self._workDir, > 'grub-inst-x86_64-efi') > > > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > > > + > > > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > > > + self.assertTrue(os.path.exists(grub_i386_mods)) > > > + > > > + new_script = "" > > > + with open(mkrescue, 'r') as filehandle: > > > + orig_script = filehandle.read() > > > + new_script = re.sub('(^prefix=)(.*)', > > > + r'\1"%s"' %grub_x86_64_mods, > > > + orig_script, flags=re.M) > > > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' > %grub_i386_mods, > > > + new_script, flags=re.M) > > > + > > > + with open(mkrescue, 'w') as filehandle: > > > + filehandle.write(new_script) > > > + > > > + def generate_bits_iso(self): > > > + """ Uses grub-mkrescue to generate a fresh bits iso with the > python > > > + test scripts > > > + """ > > > + bits_dir = os.path.join(self._workDir, 'bits-%d' > %self._bitsVer) > > > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' > %self._bitsVer) > > > + mkrescue_script = os.path.join(self._workDir, > > > + 'grub-inst-x86_64-efi', 'bin', > > > + 'grub-mkrescue') > > > + > > > + self.assertTrue(os.access(mkrescue_script, > > > + os.R_OK | os.W_OK | os.X_OK)) > > > + > > > + self.fix_mkrescue(mkrescue_script) > > > + > > > + logging.info('calling grub-mkrescue to generate the biosbits > iso ...') > > > + > > > + try: > > > + if os.getenv('V'): > > > + subprocess.check_call([mkrescue_script, '-o', > > > + iso_file, bits_dir], > > > + stdout=subprocess.DEVNULL) > > > + else: > > > + subprocess.check_call([mkrescue_script, '-o', > > > + iso_file, bits_dir], > > > + stderr=subprocess.DEVNULL, > > > + stdout=subprocess.DEVNULL) > > > + except Exception as e: # pylint: disable=broad-except > > > + self.skipTest("Error while generating the bits iso. " > > > + "Pass V=1 in the environment to get more > details. " > > > + + str(e)) > > > + > > > + self.assertTrue(os.access(iso_file, os.R_OK)) > > > + > > > + logging.info('iso file %s successfully generated.', iso_file) > > > + > > > + def setUp(self): > > > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > > > + if BITS_LOC: > > > + prefix = BITS_LOC > > > + else: > > > + prefix = os.path.join(os.getcwd(), 'prebuilt') > > > + if not os.path.isdir(prefix): > > > + os.mkdir(prefix, mode=0o775) > > > + > > > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > > > + %self._bitsVer) > > > + grub_tar_file = os.path.join(prefix, > > > + 'bits-%d-grub.tar.gz' > %self._bitsVer) > > > + # if the location of the bits binaries has been specified by > the user > > > + # and they are not found in that location, skip the test. > > > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > > > + self.skipTest("test skipped since biosbits binaries " + > > > + "could not be found in the specified > location %s." \ > > > + %BITS_LOC) > > > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > > > + self.skipTest("test skipped since biosbits binaries " + > > > + "could not be found in the specified > location %s." \ > > > + %BITS_LOC) > > > + > > > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > > > + suffix='.tmp') > > > + logging.info('working dir: %s', self._workDir) > > > + > > > + localArchive = "bits-%d.zip" % self._bitsVer > > > + if not os.access(bits_zip_file, os.F_OK): > > > + logging.info("archive %s not found in %s, downloading > ...", > > > + localArchive, bits_zip_file) > > > + try: > > > + req = request.urlopen(self._bitsLoc + localArchive) > > > + with open(os.path.join(prefix, localArchive), > > > + 'wb') as archivef: > > > + archivef.write(req.read()) > > > + except Exception as e: # pylint: disable=broad-except > > > + self.skipTest("test skipped since biosbits binaries " > + > > > + "could not be obtained." + str(e)) > > > + else: > > > + logging.info('using locally found %s', localArchive) > > > > so you skip downlaod if it already exists locally. IIUC it is looking > > in the CWD, which is presumably the directory the QEMU build is > > performed in ? > > Yes, build/test/pytest/bits-test > > > So if dev cleans their build tree, the cache is lost ? > > Yes. > > > > > Avocado has a more persistent cache outside the build tree IIUC. To be honest I’m not sure if I like that. If I clear out my build directory I would want all build related artifacts to go away, including downloaded build stuff from the cache. > > > > So that is why I have the environment variable mechanism for passing to > the test that will point to the location of the archives. The developer > can download the files there and point to the test. Or I can change the > scriprt accordingly if we know where we were downloading. We were > discussing submodules and it was fiercely hated. So mst suggested another > repo where to keep the binaries. My script that check out that repo > somewhere outside the build directory and point the test to that location. > > I have kept several options open. We just need to make some decisions. > > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > > > >
On Sun, Jul 10, 2022 at 10:30:10PM +0530, Ani Sinha wrote: > This change adds python based test environment that can be used to run pytest > from within a virtual environment. A bash script sets up a virtual environment > and then runs the python based tests from within that environment. > All dependent python packages are installed in the virtual environment using > pip python module. QEMU python test modules are also available in the environment > for spawning the QEMU based VMs. > > It also introduces QEMU acpi/smbios biosbits python test script which is run > from within the python virtual environment. When the bios bits tests are run, > bios bits binaries are downloaded from an external repo/location. > Currently, the test points to an external private github repo where the bits > archives are checked in. > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > tests/pytest/acpi-bits/meson.build | 33 ++ > tests/pytest/acpi-bits/requirements.txt | 1 + > 4 files changed, 475 insertions(+) > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > create mode 100644 tests/pytest/acpi-bits/meson.build > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > new file mode 100644 > index 0000000000..186395473b > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > @@ -0,0 +1,59 @@ > +#!/usr/bin/env bash > +# Generates a python virtual environment for the test to run. > +# Then runs python test scripts from within that virtual environment. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Author: Ani Sinha <ani@anisinha.ca> > + > +set -e > + > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > +MYDIR=$(dirname $MYPATH) > + > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > + echo " to the root of the qemu source tree." > + echo -n "This is required so that the test can find the " > + echo "python modules that it needs for execution." > + exit 1 > +fi > +SRCDIR=$PYTEST_SOURCE_ROOT > +TESTSCRIPTS=("acpi-bits-test.py") > +PIPCMD="-m pip -q --disable-pip-version-check" > +# we need to save the old value of PWD before we do a change-dir later > +PYTEST_PWD=$PWD > + > +TESTS_PYTHON=/usr/bin/python3 > +TESTS_VENV_REQ=requirements.txt > + > +# sadly for pip -e and -t options do not work together. > +# please see https://github.com/pypa/pip/issues/562 > +cd $MYDIR > + > +$TESTS_PYTHON -m venv . > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > +[ -f $TESTS_VENV_REQ ] && \ > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > + > +# venv is activated at this point. > + > +# run the test > +for testscript in ${TESTSCRIPTS[@]} ; do > + export PYTEST_PWD; python3 $testscript > +done > + > +cd $PYTEST_PWD > + > +exit 0 > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py > new file mode 100644 > index 0000000000..97e61eb709 > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > @@ -0,0 +1,382 @@ > +#!/usr/bin/env python3 > +# group: rw quick > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > +# https://biosbits.org/ > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Some parts are slightly taken from qtest.py and iotests.py > +# > +# Authors: > +# Ani Sinha <ani@anisinha.ca> > + > +# pylint: disable=invalid-name > + > +""" > +QEMU bios tests using biosbits available at > +https://biosbits.org/. > +""" > + > +import logging > +import os > +import re > +import shutil > +import subprocess > +import sys > +import tarfile > +import tempfile > +import time > +import unittest > +from urllib import request > +import zipfile > +from typing import ( > + List, > + Optional, > + Sequence, > +) > +from tap import TAPTestRunner > +from qemu.machine import QEMUMachine > + > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > +PYTEST_PWD = os.getenv('PYTEST_PWD') > + > +def get_arch(): > + """finds the arch from the qemu binary name""" > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > + if match: > + return match.group(1) > + return 'x86_64' > + > +ARCH = get_arch() > + > +class QEMUBitsMachine(QEMUMachine): > + """ > + A QEMU VM, with isa-debugcon enabled and bits iso passed > + using -cdrom to QEMU commandline. > + """ > + def __init__(self, > + binary: str, > + args: Sequence[str] = (), > + wrapper: Sequence[str] = (), > + name: Optional[str] = None, > + base_temp_dir: str = "/var/tmp", > + debugcon_log: str = "debugcon-log.txt", > + debugcon_addr: str = "0x403", > + sock_dir: Optional[str] = None, > + qmp_timer: Optional[float] = None): > + # pylint: disable=too-many-arguments > + > + if name is None: > + name = "qemu-bits-%d" % os.getpid() > + if sock_dir is None: > + sock_dir = base_temp_dir > + super().__init__(binary, args, wrapper=wrapper, name=name, > + base_temp_dir=base_temp_dir, > + sock_dir=sock_dir, qmp_timer=qmp_timer) > + self.debugcon_log = debugcon_log > + self.debugcon_addr = debugcon_addr > + self.base_temp_dir = base_temp_dir > + > + @property > + def _base_args(self) -> List[str]: > + args = super()._base_args > + args.extend([ > + '-chardev', > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > + self.debugcon_log), > + '-device', > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > + ]) > + return args > + > + def base_args(self): > + """return the base argument to QEMU binary""" > + return self._base_args > + > +class AcpiBitsTest(unittest.TestCase): > + """ACPI and SMBIOS tests using biosbits.""" > + def __init__(self, *args, **kwargs): > + super().__init__(*args, **kwargs) > + self._vm = None > + self._workDir = None > + self._bitsVer = 2100 > + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" > + self._debugcon_addr = '0x403' > + self._debugcon_log = 'debugcon-log.txt' > + logging.basicConfig(level=logging.INFO) > + > + def copy_bits_config(self): > + """ copies the bios bits config file into bits. > + """ > + config_file = 'bits-cfg.txt' > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > + target_config_dir = os.path.join(self._workDir, > + 'bits-%d' %self._bitsVer, 'boot') > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > + self.assertTrue(os.path.exists(target_config_dir)) > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > + config_file), os.R_OK)) > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > + target_config_dir) > + logging.info('copied config file %s to %s', > + config_file, target_config_dir) > + > + def copy_test_scripts(self): > + """copies the python test scripts into bits. """ > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, > + 'boot', 'python') > + > + self.assertTrue(os.path.exists(qemu_test_dir)) > + self.assertTrue(os.path.exists(target_test_dir)) > + > + for filename in os.listdir(qemu_test_dir): > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > + filename.endswith('.py'): > + shutil.copy2(os.path.join(qemu_test_dir, filename), > + target_test_dir) > + logging.info('copied test file %s to %s', > + filename, target_test_dir) > + > + # now remove the pyc test file if it exists, otherwise the > + # changes in the python test script won't be executed. > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > + if os.access(os.path.join(target_test_dir, testfile_pyc), > + os.F_OK): > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > + logging.info('removed compiled file %s', > + os.path.join(target_test_dir, testfile_pyc)) > + > + def fix_mkrescue(self, mkrescue): > + """ grub-mkrescue is a bash script with two variables, 'prefix' and > + 'libdir'. They must be pointed to the right location so that the > + iso can be generated appropriately. We point the two variables to > + the directory where we have extracted our pre-built bits grub > + tarball. > + """ > + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > + > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > + self.assertTrue(os.path.exists(grub_i386_mods)) > + > + new_script = "" > + with open(mkrescue, 'r') as filehandle: > + orig_script = filehandle.read() > + new_script = re.sub('(^prefix=)(.*)', > + r'\1"%s"' %grub_x86_64_mods, > + orig_script, flags=re.M) > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, > + new_script, flags=re.M) > + > + with open(mkrescue, 'w') as filehandle: > + filehandle.write(new_script) > + > + def generate_bits_iso(self): > + """ Uses grub-mkrescue to generate a fresh bits iso with the python > + test scripts > + """ > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > + mkrescue_script = os.path.join(self._workDir, > + 'grub-inst-x86_64-efi', 'bin', > + 'grub-mkrescue') > + > + self.assertTrue(os.access(mkrescue_script, > + os.R_OK | os.W_OK | os.X_OK)) > + > + self.fix_mkrescue(mkrescue_script) > + > + logging.info('calling grub-mkrescue to generate the biosbits iso ...') > + > + try: > + if os.getenv('V'): > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stdout=subprocess.DEVNULL) > + else: > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stderr=subprocess.DEVNULL, > + stdout=subprocess.DEVNULL) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("Error while generating the bits iso. " > + "Pass V=1 in the environment to get more details. " > + + str(e)) > + > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + logging.info('iso file %s successfully generated.', iso_file) > + > + def setUp(self): > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > + if BITS_LOC: > + prefix = BITS_LOC > + else: > + prefix = os.path.join(os.getcwd(), 'prebuilt') > + if not os.path.isdir(prefix): > + os.mkdir(prefix, mode=0o775) > + > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > + %self._bitsVer) > + grub_tar_file = os.path.join(prefix, > + 'bits-%d-grub.tar.gz' %self._bitsVer) > + # if the location of the bits binaries has been specified by the user > + # and they are not found in that location, skip the test. > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location %s." \ > + %BITS_LOC) > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location %s." \ > + %BITS_LOC) > + > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > + suffix='.tmp') > + logging.info('working dir: %s', self._workDir) > + > + localArchive = "bits-%d.zip" % self._bitsVer > + if not os.access(bits_zip_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) Poking at URLs from python is really gross IMHO. Instead of all this mess, can't we just spawn e.g. "git clone --depth 1"? And if the directory exists I would fetch and checkout. Fundamentally people hate submodules so we are recreating them ok, but at least we can use git and not open-code http and file management ... > + > + localArchive = "bits-%d-grub.tar.gz" % self._bitsVer > + if not os.access(grub_tar_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) > + > + # extract the bits software in the temp working directory > + with zipfile.ZipFile(bits_zip_file, 'r') as zref: > + zref.extractall(self._workDir) > + > + with tarfile.open(grub_tar_file, 'r') as tarball: > + tarball.extractall(self._workDir) > + > + self.copy_test_scripts() > + self.copy_bits_config() > + self.generate_bits_iso() > + > + def parse_log(self): > + """parse the log generated by running bits tests and > + check for failures. > + """ > + debugconf = os.path.join(self._workDir, self._debugcon_log) > + log = "" > + with open(debugconf, 'r') as filehandle: > + log = filehandle.read() > + > + if os.getenv('V'): > + print('\nlogs from biosbits follows:') > + print('==========================================\n') > + print(log) > + print('==========================================\n') > + > + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*', > + log) > + for match in matchiter: > + # verify that no test cases failed. > + self.assertEqual(match.group(3).split()[0], '0', > + 'Some bits tests seems to have failed. ' \ > + 'Set V=1 in the environment to get the entire ' \ > + 'log from bits.') > + > + def tearDown(self): > + if self._vm: > + self.assertFalse(not self._vm.is_running) > + logging.info('removing the work directory %s', self._workDir) > + shutil.rmtree(self._workDir) > + > + def test_acpi_smbios_bits(self): > + """The main test case implementaion.""" > + > + qemu_bin = PYTESTQEMUBIN > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > + > + # PYTESTQEMUBIN could be relative to the current directory > + if not os.access(PYTESTQEMUBIN, os.X_OK) and PYTEST_PWD: > + qemu_bin = os.path.join(PYTEST_PWD, PYTESTQEMUBIN) > + > + logging.info('QEMU binary used: %s', qemu_bin) > + > + self.assertTrue(os.access(qemu_bin, os.X_OK)) > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + self._vm = QEMUBitsMachine(binary=qemu_bin, > + base_temp_dir=self._workDir, > + debugcon_log=self._debugcon_log, > + debugcon_addr=self._debugcon_addr) > + > + self._vm.add_args('-cdrom', '%s' %iso_file) > + > + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ > + " " + " ".join(str(arg) for arg in self._vm.args) > + > + logging.info("launching QEMU vm with the following arguments: %s", > + args) > + > + self._vm.launch() > + # biosbits has been configured to run all the specified test suites > + # in batch mode and then automatically initiate a vm shutdown. > + # sleep for maximum of one minute > + max_sleep_time = time.monotonic() + 60 > + while self._vm.is_running() and time.monotonic() < max_sleep_time: > + time.sleep(1) > + > + self.assertFalse(time.monotonic() > max_sleep_time, > + 'The VM seems to have failed to shutdown in time') > + > + self.parse_log() > + > +def execute_unittest(argv: List[str], debug: bool = False, > + runner: TAPTestRunner = None) -> None: > + """Executes unittests within the calling module.""" > + > + unittest.main(argv=argv, > + testRunner=runner, > + verbosity=2 if debug else 1, > + warnings=None if sys.warnoptions else 'ignore') > + > +def main(): > + """ The main function where execution begins. """ > + > + assert PYTESTQEMUBIN is not None, \ > + "Environment variable PYTEST_QEMU_BINARY required." > + > + runner = TAPTestRunner() > + runner.set_stream(True) > + runner.set_format("%s/acpi-bits-test" %ARCH) > + execute_unittest(sys.argv, False, runner) > + > +main() > diff --git a/tests/pytest/acpi-bits/meson.build b/tests/pytest/acpi-bits/meson.build > new file mode 100644 > index 0000000000..099c191d57 > --- /dev/null > +++ b/tests/pytest/acpi-bits/meson.build > @@ -0,0 +1,33 @@ > +xorriso = find_program('xorriso', required: true) > +if not xorriso.found() > + message('xorriso not found ... disabled bits acpi tests.') > + subdir_done() > +endif > + > +subdir('bits-tests') > +subdir('bits-config') > + > +test_files = ['acpi-bits-test.py'] > +requirements = 'requirements.txt' > + > +copytestfiles = custom_target('copy test files', > + input : test_files, > + output : test_files, > + command : ['cp', '@INPUT@', '@OUTPUT@'], > + install : false, > + build_by_default : true) > + > +requirementsfiles = custom_target('copy py req files', > + input : requirements, > + output : requirements, > + command : ['cp', '@INPUT@', '@OUTPUT@'], > + install : false, > + build_by_default : true) > + > +other_deps += [copytestfiles,requirementsfiles] > + > +pytest_executables += { > + 'acpi-bits-test': configure_file(copy:true, > + input:'acpi-bits-test-venv.sh', > + output:'acpi-bits-test') > +} > diff --git a/tests/pytest/acpi-bits/requirements.txt b/tests/pytest/acpi-bits/requirements.txt > new file mode 100644 > index 0000000000..00cdad09ef > --- /dev/null > +++ b/tests/pytest/acpi-bits/requirements.txt > @@ -0,0 +1 @@ > +tap.py > -- > 2.25.1
On Thu, Jul 14, 2022 at 07:49:36PM +0530, Ani Sinha wrote: > > so you skip downlaod if it already exists locally. IIUC it is looking > > in the CWD, which is presumably the directory the QEMU build is > > performed in ? > > Yes, build/test/pytest/bits-test > > > So if dev cleans their build tree, the cache is lost ? > > Yes. > > > > > Avocado has a more persistent cache outside the build tree IIUC. > > > > So that is why I have the environment variable mechanism for passing to > the test that will point to the location of the archives. The developer > can download the files there and point to the test. Or I can change the > scriprt accordingly if we know where we were downloading. We were > discussing submodules and it was fiercely hated. So mst suggested another > repo where to keep the binaries. My script that check out that repo > somewhere outside the build directory and point the test to that location. > > I have kept several options open. We just need to make some decisions. > For now I would probably just have a script to fetch into source tree.
On Thu, 14 Jul 2022, Michael S. Tsirkin wrote: > On Sun, Jul 10, 2022 at 10:30:10PM +0530, Ani Sinha wrote: > > This change adds python based test environment that can be used to run pytest > > from within a virtual environment. A bash script sets up a virtual environment > > and then runs the python based tests from within that environment. > > All dependent python packages are installed in the virtual environment using > > pip python module. QEMU python test modules are also available in the environment > > for spawning the QEMU based VMs. > > > > It also introduces QEMU acpi/smbios biosbits python test script which is run > > from within the python virtual environment. When the bios bits tests are run, > > bios bits binaries are downloaded from an external repo/location. > > Currently, the test points to an external private github repo where the bits > > archives are checked in. > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > > tests/pytest/acpi-bits/meson.build | 33 ++ > > tests/pytest/acpi-bits/requirements.txt | 1 + > > 4 files changed, 475 insertions(+) > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > > create mode 100644 tests/pytest/acpi-bits/meson.build > > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > new file mode 100644 > > index 0000000000..186395473b > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > > @@ -0,0 +1,59 @@ > > +#!/usr/bin/env bash > > +# Generates a python virtual environment for the test to run. > > +# Then runs python test scripts from within that virtual environment. > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Author: Ani Sinha <ani@anisinha.ca> > > + > > +set -e > > + > > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > > +MYDIR=$(dirname $MYPATH) > > + > > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > > + echo " to the root of the qemu source tree." > > + echo -n "This is required so that the test can find the " > > + echo "python modules that it needs for execution." > > + exit 1 > > +fi > > +SRCDIR=$PYTEST_SOURCE_ROOT > > +TESTSCRIPTS=("acpi-bits-test.py") > > +PIPCMD="-m pip -q --disable-pip-version-check" > > +# we need to save the old value of PWD before we do a change-dir later > > +PYTEST_PWD=$PWD > > + > > +TESTS_PYTHON=/usr/bin/python3 > > +TESTS_VENV_REQ=requirements.txt > > + > > +# sadly for pip -e and -t options do not work together. > > +# please see https://github.com/pypa/pip/issues/562 > > +cd $MYDIR > > + > > +$TESTS_PYTHON -m venv . > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > > +[ -f $TESTS_VENV_REQ ] && \ > > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > > + > > +# venv is activated at this point. > > + > > +# run the test > > +for testscript in ${TESTSCRIPTS[@]} ; do > > + export PYTEST_PWD; python3 $testscript > > +done > > + > > +cd $PYTEST_PWD > > + > > +exit 0 > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py > > new file mode 100644 > > index 0000000000..97e61eb709 > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > > @@ -0,0 +1,382 @@ > > +#!/usr/bin/env python3 > > +# group: rw quick > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > > +# https://biosbits.org/ > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > > +# > > +# Some parts are slightly taken from qtest.py and iotests.py > > +# > > +# Authors: > > +# Ani Sinha <ani@anisinha.ca> > > + > > +# pylint: disable=invalid-name > > + > > +""" > > +QEMU bios tests using biosbits available at > > +https://biosbits.org/. > > +""" > > + > > +import logging > > +import os > > +import re > > +import shutil > > +import subprocess > > +import sys > > +import tarfile > > +import tempfile > > +import time > > +import unittest > > +from urllib import request > > +import zipfile > > +from typing import ( > > + List, > > + Optional, > > + Sequence, > > +) > > +from tap import TAPTestRunner > > +from qemu.machine import QEMUMachine > > + > > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > > +PYTEST_PWD = os.getenv('PYTEST_PWD') > > + > > +def get_arch(): > > + """finds the arch from the qemu binary name""" > > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > > + if match: > > + return match.group(1) > > + return 'x86_64' > > + > > +ARCH = get_arch() > > + > > +class QEMUBitsMachine(QEMUMachine): > > + """ > > + A QEMU VM, with isa-debugcon enabled and bits iso passed > > + using -cdrom to QEMU commandline. > > + """ > > + def __init__(self, > > + binary: str, > > + args: Sequence[str] = (), > > + wrapper: Sequence[str] = (), > > + name: Optional[str] = None, > > + base_temp_dir: str = "/var/tmp", > > + debugcon_log: str = "debugcon-log.txt", > > + debugcon_addr: str = "0x403", > > + sock_dir: Optional[str] = None, > > + qmp_timer: Optional[float] = None): > > + # pylint: disable=too-many-arguments > > + > > + if name is None: > > + name = "qemu-bits-%d" % os.getpid() > > + if sock_dir is None: > > + sock_dir = base_temp_dir > > + super().__init__(binary, args, wrapper=wrapper, name=name, > > + base_temp_dir=base_temp_dir, > > + sock_dir=sock_dir, qmp_timer=qmp_timer) > > + self.debugcon_log = debugcon_log > > + self.debugcon_addr = debugcon_addr > > + self.base_temp_dir = base_temp_dir > > + > > + @property > > + def _base_args(self) -> List[str]: > > + args = super()._base_args > > + args.extend([ > > + '-chardev', > > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > > + self.debugcon_log), > > + '-device', > > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > > + ]) > > + return args > > + > > + def base_args(self): > > + """return the base argument to QEMU binary""" > > + return self._base_args > > + > > +class AcpiBitsTest(unittest.TestCase): > > + """ACPI and SMBIOS tests using biosbits.""" > > + def __init__(self, *args, **kwargs): > > + super().__init__(*args, **kwargs) > > + self._vm = None > > + self._workDir = None > > + self._bitsVer = 2100 > > + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" > > + self._debugcon_addr = '0x403' > > + self._debugcon_log = 'debugcon-log.txt' > > + logging.basicConfig(level=logging.INFO) > > + > > + def copy_bits_config(self): > > + """ copies the bios bits config file into bits. > > + """ > > + config_file = 'bits-cfg.txt' > > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > > + target_config_dir = os.path.join(self._workDir, > > + 'bits-%d' %self._bitsVer, 'boot') > > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > > + self.assertTrue(os.path.exists(target_config_dir)) > > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > > + config_file), os.R_OK)) > > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > > + target_config_dir) > > + logging.info('copied config file %s to %s', > > + config_file, target_config_dir) > > + > > + def copy_test_scripts(self): > > + """copies the python test scripts into bits. """ > > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > > + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, > > + 'boot', 'python') > > + > > + self.assertTrue(os.path.exists(qemu_test_dir)) > > + self.assertTrue(os.path.exists(target_test_dir)) > > + > > + for filename in os.listdir(qemu_test_dir): > > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > > + filename.endswith('.py'): > > + shutil.copy2(os.path.join(qemu_test_dir, filename), > > + target_test_dir) > > + logging.info('copied test file %s to %s', > > + filename, target_test_dir) > > + > > + # now remove the pyc test file if it exists, otherwise the > > + # changes in the python test script won't be executed. > > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > > + if os.access(os.path.join(target_test_dir, testfile_pyc), > > + os.F_OK): > > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > > + logging.info('removed compiled file %s', > > + os.path.join(target_test_dir, testfile_pyc)) > > + > > + def fix_mkrescue(self, mkrescue): > > + """ grub-mkrescue is a bash script with two variables, 'prefix' and > > + 'libdir'. They must be pointed to the right location so that the > > + iso can be generated appropriately. We point the two variables to > > + the directory where we have extracted our pre-built bits grub > > + tarball. > > + """ > > + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') > > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > > + > > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > > + self.assertTrue(os.path.exists(grub_i386_mods)) > > + > > + new_script = "" > > + with open(mkrescue, 'r') as filehandle: > > + orig_script = filehandle.read() > > + new_script = re.sub('(^prefix=)(.*)', > > + r'\1"%s"' %grub_x86_64_mods, > > + orig_script, flags=re.M) > > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, > > + new_script, flags=re.M) > > + > > + with open(mkrescue, 'w') as filehandle: > > + filehandle.write(new_script) > > + > > + def generate_bits_iso(self): > > + """ Uses grub-mkrescue to generate a fresh bits iso with the python > > + test scripts > > + """ > > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > > + mkrescue_script = os.path.join(self._workDir, > > + 'grub-inst-x86_64-efi', 'bin', > > + 'grub-mkrescue') > > + > > + self.assertTrue(os.access(mkrescue_script, > > + os.R_OK | os.W_OK | os.X_OK)) > > + > > + self.fix_mkrescue(mkrescue_script) > > + > > + logging.info('calling grub-mkrescue to generate the biosbits iso ...') > > + > > + try: > > + if os.getenv('V'): > > + subprocess.check_call([mkrescue_script, '-o', > > + iso_file, bits_dir], > > + stdout=subprocess.DEVNULL) > > + else: > > + subprocess.check_call([mkrescue_script, '-o', > > + iso_file, bits_dir], > > + stderr=subprocess.DEVNULL, > > + stdout=subprocess.DEVNULL) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("Error while generating the bits iso. " > > + "Pass V=1 in the environment to get more details. " > > + + str(e)) > > + > > + self.assertTrue(os.access(iso_file, os.R_OK)) > > + > > + logging.info('iso file %s successfully generated.', iso_file) > > + > > + def setUp(self): > > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > > + if BITS_LOC: > > + prefix = BITS_LOC > > + else: > > + prefix = os.path.join(os.getcwd(), 'prebuilt') > > + if not os.path.isdir(prefix): > > + os.mkdir(prefix, mode=0o775) > > + > > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > > + %self._bitsVer) > > + grub_tar_file = os.path.join(prefix, > > + 'bits-%d-grub.tar.gz' %self._bitsVer) > > + # if the location of the bits binaries has been specified by the user > > + # and they are not found in that location, skip the test. > > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be found in the specified location %s." \ > > + %BITS_LOC) > > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be found in the specified location %s." \ > > + %BITS_LOC) > > + > > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > > + suffix='.tmp') > > + logging.info('working dir: %s', self._workDir) > > + > > + localArchive = "bits-%d.zip" % self._bitsVer > > + if not os.access(bits_zip_file, os.F_OK): > > + logging.info("archive %s not found in %s, downloading ...", > > + localArchive, bits_zip_file) > > + try: > > + req = request.urlopen(self._bitsLoc + localArchive) > > + with open(os.path.join(prefix, localArchive), > > + 'wb') as archivef: > > + archivef.write(req.read()) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be obtained." + str(e)) > > + else: > > + logging.info('using locally found %s', localArchive) > > Poking at URLs from python is really gross IMHO. No its not gross :) Avocado framework does the same but in a more complicarted way. > Instead of all this mess, can't we just spawn e.g. "git clone --depth 1"? > And if the directory exists I would fetch and checkout. There are two reasons I can think of why I do not like this idea: (a) a git clone of a whole directory would download all versions of the binary whereas we want only a specific version. Downloading a single file by shallow cloning or creating a git archive is overkill IMHO when a wget style retrieval works just fine. (b) we may later move the binary archives to a ftp server or a google drive. git/version control mechanisms are not the best place to store binary blobs IMHO. In this case also, wget also works. > > > Fundamentally people hate submodules so we are recreating > them ok, but at least we can use git and not open-code http > and file management ... > > > > > + > > + localArchive = "bits-%d-grub.tar.gz" % self._bitsVer > > + if not os.access(grub_tar_file, os.F_OK): > > + logging.info("archive %s not found in %s, downloading ...", > > + localArchive, bits_zip_file) > > + try: > > + req = request.urlopen(self._bitsLoc + localArchive) > > + with open(os.path.join(prefix, localArchive), > > + 'wb') as archivef: > > + archivef.write(req.read()) > > + except Exception as e: # pylint: disable=broad-except > > + self.skipTest("test skipped since biosbits binaries " + > > + "could not be obtained." + str(e)) > > + else: > > + logging.info('using locally found %s', localArchive) > > + > > + # extract the bits software in the temp working directory > > + with zipfile.ZipFile(bits_zip_file, 'r') as zref: > > + zref.extractall(self._workDir) > > + > > + with tarfile.open(grub_tar_file, 'r') as tarball: > > + tarball.extractall(self._workDir) > > + > > + self.copy_test_scripts() > > + self.copy_bits_config() > > + self.generate_bits_iso() > > + > > + def parse_log(self): > > + """parse the log generated by running bits tests and > > + check for failures. > > + """ > > + debugconf = os.path.join(self._workDir, self._debugcon_log) > > + log = "" > > + with open(debugconf, 'r') as filehandle: > > + log = filehandle.read() > > + > > + if os.getenv('V'): > > + print('\nlogs from biosbits follows:') > > + print('==========================================\n') > > + print(log) > > + print('==========================================\n') > > + > > + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*', > > + log) > > + for match in matchiter: > > + # verify that no test cases failed. > > + self.assertEqual(match.group(3).split()[0], '0', > > + 'Some bits tests seems to have failed. ' \ > > + 'Set V=1 in the environment to get the entire ' \ > > + 'log from bits.') > > + > > + def tearDown(self): > > + if self._vm: > > + self.assertFalse(not self._vm.is_running) > > + logging.info('removing the work directory %s', self._workDir) > > + shutil.rmtree(self._workDir) > > + > > + def test_acpi_smbios_bits(self): > > + """The main test case implementaion.""" > > + > > + qemu_bin = PYTESTQEMUBIN > > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) > > + > > + # PYTESTQEMUBIN could be relative to the current directory > > + if not os.access(PYTESTQEMUBIN, os.X_OK) and PYTEST_PWD: > > + qemu_bin = os.path.join(PYTEST_PWD, PYTESTQEMUBIN) > > + > > + logging.info('QEMU binary used: %s', qemu_bin) > > + > > + self.assertTrue(os.access(qemu_bin, os.X_OK)) > > + self.assertTrue(os.access(iso_file, os.R_OK)) > > + > > + self._vm = QEMUBitsMachine(binary=qemu_bin, > > + base_temp_dir=self._workDir, > > + debugcon_log=self._debugcon_log, > > + debugcon_addr=self._debugcon_addr) > > + > > + self._vm.add_args('-cdrom', '%s' %iso_file) > > + > > + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ > > + " " + " ".join(str(arg) for arg in self._vm.args) > > + > > + logging.info("launching QEMU vm with the following arguments: %s", > > + args) > > + > > + self._vm.launch() > > + # biosbits has been configured to run all the specified test suites > > + # in batch mode and then automatically initiate a vm shutdown. > > + # sleep for maximum of one minute > > + max_sleep_time = time.monotonic() + 60 > > + while self._vm.is_running() and time.monotonic() < max_sleep_time: > > + time.sleep(1) > > + > > + self.assertFalse(time.monotonic() > max_sleep_time, > > + 'The VM seems to have failed to shutdown in time') > > + > > + self.parse_log() > > + > > +def execute_unittest(argv: List[str], debug: bool = False, > > + runner: TAPTestRunner = None) -> None: > > + """Executes unittests within the calling module.""" > > + > > + unittest.main(argv=argv, > > + testRunner=runner, > > + verbosity=2 if debug else 1, > > + warnings=None if sys.warnoptions else 'ignore') > > + > > +def main(): > > + """ The main function where execution begins. """ > > + > > + assert PYTESTQEMUBIN is not None, \ > > + "Environment variable PYTEST_QEMU_BINARY required." > > + > > + runner = TAPTestRunner() > > + runner.set_stream(True) > > + runner.set_format("%s/acpi-bits-test" %ARCH) > > + execute_unittest(sys.argv, False, runner) > > + > > +main() > > diff --git a/tests/pytest/acpi-bits/meson.build b/tests/pytest/acpi-bits/meson.build > > new file mode 100644 > > index 0000000000..099c191d57 > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/meson.build > > @@ -0,0 +1,33 @@ > > +xorriso = find_program('xorriso', required: true) > > +if not xorriso.found() > > + message('xorriso not found ... disabled bits acpi tests.') > > + subdir_done() > > +endif > > + > > +subdir('bits-tests') > > +subdir('bits-config') > > + > > +test_files = ['acpi-bits-test.py'] > > +requirements = 'requirements.txt' > > + > > +copytestfiles = custom_target('copy test files', > > + input : test_files, > > + output : test_files, > > + command : ['cp', '@INPUT@', '@OUTPUT@'], > > + install : false, > > + build_by_default : true) > > + > > +requirementsfiles = custom_target('copy py req files', > > + input : requirements, > > + output : requirements, > > + command : ['cp', '@INPUT@', '@OUTPUT@'], > > + install : false, > > + build_by_default : true) > > + > > +other_deps += [copytestfiles,requirementsfiles] > > + > > +pytest_executables += { > > + 'acpi-bits-test': configure_file(copy:true, > > + input:'acpi-bits-test-venv.sh', > > + output:'acpi-bits-test') > > +} > > diff --git a/tests/pytest/acpi-bits/requirements.txt b/tests/pytest/acpi-bits/requirements.txt > > new file mode 100644 > > index 0000000000..00cdad09ef > > --- /dev/null > > +++ b/tests/pytest/acpi-bits/requirements.txt > > @@ -0,0 +1 @@ > > +tap.py > > -- > > 2.25.1 > >
On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > Instead of all this mess, can't we just spawn e.g. "git clone --depth 1"? > > And if the directory exists I would fetch and checkout. > > There are two reasons I can think of why I do not like this idea: > > (a) a git clone of a whole directory would download all versions of the > binary whereas we want only a specific version. You mention shallow clone yourself, and I used --depth 1 above. > Downloading a single file > by shallow cloning or creating a git archive is overkill IMHO when a wget > style retrieval works just fine. However, it does not provide for versioning, tagging etc so you have to implement your own schema. > (b) we may later move the binary archives to a ftp server or a google > drive. git/version control mechanisms are not the best place to store > binary blobs IMHO. In this case also, wget also works. surely neither ftp nor google drive are reasonable dependencies for a free software project. But qemu does maintain an http server already so that't a plus. I am not insisting on git, but I do not like it that security, mirroring, caching, versioning all have to be hand rolled and then figured out by users and maintainers. Who frankly have other things to do besides learning yet another boutique configuration language. And I worry that after a while we come up with a new organization schema for the files, old ones are moved around and nothing relying on the URL works. git is kind of good in that it enforces the idea that history is immutable. If not vanilla git can we find another utility we can reuse? git lfs? It seems to be supported by both github and gitlab though bizarrely github has bandwidth limits on git lfs but apparently not on vanilla git. Hosting on qemu.org will require maintaining a server there though. All that said maybe we should just run with it as it is, just so we get *something* in the door, and then worry about getting the storage side straight before making this test a requirement for all acpi developers.
On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > 1"? > > > And if the directory exists I would fetch and checkout. > > > > There are two reasons I can think of why I do not like this idea: > > > > (a) a git clone of a whole directory would download all versions of the > > binary whereas we want only a specific version. > > You mention shallow clone yourself, and I used --depth 1 above. > > > Downloading a single file > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > style retrieval works just fine. > > However, it does not provide for versioning, tagging etc so you have > to implement your own schema. Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and I think all we need to do is maintain the same versioning logic and maintain binaries of different versions. Do we really need the power of git/version control here? Dunno. > > > > (b) we may later move the binary archives to a ftp server or a google > > drive. git/version control mechanisms are not the best place to store > > binary blobs IMHO. In this case also, wget also works. > > surely neither ftp nor google drive are reasonable dependencies > for a free software project. But qemu does maintain an http server > already so that't a plus. > > > > I am not insisting on git, but I do not like it that security, > mirroring, caching, versioning all have to be hand rolled and then > figured out by users and maintainers. Who frankly have other things to > do besides learning yet another boutique configuration language. Yeah we do not want to reinvent the wheel all over again. > > And I worry that after a while we come up with a new organization schema > for the files, old ones are moved around and nothing relying on the URL > works. git is kind of good in that it enforces the idea that history is > immutable. Ah I see your point here. > > If not vanilla git can we find another utility we can reuse? > > git lfs? It seems to be supported by both github and gitlab though > bizarrely github has bandwidth limits on git lfs but apparently not on > vanilla git. Hosting on qemu.org will require maintaining a server > there though. > > > > All that said maybe we should just run with it as it is, just so we get > *something* in the door, and then worry about getting the storage side > straight before making this test a requirement for all acpi developers. >
On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > 1"? > > > And if the directory exists I would fetch and checkout. > > > > There are two reasons I can think of why I do not like this idea: > > > > (a) a git clone of a whole directory would download all versions of the > > binary whereas we want only a specific version. > > You mention shallow clone yourself, and I used --depth 1 above. > > > Downloading a single file > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > style retrieval works just fine. > > However, it does not provide for versioning, tagging etc so you have > to implement your own schema. > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > I think all we need to do is maintain the same versioning logic and maintain > binaries of different  versions. Do we really need the power of git/version > control here? Dunno. Well we need some schema. Given we are not using official bits releases I don't think we can reuse theirs. > > > > > > (b) we may later move the binary archives to a ftp server or a google > > drive. git/version control mechanisms are not the best place to store > > binary blobs IMHO. In this case also, wget also works. > > surely neither ftp nor google drive are reasonable dependencies > for a free software project. But qemu does maintain an http server > already so that't a plus. > > > > I am not insisting on git, but I do not like it that security, > mirroring, caching, versioning all have to be hand rolled and then > figured out by users and maintainers. Who frankly have other things to > do besides learning yet another boutique configuration language. > > > Yeah we do not want to reinvent the wheel all over again. > > > > > And I worry that after a while we come up with a new organization schema > for the files, old ones are moved around and nothing relying on the URL > works. git is kind of good in that it enforces the idea that history is > immutable. > > > Ah I see your point here. > > > > > If not vanilla git can we find another utility we can reuse? > > git lfs? It seems to be supported by both github and gitlab though > bizarrely github has bandwidth limits on git lfs but apparently not on > vanilla git. Hosting on qemu.org will require maintaining a server > there though. > > > > All that said maybe we should just run with it as it is, just so we get > *something* in the door, and then worry about getting the storage side > straight before making this test a requirement for all acpi developers. > > > > >
On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > 1"? > > > > And if the directory exists I would fetch and checkout. > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > (a) a git clone of a whole directory would download all versions of the > > > binary whereas we want only a specific version. > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > Downloading a single file > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > style retrieval works just fine. > > > > However, it does not provide for versioning, tagging etc so you have > > to implement your own schema. > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > I think all we need to do is maintain the same versioning logic and maintain > > binaries of different  versions. Do we really need the power of git/version > > control here? Dunno. > > Well we need some schema. Given we are not using official bits releases > I don't think we can reuse theirs. OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and the binaries in some other repo first. Everything else hinges on that. We can fix the rest of the bits later incrementally. > > > > > > > > > > > > (b) we may later move the binary archives to a ftp server or a google > > > drive. git/version control mechanisms are not the best place to store > > > binary blobs IMHO. In this case also, wget also works. > > > > surely neither ftp nor google drive are reasonable dependencies > > for a free software project. But qemu does maintain an http server > > already so that't a plus. > > > > > > > > I am not insisting on git, but I do not like it that security, > > mirroring, caching, versioning all have to be hand rolled and then > > figured out by users and maintainers. Who frankly have other things to > > do besides learning yet another boutique configuration language. > > > > > > Yeah we do not want to reinvent the wheel all over again. > > > > > > > > > > And I worry that after a while we come up with a new organization schema > > for the files, old ones are moved around and nothing relying on the URL > > works. git is kind of good in that it enforces the idea that history is > > immutable. > > > > > > Ah I see your point here. > > > > > > > > > > If not vanilla git can we find another utility we can reuse? > > > > git lfs? It seems to be supported by both github and gitlab though > > bizarrely github has bandwidth limits on git lfs but apparently not on > > vanilla git. Hosting on qemu.org will require maintaining a server > > there though. > > > > > > > > All that said maybe we should just run with it as it is, just so we get > > *something* in the door, and then worry about getting the storage side > > straight before making this test a requirement for all acpi developers. > > > > > > > > > > > >
On Mon, 25 Jul 2022, Ani Sinha wrote: > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > 1"? > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > binary whereas we want only a specific version. > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > Downloading a single file > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > style retrieval works just fine. > > > > > > However, it does not provide for versioning, tagging etc so you have > > > to implement your own schema. > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > I think all we need to do is maintain the same versioning logic and maintain > > > binaries of different  versions. Do we really need the power of git/version > > > control here? Dunno. > > > > Well we need some schema. Given we are not using official bits releases > > I don't think we can reuse theirs. > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > the binaries in some other repo first. Everything else hinges on that. We > can fix the rest of the bits later incrementally. DanPB, any thoughts on putting bits on git.qemu.org or where and how to keep the binaries?
On Tue, Jul 12, 2022 at 12:45:59PM +0530, Ani Sinha wrote: > > > On Mon, 11 Jul 2022, John Snow wrote: > > > On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > This change adds python based test environment that can be used to run pytest > > > from within a virtual environment. A bash script sets up a virtual environment > > > and then runs the python based tests from within that environment. > > > All dependent python packages are installed in the virtual environment using > > > pip python module. QEMU python test modules are also available in the environment > > > for spawning the QEMU based VMs. > > > > > > It also introduces QEMU acpi/smbios biosbits python test script which is run > > > from within the python virtual environment. When the bios bits tests are run, > > > bios bits binaries are downloaded from an external repo/location. > > > Currently, the test points to an external private github repo where the bits > > > archives are checked in. > > > > > > > Oh, I see -- requirements are handled here in this patch. > > > > Is this test designed to run the host/build system? I'm a little > > confused about the environment here. > > yes it is supposed to run on the build system where "make check" is > executed. > > > > > Is this test going to be run "by default" or will users have to opt > > into running it using a special invocation? > > run by default through "make check" or "make check-pytest" or "make > check-pytest-x86_64". I don't think that a default 'make check' should be downloading disk images, especially when they're stored in the local build dir by default and so lost when the build dir is cleaned. The downloading of disk images is one of the reasons why avocado is not set to run by default with 'make check', and I think this should be the same. This again takes me to back to believing that we should simply be using Avocado here instead of creating something that has to solve the same problems. With regards, Daniel
On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > 1"? > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > binary whereas we want only a specific version. > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > Downloading a single file > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > style retrieval works just fine. > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > to implement your own schema. > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > binaries of different versions. Do we really need the power of git/version > > > > control here? Dunno. > > > > > > Well we need some schema. Given we are not using official bits releases > > > I don't think we can reuse theirs. > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > the binaries in some other repo first. Everything else hinges on that. We > > can fix the rest of the bits later incrementally. > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > keep the binaries? Can we please conclude on this? Peter, can you please fork the repo? I have tried many times to reach you on IRC but failed.
On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > 1"? > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > binary whereas we want only a specific version. > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > Downloading a single file > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > style retrieval works just fine. > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > to implement your own schema. > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > binaries of different versions. Do we really need the power of git/version > > > > > control here? Dunno. > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > I don't think we can reuse theirs. > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > the binaries in some other repo first. Everything else hinges on that. We > > > can fix the rest of the bits later incrementally. > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > keep the binaries? > > Can we please conclude on this? > Peter, can you please fork the repo? I have tried many times to reach > you on IRC but failed. Probably because of travel around KVM forum. I think given our CI is under pressure again due to gitlab free tier limits, tying binaries to CI isn't a great idea at this stage. Can Ani just upload binaies to qemu.org for now?
On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > 1"? > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > Downloading a single file > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > style retrieval works just fine. > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > control here? Dunno. > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > I don't think we can reuse theirs. > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > can fix the rest of the bits later incrementally. > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > keep the binaries? > > > > Can we please conclude on this? > > Peter, can you please fork the repo? I have tried many times to reach > > you on IRC but failed. > > Probably because of travel around KVM forum. > > I think given our CI is under pressure again due to gitlab free tier > limits, tying binaries to CI isn't a great idea at this stage. > Can Ani just upload binaies to qemu.org for now? I agree with Michael here. Having a full ci/cd job for this is overkill IMHO. We should create a repo just for the binaries, have a README there to explain how we generate them and check in new versions as and when needed (it won't be frequent). How about biosbits-bin repo? > > -- > MST >
On Tue, Sep 27, 2022 at 01:43:15PM +0530, Ani Sinha wrote: > On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > > 1"? > > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > > > Downloading a single file > > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > > style retrieval works just fine. > > > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > > control here? Dunno. > > > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > > I don't think we can reuse theirs. > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > can fix the rest of the bits later incrementally. > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > keep the binaries? > > > > > > Can we please conclude on this? > > > Peter, can you please fork the repo? I have tried many times to reach > > > you on IRC but failed. > > > > Probably because of travel around KVM forum. > > > > I think given our CI is under pressure again due to gitlab free tier > > limits, tying binaries to CI isn't a great idea at this stage. > > Can Ani just upload binaies to qemu.org for now? > > I agree with Michael here. Having a full ci/cd job for this is > overkill IMHO. We should create a repo just for the binaries, have a > README there to explain how we generate them and check in new versions > as and when needed (it won't be frequent). > How about biosbits-bin repo? If QEMU is hosting binaries, where any part contains GPL code, then we need to be providing the full and corresponding source and the build scripts needed to re-create the binary. Once we have such scripts it should be trivial to trigger that from a CI job. If it isn't then we're doing something wrong. The CI quota is not an issue, because this is not a job that we need to run continuously. It can be triggered manually as & when we decide we need to refresh the binary, so would be a small one-off CI quota hit. Also note that gitlab is intending to start enforcing storage quota on projects in the not too distant future. This makes it unappealing to store binaries in git repos, unless we genuinely need the ability to access historical versions of the binary. I don't believe we need that for biosbits. The binary can be published as a CI artifact and accessed directly from the latest artifact download link. This ensures we only consume quota for the most recently published binary artifact. So I don't see a compelling reason to upload binaries into git. With regards, Daniel
> > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > keep the binaries? > > > > > > > > Can we please conclude on this? > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > you on IRC but failed. > > > > > > Probably because of travel around KVM forum. > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > limits, tying binaries to CI isn't a great idea at this stage. > > > Can Ani just upload binaies to qemu.org for now? > > > > I agree with Michael here. Having a full ci/cd job for this is > > overkill IMHO. We should create a repo just for the binaries, have a > > README there to explain how we generate them and check in new versions > > as and when needed (it won't be frequent). > > How about biosbits-bin repo? > > If QEMU is hosting binaries, where any part contains GPL code, then we > need to be providing the full and corresponding source and the build > scripts needed to re-create the binary. Once we have such scripts it > should be trivial to trigger that from a CI job. If it isn't then > we're doing something wrong. I was thinking of commiting the build scripts in a branch of https://gitlab.com/qemu-project/biosbits-bits. This would separate them from the main source. The scripts would build a version of qemu-bits based on the version information passed to it from the environment. Before I committed the scripts, I wanted to check whether we would want to do that or have a separate repo containing the binaries and the build scripts. Seems we want the former. As for the gitlab-ci, I looked at the yaml file and the qemu ones looks quite complicated. Can someone help me generate one based on the build script here? https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > The CI quota is not an issue, because > this is not a job that we need to run continuously. It can be triggered > manually as & when we decide we need to refresh the binary, so would > be a small one-off CI quota hit. > > Also note that gitlab is intending to start enforcing storage quota > on projects in the not too distant future. This makes it unappealing > to store binaries in git repos, unless we genuinely need the ability > to access historical versions of the binary. I don't believe we need > that for biosbits. > > The binary can be published as a CI artifact and accessed directly > from the latest artifact download link. This ensures we only consume > quota for the most recently published binary artifact. So I don't see > a compelling reason to upload binaries into git. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > keep the binaries? > > > > > > > > > > Can we please conclude on this? > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > you on IRC but failed. > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > overkill IMHO. We should create a repo just for the binaries, have a > > > README there to explain how we generate them and check in new versions > > > as and when needed (it won't be frequent). > > > How about biosbits-bin repo? > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > need to be providing the full and corresponding source and the build > > scripts needed to re-create the binary. Once we have such scripts it > > should be trivial to trigger that from a CI job. If it isn't then > > we're doing something wrong. > > I was thinking of commiting the build scripts in a branch of > https://gitlab.com/qemu-project/biosbits-bits. > This would separate them from the main source. The scripts would build > a version of qemu-bits based on the version information passed to it > from the environment. > Before I committed the scripts, I wanted to check whether we would > want to do that or have a separate repo containing the binaries and > the build scripts. > Seems we want the former. > > As for the gitlab-ci, I looked at the yaml file and the qemu ones > looks quite complicated. Can someone help me generate one based on the > build script here? > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh Yes, QEMU's rules aren't a good place to start if you're trying to learn gitlab CI, as they're very advanced. The simple case though is quite simple. * You need a container image to act as the build env * In 'before_script' install any packages you need on top of the base container image for build deps * In 'script' run whatever shell commands you need in order to build the project * Add an 'artifacts' section to publish one (or more) files/dirs as output The simplest example would be something like mybuild: image: fedora:36 before_script: - dnf install -y gcc script: - gcc -o myapp myapp.c artifacts paths: - myapp So essentially use 'docker run yourimage:tag' locally to bring up the build env, and interactively work out what packags yuo need to install, and what scripts to invoke to build it. When you have it figured out, just copy those commands into the before_script/script sections in .gitlab-ci.yml Note, the .gitlab-ci.yml file would be in the git repo of the project you're building typically, which would be the main 'biosbits' project i presume. There's a fairly decent reference of syntax here: https://docs.gitlab.com/ee/ci/yaml/ And broader set of docs https://docs.gitlab.com/ee/ci/ With regards, Daniel
On Tue, Sep 27, 2022 at 15:48 Daniel P. Berrangé <berrange@redhat.com> wrote: > On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in > git.qemu.org and > > > > > > > > the binaries in some other repo first. Everything else > hinges on that. We > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where > and how to > > > > > > > keep the binaries? > > > > > > > > > > > > Can we please conclude on this? > > > > > > Peter, can you please fork the repo? I have tried many times to > reach > > > > > > you on IRC but failed. > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > I think given our CI is under pressure again due to gitlab free > tier > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > README there to explain how we generate them and check in new > versions > > > > as and when needed (it won't be frequent). > > > > How about biosbits-bin repo? > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > need to be providing the full and corresponding source and the build > > > scripts needed to re-create the binary. Once we have such scripts it > > > should be trivial to trigger that from a CI job. If it isn't then > > > we're doing something wrong. > > > > I was thinking of commiting the build scripts in a branch of > > https://gitlab.com/qemu-project/biosbits-bits. > > This would separate them from the main source. The scripts would build > > a version of qemu-bits based on the version information passed to it > > from the environment. > > Before I committed the scripts, I wanted to check whether we would > > want to do that or have a separate repo containing the binaries and > > the build scripts. > > Seems we want the former. > > > > As for the gitlab-ci, I looked at the yaml file and the qemu ones > > looks quite complicated. Can someone help me generate one based on the > > build script here? > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > Yes, QEMU's rules aren't a good place to start if you're trying > to learn gitlab CI, as they're very advanced. > > The simple case though is quite simple. > > * You need a container image to act as the build env > * In 'before_script' install any packages you need on top of the > base container image for build deps > * In 'script' run whatever shell commands you need in order > to build the project > * Add an 'artifacts' section to publish one (or more) files/dirs > as output > > The simplest example would be something like > > mybuild: > image: fedora:36 > before_script: > - dnf install -y gcc > script: > - gcc -o myapp myapp.c > artifacts > paths: > - myapp > > > So essentially use 'docker run yourimage:tag' locally to bring > up the build env, and interactively work out what packags yuo > need to install, and what scripts to invoke to build it. When > you have it figured out, just copy those commands into the > before_script/script sections in .gitlab-ci.yml I already have a Dockerfile and a build script rahe uses it to generate the binaries https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > Note, the .gitlab-ci.yml file would be in the git repo of the > project you're building typically, which would be the main > 'biosbits' project i presume. > > There's a fairly decent reference of syntax here: > > https://docs.gitlab.com/ee/ci/yaml/ > > And broader set of docs > > https://docs.gitlab.com/ee/ci/ > > With regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > >
On Tue, Sep 27, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > > keep the binaries? > > > > > > > > > > > > Can we please conclude on this? > > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > > you on IRC but failed. > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > README there to explain how we generate them and check in new versions > > > > as and when needed (it won't be frequent). > > > > How about biosbits-bin repo? > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > need to be providing the full and corresponding source and the build > > > scripts needed to re-create the binary. Once we have such scripts it > > > should be trivial to trigger that from a CI job. If it isn't then > > > we're doing something wrong. > > > > I was thinking of commiting the build scripts in a branch of > > https://gitlab.com/qemu-project/biosbits-bits. > > This would separate them from the main source. The scripts would build > > a version of qemu-bits based on the version information passed to it > > from the environment. > > Before I committed the scripts, I wanted to check whether we would > > want to do that or have a separate repo containing the binaries and > > the build scripts. > > Seems we want the former. > > > > As for the gitlab-ci, I looked at the yaml file and the qemu ones > > looks quite complicated. Can someone help me generate one based on the > > build script here? > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > Yes, QEMU's rules aren't a good place to start if you're trying > to learn gitlab CI, as they're very advanced. > > The simple case though is quite simple. > > * You need a container image to act as the build env > * In 'before_script' install any packages you need on top of the > base container image for build deps > * In 'script' run whatever shell commands you need in order > to build the project > * Add an 'artifacts' section to publish one (or more) files/dirs > as output > > The simplest example would be something like > > mybuild: > image: fedora:36 > before_script: > - dnf install -y gcc > script: > - gcc -o myapp myapp.c > artifacts > paths: > - myapp > How does this look? https://pastebin.com/0YyLFmi3 > > So essentially use 'docker run yourimage:tag' locally to bring > up the build env, and interactively work out what packags yuo > need to install, and what scripts to invoke to build it. When > you have it figured out, just copy those commands into the > before_script/script sections in .gitlab-ci.yml > > > Note, the .gitlab-ci.yml file would be in the git repo of the > project you're building typically, which would be the main > 'biosbits' project i presume. > > There's a fairly decent reference of syntax here: > > https://docs.gitlab.com/ee/ci/yaml/ > > And broader set of docs > > https://docs.gitlab.com/ee/ci/ > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, Sep 27, 2022 at 5:12 PM Ani Sinha <ani@anisinha.ca> wrote: > > On Tue, Sep 27, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > > > keep the binaries? > > > > > > > > > > > > > > Can we please conclude on this? > > > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > > > you on IRC but failed. > > > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > > README there to explain how we generate them and check in new versions > > > > > as and when needed (it won't be frequent). > > > > > How about biosbits-bin repo? > > > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > > need to be providing the full and corresponding source and the build > > > > scripts needed to re-create the binary. Once we have such scripts it > > > > should be trivial to trigger that from a CI job. If it isn't then > > > > we're doing something wrong. > > > > > > I was thinking of commiting the build scripts in a branch of > > > https://gitlab.com/qemu-project/biosbits-bits. > > > This would separate them from the main source. The scripts would build > > > a version of qemu-bits based on the version information passed to it > > > from the environment. > > > Before I committed the scripts, I wanted to check whether we would > > > want to do that or have a separate repo containing the binaries and > > > the build scripts. > > > Seems we want the former. > > > > > > As for the gitlab-ci, I looked at the yaml file and the qemu ones > > > looks quite complicated. Can someone help me generate one based on the > > > build script here? > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > Yes, QEMU's rules aren't a good place to start if you're trying > > to learn gitlab CI, as they're very advanced. > > > > The simple case though is quite simple. > > > > * You need a container image to act as the build env > > * In 'before_script' install any packages you need on top of the > > base container image for build deps > > * In 'script' run whatever shell commands you need in order > > to build the project > > * Add an 'artifacts' section to publish one (or more) files/dirs > > as output > > > > The simplest example would be something like > > > > mybuild: > > image: fedora:36 > > before_script: > > - dnf install -y gcc > > script: > > - gcc -o myapp myapp.c > > artifacts > > paths: > > - myapp > > > > How does this look? > https://pastebin.com/0YyLFmi3 Alright, .gitlab-ci.yml is produced and the pipeline succeeds. However, the question still remains, where do we keep the generated artifacts?
On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > On Tue, Sep 27, 2022 at 5:12 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > On Tue, Sep 27, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > > > > keep the binaries? > > > > > > > > > > > > > > > > Can we please conclude on this? > > > > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > > > > you on IRC but failed. > > > > > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > > > README there to explain how we generate them and check in new versions > > > > > > as and when needed (it won't be frequent). > > > > > > How about biosbits-bin repo? > > > > > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > > > need to be providing the full and corresponding source and the build > > > > > scripts needed to re-create the binary. Once we have such scripts it > > > > > should be trivial to trigger that from a CI job. If it isn't then > > > > > we're doing something wrong. > > > > > > > > I was thinking of commiting the build scripts in a branch of > > > > https://gitlab.com/qemu-project/biosbits-bits. > > > > This would separate them from the main source. The scripts would build > > > > a version of qemu-bits based on the version information passed to it > > > > from the environment. > > > > Before I committed the scripts, I wanted to check whether we would > > > > want to do that or have a separate repo containing the binaries and > > > > the build scripts. > > > > Seems we want the former. > > > > > > > > As for the gitlab-ci, I looked at the yaml file and the qemu ones > > > > looks quite complicated. Can someone help me generate one based on the > > > > build script here? > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > > > Yes, QEMU's rules aren't a good place to start if you're trying > > > to learn gitlab CI, as they're very advanced. > > > > > > The simple case though is quite simple. > > > > > > * You need a container image to act as the build env > > > * In 'before_script' install any packages you need on top of the > > > base container image for build deps > > > * In 'script' run whatever shell commands you need in order > > > to build the project > > > * Add an 'artifacts' section to publish one (or more) files/dirs > > > as output > > > > > > The simplest example would be something like > > > > > > mybuild: > > > image: fedora:36 > > > before_script: > > > - dnf install -y gcc > > > script: > > > - gcc -o myapp myapp.c > > > artifacts > > > paths: > > > - myapp > > > > > > > How does this look? > > https://pastebin.com/0YyLFmi3 > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > However, the question still remains, where do we keep the generated > artifacts? The following link will always reflect the published artifacts from the most recently fully successful CI pipeline, on the 'qemu-bits' branch, and 'qemu-bits-build' CI job: https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build Tweak as needed if you push the CI to master branch instead. This link can be considered the permanent home of the artifact. I'd just suggest that the QEMU job automatically skip if it fails to download the artifact, as occassionally transient infra errors can impact it. With regards, Daniel
On Tue, Sep 27, 2022 at 09:33:27AM +0100, Daniel P. Berrangé wrote: > On Tue, Sep 27, 2022 at 01:43:15PM +0530, Ani Sinha wrote: > > On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > > > 1"? > > > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > > > > > Downloading a single file > > > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > > > style retrieval works just fine. > > > > > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > > > control here? Dunno. > > > > > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > > > I don't think we can reuse theirs. > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > keep the binaries? > > > > > > > > Can we please conclude on this? > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > you on IRC but failed. > > > > > > Probably because of travel around KVM forum. > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > limits, tying binaries to CI isn't a great idea at this stage. > > > Can Ani just upload binaies to qemu.org for now? > > > > I agree with Michael here. Having a full ci/cd job for this is > > overkill IMHO. We should create a repo just for the binaries, have a > > README there to explain how we generate them and check in new versions > > as and when needed (it won't be frequent). > > How about biosbits-bin repo? > > If QEMU is hosting binaries, where any part contains GPL code, then we > need to be providing the full and corresponding source and the build > scripts needed to re-create the binary. Once we have such scripts it > should be trivial to trigger that from a CI job. If it isn't then > we're doing something wrong. The CI quota is not an issue, because > this is not a job that we need to run continuously. It can be triggered > manually as & when we decide we need to refresh the binary, so would > be a small one-off CI quota hit. > > Also note that gitlab is intending to start enforcing storage quota > on projects in the not too distant future. This makes it unappealing > to store binaries in git repos, unless we genuinely need the ability > to access historical versions of the binary. I don't believe we need > that for biosbits. > > The binary can be published as a CI artifact and accessed directly > from the latest artifact download link. This ensures we only consume > quota for the most recently published binary artifact. So I don't see > a compelling reason to upload binaries into git. > > With regards, > Daniel I don't really care where we upload them but only having the latest version is just going to break anything expecting the old binary. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > keep the binaries? > > > > > > > > > > Can we please conclude on this? > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > you on IRC but failed. > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > overkill IMHO. We should create a repo just for the binaries, have a > > > README there to explain how we generate them and check in new versions > > > as and when needed (it won't be frequent). > > > How about biosbits-bin repo? > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > need to be providing the full and corresponding source and the build > > scripts needed to re-create the binary. Once we have such scripts it > > should be trivial to trigger that from a CI job. If it isn't then > > we're doing something wrong. > > I was thinking of commiting the build scripts in a branch of > https://gitlab.com/qemu-project/biosbits-bits. > This would separate them from the main source. The scripts would build > a version of qemu-bits based on the version information passed to it > from the environment. > Before I committed the scripts, I wanted to check whether we would > want to do that or have a separate repo containing the binaries and > the build scripts. > Seems we want the former. A separate repo is standard imho. Don't see any advantages to abusing git branches like that. > As for the gitlab-ci, I looked at the yaml file and the qemu ones > looks quite complicated. Can someone help me generate one based on the > build script here? > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > The CI quota is not an issue, because > > this is not a job that we need to run continuously. It can be triggered > > manually as & when we decide we need to refresh the binary, so would > > be a small one-off CI quota hit. > > > > Also note that gitlab is intending to start enforcing storage quota > > on projects in the not too distant future. This makes it unappealing > > to store binaries in git repos, unless we genuinely need the ability > > to access historical versions of the binary. I don't believe we need > > that for biosbits. > > > > The binary can be published as a CI artifact and accessed directly > > from the latest artifact download link. This ensures we only consume > > quota for the most recently published binary artifact. So I don't see > > a compelling reason to upload binaries into git. > > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > > On Tue, Sep 27, 2022 at 5:12 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > On Tue, Sep 27, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > On Tue, Sep 27, 2022 at 03:37:39PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > > > > > keep the binaries? > > > > > > > > > > > > > > > > > > Can we please conclude on this? > > > > > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > > > > > you on IRC but failed. > > > > > > > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > > > > README there to explain how we generate them and check in new versions > > > > > > > as and when needed (it won't be frequent). > > > > > > > How about biosbits-bin repo? > > > > > > > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > > > > need to be providing the full and corresponding source and the build > > > > > > scripts needed to re-create the binary. Once we have such scripts it > > > > > > should be trivial to trigger that from a CI job. If it isn't then > > > > > > we're doing something wrong. > > > > > > > > > > I was thinking of commiting the build scripts in a branch of > > > > > https://gitlab.com/qemu-project/biosbits-bits. > > > > > This would separate them from the main source. The scripts would build > > > > > a version of qemu-bits based on the version information passed to it > > > > > from the environment. > > > > > Before I committed the scripts, I wanted to check whether we would > > > > > want to do that or have a separate repo containing the binaries and > > > > > the build scripts. > > > > > Seems we want the former. > > > > > > > > > > As for the gitlab-ci, I looked at the yaml file and the qemu ones > > > > > looks quite complicated. Can someone help me generate one based on the > > > > > build script here? > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > > > > > Yes, QEMU's rules aren't a good place to start if you're trying > > > > to learn gitlab CI, as they're very advanced. > > > > > > > > The simple case though is quite simple. > > > > > > > > * You need a container image to act as the build env > > > > * In 'before_script' install any packages you need on top of the > > > > base container image for build deps > > > > * In 'script' run whatever shell commands you need in order > > > > to build the project > > > > * Add an 'artifacts' section to publish one (or more) files/dirs > > > > as output > > > > > > > > The simplest example would be something like > > > > > > > > mybuild: > > > > image: fedora:36 > > > > before_script: > > > > - dnf install -y gcc > > > > script: > > > > - gcc -o myapp myapp.c > > > > artifacts > > > > paths: > > > > - myapp > > > > > > > > > > How does this look? > > > https://pastebin.com/0YyLFmi3 > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > However, the question still remains, where do we keep the generated > > artifacts? > > The following link will always reflect the published artifacts from > the most recently fully successful CI pipeline, on the 'qemu-bits' > branch, and 'qemu-bits-build' CI job: > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > Tweak as needed if you push the CI to master branch instead. This > link can be considered the permanent home of the artifact. I'd just > suggest that the QEMU job automatically skip if it fails to download > the artifact, as occassionally transient infra errors can impact > it. > > With regards, > Daniel This just means once we change the test old qemu source can no longer use it. Why is this a good idea? Are we so short on disk space? I thought CPU is the limiting factor? > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Il dom 10 lug 2022, 19:01 Ani Sinha <ani@anisinha.ca> ha scritto: > This change adds python based test environment that can be used to run > pytest > from within a virtual environment. A bash script sets up a virtual > environment > and then runs the python based tests from within that environment. > All dependent python packages are installed in the virtual environment > using > pip python module. QEMU python test modules are also available in the > environment > for spawning the QEMU based VMs. > > It also introduces QEMU acpi/smbios biosbits python test script which is > run > from within the python virtual environment. When the bios bits tests are > run, > bios bits binaries are downloaded from an external repo/location. > Currently, the test points to an external private github repo where the > bits > archives are checked in. The virtual environment should be set up from configure, similar to git submodules. John was working on it and probably can point you at some earlier discussions in the archives about how to do it. I also second the idea of using avocado instead of pytest, by the way. Paolo > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ > tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ > tests/pytest/acpi-bits/meson.build | 33 ++ > tests/pytest/acpi-bits/requirements.txt | 1 + > 4 files changed, 475 insertions(+) > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > create mode 100644 tests/pytest/acpi-bits/meson.build > create mode 100644 tests/pytest/acpi-bits/requirements.txt > > diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > new file mode 100644 > index 0000000000..186395473b > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh > @@ -0,0 +1,59 @@ > +#!/usr/bin/env bash > +# Generates a python virtual environment for the test to run. > +# Then runs python test scripts from within that virtual environment. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Author: Ani Sinha <ani@anisinha.ca> > + > +set -e > + > +MYPATH=$(realpath ${BASH_SOURCE:-$0}) > +MYDIR=$(dirname $MYPATH) > + > +if [ -z "$PYTEST_SOURCE_ROOT" ]; then > + echo -n "Please set QTEST_SOURCE_ROOT env pointing" > + echo " to the root of the qemu source tree." > + echo -n "This is required so that the test can find the " > + echo "python modules that it needs for execution." > + exit 1 > +fi > +SRCDIR=$PYTEST_SOURCE_ROOT > +TESTSCRIPTS=("acpi-bits-test.py") > +PIPCMD="-m pip -q --disable-pip-version-check" > +# we need to save the old value of PWD before we do a change-dir later > +PYTEST_PWD=$PWD > + > +TESTS_PYTHON=/usr/bin/python3 > +TESTS_VENV_REQ=requirements.txt > + > +# sadly for pip -e and -t options do not work together. > +# please see https://github.com/pypa/pip/issues/562 > +cd $MYDIR > + > +$TESTS_PYTHON -m venv . > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ > +[ -f $TESTS_VENV_REQ ] && \ > + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 > + > +# venv is activated at this point. > + > +# run the test > +for testscript in ${TESTSCRIPTS[@]} ; do > + export PYTEST_PWD; python3 $testscript > +done > + > +cd $PYTEST_PWD > + > +exit 0 > diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py > b/tests/pytest/acpi-bits/acpi-bits-test.py > new file mode 100644 > index 0000000000..97e61eb709 > --- /dev/null > +++ b/tests/pytest/acpi-bits/acpi-bits-test.py > @@ -0,0 +1,382 @@ > +#!/usr/bin/env python3 > +# group: rw quick > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, > +# https://biosbits.org/ > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > +# > +# Some parts are slightly taken from qtest.py and iotests.py > +# > +# Authors: > +# Ani Sinha <ani@anisinha.ca> > + > +# pylint: disable=invalid-name > + > +""" > +QEMU bios tests using biosbits available at > +https://biosbits.org/. > +""" > + > +import logging > +import os > +import re > +import shutil > +import subprocess > +import sys > +import tarfile > +import tempfile > +import time > +import unittest > +from urllib import request > +import zipfile > +from typing import ( > + List, > + Optional, > + Sequence, > +) > +from tap import TAPTestRunner > +from qemu.machine import QEMUMachine > + > +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') > +PYTEST_PWD = os.getenv('PYTEST_PWD') > + > +def get_arch(): > + """finds the arch from the qemu binary name""" > + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) > + if match: > + return match.group(1) > + return 'x86_64' > + > +ARCH = get_arch() > + > +class QEMUBitsMachine(QEMUMachine): > + """ > + A QEMU VM, with isa-debugcon enabled and bits iso passed > + using -cdrom to QEMU commandline. > + """ > + def __init__(self, > + binary: str, > + args: Sequence[str] = (), > + wrapper: Sequence[str] = (), > + name: Optional[str] = None, > + base_temp_dir: str = "/var/tmp", > + debugcon_log: str = "debugcon-log.txt", > + debugcon_addr: str = "0x403", > + sock_dir: Optional[str] = None, > + qmp_timer: Optional[float] = None): > + # pylint: disable=too-many-arguments > + > + if name is None: > + name = "qemu-bits-%d" % os.getpid() > + if sock_dir is None: > + sock_dir = base_temp_dir > + super().__init__(binary, args, wrapper=wrapper, name=name, > + base_temp_dir=base_temp_dir, > + sock_dir=sock_dir, qmp_timer=qmp_timer) > + self.debugcon_log = debugcon_log > + self.debugcon_addr = debugcon_addr > + self.base_temp_dir = base_temp_dir > + > + @property > + def _base_args(self) -> List[str]: > + args = super()._base_args > + args.extend([ > + '-chardev', > + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, > + self.debugcon_log), > + '-device', > + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, > + ]) > + return args > + > + def base_args(self): > + """return the base argument to QEMU binary""" > + return self._base_args > + > +class AcpiBitsTest(unittest.TestCase): > + """ACPI and SMBIOS tests using biosbits.""" > + def __init__(self, *args, **kwargs): > + super().__init__(*args, **kwargs) > + self._vm = None > + self._workDir = None > + self._bitsVer = 2100 > + self._bitsLoc = " > https://github.com/ani-sinha/bits/raw/bits-builds/" > + self._debugcon_addr = '0x403' > + self._debugcon_log = 'debugcon-log.txt' > + logging.basicConfig(level=logging.INFO) > + > + def copy_bits_config(self): > + """ copies the bios bits config file into bits. > + """ > + config_file = 'bits-cfg.txt' > + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') > + target_config_dir = os.path.join(self._workDir, > + 'bits-%d' %self._bitsVer, 'boot') > + self.assertTrue(os.path.exists(qemu_bits_config_dir)) > + self.assertTrue(os.path.exists(target_config_dir)) > + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, > + config_file), os.R_OK)) > + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), > + target_config_dir) > + logging.info('copied config file %s to %s', > + config_file, target_config_dir) > + > + def copy_test_scripts(self): > + """copies the python test scripts into bits. """ > + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') > + target_test_dir = os.path.join(self._workDir, 'bits-%d' > %self._bitsVer, > + 'boot', 'python') > + > + self.assertTrue(os.path.exists(qemu_test_dir)) > + self.assertTrue(os.path.exists(target_test_dir)) > + > + for filename in os.listdir(qemu_test_dir): > + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ > + filename.endswith('.py'): > + shutil.copy2(os.path.join(qemu_test_dir, filename), > + target_test_dir) > + logging.info('copied test file %s to %s', > + filename, target_test_dir) > + > + # now remove the pyc test file if it exists, otherwise the > + # changes in the python test script won't be executed. > + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' > + if os.access(os.path.join(target_test_dir, testfile_pyc), > + os.F_OK): > + os.remove(os.path.join(target_test_dir, testfile_pyc)) > + logging.info('removed compiled file %s', > + os.path.join(target_test_dir, > testfile_pyc)) > + > + def fix_mkrescue(self, mkrescue): > + """ grub-mkrescue is a bash script with two variables, 'prefix' > and > + 'libdir'. They must be pointed to the right location so that > the > + iso can be generated appropriately. We point the two > variables to > + the directory where we have extracted our pre-built bits grub > + tarball. > + """ > + grub_x86_64_mods = os.path.join(self._workDir, > 'grub-inst-x86_64-efi') > + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') > + > + self.assertTrue(os.path.exists(grub_x86_64_mods)) > + self.assertTrue(os.path.exists(grub_i386_mods)) > + > + new_script = "" > + with open(mkrescue, 'r') as filehandle: > + orig_script = filehandle.read() > + new_script = re.sub('(^prefix=)(.*)', > + r'\1"%s"' %grub_x86_64_mods, > + orig_script, flags=re.M) > + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' > %grub_i386_mods, > + new_script, flags=re.M) > + > + with open(mkrescue, 'w') as filehandle: > + filehandle.write(new_script) > + > + def generate_bits_iso(self): > + """ Uses grub-mkrescue to generate a fresh bits iso with the > python > + test scripts > + """ > + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' > %self._bitsVer) > + mkrescue_script = os.path.join(self._workDir, > + 'grub-inst-x86_64-efi', 'bin', > + 'grub-mkrescue') > + > + self.assertTrue(os.access(mkrescue_script, > + os.R_OK | os.W_OK | os.X_OK)) > + > + self.fix_mkrescue(mkrescue_script) > + > + logging.info('calling grub-mkrescue to generate the biosbits iso > ...') > + > + try: > + if os.getenv('V'): > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stdout=subprocess.DEVNULL) > + else: > + subprocess.check_call([mkrescue_script, '-o', > + iso_file, bits_dir], > + stderr=subprocess.DEVNULL, > + stdout=subprocess.DEVNULL) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("Error while generating the bits iso. " > + "Pass V=1 in the environment to get more > details. " > + + str(e)) > + > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + logging.info('iso file %s successfully generated.', iso_file) > + > + def setUp(self): > + BITS_LOC = os.getenv("PYTEST_BITSLOC") > + if BITS_LOC: > + prefix = BITS_LOC > + else: > + prefix = os.path.join(os.getcwd(), 'prebuilt') > + if not os.path.isdir(prefix): > + os.mkdir(prefix, mode=0o775) > + > + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' > + %self._bitsVer) > + grub_tar_file = os.path.join(prefix, > + 'bits-%d-grub.tar.gz' %self._bitsVer) > + # if the location of the bits binaries has been specified by the > user > + # and they are not found in that location, skip the test. > + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location > %s." \ > + %BITS_LOC) > + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): > + self.skipTest("test skipped since biosbits binaries " + > + "could not be found in the specified location > %s." \ > + %BITS_LOC) > + > + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', > + suffix='.tmp') > + logging.info('working dir: %s', self._workDir) > + > + localArchive = "bits-%d.zip" % self._bitsVer > + if not os.access(bits_zip_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) > + > + localArchive = "bits-%d-grub.tar.gz" % self._bitsVer > + if not os.access(grub_tar_file, os.F_OK): > + logging.info("archive %s not found in %s, downloading ...", > + localArchive, bits_zip_file) > + try: > + req = request.urlopen(self._bitsLoc + localArchive) > + with open(os.path.join(prefix, localArchive), > + 'wb') as archivef: > + archivef.write(req.read()) > + except Exception as e: # pylint: disable=broad-except > + self.skipTest("test skipped since biosbits binaries " + > + "could not be obtained." + str(e)) > + else: > + logging.info('using locally found %s', localArchive) > + > + # extract the bits software in the temp working directory > + with zipfile.ZipFile(bits_zip_file, 'r') as zref: > + zref.extractall(self._workDir) > + > + with tarfile.open(grub_tar_file, 'r') as tarball: > + tarball.extractall(self._workDir) > + > + self.copy_test_scripts() > + self.copy_bits_config() > + self.generate_bits_iso() > + > + def parse_log(self): > + """parse the log generated by running bits tests and > + check for failures. > + """ > + debugconf = os.path.join(self._workDir, self._debugcon_log) > + log = "" > + with open(debugconf, 'r') as filehandle: > + log = filehandle.read() > + > + if os.getenv('V'): > + print('\nlogs from biosbits follows:') > + print('==========================================\n') > + print(log) > + print('==========================================\n') > + > + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ > failed).*', > + log) > + for match in matchiter: > + # verify that no test cases failed. > + self.assertEqual(match.group(3).split()[0], '0', > + 'Some bits tests seems to have failed. ' \ > + 'Set V=1 in the environment to get the > entire ' \ > + 'log from bits.') > + > + def tearDown(self): > + if self._vm: > + self.assertFalse(not self._vm.is_running) > + logging.info('removing the work directory %s', self._workDir) > + shutil.rmtree(self._workDir) > + > + def test_acpi_smbios_bits(self): > + """The main test case implementaion.""" > + > + qemu_bin = PYTESTQEMUBIN > + iso_file = os.path.join(self._workDir, 'bits-%d.iso' > %self._bitsVer) > + > + # PYTESTQEMUBIN could be relative to the current directory > + if not os.access(PYTESTQEMUBIN, os.X_OK) and PYTEST_PWD: > + qemu_bin = os.path.join(PYTEST_PWD, PYTESTQEMUBIN) > + > + logging.info('QEMU binary used: %s', qemu_bin) > + > + self.assertTrue(os.access(qemu_bin, os.X_OK)) > + self.assertTrue(os.access(iso_file, os.R_OK)) > + > + self._vm = QEMUBitsMachine(binary=qemu_bin, > + base_temp_dir=self._workDir, > + debugcon_log=self._debugcon_log, > + debugcon_addr=self._debugcon_addr) > + > + self._vm.add_args('-cdrom', '%s' %iso_file) > + > + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ > + " " + " ".join(str(arg) for arg in self._vm.args) > + > + logging.info("launching QEMU vm with the following arguments: > %s", > + args) > + > + self._vm.launch() > + # biosbits has been configured to run all the specified test > suites > + # in batch mode and then automatically initiate a vm shutdown. > + # sleep for maximum of one minute > + max_sleep_time = time.monotonic() + 60 > + while self._vm.is_running() and time.monotonic() < max_sleep_time: > + time.sleep(1) > + > + self.assertFalse(time.monotonic() > max_sleep_time, > + 'The VM seems to have failed to shutdown in > time') > + > + self.parse_log() > + > +def execute_unittest(argv: List[str], debug: bool = False, > + runner: TAPTestRunner = None) -> None: > + """Executes unittests within the calling module.""" > + > + unittest.main(argv=argv, > + testRunner=runner, > + verbosity=2 if debug else 1, > + warnings=None if sys.warnoptions else 'ignore') > + > +def main(): > + """ The main function where execution begins. """ > + > + assert PYTESTQEMUBIN is not None, \ > + "Environment variable PYTEST_QEMU_BINARY required." > + > + runner = TAPTestRunner() > + runner.set_stream(True) > + runner.set_format("%s/acpi-bits-test" %ARCH) > + execute_unittest(sys.argv, False, runner) > + > +main() > diff --git a/tests/pytest/acpi-bits/meson.build > b/tests/pytest/acpi-bits/meson.build > new file mode 100644 > index 0000000000..099c191d57 > --- /dev/null > +++ b/tests/pytest/acpi-bits/meson.build > @@ -0,0 +1,33 @@ > +xorriso = find_program('xorriso', required: true) > +if not xorriso.found() > + message('xorriso not found ... disabled bits acpi tests.') > + subdir_done() > +endif > + > +subdir('bits-tests') > +subdir('bits-config') > + > +test_files = ['acpi-bits-test.py'] > +requirements = 'requirements.txt' > + > +copytestfiles = custom_target('copy test files', > + input : test_files, > + output : test_files, > + command : ['cp', '@INPUT@', '@OUTPUT@'], > + install : false, > + build_by_default : true) > + > +requirementsfiles = custom_target('copy py req files', > + input : requirements, > + output : requirements, > + command : ['cp', '@INPUT@', '@OUTPUT@'], > + install : false, > + build_by_default : true) > + > +other_deps += [copytestfiles,requirementsfiles] > + > +pytest_executables += { > + 'acpi-bits-test': configure_file(copy:true, > + input:'acpi-bits-test-venv.sh', > + output:'acpi-bits-test') > +} > diff --git a/tests/pytest/acpi-bits/requirements.txt > b/tests/pytest/acpi-bits/requirements.txt > new file mode 100644 > index 0000000000..00cdad09ef > --- /dev/null > +++ b/tests/pytest/acpi-bits/requirements.txt > @@ -0,0 +1 @@ > +tap.py > -- > 2.25.1 > >
On Tue, Sep 27, 2022 at 11:44:56PM +0200, Paolo Bonzini wrote: > I also second the idea of using avocado instead of pytest, by the way. > > Paolo I do not think this is a good fit for bios tests. bios tests are intended for a wide audience of ACPI developers across a variety of host systems. They basically do not need anything from the host and they need to be super easy to configure since we have lots of drive through contributors. Problem is I don't think avocado is yet at the level where I can ask random developers to use it to check their ACPI patches. I just went ahead and rechecked and the situation isn't much better yet. I think the focus of avocado is system testing of full guests with KVM, not unit testing of ACPI. Let's start with installation on a clean box: following https://avocado-framework.readthedocs.io/en/98.0/guides/user/chapters/installing.html Ugh pip, will install a bunch of stuff in ~/.local and ask me to tweak PATH ... and what about security? No thanks! So ... do I want LTS or latest? Well I donnu .... let's try LTS? $ dnf module enable avocado:82lts [sudo] password for mst: Last metadata expiration check: 6 days, 15:20:21 ago on Wed 21 Sep 2022 02:33:31 AM EDT. Dependencies resolved. ========================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================== Enabling module streams: avocado 82lts Transaction Summary ========================================================================================================================================== Is this ok [y/N]: y Complete! [mst@tuck linux]$ dnf module install avocado Last metadata expiration check: 6 days, 15:20:41 ago on Wed 21 Sep 2022 02:33:31 AM EDT. No default profiles for module avocado:82lts. Available profiles: default, minimal Error: Problems in request: broken groups or modules: avocado Ugh I guess latest then? [mst@tuck linux]$ dnf module enable avocado:latest Last metadata expiration check: 6 days, 15:25:21 ago on Wed 21 Sep 2022 02:33:31 AM EDT. Dependencies resolved. The operation would result in switching of module 'avocado' stream '82lts' to stream 'latest' Error: It is not possible to switch enabled streams of a module unless explicitly enabled via configuration option module_stream_switch. It is recommended to rather remove all installed content from the module, and reset the module using 'dnf module reset <module_name>' command. After you reset the module, you can install the other stream. Scary ... I don't really know what are streams and I am guessing module is avocado here? and what will this affect. Oh well, I'll risk this: [mst@tuck linux]$ dnf module reset avocado Last metadata expiration check: 6 days, 15:25:46 ago on Wed 21 Sep 2022 02:33:31 AM EDT. Dependencies resolved. ========================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================== Resetting modules: avocado Transaction Summary ========================================================================================================================================== Is this ok [y/N]: y Complete! [mst@tuck linux]$ dnf module enable avocado:latest Last metadata expiration check: 6 days, 15:25:55 ago on Wed 21 Sep 2022 02:33:31 AM EDT. Dependencies resolved. ========================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================== Enabling module streams: avocado latest Transaction Summary ========================================================================================================================================== Is this ok [y/N]: y Complete! [mst@tuck linux]$ dnf module install avocado Last metadata expiration check: 6 days, 15:26:03 ago on Wed 21 Sep 2022 02:33:31 AM EDT. Dependencies resolved. ========================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================== Installing group/module packages: python3-avocado noarch 97.0-1.module_f35+14550+3bc43fee updates-modular 603 k python3-avocado-plugins-output-html noarch 97.0-1.module_f35+14550+3bc43fee updates-modular 99 k python3-avocado-plugins-varianter-yaml-to-mux noarch 97.0-1.module_f35+14550+3bc43fee updates-modular 29 k Installing dependencies: gdb-gdbserver x86_64 12.1-1.fc35 updates 300 k python-avocado-common noarch 97.0-1.module_f35+14550+3bc43fee updates-modular 29 k Installing module profiles: avocado/default Transaction Summary ========================================================================================================================================== Install 5 Packages Total download size: 1.0 M Installed size: 3.5 M Is this ok [y/N]: y Downloading Packages: (1/5): python-avocado-common-97.0-1.module_f35+14550+3bc43fee.noarch.rpm 40 kB/s | 29 kB 00:00 (2/5): python3-avocado-plugins-output-html-97.0-1.module_f35+14550+3bc43fee.noarch.rpm 109 kB/s | 99 kB 00:00 (3/5): python3-avocado-plugins-varianter-yaml-to-mux-97.0-1.module_f35+14550+3bc43fee.noarch.rpm 28 kB/s | 29 kB 00:01 (4/5): python3-avocado-97.0-1.module_f35+14550+3bc43fee.noarch.rpm 479 kB/s | 603 kB 00:01 (5/5): gdb-gdbserver-12.1-1.fc35.x86_64.rpm 232 kB/s | 300 kB 00:01 ------------------------------------------------------------------------------------------------------------------------------------------ Total 204 kB/s | 1.0 MB 00:05 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Regex version mismatch, expected: 10.40 2022-04-14 actual: 10.35 2020-05-09 Preparing : 1/1 Installing : python-avocado-common-97.0-1.module_f35+14550+3bc43fee.noarch 1/5 Installing : gdb-gdbserver-12.1-1.fc35.x86_64 2/5 Installing : python3-avocado-97.0-1.module_f35+14550+3bc43fee.noarch 3/5 Installing : python3-avocado-plugins-output-html-97.0-1.module_f35+14550+3bc43fee.noarch 4/5 Installing : python3-avocado-plugins-varianter-yaml-to-mux-97.0-1.module_f35+14550+3bc43fee.noarch 5/5 Running scriptlet: python3-avocado-plugins-varianter-yaml-to-mux-97.0-1.module_f35+14550+3bc43fee.noarch 5/5 Verifying : gdb-gdbserver-12.1-1.fc35.x86_64 1/5 Verifying : python-avocado-common-97.0-1.module_f35+14550+3bc43fee.noarch 2/5 Verifying : python3-avocado-97.0-1.module_f35+14550+3bc43fee.noarch 3/5 Verifying : python3-avocado-plugins-output-html-97.0-1.module_f35+14550+3bc43fee.noarch 4/5 Verifying : python3-avocado-plugins-varianter-yaml-to-mux-97.0-1.module_f35+14550+3bc43fee.noarch 5/5 Installed: gdb-gdbserver-12.1-1.fc35.x86_64 python-avocado-common-97.0-1.module_f35+14550+3bc43fee.noarch python3-avocado-97.0-1.module_f35+14550+3bc43fee.noarch python3-avocado-plugins-output-html-97.0-1.module_f35+14550+3bc43fee.noarch python3-avocado-plugins-varianter-yaml-to-mux-97.0-1.module_f35+14550+3bc43fee.noarch Complete! Great! Except I want avocado-vt of course. I will pretend someone told me that I need it. Installing Avocado-VT on Fedora or Enterprise Linux is a matter of installing the avocado-plugins-vt package. Install it with: $ yum install avocado-plugins-vt I guess I will replace yum with dnf then? [mst@tuck linux]$ dnf install avocado-plugins-vt Last metadata expiration check: 6 days, 15:31:16 ago on Wed 21 Sep 2022 02:33:31 AM EDT. No match for argument: avocado-plugins-vt Error: Unable to find a match: avocado-plugins-vt And this is Fedora. What do people do on debian? non Linux?
Il mer 28 set 2022, 00:09 Michael S. Tsirkin <mst@redhat.com> ha scritto: > On Tue, Sep 27, 2022 at 11:44:56PM +0200, Paolo Bonzini wrote: > > I also second the idea of using avocado instead of pytest, by the way. > > > > Paolo > > I do not think this is a good fit for bios tests. > bios tests are intended for a wide audience of ACPI developers > across a variety of host systems. They basically do not need anything > from the host and they need to be super easy to configure > since we have lots of drive through contributors. > The setup would be the same, with avocado installed in a virtual environment via pip. It doesn't need to be set up outside, neither with distro packages nor in ~/.local, and especially it is not necessary to deal with avocado-vt. Paolo
On Wed, Sep 28, 2022 at 2:48 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Sep 27, 2022 at 09:33:27AM +0100, Daniel P. Berrangé wrote: > > On Tue, Sep 27, 2022 at 01:43:15PM +0530, Ani Sinha wrote: > > > On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > > > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > > > > 1"? > > > > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > > > > > > > Downloading a single file > > > > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > > > > style retrieval works just fine. > > > > > > > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > > > > control here? Dunno. > > > > > > > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > > > > I don't think we can reuse theirs. > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > keep the binaries? > > > > > > > > > > Can we please conclude on this? > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > you on IRC but failed. > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > overkill IMHO. We should create a repo just for the binaries, have a > > > README there to explain how we generate them and check in new versions > > > as and when needed (it won't be frequent). > > > How about biosbits-bin repo? > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > need to be providing the full and corresponding source and the build > > scripts needed to re-create the binary. Once we have such scripts it > > should be trivial to trigger that from a CI job. If it isn't then > > we're doing something wrong. The CI quota is not an issue, because > > this is not a job that we need to run continuously. It can be triggered > > manually as & when we decide we need to refresh the binary, so would > > be a small one-off CI quota hit. > > > > Also note that gitlab is intending to start enforcing storage quota > > on projects in the not too distant future. This makes it unappealing > > to store binaries in git repos, unless we genuinely need the ability > > to access historical versions of the binary. I don't believe we need > > that for biosbits. > > > > The binary can be published as a CI artifact and accessed directly > > from the latest artifact download link. This ensures we only consume > > quota for the most recently published binary artifact. So I don't see > > a compelling reason to upload binaries into git. > > > > With regards, > > Daniel > > I don't really care where we upload them but only having the > latest version is just going to break anything expecting > the old binary. In fairness, I am not entirely certain if there is a tight coupling between the qemu tests and the bits binaries. I have written the test framework in a way such that test modifications and new tests can be pushed into the bits binaries and the iso gets regenerated with the new tests from QEMU itself before running the tests. Only when we need bits bugfixes or say upgrade to new acpica that we would need to regenerate the bits binaries. > > > > > -- > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Wed, Sep 28, 2022 at 08:38:54AM +0530, Ani Sinha wrote: > > I don't really care where we upload them but only having the > > latest version is just going to break anything expecting > > the old binary. > > In fairness, I am not entirely certain if there is a tight coupling > between the qemu tests and the bits binaries. I have written the test > framework in a way such that test modifications and new tests can be > pushed into the bits binaries and the iso gets regenerated with the > new tests from QEMU itself before running the tests. Only when we need > bits bugfixes or say upgrade to new acpica that we would need to > regenerate the bits binaries. Theoretically, that's correct. But if we did not have bugs we would not need tests.
On Wed, Sep 28, 2022 at 01:10:15AM +0200, Paolo Bonzini wrote: > > > Il mer 28 set 2022, 00:09 Michael S. Tsirkin <mst@redhat.com> ha scritto: > > On Tue, Sep 27, 2022 at 11:44:56PM +0200, Paolo Bonzini wrote: > > I also second the idea of using avocado instead of pytest, by the way. > > > > Paolo > > I do not think this is a good fit for bios tests. > bios tests are intended for a wide audience of ACPI developers > across a variety of host systems. They basically do not need anything > from the host and they need to be super easy to configure > since we have lots of drive through contributors. > > > The setup would be the same, with avocado installed in a virtual environment > via pip. It doesn't need to be set up outside, neither with distro packages nor > in ~/.local, and especially it is not necessary to deal with avocado-vt. > > Paolo Hmm, good point.
On Wed, Sep 28, 2022 at 9:28 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Sep 28, 2022 at 08:38:54AM +0530, Ani Sinha wrote: > > > I don't really care where we upload them but only having the > > > latest version is just going to break anything expecting > > > the old binary. > > > > In fairness, I am not entirely certain if there is a tight coupling > > between the qemu tests and the bits binaries. I have written the test > > framework in a way such that test modifications and new tests can be > > pushed into the bits binaries and the iso gets regenerated with the > > new tests from QEMU itself before running the tests. Only when we need > > bits bugfixes or say upgrade to new acpica that we would need to > > regenerate the bits binaries. > > Theoretically, that's correct. But if we did not have bugs we would > not need tests. Hmm, you might have a point. Curious, do we keep versioned binaries of edk for example? If so, why we can't do the same for bits?
On Tue, Sep 27, 2022 at 05:18:10PM -0400, Michael S. Tsirkin wrote: > On Tue, Sep 27, 2022 at 09:33:27AM +0100, Daniel P. Berrangé wrote: > > On Tue, Sep 27, 2022 at 01:43:15PM +0530, Ani Sinha wrote: > > > On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > > > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > > > > 1"? > > > > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > > > > > > > Downloading a single file > > > > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > > > > style retrieval works just fine. > > > > > > > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > > > > control here? Dunno. > > > > > > > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > > > > I don't think we can reuse theirs. > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > keep the binaries? > > > > > > > > > > Can we please conclude on this? > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > you on IRC but failed. > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > overkill IMHO. We should create a repo just for the binaries, have a > > > README there to explain how we generate them and check in new versions > > > as and when needed (it won't be frequent). > > > How about biosbits-bin repo? > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > need to be providing the full and corresponding source and the build > > scripts needed to re-create the binary. Once we have such scripts it > > should be trivial to trigger that from a CI job. If it isn't then > > we're doing something wrong. The CI quota is not an issue, because > > this is not a job that we need to run continuously. It can be triggered > > manually as & when we decide we need to refresh the binary, so would > > be a small one-off CI quota hit. > > > > Also note that gitlab is intending to start enforcing storage quota > > on projects in the not too distant future. This makes it unappealing > > to store binaries in git repos, unless we genuinely need the ability > > to access historical versions of the binary. I don't believe we need > > that for biosbits. > > > > The binary can be published as a CI artifact and accessed directly > > from the latest artifact download link. This ensures we only consume > > quota for the most recently published binary artifact. So I don't see > > a compelling reason to upload binaries into git. > > I don't really care where we upload them but only having the > latest version is just going to break anything expecting > the old binary. biosbits isn't tied to QEMU versions, it is an entirely separate 3rd party project. This binary is just providing the test env, and IIUC, control over what executes in this env is still done by the QEMU side test scripts. I'm not seeing a coupling here that requires precise matching. In any case biosbit is a dead project so does not look likely to have any changes. If we did want to have different versions though, we can stil publish artifacts from different branches of biosbits code. Gitlab will preserve & publish the latest artifacts from each branch in parallel. With regards, Daniel
On Tue, Sep 27, 2022 at 06:09:22PM -0400, Michael S. Tsirkin wrote: > On Tue, Sep 27, 2022 at 11:44:56PM +0200, Paolo Bonzini wrote: > > I also second the idea of using avocado instead of pytest, by the way. snip > Problem is I don't think avocado is yet at the level where I can > ask random developers to use it to check their ACPI patches. > > I just went ahead and rechecked and the situation isn't much better > yet. I think the focus of avocado is system testing of full guests with > KVM, not unit testing of ACPI. > > Let's start with installation on a clean box: ...snip... Do not do any of this stuff, it is irrelevant to QEMU's needs. A developer using Avocado with QEMU does nothing more than: make check-avocado The avocado framework itself is setup in a virtual env on first run by the makefile. Thus the developers don't need to install nor interact with avocado at all, merely write a test script and put it in tests/avocado/ in QEMU git. With regards, Daniel
On Wed, Sep 28, 2022 at 12:28 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Sep 27, 2022 at 05:18:10PM -0400, Michael S. Tsirkin wrote: > > On Tue, Sep 27, 2022 at 09:33:27AM +0100, Daniel P. Berrangé wrote: > > > On Tue, Sep 27, 2022 at 01:43:15PM +0530, Ani Sinha wrote: > > > > On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > > > > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > > > > > 1"? > > > > > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > > > > > > > > > Downloading a single file > > > > > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > > > > > style retrieval works just fine. > > > > > > > > > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > > > > > control here? Dunno. > > > > > > > > > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > > > > > I don't think we can reuse theirs. > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > > keep the binaries? > > > > > > > > > > > > Can we please conclude on this? > > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > > you on IRC but failed. > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > README there to explain how we generate them and check in new versions > > > > as and when needed (it won't be frequent). > > > > How about biosbits-bin repo? > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > need to be providing the full and corresponding source and the build > > > scripts needed to re-create the binary. Once we have such scripts it > > > should be trivial to trigger that from a CI job. If it isn't then > > > we're doing something wrong. The CI quota is not an issue, because > > > this is not a job that we need to run continuously. It can be triggered > > > manually as & when we decide we need to refresh the binary, so would > > > be a small one-off CI quota hit. > > > > > > Also note that gitlab is intending to start enforcing storage quota > > > on projects in the not too distant future. This makes it unappealing > > > to store binaries in git repos, unless we genuinely need the ability > > > to access historical versions of the binary. I don't believe we need > > > that for biosbits. > > > > > > The binary can be published as a CI artifact and accessed directly > > > from the latest artifact download link. This ensures we only consume > > > quota for the most recently published binary artifact. So I don't see > > > a compelling reason to upload binaries into git. > > > > I don't really care where we upload them but only having the > > latest version is just going to break anything expecting > > the old binary. > > biosbits isn't tied to QEMU versions, it is an entirely separate 3rd > party project. This binary is just providing the test env, and IIUC, > control over what executes in this env is still done by the QEMU side > test scripts. I'm not seeing a coupling here that requires precise > matching. In any case biosbit is a dead project so does not look > likely to have any changes. > > If we did want to have different versions though, we can stil > publish artifacts from different branches of biosbits code. No, that is just ridiculous. Say we have a bug in bits that we fixed and released a new version. Do we now create a new branch for that? Multiple branches makes things needlessly complicated. We have one branch, qemu-bits and all fixes go into that branch. We can have different tags if we need. Nothing beyond that. Gitlab > will preserve & publish the latest artifacts from each branch in > parallel. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Wed, Sep 28, 2022 at 12:45:46PM +0530, Ani Sinha wrote: > On Wed, Sep 28, 2022 at 12:28 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Tue, Sep 27, 2022 at 05:18:10PM -0400, Michael S. Tsirkin wrote: > > > On Tue, Sep 27, 2022 at 09:33:27AM +0100, Daniel P. Berrangé wrote: > > > > On Tue, Sep 27, 2022 at 01:43:15PM +0530, Ani Sinha wrote: > > > > > On Sun, Sep 18, 2022 at 1:58 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Fri, Sep 16, 2022 at 09:30:42PM +0530, Ani Sinha wrote: > > > > > > > On Thu, Jul 28, 2022 at 12:08 AM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 25 Jul 2022, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 16 Jul 2022, Michael S. Tsirkin wrote: > > > > > > > > > > > > > > > > > > > On Sat, Jul 16, 2022 at 12:06:00PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 11:20 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 15, 2022 at 09:47:27AM +0530, Ani Sinha wrote: > > > > > > > > > > > > > Instead of all this mess, can't we just spawn e.g. "git clone --depth > > > > > > > > > > > 1"? > > > > > > > > > > > > > And if the directory exists I would fetch and checkout. > > > > > > > > > > > > > > > > > > > > > > > > There are two reasons I can think of why I do not like this idea: > > > > > > > > > > > > > > > > > > > > > > > > (a) a git clone of a whole directory would download all versions of the > > > > > > > > > > > > binary whereas we want only a specific version. > > > > > > > > > > > > > > > > > > > > > > You mention shallow clone yourself, and I used --depth 1 above. > > > > > > > > > > > > > > > > > > > > > > > Downloading a single file > > > > > > > > > > > > by shallow cloning or creating a git archive is overkill IMHO when a wget > > > > > > > > > > > > style retrieval works just fine. > > > > > > > > > > > > > > > > > > > > > > However, it does not provide for versioning, tagging etc so you have > > > > > > > > > > > to implement your own schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hmm I’m not sure if we need all that. Bits has its own versioning mechanism and > > > > > > > > > > > I think all we need to do is maintain the same versioning logic and maintain > > > > > > > > > > > binaries of different versions. Do we really need the power of git/version > > > > > > > > > > > control here? Dunno. > > > > > > > > > > > > > > > > > > > > Well we need some schema. Given we are not using official bits releases > > > > > > > > > > I don't think we can reuse theirs. > > > > > > > > > > > > > > > > > > OK fine. Lets figuire out how to push bits somewhere in git.qemu.org and > > > > > > > > > the binaries in some other repo first. Everything else hinges on that. We > > > > > > > > > can fix the rest of the bits later incrementally. > > > > > > > > > > > > > > > > DanPB, any thoughts on putting bits on git.qemu.org or where and how to > > > > > > > > keep the binaries? > > > > > > > > > > > > > > Can we please conclude on this? > > > > > > > Peter, can you please fork the repo? I have tried many times to reach > > > > > > > you on IRC but failed. > > > > > > > > > > > > Probably because of travel around KVM forum. > > > > > > > > > > > > I think given our CI is under pressure again due to gitlab free tier > > > > > > limits, tying binaries to CI isn't a great idea at this stage. > > > > > > Can Ani just upload binaies to qemu.org for now? > > > > > > > > > > I agree with Michael here. Having a full ci/cd job for this is > > > > > overkill IMHO. We should create a repo just for the binaries, have a > > > > > README there to explain how we generate them and check in new versions > > > > > as and when needed (it won't be frequent). > > > > > How about biosbits-bin repo? > > > > > > > > If QEMU is hosting binaries, where any part contains GPL code, then we > > > > need to be providing the full and corresponding source and the build > > > > scripts needed to re-create the binary. Once we have such scripts it > > > > should be trivial to trigger that from a CI job. If it isn't then > > > > we're doing something wrong. The CI quota is not an issue, because > > > > this is not a job that we need to run continuously. It can be triggered > > > > manually as & when we decide we need to refresh the binary, so would > > > > be a small one-off CI quota hit. > > > > > > > > Also note that gitlab is intending to start enforcing storage quota > > > > on projects in the not too distant future. This makes it unappealing > > > > to store binaries in git repos, unless we genuinely need the ability > > > > to access historical versions of the binary. I don't believe we need > > > > that for biosbits. > > > > > > > > The binary can be published as a CI artifact and accessed directly > > > > from the latest artifact download link. This ensures we only consume > > > > quota for the most recently published binary artifact. So I don't see > > > > a compelling reason to upload binaries into git. > > > > > > I don't really care where we upload them but only having the > > > latest version is just going to break anything expecting > > > the old binary. > > > > biosbits isn't tied to QEMU versions, it is an entirely separate 3rd > > party project. This binary is just providing the test env, and IIUC, > > control over what executes in this env is still done by the QEMU side > > test scripts. I'm not seeing a coupling here that requires precise > > matching. In any case biosbit is a dead project so does not look > > likely to have any changes. > > > > If we did want to have different versions though, we can stil > > publish artifacts from different branches of biosbits code. > > No, that is just ridiculous. Say we have a bug in bits that we fixed > and released a new version. Do we now create a new branch for that? > Multiple branches makes things needlessly complicated. We have one > branch, qemu-bits and all fixes go into that branch. We can have > different tags if we need. Nothing beyond that. I didn't mean we needed different branches for bug fixes. We would only need different branches if biosbits framework changed in some way that was incompatible with how QEMU used it previously. I find that unlikely to happen given its a dead project. We'll almost certainly be fine with one branch taking bug fixes. We do have the option for multiple branches in the unlikely even we find we need it. With regards, Daniel
On 28/09/2022 09.06, Daniel P. Berrangé wrote: > On Tue, Sep 27, 2022 at 06:09:22PM -0400, Michael S. Tsirkin wrote: >> On Tue, Sep 27, 2022 at 11:44:56PM +0200, Paolo Bonzini wrote: >>> I also second the idea of using avocado instead of pytest, by the way. > > snip > >> Problem is I don't think avocado is yet at the level where I can >> ask random developers to use it to check their ACPI patches. >> >> I just went ahead and rechecked and the situation isn't much better >> yet. I think the focus of avocado is system testing of full guests with >> KVM, not unit testing of ACPI. >> >> Let's start with installation on a clean box: > > ...snip... > > Do not do any of this stuff, it is irrelevant to QEMU's needs. > A developer using Avocado with QEMU does nothing more than: > > make check-avocado Right. And if you want to run individual tests, you can also do it like this: make check-venv # Only for the first time ./tests/venv/bin/avocado run tests/avocado/boot_linux.py Or run tests via tags (very convenient for maintainers): ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/ HTH, Thomas
On 27/09/2022 23.21, Michael S. Tsirkin wrote: > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: >> On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: ... >>> Alright, .gitlab-ci.yml is produced and the pipeline succeeds. >>> However, the question still remains, where do we keep the generated >>> artifacts? >> >> The following link will always reflect the published artifacts from >> the most recently fully successful CI pipeline, on the 'qemu-bits' >> branch, and 'qemu-bits-build' CI job: >> >> https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build >> >> Tweak as needed if you push the CI to master branch instead. This >> link can be considered the permanent home of the artifact. I'd just >> suggest that the QEMU job automatically skip if it fails to download >> the artifact, as occassionally transient infra errors can impact >> it. > > This just means once we change the test old qemu source can no longer use it. > Why is this a good idea? Are we so short on disk space? I thought CPU > is the limiting factor? FYI, we'll soon be short on disk space, gitlab plans to introduce storage limits: https://about.gitlab.com/pricing/faq-paid-storage-transfer/ Thomas
On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > ... > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > However, the question still remains, where do we keep the generated > > > > artifacts? > > > > > > The following link will always reflect the published artifacts from > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > branch, and 'qemu-bits-build' CI job: > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > link can be considered the permanent home of the artifact. I'd just > > > suggest that the QEMU job automatically skip if it fails to download > > > the artifact, as occassionally transient infra errors can impact > > > it. > > > > This just means once we change the test old qemu source can no longer use it. > > Why is this a good idea? Are we so short on disk space? I thought CPU > > is the limiting factor? > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > limits: > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ That's the key reason I prefer the binary as CI artifact rather than in Git. Once checked into git, you can never reclaim that storage usage, as the git repo is append only, only option is to delete the repo and recreate. With CI artifacts we can control exactly which binaries consume storage quota at any time. With regards, Daniel
On Wed, Sep 28, 2022 at 07:58:46AM +0100, Daniel P. Berrangé wrote: > biosbits isn't tied to QEMU versions, it is an entirely separate 3rd > party project. This binary is just providing the test env, and IIUC, > control over what executes in this env is still done by the QEMU side > test scripts. I'm not seeing a coupling here that requires precise > matching. In any case biosbit is a dead project so does not look > likely to have any changes. > > If we did want to have different versions though, we can stil > publish artifacts from different branches of biosbits code. Gitlab > will preserve & publish the latest artifacts from each branch in > parallel. > > With regards, > Daniel The issue is bugs, testing, support. If biosbits needs to support old qemu versions that's a ton of work for no real benefit. CI is CI, it's not a binary distribution mechanism, abusing it will just lead to problems down the road... my $.02
On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > ... > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > However, the question still remains, where do we keep the generated > > > > artifacts? > > > > > > The following link will always reflect the published artifacts from > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > branch, and 'qemu-bits-build' CI job: > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > link can be considered the permanent home of the artifact. I'd just > > > suggest that the QEMU job automatically skip if it fails to download > > > the artifact, as occassionally transient infra errors can impact > > > it. > > > > This just means once we change the test old qemu source can no longer use it. > > Why is this a good idea? Are we so short on disk space? I thought CPU > > is the limiting factor? > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > limits: > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ > > Thomas A good reason not to use CI artifacts to store images maybe? I was proposing storing binaries on qemu.org not on gitlab.
On 28/09/2022 11.35, Michael S. Tsirkin wrote: > On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: >> On 27/09/2022 23.21, Michael S. Tsirkin wrote: >>> On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: >>>> On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: >> ... >>>>> Alright, .gitlab-ci.yml is produced and the pipeline succeeds. >>>>> However, the question still remains, where do we keep the generated >>>>> artifacts? >>>> >>>> The following link will always reflect the published artifacts from >>>> the most recently fully successful CI pipeline, on the 'qemu-bits' >>>> branch, and 'qemu-bits-build' CI job: >>>> >>>> https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build >>>> >>>> Tweak as needed if you push the CI to master branch instead. This >>>> link can be considered the permanent home of the artifact. I'd just >>>> suggest that the QEMU job automatically skip if it fails to download >>>> the artifact, as occassionally transient infra errors can impact >>>> it. >>> >>> This just means once we change the test old qemu source can no longer use it. >>> Why is this a good idea? Are we so short on disk space? I thought CPU >>> is the limiting factor? >> >> FYI, we'll soon be short on disk space, gitlab plans to introduce storage >> limits: >> >> https://about.gitlab.com/pricing/faq-paid-storage-transfer/ >> >> Thomas > > A good reason not to use CI artifacts to store images maybe? > I was proposing storing binaries on qemu.org not on gitlab. For qemu.org, you should maybe talk to Paolo and Stefan first, I'm not sure whether we could allow additional network traffic beside the normal release tarballs there... Thomas
On Wed, Sep 28, 2022 at 09:35:59AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > > ... > > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > > However, the question still remains, where do we keep the generated > > > > > artifacts? > > > > > > > > The following link will always reflect the published artifacts from > > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > > branch, and 'qemu-bits-build' CI job: > > > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > > link can be considered the permanent home of the artifact. I'd just > > > > suggest that the QEMU job automatically skip if it fails to download > > > > the artifact, as occassionally transient infra errors can impact > > > > it. > > > > > > This just means once we change the test old qemu source can no longer use it. > > > Why is this a good idea? Are we so short on disk space? I thought CPU > > > is the limiting factor? > > > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > > limits: > > > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ > > That's the key reason I prefer the binary as CI artifact rather than > in Git. Once checked into git, you can never reclaim that storage > usage, as the git repo is append only, only option is to delete the > repo and recreate. With CI artifacts we can control exactly which > binaries consume storage quota at any time. > > With regards, > Daniel I agree binaries in git are a bit of a hack. But I also feel managing files as part of a test tool is a hack too, it's an SCM issue. How about e.g. git-lfs? Seems to be reasonably well supported on gitlab. There's also gitlab but that seems to be older. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Sep 28, 2022 at 11:39:36AM +0200, Thomas Huth wrote: > On 28/09/2022 11.35, Michael S. Tsirkin wrote: > > On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > > > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > > > ... > > > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > > > However, the question still remains, where do we keep the generated > > > > > > artifacts? > > > > > > > > > > The following link will always reflect the published artifacts from > > > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > > > branch, and 'qemu-bits-build' CI job: > > > > > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > > > link can be considered the permanent home of the artifact. I'd just > > > > > suggest that the QEMU job automatically skip if it fails to download > > > > > the artifact, as occassionally transient infra errors can impact > > > > > it. > > > > > > > > This just means once we change the test old qemu source can no longer use it. > > > > Why is this a good idea? Are we so short on disk space? I thought CPU > > > > is the limiting factor? > > > > > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > > > limits: > > > > > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ > > > > > > Thomas > > > > A good reason not to use CI artifacts to store images maybe? > > I was proposing storing binaries on qemu.org not on gitlab. > > For qemu.org, you should maybe talk to Paolo and Stefan first, I'm not sure > whether we could allow additional network traffic > beside the normal release tarballs there... > > Thomas I guess we need to design this sensibly to checksum local files and only fetch if there's change, and that only for people who work on ACPI.
On Wed, 28 Sep 2022, Michael S. Tsirkin wrote: > On Wed, Sep 28, 2022 at 11:39:36AM +0200, Thomas Huth wrote: > > On 28/09/2022 11.35, Michael S. Tsirkin wrote: > > > On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > > > > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > > > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > > > > ... > > > > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > > > > However, the question still remains, where do we keep the generated > > > > > > > artifacts? > > > > > > > > > > > > The following link will always reflect the published artifacts from > > > > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > > > > branch, and 'qemu-bits-build' CI job: > > > > > > > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > > > > link can be considered the permanent home of the artifact. I'd just > > > > > > suggest that the QEMU job automatically skip if it fails to download > > > > > > the artifact, as occassionally transient infra errors can impact > > > > > > it. > > > > > > > > > > This just means once we change the test old qemu source can no longer use it. > > > > > Why is this a good idea? Are we so short on disk space? I thought CPU > > > > > is the limiting factor? I did some expriments and it seems we can keep latest artifacts for every tagged release of bits. So I have adjusted the yaml file so that everytime I push a new tag, a build is triggered and the artifacts are preserved without expiry. Ofcourse for non-tagged changes, one can trigger the build manually from the web UI as well. For exmaple, this link https://gitlab.com/qemu-project/biosbits-bits/-/jobs/3134519120/artifacts/download?file_type=archive should download the current artifacts for the tag qemu-bits-latest. What I am not sure is how to get a downloadable link for the latest build for a particular tag without the numeric job ID (which can change across builds)? So for example, we can have a consistent URLs to download archives for every tagged releases and then the test can choose which tagged release to use. If we can have this then its as good as keeping binaries in a version control system like git. > > > > > > > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > > > > limits: > > > > > > > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ > > > > > > > > Thomas > > > > > > A good reason not to use CI artifacts to store images maybe? > > > I was proposing storing binaries on qemu.org not on gitlab. > > > > For qemu.org, you should maybe talk to Paolo and Stefan first, I'm not sure > > whether we could allow additional network traffic > > beside the normal release tarballs there... > > > > Thomas > > I guess we need to design this sensibly to checksum local files > and only fetch if there's change, and that only for > people who work on ACPI. > > -- > MST > >
On Thu, 6 Oct 2022, Ani Sinha wrote: > > > On Wed, 28 Sep 2022, Michael S. Tsirkin wrote: > > > On Wed, Sep 28, 2022 at 11:39:36AM +0200, Thomas Huth wrote: > > > On 28/09/2022 11.35, Michael S. Tsirkin wrote: > > > > On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > > > > > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > > > > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > > > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > > > > > ... > > > > > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > > > > > However, the question still remains, where do we keep the generated > > > > > > > > artifacts? > > > > > > > > > > > > > > The following link will always reflect the published artifacts from > > > > > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > > > > > branch, and 'qemu-bits-build' CI job: > > > > > > > > > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > > > > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > > > > > link can be considered the permanent home of the artifact. I'd just > > > > > > > suggest that the QEMU job automatically skip if it fails to download > > > > > > > the artifact, as occassionally transient infra errors can impact > > > > > > > it. > > > > > > > > > > > > This just means once we change the test old qemu source can no longer use it. > > > > > > Why is this a good idea? Are we so short on disk space? I thought CPU > > > > > > is the limiting factor? > > > I did some expriments and it seems we can keep latest artifacts for every > tagged release of bits. So I have adjusted the yaml file so that everytime > I push a new tag, a build is > triggered and the artifacts are preserved without expiry. Ofcourse for > non-tagged changes, one can trigger the build manually from the web UI as > well. > > For exmaple, this link > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/3134519120/artifacts/download?file_type=archive > should download the current artifacts for the tag qemu-bits-latest. > > What I am not sure is how to get a downloadable link for the latest build > for a particular tag without the numeric job ID (which can change across > builds)? So for example, we can have a consistent URLs to download > archives > for every tagged releases and then the test can choose which tagged > release to > use. If we can have this then its as good as keeping binaries in a version > control system like git. To answer my own question, this is the URL for the qemu-bits-latest tag: https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits-latest/download?job=qemu-bits-build which is the same as https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits-09272022/download?job=qemu-bits-build currently. If the latest version of bits changes, we can make "qemu-bits-latest" tag always point to the latest version while artifacts for the older tagged releases will continue to be available. danPB, please correct if I am mistaken. > > > > > > > > > > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > > > > > limits: > > > > > > > > > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ > > > > > > > > > > Thomas > > > > > > > > A good reason not to use CI artifacts to store images maybe? > > > > I was proposing storing binaries on qemu.org not on gitlab. > > > > > > For qemu.org, you should maybe talk to Paolo and Stefan first, I'm not sure > > > whether we could allow additional network traffic > > > beside the normal release tarballs there... > > > > > > Thomas > > > > I guess we need to design this sensibly to checksum local files > > and only fetch if there's change, and that only for > > people who work on ACPI. > > > > -- > > MST > > > >
On Thu, 6 Oct 2022, Ani Sinha wrote: > > > On Thu, 6 Oct 2022, Ani Sinha wrote: > > > > > > > On Wed, 28 Sep 2022, Michael S. Tsirkin wrote: > > > > > On Wed, Sep 28, 2022 at 11:39:36AM +0200, Thomas Huth wrote: > > > > On 28/09/2022 11.35, Michael S. Tsirkin wrote: > > > > > On Wed, Sep 28, 2022 at 10:31:39AM +0200, Thomas Huth wrote: > > > > > > On 27/09/2022 23.21, Michael S. Tsirkin wrote: > > > > > > > On Tue, Sep 27, 2022 at 04:45:09PM +0100, Daniel P. Berrangé wrote: > > > > > > > > On Tue, Sep 27, 2022 at 07:35:13PM +0530, Ani Sinha wrote: > > > > > > ... > > > > > > > > > Alright, .gitlab-ci.yml is produced and the pipeline succeeds. > > > > > > > > > However, the question still remains, where do we keep the generated > > > > > > > > > artifacts? > > > > > > > > > > > > > > > > The following link will always reflect the published artifacts from > > > > > > > > the most recently fully successful CI pipeline, on the 'qemu-bits' > > > > > > > > branch, and 'qemu-bits-build' CI job: > > > > > > > > > > > > > > > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits/download?job=qemu-bits-build > > > > > > > > > > > > > > > > Tweak as needed if you push the CI to master branch instead. This > > > > > > > > link can be considered the permanent home of the artifact. I'd just > > > > > > > > suggest that the QEMU job automatically skip if it fails to download > > > > > > > > the artifact, as occassionally transient infra errors can impact > > > > > > > > it. > > > > > > > > > > > > > > This just means once we change the test old qemu source can no longer use it. > > > > > > > Why is this a good idea? Are we so short on disk space? I thought CPU > > > > > > > is the limiting factor? > > > > > > I did some expriments and it seems we can keep latest artifacts for every > > tagged release of bits. So I have adjusted the yaml file so that everytime > > I push a new tag, a build is > > triggered and the artifacts are preserved without expiry. Ofcourse for > > non-tagged changes, one can trigger the build manually from the web UI as > > well. > > > > For exmaple, this link > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/3134519120/artifacts/download?file_type=archive > > should download the current artifacts for the tag qemu-bits-latest. > > > > What I am not sure is how to get a downloadable link for the latest build > > for a particular tag without the numeric job ID (which can change across > > builds)? So for example, we can have a consistent URLs to download > > archives > > for every tagged releases and then the test can choose which tagged > > release to > > use. If we can have this then its as good as keeping binaries in a version > > control system like git. > > > To answer my own question, this is the URL for the qemu-bits-latest tag: > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits-latest/download?job=qemu-bits-build > > which is the same as > > https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits-09272022/download?job=qemu-bits-build > > currently. > > If the latest version of bits changes, we can make "qemu-bits-latest" tag > always point to the latest version while artifacts for the older tagged > releases will continue to be available. > > danPB, please correct if I am mistaken. Seems this answers it: https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#access-the-latest-job-artifacts-by-url So even we can download individual files from the artifact: https://gitlab.com/qemu-project/biosbits-bits/-/jobs/artifacts/qemu-bits-latest/raw/bits-2020-e40af4a7-grub.tar.gz?job=qemu-bits-build where the ref can be a tag or a branch. This makes me happy. > > > > > > > > > > > > > > > FYI, we'll soon be short on disk space, gitlab plans to introduce storage > > > > > > limits: > > > > > > > > > > > > https://about.gitlab.com/pricing/faq-paid-storage-transfer/ > > > > > > > > > > > > Thomas > > > > > > > > > > A good reason not to use CI artifacts to store images maybe? > > > > > I was proposing storing binaries on qemu.org not on gitlab. > > > > > > > > For qemu.org, you should maybe talk to Paolo and Stefan first, I'm not sure > > > > whether we could allow additional network traffic > > > > beside the normal release tarballs there... > > > > > > > > Thomas > > > > > > I guess we need to design this sensibly to checksum local files > > > and only fetch if there's change, and that only for > > > people who work on ACPI. > > > > > > -- > > > MST > > > > > >
On Wed, Sep 28, 2022 at 11:36:51AM +0530, Ani Sinha wrote: > On Wed, Sep 28, 2022 at 9:28 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Sep 28, 2022 at 08:38:54AM +0530, Ani Sinha wrote: > > > > I don't really care where we upload them but only having the > > > > latest version is just going to break anything expecting > > > > the old binary. > > > > > > In fairness, I am not entirely certain if there is a tight coupling > > > between the qemu tests and the bits binaries. I have written the test > > > framework in a way such that test modifications and new tests can be > > > pushed into the bits binaries and the iso gets regenerated with the > > > new tests from QEMU itself before running the tests. Only when we need > > > bits bugfixes or say upgrade to new acpica that we would need to > > > regenerate the bits binaries. > > > > Theoretically, that's correct. But if we did not have bugs we would > > not need tests. > > Hmm, you might have a point. Curious, do we keep versioned binaries of > edk for example? We keep them in git, don't we? Under pc-bios. > If so, why we can't do the same for bits? What can I say - let's get this upstream in some form and then worry about improving the infrastructure.
On Wed, Sep 28, 2022 at 1:14 PM Thomas Huth <thuth@redhat.com> wrote: > > On 28/09/2022 09.06, Daniel P. Berrangé wrote: > > On Tue, Sep 27, 2022 at 06:09:22PM -0400, Michael S. Tsirkin wrote: > >> On Tue, Sep 27, 2022 at 11:44:56PM +0200, Paolo Bonzini wrote: > >>> I also second the idea of using avocado instead of pytest, by the way. > > > > snip > > > >> Problem is I don't think avocado is yet at the level where I can > >> ask random developers to use it to check their ACPI patches. > >> > >> I just went ahead and rechecked and the situation isn't much better > >> yet. I think the focus of avocado is system testing of full guests with > >> KVM, not unit testing of ACPI. > >> > >> Let's start with installation on a clean box: > > > > ...snip... > > > > Do not do any of this stuff, it is irrelevant to QEMU's needs. > > A developer using Avocado with QEMU does nothing more than: > > > > make check-avocado > > Right. And if you want to run individual tests, you can also do it like this: > > make check-venv # Only for the first time > ./tests/venv/bin/avocado run tests/avocado/boot_linux.py Ok this seems to work after I did a pip3 install of avocado in the host. ./tests/venv/bin/avocado run tests/avocado/version.py JOB ID : 8dd90b1cb5baf3780cc764ca4a1ae838374a0a5f JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-09T10.48-8dd90b1/job.log (1/1) tests/avocado/version.py:Version.test_qmp_human_info_version: PASS (0.04 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 3.51 s I see that the output is not tap compliant like the qtests tests are. how do I choose tap? ./tests/venv/bin/avocado-runner-tap --help usage: avocado-runner-tap [-h] {capabilities,runnable-run,runnable-run-recipe,task-run,task-run-recipe} ... nrunner application for executable tests that produce TAP positional arguments: {capabilities,runnable-run,runnable-run-recipe,task-run,task-run-recipe} capabilities Outputs capabilities, including runnables and commands runnable-run Runs a runnable definition from arguments runnable-run-recipe Runs a runnable definition from a recipe task-run Runs a task from arguments task-run-recipe Runs a task from a recipe options: -h, --help show this help message and exit > > Or run tests via tags (very convenient for maintainers): > > ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/ > > HTH, > Thomas >
On Sun, Oct 9, 2022 at 10:51 AM Ani Sinha <ani@anisinha.ca> wrote: > > On Wed, Sep 28, 2022 at 1:14 PM Thomas Huth <thuth@redhat.com> wrote: > > > > > > > Do not do any of this stuff, it is irrelevant to QEMU's needs. > > > A developer using Avocado with QEMU does nothing more than: > > > > > > make check-avocado > > > > Right. And if you want to run individual tests, you can also do it like this: > > > > make check-venv # Only for the first time > > ./tests/venv/bin/avocado run tests/avocado/boot_linux.py > > Ok this seems to work after I did a pip3 install of avocado in the host. > > ./tests/venv/bin/avocado run tests/avocado/version.py > JOB ID : 8dd90b1cb5baf3780cc764ca4a1ae838374a0a5f > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-09T10.48-8dd90b1/job.log > (1/1) tests/avocado/version.py:Version.test_qmp_human_info_version: > PASS (0.04 s) > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > | CANCEL 0 > JOB TIME : 3.51 s > > I see that the output is not tap compliant like the qtests tests are. > how do I choose tap? > > ./tests/venv/bin/avocado-runner-tap --help > usage: avocado-runner-tap [-h] > {capabilities,runnable-run,runnable-run-recipe,task-run,task-run-recipe} > ... > > nrunner application for executable tests that produce TAP > > positional arguments: > {capabilities,runnable-run,runnable-run-recipe,task-run,task-run-recipe} > capabilities Outputs capabilities, including runnables and commands > runnable-run Runs a runnable definition from arguments > runnable-run-recipe > Runs a runnable definition from a recipe > task-run Runs a task from arguments > task-run-recipe Runs a task from a recipe > > options: > -h, --help show this help message and exit > Never mind $ ./tests/venv/bin/avocado run tests/avocado/version.py --tap - 1..1 ok 1 tests/avocado/version.py:Version.test_qmp_human_info_version from https://avocado-framework.readthedocs.io/en/52.0/ResultFormats.html .
On Wed, Sep 28, 2022 at 1:14 PM Thomas Huth <thuth@redhat.com> wrote: > <snip> > > > > Do not do any of this stuff, it is irrelevant to QEMU's needs. > > A developer using Avocado with QEMU does nothing more than: > > > > make check-avocado > > Right. And if you want to run individual tests, you can also do it like this: > > make check-venv # Only for the first time > ./tests/venv/bin/avocado run tests/avocado/boot_linux.py > > Or run tests via tags (very convenient for maintainers): > > ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/ yeah this seems to work! $ ./tests/venv/bin/avocado run -t acpi tests/avocado ERROR: Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92) ERROR: Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47) ERROR: Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158) Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : 328a83d7d0ea628ea8054f16fe2065826d4481e9 JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-09T21.31-328a83d/job.log (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (37.15 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 43.39 s $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap - ERROR: Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92) ERROR: Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47) ERROR: Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158) 1..1 ok 1 tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
diff --git a/tests/pytest/acpi-bits/acpi-bits-test-venv.sh b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh new file mode 100644 index 0000000000..186395473b --- /dev/null +++ b/tests/pytest/acpi-bits/acpi-bits-test-venv.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Generates a python virtual environment for the test to run. +# Then runs python test scripts from within that virtual environment. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Author: Ani Sinha <ani@anisinha.ca> + +set -e + +MYPATH=$(realpath ${BASH_SOURCE:-$0}) +MYDIR=$(dirname $MYPATH) + +if [ -z "$PYTEST_SOURCE_ROOT" ]; then + echo -n "Please set QTEST_SOURCE_ROOT env pointing" + echo " to the root of the qemu source tree." + echo -n "This is required so that the test can find the " + echo "python modules that it needs for execution." + exit 1 +fi +SRCDIR=$PYTEST_SOURCE_ROOT +TESTSCRIPTS=("acpi-bits-test.py") +PIPCMD="-m pip -q --disable-pip-version-check" +# we need to save the old value of PWD before we do a change-dir later +PYTEST_PWD=$PWD + +TESTS_PYTHON=/usr/bin/python3 +TESTS_VENV_REQ=requirements.txt + +# sadly for pip -e and -t options do not work together. +# please see https://github.com/pypa/pip/issues/562 +cd $MYDIR + +$TESTS_PYTHON -m venv . +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/ +[ -f $TESTS_VENV_REQ ] && \ + $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ || exit 0 + +# venv is activated at this point. + +# run the test +for testscript in ${TESTSCRIPTS[@]} ; do + export PYTEST_PWD; python3 $testscript +done + +cd $PYTEST_PWD + +exit 0 diff --git a/tests/pytest/acpi-bits/acpi-bits-test.py b/tests/pytest/acpi-bits/acpi-bits-test.py new file mode 100644 index 0000000000..97e61eb709 --- /dev/null +++ b/tests/pytest/acpi-bits/acpi-bits-test.py @@ -0,0 +1,382 @@ +#!/usr/bin/env python3 +# group: rw quick +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits, +# https://biosbits.org/ +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Some parts are slightly taken from qtest.py and iotests.py +# +# Authors: +# Ani Sinha <ani@anisinha.ca> + +# pylint: disable=invalid-name + +""" +QEMU bios tests using biosbits available at +https://biosbits.org/. +""" + +import logging +import os +import re +import shutil +import subprocess +import sys +import tarfile +import tempfile +import time +import unittest +from urllib import request +import zipfile +from typing import ( + List, + Optional, + Sequence, +) +from tap import TAPTestRunner +from qemu.machine import QEMUMachine + +PYTESTQEMUBIN = os.getenv('PYTEST_QEMU_BINARY') +PYTEST_PWD = os.getenv('PYTEST_PWD') + +def get_arch(): + """finds the arch from the qemu binary name""" + match = re.search('.*qemu-system-(.*)', PYTESTQEMUBIN) + if match: + return match.group(1) + return 'x86_64' + +ARCH = get_arch() + +class QEMUBitsMachine(QEMUMachine): + """ + A QEMU VM, with isa-debugcon enabled and bits iso passed + using -cdrom to QEMU commandline. + """ + def __init__(self, + binary: str, + args: Sequence[str] = (), + wrapper: Sequence[str] = (), + name: Optional[str] = None, + base_temp_dir: str = "/var/tmp", + debugcon_log: str = "debugcon-log.txt", + debugcon_addr: str = "0x403", + sock_dir: Optional[str] = None, + qmp_timer: Optional[float] = None): + # pylint: disable=too-many-arguments + + if name is None: + name = "qemu-bits-%d" % os.getpid() + if sock_dir is None: + sock_dir = base_temp_dir + super().__init__(binary, args, wrapper=wrapper, name=name, + base_temp_dir=base_temp_dir, + sock_dir=sock_dir, qmp_timer=qmp_timer) + self.debugcon_log = debugcon_log + self.debugcon_addr = debugcon_addr + self.base_temp_dir = base_temp_dir + + @property + def _base_args(self) -> List[str]: + args = super()._base_args + args.extend([ + '-chardev', + 'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir, + self.debugcon_log), + '-device', + 'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr, + ]) + return args + + def base_args(self): + """return the base argument to QEMU binary""" + return self._base_args + +class AcpiBitsTest(unittest.TestCase): + """ACPI and SMBIOS tests using biosbits.""" + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self._vm = None + self._workDir = None + self._bitsVer = 2100 + self._bitsLoc = "https://github.com/ani-sinha/bits/raw/bits-builds/" + self._debugcon_addr = '0x403' + self._debugcon_log = 'debugcon-log.txt' + logging.basicConfig(level=logging.INFO) + + def copy_bits_config(self): + """ copies the bios bits config file into bits. + """ + config_file = 'bits-cfg.txt' + qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config') + target_config_dir = os.path.join(self._workDir, + 'bits-%d' %self._bitsVer, 'boot') + self.assertTrue(os.path.exists(qemu_bits_config_dir)) + self.assertTrue(os.path.exists(target_config_dir)) + self.assertTrue(os.access(os.path.join(qemu_bits_config_dir, + config_file), os.R_OK)) + shutil.copy2(os.path.join(qemu_bits_config_dir, config_file), + target_config_dir) + logging.info('copied config file %s to %s', + config_file, target_config_dir) + + def copy_test_scripts(self): + """copies the python test scripts into bits. """ + qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests') + target_test_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer, + 'boot', 'python') + + self.assertTrue(os.path.exists(qemu_test_dir)) + self.assertTrue(os.path.exists(target_test_dir)) + + for filename in os.listdir(qemu_test_dir): + if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \ + filename.endswith('.py'): + shutil.copy2(os.path.join(qemu_test_dir, filename), + target_test_dir) + logging.info('copied test file %s to %s', + filename, target_test_dir) + + # now remove the pyc test file if it exists, otherwise the + # changes in the python test script won't be executed. + testfile_pyc = os.path.splitext(filename)[0] + '.pyc' + if os.access(os.path.join(target_test_dir, testfile_pyc), + os.F_OK): + os.remove(os.path.join(target_test_dir, testfile_pyc)) + logging.info('removed compiled file %s', + os.path.join(target_test_dir, testfile_pyc)) + + def fix_mkrescue(self, mkrescue): + """ grub-mkrescue is a bash script with two variables, 'prefix' and + 'libdir'. They must be pointed to the right location so that the + iso can be generated appropriately. We point the two variables to + the directory where we have extracted our pre-built bits grub + tarball. + """ + grub_x86_64_mods = os.path.join(self._workDir, 'grub-inst-x86_64-efi') + grub_i386_mods = os.path.join(self._workDir, 'grub-inst') + + self.assertTrue(os.path.exists(grub_x86_64_mods)) + self.assertTrue(os.path.exists(grub_i386_mods)) + + new_script = "" + with open(mkrescue, 'r') as filehandle: + orig_script = filehandle.read() + new_script = re.sub('(^prefix=)(.*)', + r'\1"%s"' %grub_x86_64_mods, + orig_script, flags=re.M) + new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods, + new_script, flags=re.M) + + with open(mkrescue, 'w') as filehandle: + filehandle.write(new_script) + + def generate_bits_iso(self): + """ Uses grub-mkrescue to generate a fresh bits iso with the python + test scripts + """ + bits_dir = os.path.join(self._workDir, 'bits-%d' %self._bitsVer) + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) + mkrescue_script = os.path.join(self._workDir, + 'grub-inst-x86_64-efi', 'bin', + 'grub-mkrescue') + + self.assertTrue(os.access(mkrescue_script, + os.R_OK | os.W_OK | os.X_OK)) + + self.fix_mkrescue(mkrescue_script) + + logging.info('calling grub-mkrescue to generate the biosbits iso ...') + + try: + if os.getenv('V'): + subprocess.check_call([mkrescue_script, '-o', + iso_file, bits_dir], + stdout=subprocess.DEVNULL) + else: + subprocess.check_call([mkrescue_script, '-o', + iso_file, bits_dir], + stderr=subprocess.DEVNULL, + stdout=subprocess.DEVNULL) + except Exception as e: # pylint: disable=broad-except + self.skipTest("Error while generating the bits iso. " + "Pass V=1 in the environment to get more details. " + + str(e)) + + self.assertTrue(os.access(iso_file, os.R_OK)) + + logging.info('iso file %s successfully generated.', iso_file) + + def setUp(self): + BITS_LOC = os.getenv("PYTEST_BITSLOC") + if BITS_LOC: + prefix = BITS_LOC + else: + prefix = os.path.join(os.getcwd(), 'prebuilt') + if not os.path.isdir(prefix): + os.mkdir(prefix, mode=0o775) + + bits_zip_file = os.path.join(prefix, 'bits-%d.zip' + %self._bitsVer) + grub_tar_file = os.path.join(prefix, + 'bits-%d-grub.tar.gz' %self._bitsVer) + # if the location of the bits binaries has been specified by the user + # and they are not found in that location, skip the test. + if BITS_LOC and not os.access(bits_zip_file, os.F_OK): + self.skipTest("test skipped since biosbits binaries " + + "could not be found in the specified location %s." \ + %BITS_LOC) + if BITS_LOC and not os.access(grub_tar_file, os.F_OK): + self.skipTest("test skipped since biosbits binaries " + + "could not be found in the specified location %s." \ + %BITS_LOC) + + self._workDir = tempfile.mkdtemp(prefix='acpi-bits-', + suffix='.tmp') + logging.info('working dir: %s', self._workDir) + + localArchive = "bits-%d.zip" % self._bitsVer + if not os.access(bits_zip_file, os.F_OK): + logging.info("archive %s not found in %s, downloading ...", + localArchive, bits_zip_file) + try: + req = request.urlopen(self._bitsLoc + localArchive) + with open(os.path.join(prefix, localArchive), + 'wb') as archivef: + archivef.write(req.read()) + except Exception as e: # pylint: disable=broad-except + self.skipTest("test skipped since biosbits binaries " + + "could not be obtained." + str(e)) + else: + logging.info('using locally found %s', localArchive) + + localArchive = "bits-%d-grub.tar.gz" % self._bitsVer + if not os.access(grub_tar_file, os.F_OK): + logging.info("archive %s not found in %s, downloading ...", + localArchive, bits_zip_file) + try: + req = request.urlopen(self._bitsLoc + localArchive) + with open(os.path.join(prefix, localArchive), + 'wb') as archivef: + archivef.write(req.read()) + except Exception as e: # pylint: disable=broad-except + self.skipTest("test skipped since biosbits binaries " + + "could not be obtained." + str(e)) + else: + logging.info('using locally found %s', localArchive) + + # extract the bits software in the temp working directory + with zipfile.ZipFile(bits_zip_file, 'r') as zref: + zref.extractall(self._workDir) + + with tarfile.open(grub_tar_file, 'r') as tarball: + tarball.extractall(self._workDir) + + self.copy_test_scripts() + self.copy_bits_config() + self.generate_bits_iso() + + def parse_log(self): + """parse the log generated by running bits tests and + check for failures. + """ + debugconf = os.path.join(self._workDir, self._debugcon_log) + log = "" + with open(debugconf, 'r') as filehandle: + log = filehandle.read() + + if os.getenv('V'): + print('\nlogs from biosbits follows:') + print('==========================================\n') + print(log) + print('==========================================\n') + + matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*', + log) + for match in matchiter: + # verify that no test cases failed. + self.assertEqual(match.group(3).split()[0], '0', + 'Some bits tests seems to have failed. ' \ + 'Set V=1 in the environment to get the entire ' \ + 'log from bits.') + + def tearDown(self): + if self._vm: + self.assertFalse(not self._vm.is_running) + logging.info('removing the work directory %s', self._workDir) + shutil.rmtree(self._workDir) + + def test_acpi_smbios_bits(self): + """The main test case implementaion.""" + + qemu_bin = PYTESTQEMUBIN + iso_file = os.path.join(self._workDir, 'bits-%d.iso' %self._bitsVer) + + # PYTESTQEMUBIN could be relative to the current directory + if not os.access(PYTESTQEMUBIN, os.X_OK) and PYTEST_PWD: + qemu_bin = os.path.join(PYTEST_PWD, PYTESTQEMUBIN) + + logging.info('QEMU binary used: %s', qemu_bin) + + self.assertTrue(os.access(qemu_bin, os.X_OK)) + self.assertTrue(os.access(iso_file, os.R_OK)) + + self._vm = QEMUBitsMachine(binary=qemu_bin, + base_temp_dir=self._workDir, + debugcon_log=self._debugcon_log, + debugcon_addr=self._debugcon_addr) + + self._vm.add_args('-cdrom', '%s' %iso_file) + + args = " ".join(str(arg) for arg in self._vm.base_args()) + \ + " " + " ".join(str(arg) for arg in self._vm.args) + + logging.info("launching QEMU vm with the following arguments: %s", + args) + + self._vm.launch() + # biosbits has been configured to run all the specified test suites + # in batch mode and then automatically initiate a vm shutdown. + # sleep for maximum of one minute + max_sleep_time = time.monotonic() + 60 + while self._vm.is_running() and time.monotonic() < max_sleep_time: + time.sleep(1) + + self.assertFalse(time.monotonic() > max_sleep_time, + 'The VM seems to have failed to shutdown in time') + + self.parse_log() + +def execute_unittest(argv: List[str], debug: bool = False, + runner: TAPTestRunner = None) -> None: + """Executes unittests within the calling module.""" + + unittest.main(argv=argv, + testRunner=runner, + verbosity=2 if debug else 1, + warnings=None if sys.warnoptions else 'ignore') + +def main(): + """ The main function where execution begins. """ + + assert PYTESTQEMUBIN is not None, \ + "Environment variable PYTEST_QEMU_BINARY required." + + runner = TAPTestRunner() + runner.set_stream(True) + runner.set_format("%s/acpi-bits-test" %ARCH) + execute_unittest(sys.argv, False, runner) + +main() diff --git a/tests/pytest/acpi-bits/meson.build b/tests/pytest/acpi-bits/meson.build new file mode 100644 index 0000000000..099c191d57 --- /dev/null +++ b/tests/pytest/acpi-bits/meson.build @@ -0,0 +1,33 @@ +xorriso = find_program('xorriso', required: true) +if not xorriso.found() + message('xorriso not found ... disabled bits acpi tests.') + subdir_done() +endif + +subdir('bits-tests') +subdir('bits-config') + +test_files = ['acpi-bits-test.py'] +requirements = 'requirements.txt' + +copytestfiles = custom_target('copy test files', + input : test_files, + output : test_files, + command : ['cp', '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) + +requirementsfiles = custom_target('copy py req files', + input : requirements, + output : requirements, + command : ['cp', '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) + +other_deps += [copytestfiles,requirementsfiles] + +pytest_executables += { + 'acpi-bits-test': configure_file(copy:true, + input:'acpi-bits-test-venv.sh', + output:'acpi-bits-test') +} diff --git a/tests/pytest/acpi-bits/requirements.txt b/tests/pytest/acpi-bits/requirements.txt new file mode 100644 index 0000000000..00cdad09ef --- /dev/null +++ b/tests/pytest/acpi-bits/requirements.txt @@ -0,0 +1 @@ +tap.py
This change adds python based test environment that can be used to run pytest from within a virtual environment. A bash script sets up a virtual environment and then runs the python based tests from within that environment. All dependent python packages are installed in the virtual environment using pip python module. QEMU python test modules are also available in the environment for spawning the QEMU based VMs. It also introduces QEMU acpi/smbios biosbits python test script which is run from within the python virtual environment. When the bios bits tests are run, bios bits binaries are downloaded from an external repo/location. Currently, the test points to an external private github repo where the bits archives are checked in. Signed-off-by: Ani Sinha <ani@anisinha.ca> --- tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 +++ tests/pytest/acpi-bits/acpi-bits-test.py | 382 ++++++++++++++++++ tests/pytest/acpi-bits/meson.build | 33 ++ tests/pytest/acpi-bits/requirements.txt | 1 + 4 files changed, 475 insertions(+) create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py create mode 100644 tests/pytest/acpi-bits/meson.build create mode 100644 tests/pytest/acpi-bits/requirements.txt