diff mbox series

[11/12] acpi/tests/bits: add README file for bits qtests

Message ID 20220627072856.1529357-12-ani@anisinha.ca (mailing list archive)
State New, archived
Headers show
Series Introduce new acpi/smbios qtests using biosbits | expand

Commit Message

Ani Sinha June 27, 2022, 7:28 a.m. UTC
The README file is added describing the directory structure and the purpose
of every file it contains. It also describes how to add new tests, make changes
to existing tests or bits config files or regenerate the bits software.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
 1 file changed, 168 insertions(+)
 create mode 100644 tests/qtest/acpi-bits/README

Comments

Michael S. Tsirkin June 27, 2022, 10:26 p.m. UTC | #1
On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> The README file is added describing the directory structure and the purpose
> of every file it contains. It also describes how to add new tests, make changes
> to existing tests or bits config files or regenerate the bits software.
> 
> Signed-off-by: Ani Sinha <ani@anisinha.ca>
> ---
>  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
>  1 file changed, 168 insertions(+)
>  create mode 100644 tests/qtest/acpi-bits/README
> 
> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> new file mode 100644
> index 0000000000..97b15f1665
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/README
> @@ -0,0 +1,168 @@
> +=============================================================================
> +ACPI/SMBIOS QTESTS USING BIOSBITS
> +=============================================================================
> +
> +Biosbits is a software written by Josh Triplett that can be downloaded by
> +visiting https://biosbits.org/. The github codebase can be found here:
> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> +the bios components such as acpi and smbios tables directly through acpica
> +bios interpreter (a freely available C based library written by Intel,
> +downloadable from https://acpica.org/ and is included with biosbits) without an
> +operating system getting involved in between.
> +There are several advantages to directly testing the bios in a real physical
> +machine or VM as opposed to indirectly discovering bios issues through the
> +operating system. For one thing, the OSes tend to hide bios problems from the
> +end user. The other is that we have more control of what we wanted to test
> +and how by directly using acpica interpreter on top of the bios on a running
> +system. More details on the inspiration for developing biosbits and its real
> +life uses can be found in (a) and (b).
> +This directory contains QEMU qtests written in python that exercizes the QEMU
> +bios components using biosbits and reports test failures.
> +
> +These tests use python virtual environment. In debian/ubuntu system, the tests
> +would require python3.8-venv and python3-pip packages to be installed.

Why do we mess with venv and pip? Certainly possible but
what's wrong with using distro provided packages?

> +
> +A brief description of the contents of this directory follows:
> +
> +├── acpi-bits-test.py
> +├── acpi-bits-test-venv.sh
> +├── bits-config
> +│   ├── bits-cfg.txt
> +│   └── meson.build
> +├── bits-tests
> +│   ├── meson.build
> +│   ├── smbios.py
> +│   ├── smilatency.py
> +│   ├── testacpi.py
> +│   └── testcpuid.py
> +├── meson.build
> +├── prebuilt
> +│   ├── bits-2095-grub.tar.gz
> +│   ├── bits-2095.zip
> +│   └── meson.build
> +├── README
> +└── requirements.txt
> +
> +acpi-bits:
> + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> +   environment necessary for the actual python test script to run. Amongst
> +   other things, it makes sure that QEMU python library is available within
> +   that venv so that QEMU machines can be forked. QEMU python library can be
> +   found within python/ directory in QEMU source.
> +   After setting up the virtual environment, it runs the python test script
> +   from within that environment.
> +   If you want to enable verbose mode only for bits test and run make check,
> +   one trick is to add V=1 before the call to execute the python script in
> +   this file.
> + - acpi-bits-test.py: This is the main python test script that generates a
> +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> +   test failures. This is the script one would be interested in if they wanted
> +   to add or change some component of the log parsing, add a new commandline to
> +   how QEMU is spawned etc. Test writers typically would not need to modify
> +   this script unless they wanted to enhance or change the log parsing for
> +   their tests.
> + - requirements.txt: This text file contains the dependent python libraries
> +   needed for the tests to run. If a new dependent library is needed, it would
> +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> +   it when setting up the python virtual environment using pip.
> + - README: This text file.
> +
> +acpi-bits/bits-config:
> +   This location contains biosbits config files that determine how the software
> +   runs the tests.
> + - bits-config.txt: this is the biosbits config file that determines what tests
> +   or actions are performed by bits. The description of the config options are
> +   provided in the file itself.
> +
> +acpi-bits/prebuilt:
> +   This location contains prebuilt biosbits binaries that are used to generate
> +   the bits iso file for testing.
> + - bits-2095.zip: The contents from this zip archive are the main contents of
> +    the iso file that are used for testing. This binary zip archive also
> +    contains the full source of the bits software including the full text of
> +    the license agreement listed here:
> +    https://github.com/biosbits/bits/blob/master/COPYING
> +    The source tarball can be found in this location in the zip file:
> +    boot/src/bits-2095.tar.gz
> +    The additional changes beyond those that are present in the official
> +    biosbits github repository can be found here:
> +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> +
> +    Basically these changes can be categorized into the following:
> +    (a) changes to make sure biosbits builds with the latest gcc compiler
> +    (gcc 9.4).
> +    (b) upgrade of acpica to the latest version (march 2022).
> +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> +    debugcon can be used to collect the logs.
> +    (d) send a gub halt command to shutdown the VM once all the tests have been
> +    executed.
> +
> +    This zip file is automatically generated by the bits build system. It can
> +    be found in the bits build directory and it is suffixed by the bits build
> +    number.
> +    Normally, there is no need to make any alteration to this zip archive
> +    unless one wanted to change the bits software itself (for example to add
> +    a new config option or change actions of existing options or change the
> +    debug IO port etc). When such changes are made and a new biosbits software
> +    is needed to be generated, please refresh this zip archive as well as the
> +    grub tarball at the same time. Tests will break if changes are made to bits
> +    that are incompatible with existing behavior. So care must be taken to make
> +    sure that the change is compatible with existing bits software as much as
> +    possible. When a new config option is introduced for example, bits must
> +    be upadated here first before introducing the new config option through
> +    a later commit.

I don't think playing with source tarballs is a reasonable work environment.
Let's use submodules just like e.g. firmware does?



> + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> +    scripts and modules to the prebuilt directory. These prebuilt grub
> +    artifacts are required in order to generate a bootable bits iso file that
> +    can run tests.
> +    In order to generate this tar archive, please put the following two
> +    directories that can be found in the bits build directory in a single
> +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> +    bits that generated the archive:
> +
> +    grub-inst-x86_64-efi
> +    grub-inst
> +
> +    This version should be the same as the version number of bits that generated
> +    bits-<n>.zip file. In other words, the two files must be in sync and should
> +    come from the same build of biosbits of the same version. Things may still
> +    work if they come from different versions but mixing version numbers are
> +    not recommended.
> +    There is normally no need to regenerate this archive unless some fixes or
> +    changes have gone into the grub that are part of biosbits.
> +
> +acpi-bits/bits-tests:
> +   This directory contains biosbits python based tests that are run from within
> +   the biosbits environment in the spawned VM. New additions of test cases can
> +   be made in the appropriate test file. For example, new acpi tests can go
> +   into testacpi.py and one would call testsuite.add_test() to register the new
> +   test so that it gets executed as a part of the ACPI tests.
> +   It might be occassionally necessary to disable some subtests or add a new
> +   test that belongs to a test suite not already present in this directory. To
> +   do this, please extract the bits source from the zip file mentioned above.
> +   Copy the test suite/script that needs modification (addition of new tests
> +   or disabling them) from boot/python location of the extracted directory
> +   into this directory.
> +
> +   step (a): copy unmodified test script to this directory.
> +   step (b): update meson.build and add this file to the list.
> +   Commit (a) and (b) together in the same commit.
> +
> +   step (c): perform modifications to the test.
> +   Commit (c) in a separate commit.
> +
> +   The test framework will then use your modified test script to run the test.
> +   No further changes would be needed. Please check the logs to make sure that
> +   appropriate changes have taken effect.
> +
> +meson.build files makes sure that the bits qtests are appropriately integrated
> +into the QEMU qtest framework and are run with "make check-qtest".
> +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> +has been built only for this architecture.
> +
> +
> +Author: Ani Sinha <ani@anisinha.ca>
> +
> +References:
> +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> -- 
> 2.25.1
Ani Sinha June 28, 2022, 4:57 a.m. UTC | #2
On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > The README file is added describing the directory structure and the purpose
> > of every file it contains. It also describes how to add new tests, make changes
> > to existing tests or bits config files or regenerate the bits software.
> >
> > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > ---
> >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> >  1 file changed, 168 insertions(+)
> >  create mode 100644 tests/qtest/acpi-bits/README
> >
> > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > new file mode 100644
> > index 0000000000..97b15f1665
> > --- /dev/null
> > +++ b/tests/qtest/acpi-bits/README
> > @@ -0,0 +1,168 @@
> > +=============================================================================
> > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > +=============================================================================
> > +
> > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > +visiting https://biosbits.org/. The github codebase can be found here:
> > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > +the bios components such as acpi and smbios tables directly through acpica
> > +bios interpreter (a freely available C based library written by Intel,
> > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > +operating system getting involved in between.
> > +There are several advantages to directly testing the bios in a real physical
> > +machine or VM as opposed to indirectly discovering bios issues through the
> > +operating system. For one thing, the OSes tend to hide bios problems from the
> > +end user. The other is that we have more control of what we wanted to test
> > +and how by directly using acpica interpreter on top of the bios on a running
> > +system. More details on the inspiration for developing biosbits and its real
> > +life uses can be found in (a) and (b).
> > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > +bios components using biosbits and reports test failures.
> > +
> > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > +would require python3.8-venv and python3-pip packages to be installed.
>
> Why do we mess with venv and pip? Certainly possible but
> what's wrong with using distro provided packages?

There are two things:
(a) We are already using pip and venv for our avocado based
integration tests. Look for TESTS_VENV_DIR in Makefile.include under
tests.
(b) the venv is primarily needed because I wanted to take advantage of
our rich python library that handles QEMU based machines. There are
python qtest libraries as well. These are well tested and used with
integration tests and I wanted to keep the test part of the code
simple by simply reusing them. however, in order to use them, we need
a venv environment within which these qemu python libraries are
installed. Integration tests does the same thing.

A note about my language of choice - python. I gave a lot of thoughts
on this. We do not do a lot of stuff here. All we do is:
(a) generate bits iso.
(b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
(c) collect and analyze logs.

We are not inspecting guest memory or manipulating devices or pci
buses. We do not need the power of C here. We need something that is
simple to write, easy to maintain and understand and can deal with
things like manipulating text files and configs easily. Python seems a
better fit for the role.

>
> > +
> > +A brief description of the contents of this directory follows:
> > +
> > +├── acpi-bits-test.py
> > +├── acpi-bits-test-venv.sh
> > +├── bits-config
> > +│   ├── bits-cfg.txt
> > +│   └── meson.build
> > +├── bits-tests
> > +│   ├── meson.build
> > +│   ├── smbios.py
> > +│   ├── smilatency.py
> > +│   ├── testacpi.py
> > +│   └── testcpuid.py
> > +├── meson.build
> > +├── prebuilt
> > +│   ├── bits-2095-grub.tar.gz
> > +│   ├── bits-2095.zip
> > +│   └── meson.build
> > +├── README
> > +└── requirements.txt
> > +
> > +acpi-bits:
> > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > +   environment necessary for the actual python test script to run. Amongst
> > +   other things, it makes sure that QEMU python library is available within
> > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > +   found within python/ directory in QEMU source.
> > +   After setting up the virtual environment, it runs the python test script
> > +   from within that environment.
> > +   If you want to enable verbose mode only for bits test and run make check,
> > +   one trick is to add V=1 before the call to execute the python script in
> > +   this file.
> > + - acpi-bits-test.py: This is the main python test script that generates a
> > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > +   test failures. This is the script one would be interested in if they wanted
> > +   to add or change some component of the log parsing, add a new commandline to
> > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > +   this script unless they wanted to enhance or change the log parsing for
> > +   their tests.
> > + - requirements.txt: This text file contains the dependent python libraries
> > +   needed for the tests to run. If a new dependent library is needed, it would
> > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > +   it when setting up the python virtual environment using pip.
> > + - README: This text file.
> > +
> > +acpi-bits/bits-config:
> > +   This location contains biosbits config files that determine how the software
> > +   runs the tests.
> > + - bits-config.txt: this is the biosbits config file that determines what tests
> > +   or actions are performed by bits. The description of the config options are
> > +   provided in the file itself.
> > +
> > +acpi-bits/prebuilt:
> > +   This location contains prebuilt biosbits binaries that are used to generate
> > +   the bits iso file for testing.
> > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > +    the iso file that are used for testing. This binary zip archive also
> > +    contains the full source of the bits software including the full text of
> > +    the license agreement listed here:
> > +    https://github.com/biosbits/bits/blob/master/COPYING
> > +    The source tarball can be found in this location in the zip file:
> > +    boot/src/bits-2095.tar.gz
> > +    The additional changes beyond those that are present in the official
> > +    biosbits github repository can be found here:
> > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > +
> > +    Basically these changes can be categorized into the following:
> > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > +    (gcc 9.4).
> > +    (b) upgrade of acpica to the latest version (march 2022).
> > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > +    debugcon can be used to collect the logs.
> > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > +    executed.
> > +
> > +    This zip file is automatically generated by the bits build system. It can
> > +    be found in the bits build directory and it is suffixed by the bits build
> > +    number.
> > +    Normally, there is no need to make any alteration to this zip archive
> > +    unless one wanted to change the bits software itself (for example to add
> > +    a new config option or change actions of existing options or change the
> > +    debug IO port etc). When such changes are made and a new biosbits software
> > +    is needed to be generated, please refresh this zip archive as well as the
> > +    grub tarball at the same time. Tests will break if changes are made to bits
> > +    that are incompatible with existing behavior. So care must be taken to make
> > +    sure that the change is compatible with existing bits software as much as
> > +    possible. When a new config option is introduced for example, bits must
> > +    be upadated here first before introducing the new config option through
> > +    a later commit.
>
> I don't think playing with source tarballs is a reasonable work environment.

I agree. However, we do not do much with the source tarball. It is
there as part of bits to satisfy the license requirement. If we need
to manipulate any test script that is in the source file, we would
copy it over and keep it in the bits-test directory and change it
there.

> Let's use submodules just like e.g. firmware does?

Yes I also proposed this to Igor on IRC. We can/maybe need to figure
out a place to store the bits source if we think my github is not the
best place. However, we need the source only if/when we need to
rebuild bits. I suspect it won't be too often if at all.

>
>
>
> > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > +    artifacts are required in order to generate a bootable bits iso file that
> > +    can run tests.
> > +    In order to generate this tar archive, please put the following two
> > +    directories that can be found in the bits build directory in a single
> > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > +    bits that generated the archive:
> > +
> > +    grub-inst-x86_64-efi
> > +    grub-inst
> > +
> > +    This version should be the same as the version number of bits that generated
> > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > +    come from the same build of biosbits of the same version. Things may still
> > +    work if they come from different versions but mixing version numbers are
> > +    not recommended.
> > +    There is normally no need to regenerate this archive unless some fixes or
> > +    changes have gone into the grub that are part of biosbits.
> > +
> > +acpi-bits/bits-tests:
> > +   This directory contains biosbits python based tests that are run from within
> > +   the biosbits environment in the spawned VM. New additions of test cases can
> > +   be made in the appropriate test file. For example, new acpi tests can go
> > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > +   test so that it gets executed as a part of the ACPI tests.
> > +   It might be occassionally necessary to disable some subtests or add a new
> > +   test that belongs to a test suite not already present in this directory. To
> > +   do this, please extract the bits source from the zip file mentioned above.
> > +   Copy the test suite/script that needs modification (addition of new tests
> > +   or disabling them) from boot/python location of the extracted directory
> > +   into this directory.
> > +
> > +   step (a): copy unmodified test script to this directory.
> > +   step (b): update meson.build and add this file to the list.
> > +   Commit (a) and (b) together in the same commit.
> > +
> > +   step (c): perform modifications to the test.
> > +   Commit (c) in a separate commit.
> > +
> > +   The test framework will then use your modified test script to run the test.
> > +   No further changes would be needed. Please check the logs to make sure that
> > +   appropriate changes have taken effect.
> > +
> > +meson.build files makes sure that the bits qtests are appropriately integrated
> > +into the QEMU qtest framework and are run with "make check-qtest".
> > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > +has been built only for this architecture.
> > +
> > +
> > +Author: Ani Sinha <ani@anisinha.ca>
> > +
> > +References:
> > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > --
> > 2.25.1
>
Michael S. Tsirkin June 28, 2022, 6:06 a.m. UTC | #3
On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > The README file is added describing the directory structure and the purpose
> > > of every file it contains. It also describes how to add new tests, make changes
> > > to existing tests or bits config files or regenerate the bits software.
> > >
> > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > ---
> > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > >  1 file changed, 168 insertions(+)
> > >  create mode 100644 tests/qtest/acpi-bits/README
> > >
> > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > new file mode 100644
> > > index 0000000000..97b15f1665
> > > --- /dev/null
> > > +++ b/tests/qtest/acpi-bits/README
> > > @@ -0,0 +1,168 @@
> > > +=============================================================================
> > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > +=============================================================================
> > > +
> > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > +the bios components such as acpi and smbios tables directly through acpica
> > > +bios interpreter (a freely available C based library written by Intel,
> > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > +operating system getting involved in between.
> > > +There are several advantages to directly testing the bios in a real physical
> > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > +end user. The other is that we have more control of what we wanted to test
> > > +and how by directly using acpica interpreter on top of the bios on a running
> > > +system. More details on the inspiration for developing biosbits and its real
> > > +life uses can be found in (a) and (b).
> > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > +bios components using biosbits and reports test failures.
> > > +
> > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > +would require python3.8-venv and python3-pip packages to be installed.
> >
> > Why do we mess with venv and pip? Certainly possible but
> > what's wrong with using distro provided packages?
> 
> There are two things:
> (a) We are already using pip and venv for our avocado based
> integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> tests.
> (b) the venv is primarily needed because I wanted to take advantage of
> our rich python library that handles QEMU based machines. There are
> python qtest libraries as well. These are well tested and used with
> integration tests and I wanted to keep the test part of the code
> simple by simply reusing them. however, in order to use them, we need
> a venv environment within which these qemu python libraries are
> installed. Integration tests does the same thing.
> 
> A note about my language of choice - python. I gave a lot of thoughts
> on this. We do not do a lot of stuff here. All we do is:
> (a) generate bits iso.
> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> (c) collect and analyze logs.
> 
> We are not inspecting guest memory or manipulating devices or pci
> buses. We do not need the power of C here. We need something that is
> simple to write, easy to maintain and understand and can deal with
> things like manipulating text files and configs easily. Python seems a
> better fit for the role.

No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?

> >
> > > +
> > > +A brief description of the contents of this directory follows:
> > > +
> > > +├── acpi-bits-test.py
> > > +├── acpi-bits-test-venv.sh
> > > +├── bits-config
> > > +│   ├── bits-cfg.txt
> > > +│   └── meson.build
> > > +├── bits-tests
> > > +│   ├── meson.build
> > > +│   ├── smbios.py
> > > +│   ├── smilatency.py
> > > +│   ├── testacpi.py
> > > +│   └── testcpuid.py
> > > +├── meson.build
> > > +├── prebuilt
> > > +│   ├── bits-2095-grub.tar.gz
> > > +│   ├── bits-2095.zip
> > > +│   └── meson.build
> > > +├── README
> > > +└── requirements.txt
> > > +
> > > +acpi-bits:
> > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > +   environment necessary for the actual python test script to run. Amongst
> > > +   other things, it makes sure that QEMU python library is available within
> > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > +   found within python/ directory in QEMU source.
> > > +   After setting up the virtual environment, it runs the python test script
> > > +   from within that environment.
> > > +   If you want to enable verbose mode only for bits test and run make check,
> > > +   one trick is to add V=1 before the call to execute the python script in
> > > +   this file.
> > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > +   test failures. This is the script one would be interested in if they wanted
> > > +   to add or change some component of the log parsing, add a new commandline to
> > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > +   this script unless they wanted to enhance or change the log parsing for
> > > +   their tests.
> > > + - requirements.txt: This text file contains the dependent python libraries
> > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > +   it when setting up the python virtual environment using pip.
> > > + - README: This text file.
> > > +
> > > +acpi-bits/bits-config:
> > > +   This location contains biosbits config files that determine how the software
> > > +   runs the tests.
> > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > +   or actions are performed by bits. The description of the config options are
> > > +   provided in the file itself.
> > > +
> > > +acpi-bits/prebuilt:
> > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > +   the bits iso file for testing.
> > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > +    the iso file that are used for testing. This binary zip archive also
> > > +    contains the full source of the bits software including the full text of
> > > +    the license agreement listed here:
> > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > +    The source tarball can be found in this location in the zip file:
> > > +    boot/src/bits-2095.tar.gz
> > > +    The additional changes beyond those that are present in the official
> > > +    biosbits github repository can be found here:
> > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > +
> > > +    Basically these changes can be categorized into the following:
> > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > +    (gcc 9.4).
> > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > +    debugcon can be used to collect the logs.
> > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > +    executed.
> > > +
> > > +    This zip file is automatically generated by the bits build system. It can
> > > +    be found in the bits build directory and it is suffixed by the bits build
> > > +    number.
> > > +    Normally, there is no need to make any alteration to this zip archive
> > > +    unless one wanted to change the bits software itself (for example to add
> > > +    a new config option or change actions of existing options or change the
> > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > +    is needed to be generated, please refresh this zip archive as well as the
> > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > +    that are incompatible with existing behavior. So care must be taken to make
> > > +    sure that the change is compatible with existing bits software as much as
> > > +    possible. When a new config option is introduced for example, bits must
> > > +    be upadated here first before introducing the new config option through
> > > +    a later commit.
> >
> > I don't think playing with source tarballs is a reasonable work environment.
> 
> I agree. However, we do not do much with the source tarball. It is
> there as part of bits to satisfy the license requirement. If we need
> to manipulate any test script that is in the source file, we would
> copy it over and keep it in the bits-test directory and change it
> there.
> 
> > Let's use submodules just like e.g. firmware does?
> 
> Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> out a place to store the bits source if we think my github is not the
> best place. However, we need the source only if/when we need to
> rebuild bits. I suspect it won't be too often if at all.

Why not on git.qemu.org ?

> >
> >
> >
> > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > +    artifacts are required in order to generate a bootable bits iso file that
> > > +    can run tests.
> > > +    In order to generate this tar archive, please put the following two
> > > +    directories that can be found in the bits build directory in a single
> > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > +    bits that generated the archive:
> > > +
> > > +    grub-inst-x86_64-efi
> > > +    grub-inst
> > > +
> > > +    This version should be the same as the version number of bits that generated
> > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > +    come from the same build of biosbits of the same version. Things may still
> > > +    work if they come from different versions but mixing version numbers are
> > > +    not recommended.
> > > +    There is normally no need to regenerate this archive unless some fixes or
> > > +    changes have gone into the grub that are part of biosbits.
> > > +
> > > +acpi-bits/bits-tests:
> > > +   This directory contains biosbits python based tests that are run from within
> > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > +   test so that it gets executed as a part of the ACPI tests.
> > > +   It might be occassionally necessary to disable some subtests or add a new
> > > +   test that belongs to a test suite not already present in this directory. To
> > > +   do this, please extract the bits source from the zip file mentioned above.
> > > +   Copy the test suite/script that needs modification (addition of new tests
> > > +   or disabling them) from boot/python location of the extracted directory
> > > +   into this directory.
> > > +
> > > +   step (a): copy unmodified test script to this directory.
> > > +   step (b): update meson.build and add this file to the list.
> > > +   Commit (a) and (b) together in the same commit.
> > > +
> > > +   step (c): perform modifications to the test.
> > > +   Commit (c) in a separate commit.
> > > +
> > > +   The test framework will then use your modified test script to run the test.
> > > +   No further changes would be needed. Please check the logs to make sure that
> > > +   appropriate changes have taken effect.
> > > +
> > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > +has been built only for this architecture.
> > > +
> > > +
> > > +Author: Ani Sinha <ani@anisinha.ca>
> > > +
> > > +References:
> > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > --
> > > 2.25.1
> >
Ani Sinha June 28, 2022, 6:16 a.m. UTC | #4
On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > The README file is added describing the directory structure and the purpose
> > > > of every file it contains. It also describes how to add new tests, make changes
> > > > to existing tests or bits config files or regenerate the bits software.
> > > >
> > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > ---
> > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 168 insertions(+)
> > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > >
> > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > new file mode 100644
> > > > index 0000000000..97b15f1665
> > > > --- /dev/null
> > > > +++ b/tests/qtest/acpi-bits/README
> > > > @@ -0,0 +1,168 @@
> > > > +=============================================================================
> > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > +=============================================================================
> > > > +
> > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > +bios interpreter (a freely available C based library written by Intel,
> > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > +operating system getting involved in between.
> > > > +There are several advantages to directly testing the bios in a real physical
> > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > +end user. The other is that we have more control of what we wanted to test
> > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > +system. More details on the inspiration for developing biosbits and its real
> > > > +life uses can be found in (a) and (b).
> > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > +bios components using biosbits and reports test failures.
> > > > +
> > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > +would require python3.8-venv and python3-pip packages to be installed.
> > >
> > > Why do we mess with venv and pip? Certainly possible but
> > > what's wrong with using distro provided packages?
> >
> > There are two things:
> > (a) We are already using pip and venv for our avocado based
> > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > tests.
> > (b) the venv is primarily needed because I wanted to take advantage of
> > our rich python library that handles QEMU based machines. There are
> > python qtest libraries as well. These are well tested and used with
> > integration tests and I wanted to keep the test part of the code
> > simple by simply reusing them. however, in order to use them, we need
> > a venv environment within which these qemu python libraries are
> > installed. Integration tests does the same thing.
> >
> > A note about my language of choice - python. I gave a lot of thoughts
> > on this. We do not do a lot of stuff here. All we do is:
> > (a) generate bits iso.
> > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > (c) collect and analyze logs.
> >
> > We are not inspecting guest memory or manipulating devices or pci
> > buses. We do not need the power of C here. We need something that is
> > simple to write, easy to maintain and understand and can deal with
> > things like manipulating text files and configs easily. Python seems a
> > better fit for the role.
>
> No problem with that. So that's venv. But do we need pip and pulling
> packages from the net during testing?

We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))

>
> > >
> > > > +
> > > > +A brief description of the contents of this directory follows:
> > > > +
> > > > +├── acpi-bits-test.py
> > > > +├── acpi-bits-test-venv.sh
> > > > +├── bits-config
> > > > +│   ├── bits-cfg.txt
> > > > +│   └── meson.build
> > > > +├── bits-tests
> > > > +│   ├── meson.build
> > > > +│   ├── smbios.py
> > > > +│   ├── smilatency.py
> > > > +│   ├── testacpi.py
> > > > +│   └── testcpuid.py
> > > > +├── meson.build
> > > > +├── prebuilt
> > > > +│   ├── bits-2095-grub.tar.gz
> > > > +│   ├── bits-2095.zip
> > > > +│   └── meson.build
> > > > +├── README
> > > > +└── requirements.txt
> > > > +
> > > > +acpi-bits:
> > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > +   environment necessary for the actual python test script to run. Amongst
> > > > +   other things, it makes sure that QEMU python library is available within
> > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > +   found within python/ directory in QEMU source.
> > > > +   After setting up the virtual environment, it runs the python test script
> > > > +   from within that environment.
> > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > +   this file.
> > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > +   test failures. This is the script one would be interested in if they wanted
> > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > +   their tests.
> > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > +   it when setting up the python virtual environment using pip.
> > > > + - README: This text file.
> > > > +
> > > > +acpi-bits/bits-config:
> > > > +   This location contains biosbits config files that determine how the software
> > > > +   runs the tests.
> > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > +   or actions are performed by bits. The description of the config options are
> > > > +   provided in the file itself.
> > > > +
> > > > +acpi-bits/prebuilt:
> > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > +   the bits iso file for testing.
> > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > +    the iso file that are used for testing. This binary zip archive also
> > > > +    contains the full source of the bits software including the full text of
> > > > +    the license agreement listed here:
> > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > +    The source tarball can be found in this location in the zip file:
> > > > +    boot/src/bits-2095.tar.gz
> > > > +    The additional changes beyond those that are present in the official
> > > > +    biosbits github repository can be found here:
> > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > +
> > > > +    Basically these changes can be categorized into the following:
> > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > +    (gcc 9.4).
> > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > +    debugcon can be used to collect the logs.
> > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > +    executed.
> > > > +
> > > > +    This zip file is automatically generated by the bits build system. It can
> > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > +    number.
> > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > +    unless one wanted to change the bits software itself (for example to add
> > > > +    a new config option or change actions of existing options or change the
> > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > +    sure that the change is compatible with existing bits software as much as
> > > > +    possible. When a new config option is introduced for example, bits must
> > > > +    be upadated here first before introducing the new config option through
> > > > +    a later commit.
> > >
> > > I don't think playing with source tarballs is a reasonable work environment.
> >
> > I agree. However, we do not do much with the source tarball. It is
> > there as part of bits to satisfy the license requirement. If we need
> > to manipulate any test script that is in the source file, we would
> > copy it over and keep it in the bits-test directory and change it
> > there.
> >
> > > Let's use submodules just like e.g. firmware does?
> >
> > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > out a place to store the bits source if we think my github is not the
> > best place. However, we need the source only if/when we need to
> > rebuild bits. I suspect it won't be too often if at all.
>
> Why not on git.qemu.org ?

Sounds good to me. Do I get push access? :-)

>
> > >
> > >
> > >
> > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > +    can run tests.
> > > > +    In order to generate this tar archive, please put the following two
> > > > +    directories that can be found in the bits build directory in a single
> > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > +    bits that generated the archive:
> > > > +
> > > > +    grub-inst-x86_64-efi
> > > > +    grub-inst
> > > > +
> > > > +    This version should be the same as the version number of bits that generated
> > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > +    come from the same build of biosbits of the same version. Things may still
> > > > +    work if they come from different versions but mixing version numbers are
> > > > +    not recommended.
> > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > +    changes have gone into the grub that are part of biosbits.
> > > > +
> > > > +acpi-bits/bits-tests:
> > > > +   This directory contains biosbits python based tests that are run from within
> > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > +   test that belongs to a test suite not already present in this directory. To
> > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > +   or disabling them) from boot/python location of the extracted directory
> > > > +   into this directory.
> > > > +
> > > > +   step (a): copy unmodified test script to this directory.
> > > > +   step (b): update meson.build and add this file to the list.
> > > > +   Commit (a) and (b) together in the same commit.
> > > > +
> > > > +   step (c): perform modifications to the test.
> > > > +   Commit (c) in a separate commit.
> > > > +
> > > > +   The test framework will then use your modified test script to run the test.
> > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > +   appropriate changes have taken effect.
> > > > +
> > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > +has been built only for this architecture.
> > > > +
> > > > +
> > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > +
> > > > +References:
> > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > --
> > > > 2.25.1
> > >
>
Michael S. Tsirkin June 28, 2022, 6:20 a.m. UTC | #5
On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > The README file is added describing the directory structure and the purpose
> > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > to existing tests or bits config files or regenerate the bits software.
> > > > >
> > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > ---
> > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > >  1 file changed, 168 insertions(+)
> > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > >
> > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > new file mode 100644
> > > > > index 0000000000..97b15f1665
> > > > > --- /dev/null
> > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > @@ -0,0 +1,168 @@
> > > > > +=============================================================================
> > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > +=============================================================================
> > > > > +
> > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > +operating system getting involved in between.
> > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > +life uses can be found in (a) and (b).
> > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > +bios components using biosbits and reports test failures.
> > > > > +
> > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > >
> > > > Why do we mess with venv and pip? Certainly possible but
> > > > what's wrong with using distro provided packages?
> > >
> > > There are two things:
> > > (a) We are already using pip and venv for our avocado based
> > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > tests.
> > > (b) the venv is primarily needed because I wanted to take advantage of
> > > our rich python library that handles QEMU based machines. There are
> > > python qtest libraries as well. These are well tested and used with
> > > integration tests and I wanted to keep the test part of the code
> > > simple by simply reusing them. however, in order to use them, we need
> > > a venv environment within which these qemu python libraries are
> > > installed. Integration tests does the same thing.
> > >
> > > A note about my language of choice - python. I gave a lot of thoughts
> > > on this. We do not do a lot of stuff here. All we do is:
> > > (a) generate bits iso.
> > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > (c) collect and analyze logs.
> > >
> > > We are not inspecting guest memory or manipulating devices or pci
> > > buses. We do not need the power of C here. We need something that is
> > > simple to write, easy to maintain and understand and can deal with
> > > things like manipulating text files and configs easily. Python seems a
> > > better fit for the role.
> >
> > No problem with that. So that's venv. But do we need pip and pulling
> > packages from the net during testing?
> 
> We do that too. See requirements.txt in tests/
> Following two are downloaded:
> avocado-framework==88.1
> pycdlib==1.11.0
> 
> Also see this line in Makefie.include:
> 
> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))

Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?

> >
> > > >
> > > > > +
> > > > > +A brief description of the contents of this directory follows:
> > > > > +
> > > > > +├── acpi-bits-test.py
> > > > > +├── acpi-bits-test-venv.sh
> > > > > +├── bits-config
> > > > > +│   ├── bits-cfg.txt
> > > > > +│   └── meson.build
> > > > > +├── bits-tests
> > > > > +│   ├── meson.build
> > > > > +│   ├── smbios.py
> > > > > +│   ├── smilatency.py
> > > > > +│   ├── testacpi.py
> > > > > +│   └── testcpuid.py
> > > > > +├── meson.build
> > > > > +├── prebuilt
> > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > +│   ├── bits-2095.zip
> > > > > +│   └── meson.build
> > > > > +├── README
> > > > > +└── requirements.txt
> > > > > +
> > > > > +acpi-bits:
> > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > +   found within python/ directory in QEMU source.
> > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > +   from within that environment.
> > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > +   this file.
> > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > +   their tests.
> > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > +   it when setting up the python virtual environment using pip.
> > > > > + - README: This text file.
> > > > > +
> > > > > +acpi-bits/bits-config:
> > > > > +   This location contains biosbits config files that determine how the software
> > > > > +   runs the tests.
> > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > +   or actions are performed by bits. The description of the config options are
> > > > > +   provided in the file itself.
> > > > > +
> > > > > +acpi-bits/prebuilt:
> > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > +   the bits iso file for testing.
> > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > +    contains the full source of the bits software including the full text of
> > > > > +    the license agreement listed here:
> > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > +    The source tarball can be found in this location in the zip file:
> > > > > +    boot/src/bits-2095.tar.gz
> > > > > +    The additional changes beyond those that are present in the official
> > > > > +    biosbits github repository can be found here:
> > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > +
> > > > > +    Basically these changes can be categorized into the following:
> > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > +    (gcc 9.4).
> > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > +    debugcon can be used to collect the logs.
> > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > +    executed.
> > > > > +
> > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > +    number.
> > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > +    a new config option or change actions of existing options or change the
> > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > +    be upadated here first before introducing the new config option through
> > > > > +    a later commit.
> > > >
> > > > I don't think playing with source tarballs is a reasonable work environment.
> > >
> > > I agree. However, we do not do much with the source tarball. It is
> > > there as part of bits to satisfy the license requirement. If we need
> > > to manipulate any test script that is in the source file, we would
> > > copy it over and keep it in the bits-test directory and change it
> > > there.
> > >
> > > > Let's use submodules just like e.g. firmware does?
> > >
> > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > out a place to store the bits source if we think my github is not the
> > > best place. However, we need the source only if/when we need to
> > > rebuild bits. I suspect it won't be too often if at all.
> >
> > Why not on git.qemu.org ?
> 
> Sounds good to me. Do I get push access? :-)
> 
> >
> > > >
> > > >
> > > >
> > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > +    can run tests.
> > > > > +    In order to generate this tar archive, please put the following two
> > > > > +    directories that can be found in the bits build directory in a single
> > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > +    bits that generated the archive:
> > > > > +
> > > > > +    grub-inst-x86_64-efi
> > > > > +    grub-inst
> > > > > +
> > > > > +    This version should be the same as the version number of bits that generated
> > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > +    work if they come from different versions but mixing version numbers are
> > > > > +    not recommended.
> > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > +
> > > > > +acpi-bits/bits-tests:
> > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > +   into this directory.
> > > > > +
> > > > > +   step (a): copy unmodified test script to this directory.
> > > > > +   step (b): update meson.build and add this file to the list.
> > > > > +   Commit (a) and (b) together in the same commit.
> > > > > +
> > > > > +   step (c): perform modifications to the test.
> > > > > +   Commit (c) in a separate commit.
> > > > > +
> > > > > +   The test framework will then use your modified test script to run the test.
> > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > +   appropriate changes have taken effect.
> > > > > +
> > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > +has been built only for this architecture.
> > > > > +
> > > > > +
> > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > +
> > > > > +References:
> > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > --
> > > > > 2.25.1
> > > >
> >
Ani Sinha June 28, 2022, 6:36 a.m. UTC | #6
On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > The README file is added describing the directory structure and the purpose
> > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > >
> > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > ---
> > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > >  1 file changed, 168 insertions(+)
> > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > >
> > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > new file mode 100644
> > > > > > index 0000000000..97b15f1665
> > > > > > --- /dev/null
> > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > @@ -0,0 +1,168 @@
> > > > > > +=============================================================================
> > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > +=============================================================================
> > > > > > +
> > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > +operating system getting involved in between.
> > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > +life uses can be found in (a) and (b).
> > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > +bios components using biosbits and reports test failures.
> > > > > > +
> > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > >
> > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > what's wrong with using distro provided packages?
> > > >
> > > > There are two things:
> > > > (a) We are already using pip and venv for our avocado based
> > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > tests.
> > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > our rich python library that handles QEMU based machines. There are
> > > > python qtest libraries as well. These are well tested and used with
> > > > integration tests and I wanted to keep the test part of the code
> > > > simple by simply reusing them. however, in order to use them, we need
> > > > a venv environment within which these qemu python libraries are
> > > > installed. Integration tests does the same thing.
> > > >
> > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > on this. We do not do a lot of stuff here. All we do is:
> > > > (a) generate bits iso.
> > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > (c) collect and analyze logs.
> > > >
> > > > We are not inspecting guest memory or manipulating devices or pci
> > > > buses. We do not need the power of C here. We need something that is
> > > > simple to write, easy to maintain and understand and can deal with
> > > > things like manipulating text files and configs easily. Python seems a
> > > > better fit for the role.
> > >
> > > No problem with that. So that's venv. But do we need pip and pulling
> > > packages from the net during testing?
> >
> > We do that too. See requirements.txt in tests/
> > Following two are downloaded:
> > avocado-framework==88.1
> > pycdlib==1.11.0
> >
> > Also see this line in Makefie.include:
> >
> > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>
> Right but that's avocado since it pulls lots of stuff from
> the net anyway.
> Are the libraries in question not packaged on major distros?

Currently I only need this:
https://github.com/python-tap/tappy
which is the basic TAP processing library for python.

It seems its only installed through pip:
https://tappy.readthedocs.io/en/latest/

I do not think this is packaged by default. It's such a basic library
for parsing test output that maybe we can keep this somewhere within
the python src tree? Not sure ...

>
> > >
> > > > >
> > > > > > +
> > > > > > +A brief description of the contents of this directory follows:
> > > > > > +
> > > > > > +├── acpi-bits-test.py
> > > > > > +├── acpi-bits-test-venv.sh
> > > > > > +├── bits-config
> > > > > > +│   ├── bits-cfg.txt
> > > > > > +│   └── meson.build
> > > > > > +├── bits-tests
> > > > > > +│   ├── meson.build
> > > > > > +│   ├── smbios.py
> > > > > > +│   ├── smilatency.py
> > > > > > +│   ├── testacpi.py
> > > > > > +│   └── testcpuid.py
> > > > > > +├── meson.build
> > > > > > +├── prebuilt
> > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > +│   ├── bits-2095.zip
> > > > > > +│   └── meson.build
> > > > > > +├── README
> > > > > > +└── requirements.txt
> > > > > > +
> > > > > > +acpi-bits:
> > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > +   found within python/ directory in QEMU source.
> > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > +   from within that environment.
> > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > +   this file.
> > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > +   their tests.
> > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > + - README: This text file.
> > > > > > +
> > > > > > +acpi-bits/bits-config:
> > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > +   runs the tests.
> > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > +   provided in the file itself.
> > > > > > +
> > > > > > +acpi-bits/prebuilt:
> > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > +   the bits iso file for testing.
> > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > +    contains the full source of the bits software including the full text of
> > > > > > +    the license agreement listed here:
> > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > +    The additional changes beyond those that are present in the official
> > > > > > +    biosbits github repository can be found here:
> > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > +
> > > > > > +    Basically these changes can be categorized into the following:
> > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > +    (gcc 9.4).
> > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > +    debugcon can be used to collect the logs.
> > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > +    executed.
> > > > > > +
> > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > +    number.
> > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > +    a new config option or change actions of existing options or change the
> > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > +    be upadated here first before introducing the new config option through
> > > > > > +    a later commit.
> > > > >
> > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > >
> > > > I agree. However, we do not do much with the source tarball. It is
> > > > there as part of bits to satisfy the license requirement. If we need
> > > > to manipulate any test script that is in the source file, we would
> > > > copy it over and keep it in the bits-test directory and change it
> > > > there.
> > > >
> > > > > Let's use submodules just like e.g. firmware does?
> > > >
> > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > out a place to store the bits source if we think my github is not the
> > > > best place. However, we need the source only if/when we need to
> > > > rebuild bits. I suspect it won't be too often if at all.
> > >
> > > Why not on git.qemu.org ?
> >
> > Sounds good to me. Do I get push access? :-)
> >
> > >
> > > > >
> > > > >
> > > > >
> > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > +    can run tests.
> > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > +    bits that generated the archive:
> > > > > > +
> > > > > > +    grub-inst-x86_64-efi
> > > > > > +    grub-inst
> > > > > > +
> > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > +    not recommended.
> > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > +
> > > > > > +acpi-bits/bits-tests:
> > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > +   into this directory.
> > > > > > +
> > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > +
> > > > > > +   step (c): perform modifications to the test.
> > > > > > +   Commit (c) in a separate commit.
> > > > > > +
> > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > +   appropriate changes have taken effect.
> > > > > > +
> > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > +has been built only for this architecture.
> > > > > > +
> > > > > > +
> > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > +
> > > > > > +References:
> > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > --
> > > > > > 2.25.1
> > > > >
> > >
>
Michael S. Tsirkin June 28, 2022, 6:50 a.m. UTC | #7
On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > The README file is added describing the directory structure and the purpose
> > > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > > >
> > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > > ---
> > > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > > >  1 file changed, 168 insertions(+)
> > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > >
> > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > > new file mode 100644
> > > > > > > index 0000000000..97b15f1665
> > > > > > > --- /dev/null
> > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > @@ -0,0 +1,168 @@
> > > > > > > +=============================================================================
> > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > +=============================================================================
> > > > > > > +
> > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > > +operating system getting involved in between.
> > > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > > +life uses can be found in (a) and (b).
> > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > +
> > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > > >
> > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > what's wrong with using distro provided packages?
> > > > >
> > > > > There are two things:
> > > > > (a) We are already using pip and venv for our avocado based
> > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > tests.
> > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > our rich python library that handles QEMU based machines. There are
> > > > > python qtest libraries as well. These are well tested and used with
> > > > > integration tests and I wanted to keep the test part of the code
> > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > a venv environment within which these qemu python libraries are
> > > > > installed. Integration tests does the same thing.
> > > > >
> > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > (a) generate bits iso.
> > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > > (c) collect and analyze logs.
> > > > >
> > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > buses. We do not need the power of C here. We need something that is
> > > > > simple to write, easy to maintain and understand and can deal with
> > > > > things like manipulating text files and configs easily. Python seems a
> > > > > better fit for the role.
> > > >
> > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > packages from the net during testing?
> > >
> > > We do that too. See requirements.txt in tests/
> > > Following two are downloaded:
> > > avocado-framework==88.1
> > > pycdlib==1.11.0
> > >
> > > Also see this line in Makefie.include:
> > >
> > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >
> > Right but that's avocado since it pulls lots of stuff from
> > the net anyway.
> > Are the libraries in question not packaged on major distros?
> 
> Currently I only need this:
> https://github.com/python-tap/tappy
> which is the basic TAP processing library for python.
> 
> It seems its only installed through pip:
> https://tappy.readthedocs.io/en/latest/
> 
> I do not think this is packaged by default. It's such a basic library
> for parsing test output that maybe we can keep this somewhere within
> the python src tree? Not sure ...

It's pretty small for sure. Another submodule?

> >
> > > >
> > > > > >
> > > > > > > +
> > > > > > > +A brief description of the contents of this directory follows:
> > > > > > > +
> > > > > > > +├── acpi-bits-test.py
> > > > > > > +├── acpi-bits-test-venv.sh
> > > > > > > +├── bits-config
> > > > > > > +│   ├── bits-cfg.txt
> > > > > > > +│   └── meson.build
> > > > > > > +├── bits-tests
> > > > > > > +│   ├── meson.build
> > > > > > > +│   ├── smbios.py
> > > > > > > +│   ├── smilatency.py
> > > > > > > +│   ├── testacpi.py
> > > > > > > +│   └── testcpuid.py
> > > > > > > +├── meson.build
> > > > > > > +├── prebuilt
> > > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > > +│   ├── bits-2095.zip
> > > > > > > +│   └── meson.build
> > > > > > > +├── README
> > > > > > > +└── requirements.txt
> > > > > > > +
> > > > > > > +acpi-bits:
> > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > > +   found within python/ directory in QEMU source.
> > > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > > +   from within that environment.
> > > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > > +   this file.
> > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > > +   their tests.
> > > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > > + - README: This text file.
> > > > > > > +
> > > > > > > +acpi-bits/bits-config:
> > > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > > +   runs the tests.
> > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > > +   provided in the file itself.
> > > > > > > +
> > > > > > > +acpi-bits/prebuilt:
> > > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > > +   the bits iso file for testing.
> > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > > +    contains the full source of the bits software including the full text of
> > > > > > > +    the license agreement listed here:
> > > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > > +    The additional changes beyond those that are present in the official
> > > > > > > +    biosbits github repository can be found here:
> > > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > > +
> > > > > > > +    Basically these changes can be categorized into the following:
> > > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > > +    (gcc 9.4).
> > > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > > +    debugcon can be used to collect the logs.
> > > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > > +    executed.
> > > > > > > +
> > > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > > +    number.
> > > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > > +    a new config option or change actions of existing options or change the
> > > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > > +    be upadated here first before introducing the new config option through
> > > > > > > +    a later commit.
> > > > > >
> > > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > > >
> > > > > I agree. However, we do not do much with the source tarball. It is
> > > > > there as part of bits to satisfy the license requirement. If we need
> > > > > to manipulate any test script that is in the source file, we would
> > > > > copy it over and keep it in the bits-test directory and change it
> > > > > there.
> > > > >
> > > > > > Let's use submodules just like e.g. firmware does?
> > > > >
> > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > > out a place to store the bits source if we think my github is not the
> > > > > best place. However, we need the source only if/when we need to
> > > > > rebuild bits. I suspect it won't be too often if at all.
> > > >
> > > > Why not on git.qemu.org ?
> > >
> > > Sounds good to me. Do I get push access? :-)
> > >
> > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > > +    can run tests.
> > > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > > +    bits that generated the archive:
> > > > > > > +
> > > > > > > +    grub-inst-x86_64-efi
> > > > > > > +    grub-inst
> > > > > > > +
> > > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > > +    not recommended.
> > > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > > +
> > > > > > > +acpi-bits/bits-tests:
> > > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > > +   into this directory.
> > > > > > > +
> > > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > > +
> > > > > > > +   step (c): perform modifications to the test.
> > > > > > > +   Commit (c) in a separate commit.
> > > > > > > +
> > > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > > +   appropriate changes have taken effect.
> > > > > > > +
> > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > > +has been built only for this architecture.
> > > > > > > +
> > > > > > > +
> > > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > > +
> > > > > > > +References:
> > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > > --
> > > > > > > 2.25.1
> > > > > >
> > > >
> >
Ani Sinha June 28, 2022, 6:57 a.m. UTC | #8
On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > > The README file is added describing the directory structure and the purpose
> > > > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > > > >
> > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > > > ---
> > > > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > > > >  1 file changed, 168 insertions(+)
> > > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > > >
> > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000..97b15f1665
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > > @@ -0,0 +1,168 @@
> > > > > > > > +=============================================================================
> > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > > +=============================================================================
> > > > > > > > +
> > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > > > +operating system getting involved in between.
> > > > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > > > +life uses can be found in (a) and (b).
> > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > > +
> > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > > > >
> > > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > > what's wrong with using distro provided packages?
> > > > > >
> > > > > > There are two things:
> > > > > > (a) We are already using pip and venv for our avocado based
> > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > > tests.
> > > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > > our rich python library that handles QEMU based machines. There are
> > > > > > python qtest libraries as well. These are well tested and used with
> > > > > > integration tests and I wanted to keep the test part of the code
> > > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > > a venv environment within which these qemu python libraries are
> > > > > > installed. Integration tests does the same thing.
> > > > > >
> > > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > > (a) generate bits iso.
> > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > > > (c) collect and analyze logs.
> > > > > >
> > > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > > buses. We do not need the power of C here. We need something that is
> > > > > > simple to write, easy to maintain and understand and can deal with
> > > > > > things like manipulating text files and configs easily. Python seems a
> > > > > > better fit for the role.
> > > > >
> > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > packages from the net during testing?
> > > >
> > > > We do that too. See requirements.txt in tests/
> > > > Following two are downloaded:
> > > > avocado-framework==88.1
> > > > pycdlib==1.11.0
> > > >
> > > > Also see this line in Makefie.include:
> > > >
> > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >
> > > Right but that's avocado since it pulls lots of stuff from
> > > the net anyway.
> > > Are the libraries in question not packaged on major distros?
> >
> > Currently I only need this:
> > https://github.com/python-tap/tappy
> > which is the basic TAP processing library for python.
> >
> > It seems its only installed through pip:
> > https://tappy.readthedocs.io/en/latest/
> >
> > I do not think this is packaged by default. It's such a basic library
> > for parsing test output that maybe we can keep this somewhere within
> > the python src tree? Not sure ...
>
> It's pretty small for sure. Another submodule?

Unlike BITS, this one is likely going to be maintained for a while and
will receive new releases through
https://pypi.org/project/tap.py/
so forking is OK but someone has to keep this updated.

I am open to anything. Whatever feels right is fine to me.

>
> > >
> > > > >
> > > > > > >
> > > > > > > > +
> > > > > > > > +A brief description of the contents of this directory follows:
> > > > > > > > +
> > > > > > > > +├── acpi-bits-test.py
> > > > > > > > +├── acpi-bits-test-venv.sh
> > > > > > > > +├── bits-config
> > > > > > > > +│   ├── bits-cfg.txt
> > > > > > > > +│   └── meson.build
> > > > > > > > +├── bits-tests
> > > > > > > > +│   ├── meson.build
> > > > > > > > +│   ├── smbios.py
> > > > > > > > +│   ├── smilatency.py
> > > > > > > > +│   ├── testacpi.py
> > > > > > > > +│   └── testcpuid.py
> > > > > > > > +├── meson.build
> > > > > > > > +├── prebuilt
> > > > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > > > +│   ├── bits-2095.zip
> > > > > > > > +│   └── meson.build
> > > > > > > > +├── README
> > > > > > > > +└── requirements.txt
> > > > > > > > +
> > > > > > > > +acpi-bits:
> > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > > > +   found within python/ directory in QEMU source.
> > > > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > > > +   from within that environment.
> > > > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > > > +   this file.
> > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > > > +   their tests.
> > > > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > > > + - README: This text file.
> > > > > > > > +
> > > > > > > > +acpi-bits/bits-config:
> > > > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > > > +   runs the tests.
> > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > > > +   provided in the file itself.
> > > > > > > > +
> > > > > > > > +acpi-bits/prebuilt:
> > > > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > > > +   the bits iso file for testing.
> > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > > > +    contains the full source of the bits software including the full text of
> > > > > > > > +    the license agreement listed here:
> > > > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > > > +    The additional changes beyond those that are present in the official
> > > > > > > > +    biosbits github repository can be found here:
> > > > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > > > +
> > > > > > > > +    Basically these changes can be categorized into the following:
> > > > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > > > +    (gcc 9.4).
> > > > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > > > +    debugcon can be used to collect the logs.
> > > > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > > > +    executed.
> > > > > > > > +
> > > > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > > > +    number.
> > > > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > > > +    a new config option or change actions of existing options or change the
> > > > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > > > +    be upadated here first before introducing the new config option through
> > > > > > > > +    a later commit.
> > > > > > >
> > > > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > > > >
> > > > > > I agree. However, we do not do much with the source tarball. It is
> > > > > > there as part of bits to satisfy the license requirement. If we need
> > > > > > to manipulate any test script that is in the source file, we would
> > > > > > copy it over and keep it in the bits-test directory and change it
> > > > > > there.
> > > > > >
> > > > > > > Let's use submodules just like e.g. firmware does?
> > > > > >
> > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > > > out a place to store the bits source if we think my github is not the
> > > > > > best place. However, we need the source only if/when we need to
> > > > > > rebuild bits. I suspect it won't be too often if at all.
> > > > >
> > > > > Why not on git.qemu.org ?
> > > >
> > > > Sounds good to me. Do I get push access? :-)
> > > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > > > +    can run tests.
> > > > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > > > +    bits that generated the archive:
> > > > > > > > +
> > > > > > > > +    grub-inst-x86_64-efi
> > > > > > > > +    grub-inst
> > > > > > > > +
> > > > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > > > +    not recommended.
> > > > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > > > +
> > > > > > > > +acpi-bits/bits-tests:
> > > > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > > > +   into this directory.
> > > > > > > > +
> > > > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > > > +
> > > > > > > > +   step (c): perform modifications to the test.
> > > > > > > > +   Commit (c) in a separate commit.
> > > > > > > > +
> > > > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > > > +   appropriate changes have taken effect.
> > > > > > > > +
> > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > > > +has been built only for this architecture.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > > > +
> > > > > > > > +References:
> > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > > > --
> > > > > > > > 2.25.1
> > > > > > >
> > > > >
> > >
>
Thomas Huth June 28, 2022, 7:03 a.m. UTC | #9
On 28/06/2022 08.57, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
>>> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>
>>>> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
>>>>> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>
>>>>>> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
>>>>>>> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>>>
>>>>>>>> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
>>>>>>>>> The README file is added describing the directory structure and the purpose
>>>>>>>>> of every file it contains. It also describes how to add new tests, make changes
>>>>>>>>> to existing tests or bits config files or regenerate the bits software.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Ani Sinha <ani@anisinha.ca>
>>>>>>>>> ---
>>>>>>>>>   tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
>>>>>>>>>   1 file changed, 168 insertions(+)
>>>>>>>>>   create mode 100644 tests/qtest/acpi-bits/README
>>>>>>>>>
>>>>>>>>> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
>>>>>>>>> new file mode 100644
>>>>>>>>> index 0000000000..97b15f1665
>>>>>>>>> --- /dev/null
>>>>>>>>> +++ b/tests/qtest/acpi-bits/README
>>>>>>>>> @@ -0,0 +1,168 @@
>>>>>>>>> +=============================================================================
>>>>>>>>> +ACPI/SMBIOS QTESTS USING BIOSBITS
>>>>>>>>> +=============================================================================
>>>>>>>>> +
>>>>>>>>> +Biosbits is a software written by Josh Triplett that can be downloaded by
>>>>>>>>> +visiting https://biosbits.org/. The github codebase can be found here:
>>>>>>>>> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
>>>>>>>>> +the bios components such as acpi and smbios tables directly through acpica
>>>>>>>>> +bios interpreter (a freely available C based library written by Intel,
>>>>>>>>> +downloadable from https://acpica.org/ and is included with biosbits) without an
>>>>>>>>> +operating system getting involved in between.
>>>>>>>>> +There are several advantages to directly testing the bios in a real physical
>>>>>>>>> +machine or VM as opposed to indirectly discovering bios issues through the
>>>>>>>>> +operating system. For one thing, the OSes tend to hide bios problems from the
>>>>>>>>> +end user. The other is that we have more control of what we wanted to test
>>>>>>>>> +and how by directly using acpica interpreter on top of the bios on a running
>>>>>>>>> +system. More details on the inspiration for developing biosbits and its real
>>>>>>>>> +life uses can be found in (a) and (b).
>>>>>>>>> +This directory contains QEMU qtests written in python that exercizes the QEMU
>>>>>>>>> +bios components using biosbits and reports test failures.
>>>>>>>>> +
>>>>>>>>> +These tests use python virtual environment. In debian/ubuntu system, the tests
>>>>>>>>> +would require python3.8-venv and python3-pip packages to be installed.
>>>>>>>>
>>>>>>>> Why do we mess with venv and pip? Certainly possible but
>>>>>>>> what's wrong with using distro provided packages?
>>>>>>>
>>>>>>> There are two things:
>>>>>>> (a) We are already using pip and venv for our avocado based
>>>>>>> integration tests. Look for TESTS_VENV_DIR in Makefile.include under
>>>>>>> tests.
>>>>>>> (b) the venv is primarily needed because I wanted to take advantage of
>>>>>>> our rich python library that handles QEMU based machines. There are
>>>>>>> python qtest libraries as well. These are well tested and used with
>>>>>>> integration tests and I wanted to keep the test part of the code
>>>>>>> simple by simply reusing them. however, in order to use them, we need
>>>>>>> a venv environment within which these qemu python libraries are
>>>>>>> installed. Integration tests does the same thing.
>>>>>>>
>>>>>>> A note about my language of choice - python. I gave a lot of thoughts
>>>>>>> on this. We do not do a lot of stuff here. All we do is:
>>>>>>> (a) generate bits iso.
>>>>>>> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
>>>>>>> (c) collect and analyze logs.
>>>>>>>
>>>>>>> We are not inspecting guest memory or manipulating devices or pci
>>>>>>> buses. We do not need the power of C here. We need something that is
>>>>>>> simple to write, easy to maintain and understand and can deal with
>>>>>>> things like manipulating text files and configs easily. Python seems a
>>>>>>> better fit for the role.
>>>>>>
>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>> packages from the net during testing?
>>>>>
>>>>> We do that too. See requirements.txt in tests/
>>>>> Following two are downloaded:
>>>>> avocado-framework==88.1
>>>>> pycdlib==1.11.0
>>>>>
>>>>> Also see this line in Makefie.include:
>>>>>
>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>
>>>> Right but that's avocado since it pulls lots of stuff from
>>>> the net anyway.
>>>> Are the libraries in question not packaged on major distros?
>>>
>>> Currently I only need this:
>>> https://github.com/python-tap/tappy
>>> which is the basic TAP processing library for python.
>>>
>>> It seems its only installed through pip:
>>> https://tappy.readthedocs.io/en/latest/
>>>
>>> I do not think this is packaged by default. It's such a basic library
>>> for parsing test output that maybe we can keep this somewhere within
>>> the python src tree? Not sure ...
>>
>> It's pretty small for sure. Another submodule?
> 
> Unlike BITS, this one is likely going to be maintained for a while and
> will receive new releases through
> https://pypi.org/project/tap.py/
> so forking is OK but someone has to keep this updated.
> 
> I am open to anything. Whatever feels right is fine to me.

John Snow is currently working on the "Pythonification" of various QEMU 
bits, I think you should loop him into this discussion, too.

  Thomas
Ani Sinha June 28, 2022, 7:05 a.m. UTC | #10
+danPb who pointed me to the TAP stuff. Maybe he has some good ideas on this.

On Tue, Jun 28, 2022 at 12:27 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > > > The README file is added describing the directory structure and the purpose
> > > > > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > > > > ---
> > > > > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > > > > >  1 file changed, 168 insertions(+)
> > > > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > > > >
> > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000..97b15f1665
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > > > @@ -0,0 +1,168 @@
> > > > > > > > > +=============================================================================
> > > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > > > +=============================================================================
> > > > > > > > > +
> > > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > > > > +operating system getting involved in between.
> > > > > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > > > > +life uses can be found in (a) and (b).
> > > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > > > +
> > > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > > > > >
> > > > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > > > what's wrong with using distro provided packages?
> > > > > > >
> > > > > > > There are two things:
> > > > > > > (a) We are already using pip and venv for our avocado based
> > > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > > > tests.
> > > > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > > > our rich python library that handles QEMU based machines. There are
> > > > > > > python qtest libraries as well. These are well tested and used with
> > > > > > > integration tests and I wanted to keep the test part of the code
> > > > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > > > a venv environment within which these qemu python libraries are
> > > > > > > installed. Integration tests does the same thing.
> > > > > > >
> > > > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > > > (a) generate bits iso.
> > > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > > > > (c) collect and analyze logs.
> > > > > > >
> > > > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > > > buses. We do not need the power of C here. We need something that is
> > > > > > > simple to write, easy to maintain and understand and can deal with
> > > > > > > things like manipulating text files and configs easily. Python seems a
> > > > > > > better fit for the role.
> > > > > >
> > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > packages from the net during testing?
> > > > >
> > > > > We do that too. See requirements.txt in tests/
> > > > > Following two are downloaded:
> > > > > avocado-framework==88.1
> > > > > pycdlib==1.11.0
> > > > >
> > > > > Also see this line in Makefie.include:
> > > > >
> > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > >
> > > > Right but that's avocado since it pulls lots of stuff from
> > > > the net anyway.
> > > > Are the libraries in question not packaged on major distros?
> > >
> > > Currently I only need this:
> > > https://github.com/python-tap/tappy
> > > which is the basic TAP processing library for python.
> > >
> > > It seems its only installed through pip:
> > > https://tappy.readthedocs.io/en/latest/
> > >
> > > I do not think this is packaged by default. It's such a basic library
> > > for parsing test output that maybe we can keep this somewhere within
> > > the python src tree? Not sure ...
> >
> > It's pretty small for sure. Another submodule?
>
> Unlike BITS, this one is likely going to be maintained for a while and
> will receive new releases through
> https://pypi.org/project/tap.py/
> so forking is OK but someone has to keep this updated.
>
> I am open to anything. Whatever feels right is fine to me.
>
> >
> > > >
> > > > > >
> > > > > > > >
> > > > > > > > > +
> > > > > > > > > +A brief description of the contents of this directory follows:
> > > > > > > > > +
> > > > > > > > > +├── acpi-bits-test.py
> > > > > > > > > +├── acpi-bits-test-venv.sh
> > > > > > > > > +├── bits-config
> > > > > > > > > +│   ├── bits-cfg.txt
> > > > > > > > > +│   └── meson.build
> > > > > > > > > +├── bits-tests
> > > > > > > > > +│   ├── meson.build
> > > > > > > > > +│   ├── smbios.py
> > > > > > > > > +│   ├── smilatency.py
> > > > > > > > > +│   ├── testacpi.py
> > > > > > > > > +│   └── testcpuid.py
> > > > > > > > > +├── meson.build
> > > > > > > > > +├── prebuilt
> > > > > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > > > > +│   ├── bits-2095.zip
> > > > > > > > > +│   └── meson.build
> > > > > > > > > +├── README
> > > > > > > > > +└── requirements.txt
> > > > > > > > > +
> > > > > > > > > +acpi-bits:
> > > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > > > > +   found within python/ directory in QEMU source.
> > > > > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > > > > +   from within that environment.
> > > > > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > > > > +   this file.
> > > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > > > > +   their tests.
> > > > > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > > > > + - README: This text file.
> > > > > > > > > +
> > > > > > > > > +acpi-bits/bits-config:
> > > > > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > > > > +   runs the tests.
> > > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > > > > +   provided in the file itself.
> > > > > > > > > +
> > > > > > > > > +acpi-bits/prebuilt:
> > > > > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > > > > +   the bits iso file for testing.
> > > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > > > > +    contains the full source of the bits software including the full text of
> > > > > > > > > +    the license agreement listed here:
> > > > > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > > > > +    The additional changes beyond those that are present in the official
> > > > > > > > > +    biosbits github repository can be found here:
> > > > > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > > > > +
> > > > > > > > > +    Basically these changes can be categorized into the following:
> > > > > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > > > > +    (gcc 9.4).
> > > > > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > > > > +    debugcon can be used to collect the logs.
> > > > > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > > > > +    executed.
> > > > > > > > > +
> > > > > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > > > > +    number.
> > > > > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > > > > +    a new config option or change actions of existing options or change the
> > > > > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > > > > +    be upadated here first before introducing the new config option through
> > > > > > > > > +    a later commit.
> > > > > > > >
> > > > > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > > > > >
> > > > > > > I agree. However, we do not do much with the source tarball. It is
> > > > > > > there as part of bits to satisfy the license requirement. If we need
> > > > > > > to manipulate any test script that is in the source file, we would
> > > > > > > copy it over and keep it in the bits-test directory and change it
> > > > > > > there.
> > > > > > >
> > > > > > > > Let's use submodules just like e.g. firmware does?
> > > > > > >
> > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > > > > out a place to store the bits source if we think my github is not the
> > > > > > > best place. However, we need the source only if/when we need to
> > > > > > > rebuild bits. I suspect it won't be too often if at all.
> > > > > >
> > > > > > Why not on git.qemu.org ?
> > > > >
> > > > > Sounds good to me. Do I get push access? :-)
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > > > > +    can run tests.
> > > > > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > > > > +    bits that generated the archive:
> > > > > > > > > +
> > > > > > > > > +    grub-inst-x86_64-efi
> > > > > > > > > +    grub-inst
> > > > > > > > > +
> > > > > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > > > > +    not recommended.
> > > > > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > > > > +
> > > > > > > > > +acpi-bits/bits-tests:
> > > > > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > > > > +   into this directory.
> > > > > > > > > +
> > > > > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > > > > +
> > > > > > > > > +   step (c): perform modifications to the test.
> > > > > > > > > +   Commit (c) in a separate commit.
> > > > > > > > > +
> > > > > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > > > > +   appropriate changes have taken effect.
> > > > > > > > > +
> > > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > > > > +has been built only for this architecture.
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > > > > +
> > > > > > > > > +References:
> > > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > > > > --
> > > > > > > > > 2.25.1
> > > > > > > >
> > > > > >
> > > >
> >
Michael S. Tsirkin June 28, 2022, 7:10 a.m. UTC | #11
On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > packages from the net during testing?
> > > > > > 
> > > > > > We do that too. See requirements.txt in tests/
> > > > > > Following two are downloaded:
> > > > > > avocado-framework==88.1
> > > > > > pycdlib==1.11.0
> > > > > > 
> > > > > > Also see this line in Makefie.include:
> > > > > > 
> > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > 
> > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > the net anyway.
> > > > > Are the libraries in question not packaged on major distros?
> > > > 
> > > > Currently I only need this:
> > > > https://github.com/python-tap/tappy
> > > > which is the basic TAP processing library for python.
> > > > 
> > > > It seems its only installed through pip:
> > > > https://tappy.readthedocs.io/en/latest/
> > > > 
> > > > I do not think this is packaged by default. It's such a basic library
> > > > for parsing test output that maybe we can keep this somewhere within
> > > > the python src tree? Not sure ...
> > > 
> > > It's pretty small for sure. Another submodule?
> > 
> > Unlike BITS, this one is likely going to be maintained for a while and
> > will receive new releases through
> > https://pypi.org/project/tap.py/
> > so forking is OK but someone has to keep this updated.
> > 
> > I am open to anything. Whatever feels right is fine to me.
> 
> John Snow is currently working on the "Pythonification" of various QEMU
> bits, I think you should loop him into this discussion, too.
> 
>  Thomas

submodule does not mean we fork necessarily. We could have
all options: check for the module and use it if there, if not
use one from system if not there install with pip ..
But yea, I'm not sure what's best either.
Thomas Huth June 28, 2022, 7:25 a.m. UTC | #12
On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>>>> packages from the net during testing?
>>>>>>>
>>>>>>> We do that too. See requirements.txt in tests/
>>>>>>> Following two are downloaded:
>>>>>>> avocado-framework==88.1
>>>>>>> pycdlib==1.11.0
>>>>>>>
>>>>>>> Also see this line in Makefie.include:
>>>>>>>
>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>>>
>>>>>> Right but that's avocado since it pulls lots of stuff from
>>>>>> the net anyway.
>>>>>> Are the libraries in question not packaged on major distros?
>>>>>
>>>>> Currently I only need this:
>>>>> https://github.com/python-tap/tappy
>>>>> which is the basic TAP processing library for python.
>>>>>
>>>>> It seems its only installed through pip:
>>>>> https://tappy.readthedocs.io/en/latest/
>>>>>
>>>>> I do not think this is packaged by default. It's such a basic library
>>>>> for parsing test output that maybe we can keep this somewhere within
>>>>> the python src tree? Not sure ...
>>>>
>>>> It's pretty small for sure. Another submodule?
>>>
>>> Unlike BITS, this one is likely going to be maintained for a while and
>>> will receive new releases through
>>> https://pypi.org/project/tap.py/
>>> so forking is OK but someone has to keep this updated.
>>>
>>> I am open to anything. Whatever feels right is fine to me.
>>
>> John Snow is currently working on the "Pythonification" of various QEMU
>> bits, I think you should loop him into this discussion, too.
>>
>>   Thomas
> 
> submodule does not mean we fork necessarily. We could have
> all options: check for the module and use it if there, if not
> use one from system if not there install with pip ..
> But yea, I'm not sure what's best either.

submodules create a dependency on an internet connection, too. So before you 
add yet another submodule (which have a couple of other disadvantages), I 
think you could also directly use the venv here.

  Thomas
Daniel P. Berrangé June 28, 2022, 7:48 a.m. UTC | #13
On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > packages from the net during testing?
> > > > > > > > 
> > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > Following two are downloaded:
> > > > > > > > avocado-framework==88.1
> > > > > > > > pycdlib==1.11.0
> > > > > > > > 
> > > > > > > > Also see this line in Makefie.include:
> > > > > > > > 
> > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > 
> > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > the net anyway.
> > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > 
> > > > > > Currently I only need this:
> > > > > > https://github.com/python-tap/tappy
> > > > > > which is the basic TAP processing library for python.
> > > > > > 
> > > > > > It seems its only installed through pip:
> > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > 
> > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > the python src tree? Not sure ...
> > > > > 
> > > > > It's pretty small for sure. Another submodule?
> > > > 
> > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > will receive new releases through
> > > > https://pypi.org/project/tap.py/
> > > > so forking is OK but someone has to keep this updated.
> > > > 
> > > > I am open to anything. Whatever feels right is fine to me.
> > > 
> > > John Snow is currently working on the "Pythonification" of various QEMU
> > > bits, I think you should loop him into this discussion, too.
> > > 
> > >   Thomas
> > 
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
> submodules create a dependency on an internet connection, too. So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.

Definitely not submodules.

We need to get out of the mindset that submodules are needed for every new
dependancy we add. Submodules are only appropriate if the external project
is designed to be used as a copylib (eg the keycodemapdb tool), or if we
need to bundle in order to prevent a regression for previously deployed
QEMU installs where the dependancy is known not to exist on all our
supported platforms.

This does not apply in this case, because the proposed use of tappy is
merely for a test case. Meson just needs to check if tappy exists and if
it does, then use it, otherwise skip the tests that need it. The user can
arrange to install tappy, as they do with the majority of other deps.

If John's venv stuff is relevant, then we don't even need the meson checks,
just delegate to the venv setup.

Regardless, no submodules are needed or desirable.

With regards,
Daniel
Ani Sinha June 28, 2022, 7:49 a.m. UTC | #14
On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
> >>>>>>>> packages from the net during testing?
> >>>>>>>
> >>>>>>> We do that too. See requirements.txt in tests/
> >>>>>>> Following two are downloaded:
> >>>>>>> avocado-framework==88.1
> >>>>>>> pycdlib==1.11.0
> >>>>>>>
> >>>>>>> Also see this line in Makefie.include:
> >>>>>>>
> >>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >>>>>>
> >>>>>> Right but that's avocado since it pulls lots of stuff from
> >>>>>> the net anyway.
> >>>>>> Are the libraries in question not packaged on major distros?
> >>>>>
> >>>>> Currently I only need this:
> >>>>> https://github.com/python-tap/tappy
> >>>>> which is the basic TAP processing library for python.
> >>>>>
> >>>>> It seems its only installed through pip:
> >>>>> https://tappy.readthedocs.io/en/latest/
> >>>>>
> >>>>> I do not think this is packaged by default. It's such a basic library
> >>>>> for parsing test output that maybe we can keep this somewhere within
> >>>>> the python src tree? Not sure ...
> >>>>
> >>>> It's pretty small for sure. Another submodule?
> >>>
> >>> Unlike BITS, this one is likely going to be maintained for a while and
> >>> will receive new releases through
> >>> https://pypi.org/project/tap.py/
> >>> so forking is OK but someone has to keep this updated.
> >>>
> >>> I am open to anything. Whatever feels right is fine to me.
> >>
> >> John Snow is currently working on the "Pythonification" of various QEMU
> >> bits, I think you should loop him into this discussion, too.
> >>
> >>   Thomas
> >
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
>
> submodules create a dependency on an internet connection, too. So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.

Not sure what you mean by directly using the venv.

>
>   Thomas
>
Ani Sinha June 28, 2022, 7:51 a.m. UTC | #15
On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > packages from the net during testing?
> > > > > > > > >
> > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > Following two are downloaded:
> > > > > > > > > avocado-framework==88.1
> > > > > > > > > pycdlib==1.11.0
> > > > > > > > >
> > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > >
> > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > >
> > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > the net anyway.
> > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > >
> > > > > > > Currently I only need this:
> > > > > > > https://github.com/python-tap/tappy
> > > > > > > which is the basic TAP processing library for python.
> > > > > > >
> > > > > > > It seems its only installed through pip:
> > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > >
> > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > the python src tree? Not sure ...
> > > > > >
> > > > > > It's pretty small for sure. Another submodule?
> > > > >
> > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > will receive new releases through
> > > > > https://pypi.org/project/tap.py/
> > > > > so forking is OK but someone has to keep this updated.
> > > > >
> > > > > I am open to anything. Whatever feels right is fine to me.
> > > >
> > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > bits, I think you should loop him into this discussion, too.
> > > >
> > > >   Thomas
> > >
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> >
> > submodules create a dependency on an internet connection, too. So before you
> > add yet another submodule (which have a couple of other disadvantages), I
> > think you could also directly use the venv here.
>
> Definitely not submodules.
>
> We need to get out of the mindset that submodules are needed for every new
> dependancy we add. Submodules are only appropriate if the external project
> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> need to bundle in order to prevent a regression for previously deployed
> QEMU installs where the dependancy is known not to exist on all our
> supported platforms.
>
> This does not apply in this case, because the proposed use of tappy is
> merely for a test case. Meson just needs to check if tappy exists and if
> it does, then use it, otherwise skip the tests that need it. The user can
> arrange to install tappy, as they do with the majority of other deps.
>
> If John's venv stuff is relevant, then we don't even need the meson checks,
> just delegate to the venv setup.
>
> Regardless, no submodules are needed or desirable.

What about keeping biosbits stuff? Source or pre-built.


>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
Thomas Huth June 28, 2022, 7:53 a.m. UTC | #16
On 28/06/2022 09.49, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>>>>>> packages from the net during testing?
>>>>>>>>>
>>>>>>>>> We do that too. See requirements.txt in tests/
>>>>>>>>> Following two are downloaded:
>>>>>>>>> avocado-framework==88.1
>>>>>>>>> pycdlib==1.11.0
>>>>>>>>>
>>>>>>>>> Also see this line in Makefie.include:
>>>>>>>>>
>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>>>>>
>>>>>>>> Right but that's avocado since it pulls lots of stuff from
>>>>>>>> the net anyway.
>>>>>>>> Are the libraries in question not packaged on major distros?
>>>>>>>
>>>>>>> Currently I only need this:
>>>>>>> https://github.com/python-tap/tappy
>>>>>>> which is the basic TAP processing library for python.
>>>>>>>
>>>>>>> It seems its only installed through pip:
>>>>>>> https://tappy.readthedocs.io/en/latest/
>>>>>>>
>>>>>>> I do not think this is packaged by default. It's such a basic library
>>>>>>> for parsing test output that maybe we can keep this somewhere within
>>>>>>> the python src tree? Not sure ...
>>>>>>
>>>>>> It's pretty small for sure. Another submodule?
>>>>>
>>>>> Unlike BITS, this one is likely going to be maintained for a while and
>>>>> will receive new releases through
>>>>> https://pypi.org/project/tap.py/
>>>>> so forking is OK but someone has to keep this updated.
>>>>>
>>>>> I am open to anything. Whatever feels right is fine to me.
>>>>
>>>> John Snow is currently working on the "Pythonification" of various QEMU
>>>> bits, I think you should loop him into this discussion, too.
>>>>
>>>>    Thomas
>>>
>>> submodule does not mean we fork necessarily. We could have
>>> all options: check for the module and use it if there, if not
>>> use one from system if not there install with pip ..
>>> But yea, I'm not sure what's best either.
>>
>> submodules create a dependency on an internet connection, too. So before you
>> add yet another submodule (which have a couple of other disadvantages), I
>> think you could also directly use the venv here.
> 
> Not sure what you mean by directly using the venv.

I meant installing the dependency via pip in a venv, as you are currently 
doing it via the acpi-bits-test-venv.sh script.

  Thomas
Daniel P. Berrangé June 28, 2022, 8:23 a.m. UTC | #17
On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > packages from the net during testing?
> > > > > > > > > >
> > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > Following two are downloaded:
> > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > >
> > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > >
> > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > >
> > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > the net anyway.
> > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > >
> > > > > > > > Currently I only need this:
> > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > which is the basic TAP processing library for python.
> > > > > > > >
> > > > > > > > It seems its only installed through pip:
> > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > >
> > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > the python src tree? Not sure ...
> > > > > > >
> > > > > > > It's pretty small for sure. Another submodule?
> > > > > >
> > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > will receive new releases through
> > > > > > https://pypi.org/project/tap.py/
> > > > > > so forking is OK but someone has to keep this updated.
> > > > > >
> > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > >
> > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > bits, I think you should loop him into this discussion, too.
> > > > >
> > > > >   Thomas
> > > >
> > > > submodule does not mean we fork necessarily. We could have
> > > > all options: check for the module and use it if there, if not
> > > > use one from system if not there install with pip ..
> > > > But yea, I'm not sure what's best either.
> > >
> > > submodules create a dependency on an internet connection, too. So before you
> > > add yet another submodule (which have a couple of other disadvantages), I
> > > think you could also directly use the venv here.
> >
> > Definitely not submodules.
> >
> > We need to get out of the mindset that submodules are needed for every new
> > dependancy we add. Submodules are only appropriate if the external project
> > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > need to bundle in order to prevent a regression for previously deployed
> > QEMU installs where the dependancy is known not to exist on all our
> > supported platforms.
> >
> > This does not apply in this case, because the proposed use of tappy is
> > merely for a test case. Meson just needs to check if tappy exists and if
> > it does, then use it, otherwise skip the tests that need it. The user can
> > arrange to install tappy, as they do with the majority of other deps.
> >
> > If John's venv stuff is relevant, then we don't even need the meson checks,
> > just delegate to the venv setup.
> >
> > Regardless, no submodules are needed or desirable.
> 
> What about keeping biosbits stuff? Source or pre-built.

Shipping them as pre-built binaries in QEMU is not a viable option
IMHO, especially for grub as a GPL'd project we need to be extremely
clear about the exact corresponding source and build process for any
binary.

For this kind of thing I would generally expect the distro to provide
packages that we consume. Looking at biosbits I see it is itself
bundling a bunch more 3rd party projects, libffi, grub2, and including
even an ancient version of python as a submodule.

So bundling a pre-built biosbits in QEMU appears to mean that we're in
turn going to unexpectedly bundle a bunch of other 3rd party projects
too, all with dubious license compliance. I don't think this looks like
something we should have in qemu.git or qemu tarballs. It will also
make it challenging for the distro to take biosbits at all, unless
those 3rd party bundles can be eliminated in favour of using existing
builds their have packaged for grub, python, libffi, etc.

With regards,
Daniel
Thomas Huth June 28, 2022, 8:28 a.m. UTC | #18
On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
>> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>
>>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
>>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
>>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>>>>>>>> packages from the net during testing?
>>>>>>>>>>>
>>>>>>>>>>> We do that too. See requirements.txt in tests/
>>>>>>>>>>> Following two are downloaded:
>>>>>>>>>>> avocado-framework==88.1
>>>>>>>>>>> pycdlib==1.11.0
>>>>>>>>>>>
>>>>>>>>>>> Also see this line in Makefie.include:
>>>>>>>>>>>
>>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>>>>>>>
>>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
>>>>>>>>>> the net anyway.
>>>>>>>>>> Are the libraries in question not packaged on major distros?
>>>>>>>>>
>>>>>>>>> Currently I only need this:
>>>>>>>>> https://github.com/python-tap/tappy
>>>>>>>>> which is the basic TAP processing library for python.
>>>>>>>>>
>>>>>>>>> It seems its only installed through pip:
>>>>>>>>> https://tappy.readthedocs.io/en/latest/
>>>>>>>>>
>>>>>>>>> I do not think this is packaged by default. It's such a basic library
>>>>>>>>> for parsing test output that maybe we can keep this somewhere within
>>>>>>>>> the python src tree? Not sure ...
>>>>>>>>
>>>>>>>> It's pretty small for sure. Another submodule?
>>>>>>>
>>>>>>> Unlike BITS, this one is likely going to be maintained for a while and
>>>>>>> will receive new releases through
>>>>>>> https://pypi.org/project/tap.py/
>>>>>>> so forking is OK but someone has to keep this updated.
>>>>>>>
>>>>>>> I am open to anything. Whatever feels right is fine to me.
>>>>>>
>>>>>> John Snow is currently working on the "Pythonification" of various QEMU
>>>>>> bits, I think you should loop him into this discussion, too.
>>>>>>
>>>>>>    Thomas
>>>>>
>>>>> submodule does not mean we fork necessarily. We could have
>>>>> all options: check for the module and use it if there, if not
>>>>> use one from system if not there install with pip ..
>>>>> But yea, I'm not sure what's best either.
>>>>
>>>> submodules create a dependency on an internet connection, too. So before you
>>>> add yet another submodule (which have a couple of other disadvantages), I
>>>> think you could also directly use the venv here.
>>>
>>> Definitely not submodules.
>>>
>>> We need to get out of the mindset that submodules are needed for every new
>>> dependancy we add. Submodules are only appropriate if the external project
>>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
>>> need to bundle in order to prevent a regression for previously deployed
>>> QEMU installs where the dependancy is known not to exist on all our
>>> supported platforms.
>>>
>>> This does not apply in this case, because the proposed use of tappy is
>>> merely for a test case. Meson just needs to check if tappy exists and if
>>> it does, then use it, otherwise skip the tests that need it. The user can
>>> arrange to install tappy, as they do with the majority of other deps.
>>>
>>> If John's venv stuff is relevant, then we don't even need the meson checks,
>>> just delegate to the venv setup.
>>>
>>> Regardless, no submodules are needed or desirable.
>>
>> What about keeping biosbits stuff? Source or pre-built.
> 
> Shipping them as pre-built binaries in QEMU is not a viable option
> IMHO, especially for grub as a GPL'd project we need to be extremely
> clear about the exact corresponding source and build process for any
> binary.
> 
> For this kind of thing I would generally expect the distro to provide
> packages that we consume. Looking at biosbits I see it is itself
> bundling a bunch more 3rd party projects, libffi, grub2, and including
> even an ancient version of python as a submodule.
> 
> So bundling a pre-built biosbits in QEMU appears to mean that we're in
> turn going to unexpectedly bundle a bunch of other 3rd party projects
> too, all with dubious license compliance. I don't think this looks like
> something we should have in qemu.git or qemu tarballs. It will also
> make it challenging for the distro to take biosbits at all, unless
> those 3rd party bundles can be eliminated in favour of using existing
> builds their have packaged for grub, python, libffi, etc.

So if this depends on some third party binary bits, I think this is pretty 
similar to the tests in the avocado directory ... there we download third 
party binaries, too... Wouldn't it make sense to adapt your tests to that 
framework?

  Thomas
Ani Sinha June 28, 2022, 8:35 a.m. UTC | #19
On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>>
> >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
> >>>>>>>>>>>> packages from the net during testing?
> >>>>>>>>>>>
> >>>>>>>>>>> We do that too. See requirements.txt in tests/
> >>>>>>>>>>> Following two are downloaded:
> >>>>>>>>>>> avocado-framework==88.1
> >>>>>>>>>>> pycdlib==1.11.0
> >>>>>>>>>>>
> >>>>>>>>>>> Also see this line in Makefie.include:
> >>>>>>>>>>>
> >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >>>>>>>>>>
> >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
> >>>>>>>>>> the net anyway.
> >>>>>>>>>> Are the libraries in question not packaged on major distros?
> >>>>>>>>>
> >>>>>>>>> Currently I only need this:
> >>>>>>>>> https://github.com/python-tap/tappy
> >>>>>>>>> which is the basic TAP processing library for python.
> >>>>>>>>>
> >>>>>>>>> It seems its only installed through pip:
> >>>>>>>>> https://tappy.readthedocs.io/en/latest/
> >>>>>>>>>
> >>>>>>>>> I do not think this is packaged by default. It's such a basic library
> >>>>>>>>> for parsing test output that maybe we can keep this somewhere within
> >>>>>>>>> the python src tree? Not sure ...
> >>>>>>>>
> >>>>>>>> It's pretty small for sure. Another submodule?
> >>>>>>>
> >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and
> >>>>>>> will receive new releases through
> >>>>>>> https://pypi.org/project/tap.py/
> >>>>>>> so forking is OK but someone has to keep this updated.
> >>>>>>>
> >>>>>>> I am open to anything. Whatever feels right is fine to me.
> >>>>>>
> >>>>>> John Snow is currently working on the "Pythonification" of various QEMU
> >>>>>> bits, I think you should loop him into this discussion, too.
> >>>>>>
> >>>>>>    Thomas
> >>>>>
> >>>>> submodule does not mean we fork necessarily. We could have
> >>>>> all options: check for the module and use it if there, if not
> >>>>> use one from system if not there install with pip ..
> >>>>> But yea, I'm not sure what's best either.
> >>>>
> >>>> submodules create a dependency on an internet connection, too. So before you
> >>>> add yet another submodule (which have a couple of other disadvantages), I
> >>>> think you could also directly use the venv here.
> >>>
> >>> Definitely not submodules.
> >>>
> >>> We need to get out of the mindset that submodules are needed for every new
> >>> dependancy we add. Submodules are only appropriate if the external project
> >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> >>> need to bundle in order to prevent a regression for previously deployed
> >>> QEMU installs where the dependancy is known not to exist on all our
> >>> supported platforms.
> >>>
> >>> This does not apply in this case, because the proposed use of tappy is
> >>> merely for a test case. Meson just needs to check if tappy exists and if
> >>> it does, then use it, otherwise skip the tests that need it. The user can
> >>> arrange to install tappy, as they do with the majority of other deps.
> >>>
> >>> If John's venv stuff is relevant, then we don't even need the meson checks,
> >>> just delegate to the venv setup.
> >>>
> >>> Regardless, no submodules are needed or desirable.
> >>
> >> What about keeping biosbits stuff? Source or pre-built.
> >
> > Shipping them as pre-built binaries in QEMU is not a viable option
> > IMHO, especially for grub as a GPL'd project we need to be extremely
> > clear about the exact corresponding source and build process for any
> > binary.
> >
> > For this kind of thing I would generally expect the distro to provide
> > packages that we consume. Looking at biosbits I see it is itself
> > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > even an ancient version of python as a submodule.
> >
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance. I don't think this looks like
> > something we should have in qemu.git or qemu tarballs. It will also
> > make it challenging for the distro to take biosbits at all, unless
> > those 3rd party bundles can be eliminated in favour of using existing
> > builds their have packaged for grub, python, libffi, etc.
>
> So if this depends on some third party binary bits, I think this is pretty
> similar to the tests in the avocado directory ... there we download third
> party binaries, too... Wouldn't it make sense to adapt your tests to that
> framework?

I do not want to bring in the whole avocado framework because it would
unnecessarily make things complicated. I just need the qemu machine
python library and that is enough. For downloading third party stuff,
we can simply wget things from somewhere.

>
>   Thomas
>
Ani Sinha June 28, 2022, 8:49 a.m. UTC | #20
On Tue, Jun 28, 2022 at 14:05 Ani Sinha <ani@anisinha.ca> wrote:

> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <
> berrange@redhat.com> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip
> and pulling
> > >>>>>>>>>>>> packages from the net during testing?
> > >>>>>>>>>>>
> > >>>>>>>>>>> We do that too. See requirements.txt in tests/
> > >>>>>>>>>>> Following two are downloaded:
> > >>>>>>>>>>> avocado-framework==88.1
> > >>>>>>>>>>> pycdlib==1.11.0
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also see this line in Makefie.include:
> > >>>>>>>>>>>
> > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>>>>>>>>>
> > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
> > >>>>>>>>>> the net anyway.
> > >>>>>>>>>> Are the libraries in question not packaged on major distros?
> > >>>>>>>>>
> > >>>>>>>>> Currently I only need this:
> > >>>>>>>>> https://github.com/python-tap/tappy
> > >>>>>>>>> which is the basic TAP processing library for python.
> > >>>>>>>>>
> > >>>>>>>>> It seems its only installed through pip:
> > >>>>>>>>> https://tappy.readthedocs.io/en/latest/
> > >>>>>>>>>
> > >>>>>>>>> I do not think this is packaged by default. It's such a basic
> library
> > >>>>>>>>> for parsing test output that maybe we can keep this somewhere
> within
> > >>>>>>>>> the python src tree? Not sure ...
> > >>>>>>>>
> > >>>>>>>> It's pretty small for sure. Another submodule?
> > >>>>>>>
> > >>>>>>> Unlike BITS, this one is likely going to be maintained for a
> while and
> > >>>>>>> will receive new releases through
> > >>>>>>> https://pypi.org/project/tap.py/
> > >>>>>>> so forking is OK but someone has to keep this updated.
> > >>>>>>>
> > >>>>>>> I am open to anything. Whatever feels right is fine to me.
> > >>>>>>
> > >>>>>> John Snow is currently working on the "Pythonification" of
> various QEMU
> > >>>>>> bits, I think you should loop him into this discussion, too.
> > >>>>>>
> > >>>>>>    Thomas
> > >>>>>
> > >>>>> submodule does not mean we fork necessarily. We could have
> > >>>>> all options: check for the module and use it if there, if not
> > >>>>> use one from system if not there install with pip ..
> > >>>>> But yea, I'm not sure what's best either.
> > >>>>
> > >>>> submodules create a dependency on an internet connection, too. So
> before you
> > >>>> add yet another submodule (which have a couple of other
> disadvantages), I
> > >>>> think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for
> every new
> > >>> dependancy we add. Submodules are only appropriate if the external
> project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or
> if we
> > >>> need to bundle in order to prevent a regression for previously
> deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy
> is
> > >>> merely for a test case. Meson just needs to check if tappy exists
> and if
> > >>> it does, then use it, otherwise skip the tests that need it. The
> user can
> > >>> arrange to install tappy, as they do with the majority of other deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson
> checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, libffi, etc.
> >
> > So if this depends on some third party binary bits, I think this is
> pretty
> > similar to the tests in the avocado directory ... there we download third
> > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > framework?
>
> I do not want to bring in the whole avocado framework because it would
> unnecessarily make things complicated. I just need the qemu machine
> python library and that is enough. For downloading third party stuff,

we can simply wget things from somewhere.


https://pypi.org/project/wget/

That get_asset() call is an overkill for downloading two archives.


>
> >
> >   Thomas
> >
>
Michael S. Tsirkin June 28, 2022, 9:53 a.m. UTC | #21
On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > packages from the net during testing?
> > > > > > > > 
> > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > Following two are downloaded:
> > > > > > > > avocado-framework==88.1
> > > > > > > > pycdlib==1.11.0
> > > > > > > > 
> > > > > > > > Also see this line in Makefie.include:
> > > > > > > > 
> > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > 
> > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > the net anyway.
> > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > 
> > > > > > Currently I only need this:
> > > > > > https://github.com/python-tap/tappy
> > > > > > which is the basic TAP processing library for python.
> > > > > > 
> > > > > > It seems its only installed through pip:
> > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > 
> > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > the python src tree? Not sure ...
> > > > > 
> > > > > It's pretty small for sure. Another submodule?
> > > > 
> > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > will receive new releases through
> > > > https://pypi.org/project/tap.py/
> > > > so forking is OK but someone has to keep this updated.
> > > > 
> > > > I am open to anything. Whatever feels right is fine to me.
> > > 
> > > John Snow is currently working on the "Pythonification" of various QEMU
> > > bits, I think you should loop him into this discussion, too.
> > > 
> > >   Thomas
> > 
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
> submodules create a dependency on an internet connection, too.

Yes but this dependency is explicit.

> So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.
> 
>  Thomas
Michael S. Tsirkin June 28, 2022, 10 a.m. UTC | #22
On Tue, Jun 28, 2022 at 08:48:42AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > packages from the net during testing?
> > > > > > > > > 
> > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > Following two are downloaded:
> > > > > > > > > avocado-framework==88.1
> > > > > > > > > pycdlib==1.11.0
> > > > > > > > > 
> > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > 
> > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > 
> > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > the net anyway.
> > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > 
> > > > > > > Currently I only need this:
> > > > > > > https://github.com/python-tap/tappy
> > > > > > > which is the basic TAP processing library for python.
> > > > > > > 
> > > > > > > It seems its only installed through pip:
> > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > 
> > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > the python src tree? Not sure ...
> > > > > > 
> > > > > > It's pretty small for sure. Another submodule?
> > > > > 
> > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > will receive new releases through
> > > > > https://pypi.org/project/tap.py/
> > > > > so forking is OK but someone has to keep this updated.
> > > > > 
> > > > > I am open to anything. Whatever feels right is fine to me.
> > > > 
> > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > bits, I think you should loop him into this discussion, too.
> > > > 
> > > >   Thomas
> > > 
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> > 
> > submodules create a dependency on an internet connection, too. So before you
> > add yet another submodule (which have a couple of other disadvantages), I
> > think you could also directly use the venv here.
> 
> Definitely not submodules.
> 
> We need to get out of the mindset that submodules are needed for every new
> dependancy we add. Submodules are only appropriate if the external project
> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> need to bundle in order to prevent a regression for previously deployed
> QEMU installs where the dependancy is known not to exist on all our
> supported platforms.
> 
> This does not apply in this case, because the proposed use of tappy is
> merely for a test case. Meson just needs to check if tappy exists and if
> it does, then use it, otherwise skip the tests that need it. The user can
> arrange to install tappy, as they do with the majority of other deps.

I think that's reasonable, except I think we should
make configure do the checks as that is when users see the output
of what is available.

By the way, it might make sense
to have a script setting up developer dependencies for qemu
for major platforms.

> If John's venv stuff is relevant, then we don't even need the meson checks,
> just delegate to the venv setup.

IIUC venv does not install things itself. pip does.

> Regardless, no submodules are needed or desirable.
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Michael S. Tsirkin June 28, 2022, 10:03 a.m. UTC | #23
On Tue, Jun 28, 2022 at 02:19:41PM +0530, Ani Sinha wrote:
> 
> 
> On Tue, Jun 28, 2022 at 14:05 Ani Sinha <ani@anisinha.ca> wrote:
> 
>     On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
>     >
>     > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
>     > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
>     > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <
>     berrange@redhat.com> wrote:
>     > >>>
>     > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
>     > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
>     > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>     > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and
>     pulling
>     > >>>>>>>>>>>> packages from the net during testing?
>     > >>>>>>>>>>>
>     > >>>>>>>>>>> We do that too. See requirements.txt in tests/
>     > >>>>>>>>>>> Following two are downloaded:
>     > >>>>>>>>>>> avocado-framework==88.1
>     > >>>>>>>>>>> pycdlib==1.11.0
>     > >>>>>>>>>>>
>     > >>>>>>>>>>> Also see this line in Makefie.include:
>     > >>>>>>>>>>>
>     > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>     > >>>>>>>>>>
>     > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
>     > >>>>>>>>>> the net anyway.
>     > >>>>>>>>>> Are the libraries in question not packaged on major distros?
>     > >>>>>>>>>
>     > >>>>>>>>> Currently I only need this:
>     > >>>>>>>>> https://github.com/python-tap/tappy
>     > >>>>>>>>> which is the basic TAP processing library for python.
>     > >>>>>>>>>
>     > >>>>>>>>> It seems its only installed through pip:
>     > >>>>>>>>> https://tappy.readthedocs.io/en/latest/
>     > >>>>>>>>>
>     > >>>>>>>>> I do not think this is packaged by default. It's such a basic
>     library
>     > >>>>>>>>> for parsing test output that maybe we can keep this somewhere
>     within
>     > >>>>>>>>> the python src tree? Not sure ...
>     > >>>>>>>>
>     > >>>>>>>> It's pretty small for sure. Another submodule?
>     > >>>>>>>
>     > >>>>>>> Unlike BITS, this one is likely going to be maintained for a
>     while and
>     > >>>>>>> will receive new releases through
>     > >>>>>>> https://pypi.org/project/tap.py/
>     > >>>>>>> so forking is OK but someone has to keep this updated.
>     > >>>>>>>
>     > >>>>>>> I am open to anything. Whatever feels right is fine to me.
>     > >>>>>>
>     > >>>>>> John Snow is currently working on the "Pythonification" of various
>     QEMU
>     > >>>>>> bits, I think you should loop him into this discussion, too.
>     > >>>>>>
>     > >>>>>>    Thomas
>     > >>>>>
>     > >>>>> submodule does not mean we fork necessarily. We could have
>     > >>>>> all options: check for the module and use it if there, if not
>     > >>>>> use one from system if not there install with pip ..
>     > >>>>> But yea, I'm not sure what's best either.
>     > >>>>
>     > >>>> submodules create a dependency on an internet connection, too. So
>     before you
>     > >>>> add yet another submodule (which have a couple of other
>     disadvantages), I
>     > >>>> think you could also directly use the venv here.
>     > >>>
>     > >>> Definitely not submodules.
>     > >>>
>     > >>> We need to get out of the mindset that submodules are needed for
>     every new
>     > >>> dependancy we add. Submodules are only appropriate if the external
>     project
>     > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if
>     we
>     > >>> need to bundle in order to prevent a regression for previously
>     deployed
>     > >>> QEMU installs where the dependancy is known not to exist on all our
>     > >>> supported platforms.
>     > >>>
>     > >>> This does not apply in this case, because the proposed use of tappy
>     is
>     > >>> merely for a test case. Meson just needs to check if tappy exists and
>     if
>     > >>> it does, then use it, otherwise skip the tests that need it. The user
>     can
>     > >>> arrange to install tappy, as they do with the majority of other deps.
>     > >>>
>     > >>> If John's venv stuff is relevant, then we don't even need the meson
>     checks,
>     > >>> just delegate to the venv setup.
>     > >>>
>     > >>> Regardless, no submodules are needed or desirable.
>     > >>
>     > >> What about keeping biosbits stuff? Source or pre-built.
>     > >
>     > > Shipping them as pre-built binaries in QEMU is not a viable option
>     > > IMHO, especially for grub as a GPL'd project we need to be extremely
>     > > clear about the exact corresponding source and build process for any
>     > > binary.
>     > >
>     > > For this kind of thing I would generally expect the distro to provide
>     > > packages that we consume. Looking at biosbits I see it is itself
>     > > bundling a bunch more 3rd party projects, libffi, grub2, and including
>     > > even an ancient version of python as a submodule.
>     > >
>     > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
>     > > turn going to unexpectedly bundle a bunch of other 3rd party projects
>     > > too, all with dubious license compliance. I don't think this looks like
>     > > something we should have in qemu.git or qemu tarballs. It will also
>     > > make it challenging for the distro to take biosbits at all, unless
>     > > those 3rd party bundles can be eliminated in favour of using existing
>     > > builds their have packaged for grub, python, libffi, etc.
>     >
>     > So if this depends on some third party binary bits, I think this is
>     pretty
>     > similar to the tests in the avocado directory ... there we download third
>     > party binaries, too... Wouldn't it make sense to adapt your tests to that
>     > framework?
> 
>     I do not want to bring in the whole avocado framework because it would
>     unnecessarily make things complicated. I just need the qemu machine
>     python library and that is enough. For downloading third party stuff,
> 
>     we can simply wget things from somewhere.
> 
> 
> https://pypi.org/project/wget/
> 
> That get_asset() call is an overkill for downloading two archives. 
> 

For biosbits if we are going this route then I feel a submodule is much
better.  It records which version exactly each qemu version wants.


> 
> 
>     >
>     >   Thomas
>     >
>
Daniel P. Berrangé June 28, 2022, 10:04 a.m. UTC | #24
On Tue, Jun 28, 2022 at 02:05:19PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
> > >>>>>>>>>>>> packages from the net during testing?
> > >>>>>>>>>>>
> > >>>>>>>>>>> We do that too. See requirements.txt in tests/
> > >>>>>>>>>>> Following two are downloaded:
> > >>>>>>>>>>> avocado-framework==88.1
> > >>>>>>>>>>> pycdlib==1.11.0
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also see this line in Makefie.include:
> > >>>>>>>>>>>
> > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>>>>>>>>>
> > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
> > >>>>>>>>>> the net anyway.
> > >>>>>>>>>> Are the libraries in question not packaged on major distros?
> > >>>>>>>>>
> > >>>>>>>>> Currently I only need this:
> > >>>>>>>>> https://github.com/python-tap/tappy
> > >>>>>>>>> which is the basic TAP processing library for python.
> > >>>>>>>>>
> > >>>>>>>>> It seems its only installed through pip:
> > >>>>>>>>> https://tappy.readthedocs.io/en/latest/
> > >>>>>>>>>
> > >>>>>>>>> I do not think this is packaged by default. It's such a basic library
> > >>>>>>>>> for parsing test output that maybe we can keep this somewhere within
> > >>>>>>>>> the python src tree? Not sure ...
> > >>>>>>>>
> > >>>>>>>> It's pretty small for sure. Another submodule?
> > >>>>>>>
> > >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and
> > >>>>>>> will receive new releases through
> > >>>>>>> https://pypi.org/project/tap.py/
> > >>>>>>> so forking is OK but someone has to keep this updated.
> > >>>>>>>
> > >>>>>>> I am open to anything. Whatever feels right is fine to me.
> > >>>>>>
> > >>>>>> John Snow is currently working on the "Pythonification" of various QEMU
> > >>>>>> bits, I think you should loop him into this discussion, too.
> > >>>>>>
> > >>>>>>    Thomas
> > >>>>>
> > >>>>> submodule does not mean we fork necessarily. We could have
> > >>>>> all options: check for the module and use it if there, if not
> > >>>>> use one from system if not there install with pip ..
> > >>>>> But yea, I'm not sure what's best either.
> > >>>>
> > >>>> submodules create a dependency on an internet connection, too. So before you
> > >>>> add yet another submodule (which have a couple of other disadvantages), I
> > >>>> think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for every new
> > >>> dependancy we add. Submodules are only appropriate if the external project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > >>> need to bundle in order to prevent a regression for previously deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy is
> > >>> merely for a test case. Meson just needs to check if tappy exists and if
> > >>> it does, then use it, otherwise skip the tests that need it. The user can
> > >>> arrange to install tappy, as they do with the majority of other deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, libffi, etc.
> >
> > So if this depends on some third party binary bits, I think this is pretty
> > similar to the tests in the avocado directory ... there we download third
> > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > framework?
> 
> I do not want to bring in the whole avocado framework because it would
> unnecessarily make things complicated. I just need the qemu machine
> python library and that is enough. For downloading third party stuff,
> we can simply wget things from somewhere.

Well avocado is already a part of QEMU's testing framework, so it is
not bringing in a whole new framework. Arguably the introduction of
python usage in qtest is what's introducing a new maint burden here.

If it is actually booting a real guest image (from biosbits) and interacting
with it, then it does feel like the scope of this testing is more appropriate
to QEMU's avocado framework than qtest, especially given the desire to use
python for it all.

With regards,
Daniel
Michael S. Tsirkin June 28, 2022, 10:07 a.m. UTC | #25
On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> If it is actually booting a real guest image (from biosbits) and interacting
> with it, then it does feel like the scope of this testing is more appropriate
> to QEMU's avocado framework than qtest, especially given the desire to use
> python for it all.
> 
> With regards,
> Daniel

I feel avocado is directed towards booting full fledged guest OS.
It makes it much easier to figure out guest issues but it also
prone to false positives and is harder to debug as a result.
Booting a minimal image like this shouldn't require that.
Michael S. Tsirkin June 28, 2022, 10:12 a.m. UTC | #26
On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote:
> So bundling a pre-built biosbits in QEMU appears to mean that we're in
> turn going to unexpectedly bundle a bunch of other 3rd party projects
> too, all with dubious license compliance.

Well looks like classical mere aggregation to me ... license issues
need to be figured out if we are to distribute things but I think
this is basically what distros do anyway.

And I doubt we want to support arbitrary versions of grub etc,
they are very distro specific tools.
I don't see why we can't have the resulting ISOs in some submodule -
nothing requires us to distribute it in qemu tarballs.
Daniel P. Berrangé June 28, 2022, 10:14 a.m. UTC | #27
On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > 
> > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > 
> > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > 
> > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > 
> > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > 
> > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > the net anyway.
> > > > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > > > > 
> > > > > > > > > > Currently I only need this:
> > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > 
> > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > 
> > > > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > 
> > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > 
> > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > > > will receive new releases through
> > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > 
> > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > 
> > > > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > 
> > > > > > >    Thomas
> > > > > > 
> > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > all options: check for the module and use it if there, if not
> > > > > > use one from system if not there install with pip ..
> > > > > > But yea, I'm not sure what's best either.
> > > > > 
> > > > > submodules create a dependency on an internet connection, too. So before you
> > > > > add yet another submodule (which have a couple of other disadvantages), I
> > > > > think you could also directly use the venv here.
> > > > 
> > > > Definitely not submodules.
> > > > 
> > > > We need to get out of the mindset that submodules are needed for every new
> > > > dependancy we add. Submodules are only appropriate if the external project
> > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > need to bundle in order to prevent a regression for previously deployed
> > > > QEMU installs where the dependancy is known not to exist on all our
> > > > supported platforms.
> > > > 
> > > > This does not apply in this case, because the proposed use of tappy is
> > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > it does, then use it, otherwise skip the tests that need it. The user can
> > > > arrange to install tappy, as they do with the majority of other deps.
> > > > 
> > > > If John's venv stuff is relevant, then we don't even need the meson checks,
> > > > just delegate to the venv setup.
> > > > 
> > > > Regardless, no submodules are needed or desirable.
> > > 
> > > What about keeping biosbits stuff? Source or pre-built.
> > 
> > Shipping them as pre-built binaries in QEMU is not a viable option
> > IMHO, especially for grub as a GPL'd project we need to be extremely
> > clear about the exact corresponding source and build process for any
> > binary.
> > 
> > For this kind of thing I would generally expect the distro to provide
> > packages that we consume. Looking at biosbits I see it is itself
> > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > even an ancient version of python as a submodule.
> > 
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance. I don't think this looks like
> > something we should have in qemu.git or qemu tarballs. It will also
> > make it challenging for the distro to take biosbits at all, unless
> > those 3rd party bundles can be eliminated in favour of using existing
> > builds their have packaged for grub, python, libffi, etc.
> 
> So if this depends on some third party binary bits, I think this is pretty
> similar to the tests in the avocado directory ... there we download third
> party binaries, too... Wouldn't it make sense to adapt your tests to that
> framework?

Now that you mention it, avocado does feel like a more appropriate fit.
IIUC the biosbits project appears to be effectively providing a custom
guest OS ISO image. IOW this testing is quite biased towards being
integration testing which is the target of avocado, while qtest is much
more to the unit testing end of the spectrum. This would avoid all the
discussion and patches around introducing python to qtest

With regards,
Daniel
Daniel P. Berrangé June 28, 2022, 10:16 a.m. UTC | #28
On Tue, Jun 28, 2022 at 06:12:48AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote:
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance.
> 
> Well looks like classical mere aggregation to me ... license issues
> need to be figured out if we are to distribute things but I think
> this is basically what distros do anyway.
> 
> And I doubt we want to support arbitrary versions of grub etc,
> they are very distro specific tools.
> I don't see why we can't have the resulting ISOs in some submodule -
> nothing requires us to distribute it in qemu tarballs.

If we don't distribute it in the tarballs then these tests won't be
usable for testing release builds of QEMU which feels wrong to me.

With regards,
Daniel
Daniel P. Berrangé June 28, 2022, 10:18 a.m. UTC | #29
On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > If it is actually booting a real guest image (from biosbits) and interacting
> > with it, then it does feel like the scope of this testing is more appropriate
> > to QEMU's avocado framework than qtest, especially given the desire to use
> > python for it all.
> > 
> > With regards,
> > Daniel
> 
> I feel avocado is directed towards booting full fledged guest OS.

That's essentially what this is doing - its a custom guest OS rather
than a common distro IIUC

> It makes it much easier to figure out guest issues but it also
> prone to false positives and is harder to debug as a result.
> Booting a minimal image like this shouldn't require that.

Well avocado is as reliable as the tests you write for it. The problems
are largely around the images being used in avocado. If the biosbits
testing system is reliable, then avocado will be too, and if they not
reliable, then it will affect qtest too.

With regards,
Daniel
Michael S. Tsirkin June 28, 2022, 10:21 a.m. UTC | #30
On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote:
> Now that you mention it, avocado does feel like a more appropriate fit.
> IIUC the biosbits project appears to be effectively providing a custom
> guest OS ISO image. IOW this testing is quite biased towards being
> integration testing which is the target of avocado, while qtest is much
> more to the unit testing end of the spectrum. This would avoid all the
> discussion and patches around introducing python to qtest
> 
> With regards,
> Daniel

Yes it's an ISO image but not a full OS. So IMHO not really a
integration test, more like  a unit test for ACPI - current
tests we have just compare ACPI to expected file which is
okay but very limited.
Thomas Huth June 28, 2022, 10:21 a.m. UTC | #31
On 28/06/2022 12.03, Michael S. Tsirkin wrote:
[...]
> For biosbits if we are going this route then I feel a submodule is much
> better.  It records which version exactly each qemu version wants.

As far as I know, you can also specify the version when using pip, can't 
you? So that's not really an advantage here.

On the contrary, submodules have a couple of disadvantages that I really 
dislike:

- submodules do not get updated automatically when doing a "git checkout", 
we have to update them via a script instead. This causes e.g. trouble if you 
rsync your source tree to a machine that has no access to the internet and 
you forgot to update the submodule before the sync

- the content of submodules is not added to the tarballs that get created on 
the git forges automatically. There were lots of requests from users in the 
past that tried to download a tarball from github and then wondered why they 
couldn't compile QEMU.

- we include the submodule content in our release tarballs, so people get 
the impression that hte submodule content is part of the QEMU sources. This 
has two disadvantages:
  * We already got bug reports for the code in the submodule,
    where people did not understand that they should report that
    rather to the original project instead (i.e. you ship it - you
    own it)
  * People get the impression that QEMU is a huge monster
    application if they count the number of code lines, run
    their code scanner tools on the tarball contents, etc.
    Remember "nemu", for example, where one of the main complaints
    was that QEMU has too many lines of code?

- If programs includes code via submodules, this gets a higher
   burder for distro maintainers, since they have to patch each
   and every package when there is a bug, instead of being able to
   fix it in one central place.

So in my opinion we should avoid new submodules if there is an alternative.

  Thomas
Michael S. Tsirkin June 28, 2022, 10:25 a.m. UTC | #32
On Tue, Jun 28, 2022 at 11:18:14AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and interacting
> > > with it, then it does feel like the scope of this testing is more appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > > 
> > > With regards,
> > > Daniel
> > 
> > I feel avocado is directed towards booting full fledged guest OS.
> 
> That's essentially what this is doing - its a custom guest OS rather
> than a common distro IIUC

yes but then so is bios tables test - we generate the disk on the fly.

> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
> 
> Well avocado is as reliable as the tests you write for it. The problems
> are largely around the images being used in avocado.

The idea of downloading large images as part of the test as opposed to
part of setup is a large part of what makes it flaky - due to
running into unpredictable latency and errors on both the internet and
local IO side of things.

> If the biosbits
> testing system is reliable, then avocado will be too, and if they not
> reliable, then it will affect qtest too.

A larger system has more potential for bugs ;) I'm with Ani here. Using
a system that is overkill will just make debugging more painful than it
needs to be.

> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Ani Sinha June 28, 2022, 10:28 a.m. UTC | #33
On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > If it is actually booting a real guest image (from biosbits) and interacting
> > with it, then it does feel like the scope of this testing is more appropriate
> > to QEMU's avocado framework than qtest, especially given the desire to use
> > python for it all.
> >
> > With regards,
> > Daniel
>
> I feel avocado is directed towards booting full fledged guest OS.
> It makes it much easier to figure out guest issues but it also
> prone to false positives and is harder to debug as a result.
> Booting a minimal image like this shouldn't require that.

Yes 100% agree with Michael on this. Biobits is *not* booting any OS
image. It runs off grub, that is, directly from bootloader stage. The
interraction with the VM is minimal.

>
> --
> MST
>
Ani Sinha June 28, 2022, 10:30 a.m. UTC | #34
On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > >
> > > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > >
> > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > >
> > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > > the net anyway.
> > > > > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > > > > >
> > > > > > > > > > > Currently I only need this:
> > > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > >
> > > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > >
> > > > > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > >
> > > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > >
> > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > > > > will receive new releases through
> > > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > >
> > > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > >
> > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > >
> > > > > > > >    Thomas
> > > > > > >
> > > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > > all options: check for the module and use it if there, if not
> > > > > > > use one from system if not there install with pip ..
> > > > > > > But yea, I'm not sure what's best either.
> > > > > >
> > > > > > submodules create a dependency on an internet connection, too. So before you
> > > > > > add yet another submodule (which have a couple of other disadvantages), I
> > > > > > think you could also directly use the venv here.
> > > > >
> > > > > Definitely not submodules.
> > > > >
> > > > > We need to get out of the mindset that submodules are needed for every new
> > > > > dependancy we add. Submodules are only appropriate if the external project
> > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > > need to bundle in order to prevent a regression for previously deployed
> > > > > QEMU installs where the dependancy is known not to exist on all our
> > > > > supported platforms.
> > > > >
> > > > > This does not apply in this case, because the proposed use of tappy is
> > > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > > it does, then use it, otherwise skip the tests that need it. The user can
> > > > > arrange to install tappy, as they do with the majority of other deps.
> > > > >
> > > > > If John's venv stuff is relevant, then we don't even need the meson checks,
> > > > > just delegate to the venv setup.
> > > > >
> > > > > Regardless, no submodules are needed or desirable.
> > > >
> > > > What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, libffi, etc.
> >
> > So if this depends on some third party binary bits, I think this is pretty
> > similar to the tests in the avocado directory ... there we download third
> > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > framework?
>
> Now that you mention it, avocado does feel like a more appropriate fit.
> IIUC the biosbits project appears to be effectively providing a custom
> guest OS ISO image. IOW this testing is quite biased towards being
> integration testing which is the target of avocado, while qtest is much
> more to the unit testing end of the spectrum.

This is more like unit testing than integration testing, now that you
mention it. It tests only the bios, very narrowly and does not involve
any OS at all.

This would avoid all the
> discussion and patches around introducing python to qtest
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
Thomas Huth June 28, 2022, 10:30 a.m. UTC | #35
On 28/06/2022 12.21, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote:
>> Now that you mention it, avocado does feel like a more appropriate fit.
>> IIUC the biosbits project appears to be effectively providing a custom
>> guest OS ISO image. IOW this testing is quite biased towards being
>> integration testing which is the target of avocado, while qtest is much
>> more to the unit testing end of the spectrum. This would avoid all the
>> discussion and patches around introducing python to qtest
>>
>> With regards,
>> Daniel
> 
> Yes it's an ISO image but not a full OS.

The avocado framework is not meant for full OS testing only. We have a 
couple of tests there that just run a small firmware image. For example:

- tests/avocado/machine_m68k_nextcube.py : Just runs the original firmware 
of the NexT Cube machine

- tests/avocado/ppc_405.py : Just runs an U-Boot firmware image

Some of the tests even do not download anything at all and just run a QEMU 
instance via python, e.g.:

- tests/avocado/virtio_check_params.py

- tests/avocado/x86_cpu_model_versions.py

So I fail to see why biosbits should be so much different here.

Of course we could argue whether the avocado framework is really such a 
great fit for the QEMU test suite (as I said in another mail, it rather 
feels like an oddball nowadays), but that's a different argument.

  Thomas
Michael S. Tsirkin June 28, 2022, 10:30 a.m. UTC | #36
On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> [...]
> > For biosbits if we are going this route then I feel a submodule is much
> > better.  It records which version exactly each qemu version wants.
> 
> As far as I know, you can also specify the version when using pip, can't
> you? So that's not really an advantage here.

But of course if you do you do not get updates ;) You do
however rely on a 3rd party to faithfully provide you
correct code based on the version, and host it forever.

> On the contrary, submodules have a couple of disadvantages that I really
> dislike:
> 
> - submodules do not get updated automatically when doing a "git checkout",
> we have to update them via a script instead. This causes e.g. trouble if you
> rsync your source tree to a machine that has no access to the internet and
> you forgot to update the submodule before the sync

how is pip better?

> - the content of submodules is not added to the tarballs that get created on
> the git forges automatically. There were lots of requests from users in the
> past that tried to download a tarball from github and then wondered why they
> couldn't compile QEMU.

how is pip better here?

> - we include the submodule content in our release tarballs, so people get
> the impression that hte submodule content is part of the QEMU sources. This
> has two disadvantages:
>  * We already got bug reports for the code in the submodule,
>    where people did not understand that they should report that
>    rather to the original project instead (i.e. you ship it - you
>    own it)
>  * People get the impression that QEMU is a huge monster
>    application if they count the number of code lines, run
>    their code scanner tools on the tarball contents, etc.
>    Remember "nemu", for example, where one of the main complaints
>    was that QEMU has too many lines of code?

I think we can skip the checkout in the tarball if we like.
If people want to run the test they can checkout then.


> - If programs includes code via submodules, this gets a higher
>   burder for distro maintainers, since they have to patch each
>   and every package when there is a bug, instead of being able to
>   fix it in one central place.

Come on, this is just a test. We *really* don't care if an ISO
we use to test ACPI is using an exploitable version of grub.

> So in my opinion we should avoid new submodules if there is an alternative.
> 
>  Thomas

Interesting take generally, but I don't see how the logic applies in this
case. Would appreciate hearing your answers to the above.
Ani Sinha June 28, 2022, 10:41 a.m. UTC | #37
On Tue, Jun 28, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and interacting
> > > with it, then it does feel like the scope of this testing is more appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > >
> > > With regards,
> > > Daniel
> >
> > I feel avocado is directed towards booting full fledged guest OS.
>
> That's essentially what this is doing - its a custom guest OS rather

Daniel, there is no OS :-) Like there is no kernel image, no
persistent real file system (there is a fake one called pyfs), no
packages, nothing. Its very minimalistic image narrowly targeted for
one thing alone - to test the bios.

> than a common distro IIUC
>
> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
>
> Well avocado is as reliable as the tests you write for it. The problems
> are largely around the images being used in avocado. If the biosbits
> testing system is reliable, then avocado will be too, and if they not
> reliable, then it will affect qtest too.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
Daniel P. Berrangé June 28, 2022, 10:42 a.m. UTC | #38
On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and interacting
> > > with it, then it does feel like the scope of this testing is more appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > >
> > > With regards,
> > > Daniel
> >
> > I feel avocado is directed towards booting full fledged guest OS.
> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
> 
> Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> image. It runs off grub, that is, directly from bootloader stage. The
> interraction with the VM is minimal.

Just because it doesn't run a whole Linux kernel, doesn't make it
not a guest OS image. It is merely unsual in that it can do everything
it needs from grub stage, because it is just poking low level BIOS
stuff and doesn't need a full OS like Linux on top. This is still
functional integration testing IMHO and relevant to avocado in QEMU.

With regards,
Daniel
Peter Maydell June 28, 2022, 10:43 a.m. UTC | #39
On Tue, 28 Jun 2022 at 11:38, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > - we include the submodule content in our release tarballs, so people get
> > the impression that hte submodule content is part of the QEMU sources. This
> > has two disadvantages:
> >  * We already got bug reports for the code in the submodule,
> >    where people did not understand that they should report that
> >    rather to the original project instead (i.e. you ship it - you
> >    own it)
> >  * People get the impression that QEMU is a huge monster
> >    application if they count the number of code lines, run
> >    their code scanner tools on the tarball contents, etc.
> >    Remember "nemu", for example, where one of the main complaints
> >    was that QEMU has too many lines of code?
>
> I think we can skip the checkout in the tarball if we like.
> If people want to run the test they can checkout then.

For tarballs and submodules, we want to provide the code in the
cases where we're providing binary blobs, and for where it's
required to build QEMU proper.

Overall I think that the approach we use today for providing
guest binaries (submodules with the code, pre-built blobs checked
into git) is creaking at the seams and often awkward for downstream
distros (who want to rebuild the binaries anyway).

Plus submodules in general in git work really badly and awkwardly,
and I'd rather we didn't add them unless we really must.

We already have an approach for "tests that use binaries" --
the avocado test suites. Is that something we could use in this
case ?

thanks
-- PMM
Ani Sinha June 28, 2022, 10:49 a.m. UTC | #40
On Tue, Jun 28, 2022 at 4:00 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> > > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > > >
> > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > > > the net anyway.
> > > > > > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > > > > > >
> > > > > > > > > > > > Currently I only need this:
> > > > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > > >
> > > > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > > >
> > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > > >
> > > > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > > >
> > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > > > > > will receive new releases through
> > > > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > > >
> > > > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > > >
> > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > > >
> > > > > > > > >    Thomas
> > > > > > > >
> > > > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > > > all options: check for the module and use it if there, if not
> > > > > > > > use one from system if not there install with pip ..
> > > > > > > > But yea, I'm not sure what's best either.
> > > > > > >
> > > > > > > submodules create a dependency on an internet connection, too. So before you
> > > > > > > add yet another submodule (which have a couple of other disadvantages), I
> > > > > > > think you could also directly use the venv here.
> > > > > >
> > > > > > Definitely not submodules.
> > > > > >
> > > > > > We need to get out of the mindset that submodules are needed for every new
> > > > > > dependancy we add. Submodules are only appropriate if the external project
> > > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > > > need to bundle in order to prevent a regression for previously deployed
> > > > > > QEMU installs where the dependancy is known not to exist on all our
> > > > > > supported platforms.
> > > > > >
> > > > > > This does not apply in this case, because the proposed use of tappy is
> > > > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > > > it does, then use it, otherwise skip the tests that need it. The user can
> > > > > > arrange to install tappy, as they do with the majority of other deps.
> > > > > >
> > > > > > If John's venv stuff is relevant, then we don't even need the meson checks,
> > > > > > just delegate to the venv setup.
> > > > > >
> > > > > > Regardless, no submodules are needed or desirable.
> > > > >
> > > > > What about keeping biosbits stuff? Source or pre-built.
> > > >
> > > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > > clear about the exact corresponding source and build process for any
> > > > binary.
> > > >
> > > > For this kind of thing I would generally expect the distro to provide
> > > > packages that we consume. Looking at biosbits I see it is itself
> > > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > > even an ancient version of python as a submodule.
> > > >
> > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > > too, all with dubious license compliance. I don't think this looks like
> > > > something we should have in qemu.git or qemu tarballs. It will also
> > > > make it challenging for the distro to take biosbits at all, unless
> > > > those 3rd party bundles can be eliminated in favour of using existing
> > > > builds their have packaged for grub, python, libffi, etc.
> > >
> > > So if this depends on some third party binary bits, I think this is pretty
> > > similar to the tests in the avocado directory ... there we download third
> > > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > > framework?
> >
> > Now that you mention it, avocado does feel like a more appropriate fit.
> > IIUC the biosbits project appears to be effectively providing a custom
> > guest OS ISO image. IOW this testing is quite biased towards being
> > integration testing which is the target of avocado, while qtest is much
> > more to the unit testing end of the spectrum.
>
> This is more like unit testing than integration testing, now that you
> mention it. It tests only the bios, very narrowly and does not involve
> any OS at all.

Another thing to consider is that integration testing is further down
the line? Not for once when submitting patches on acpi have I run
them. However, every time I have run make check to make sure
bios-tables-test passes and I am not breaking anything. It's much more
useful to have this kind of thing part of make check before patch
submitters can quickly check for failures either in bios-tables-test
or in bits. Also its lot easier to add new acpi/smbios tests as a part
of this when bios-tables-test and this one are closer together.

>
> This would avoid all the
> > discussion and patches around introducing python to qtest
> >
> > With regards,
> > Daniel
> > --
> > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> >
Thomas Huth June 28, 2022, 10:50 a.m. UTC | #41
On 28/06/2022 12.30, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>> [...]
>>> For biosbits if we are going this route then I feel a submodule is much
>>> better.  It records which version exactly each qemu version wants.
>>
>> As far as I know, you can also specify the version when using pip, can't
>> you? So that's not really an advantage here.
> 
> But of course if you do you do not get updates ;) You do
> however rely on a 3rd party to faithfully provide you
> correct code based on the version, and host it forever.
> 
>> On the contrary, submodules have a couple of disadvantages that I really
>> dislike:
>>
>> - submodules do not get updated automatically when doing a "git checkout",
>> we have to update them via a script instead. This causes e.g. trouble if you
>> rsync your source tree to a machine that has no access to the internet and
>> you forgot to update the submodule before the sync
> 
> how is pip better?

You don't end up with an inconsistent source tree in that case (which 
happens with submodules quite easily, at least for me it happened a couple 
of times already). Either the machine has an internet connection, so that 
pip can install the missing bits, or it does not and the test has to be 
skipped. But if I copy the wrong state of a submodule around, things get 
messed up quite easily in my experience. Ok, you could say that this is just 
my special setup with rsync, but already given the fact that "git checkout" 
creates an inconsistent state of your source tree until you run the script 
for updating the submodules the next time is an indication that submodules 
are rather a shaky thing (e.g. if you'd create a tarball for somebody else 
from your source tree right after doing a "git checkout").

>> - the content of submodules is not added to the tarballs that get created on
>> the git forges automatically. There were lots of requests from users in the
>> past that tried to download a tarball from github and then wondered why they
>> couldn't compile QEMU.
> 
> how is pip better here?

You don't get incomplete/non-working tarballs in that case.

>> - we include the submodule content in our release tarballs, so people get
>> the impression that hte submodule content is part of the QEMU sources. This
>> has two disadvantages:
>>   * We already got bug reports for the code in the submodule,
>>     where people did not understand that they should report that
>>     rather to the original project instead (i.e. you ship it - you
>>     own it)
>>   * People get the impression that QEMU is a huge monster
>>     application if they count the number of code lines, run
>>     their code scanner tools on the tarball contents, etc.
>>     Remember "nemu", for example, where one of the main complaints
>>     was that QEMU has too many lines of code?
> 
> I think we can skip the checkout in the tarball if we like.
> If people want to run the test they can checkout then.

Release tarballs don't include the ".git" folder infrastructur, so everybody 
who downloads a tarball will simply never be able to run the test.

> 
>> - If programs includes code via submodules, this gets a higher
>>    burder for distro maintainers, since they have to patch each
>>    and every package when there is a bug, instead of being able to
>>    fix it in one central place.
> 
> Come on, this is just a test. We *really* don't care if an ISO
> we use to test ACPI is using an exploitable version of grub.

Wait, I thought we were only talking about tappy here? The ISO binaries 
should certainly *not* be bundled in the QEMU tarballs (they are too big 
already anyway, we should rather think of moving the firmware binaries out 
of the tarball instead).

  Thomas
Michael S. Tsirkin June 28, 2022, 11 a.m. UTC | #42
On Tue, Jun 28, 2022 at 11:43:58AM +0100, Peter Maydell wrote:
> On Tue, 28 Jun 2022 at 11:38, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >  * We already got bug reports for the code in the submodule,
> > >    where people did not understand that they should report that
> > >    rather to the original project instead (i.e. you ship it - you
> > >    own it)
> > >  * People get the impression that QEMU is a huge monster
> > >    application if they count the number of code lines, run
> > >    their code scanner tools on the tarball contents, etc.
> > >    Remember "nemu", for example, where one of the main complaints
> > >    was that QEMU has too many lines of code?
> >
> > I think we can skip the checkout in the tarball if we like.
> > If people want to run the test they can checkout then.
> 
> For tarballs and submodules, we want to provide the code in the
> cases where we're providing binary blobs, and for where it's
> required to build QEMU proper.
> 
> Overall I think that the approach we use today for providing
> guest binaries (submodules with the code, pre-built blobs checked
> into git) is creaking at the seams and often awkward for downstream
> distros (who want to rebuild the binaries anyway).
> 
> Plus submodules in general in git work really badly and awkwardly,
> and I'd rather we didn't add them unless we really must.
> 
> We already have an approach for "tests that use binaries" --
> the avocado test suites. Is that something we could use in this
> case ?
> 
> thanks
> -- PMM

From experience avocado is basically rolling its own reimplementation of
git fetch. Result is weird failures when some corner case or another
isn't handled perfectly. Kind of unavoidable since you can't have
ISOs for a ton of guest OSes reasonably in a git repo. Well maybe
with sparse checkouts you can. Haven't really looked at it.

These specific tests are weird, kind of special in that they are developer
oriented.  I *would* like anyone making changes to ACPI to run them, and
for CI to run them on patches, but I do not believe e.g. distros
necessarily need them.

git submodules are awkward basically because they are an automated wget.
I don't think an explicit wget is much better ... but
looks like I'm alone in this. Oh well.
So it will be a weird dance of wget a tarball, unpack, generate
ISO and run. God help you if you need to patch the test - it's
wget all the way down.
Michael S. Tsirkin June 28, 2022, 11:14 a.m. UTC | #43
On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> On 28/06/2022 12.30, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > [...]
> > > > For biosbits if we are going this route then I feel a submodule is much
> > > > better.  It records which version exactly each qemu version wants.
> > > 
> > > As far as I know, you can also specify the version when using pip, can't
> > > you? So that's not really an advantage here.
> > 
> > But of course if you do you do not get updates ;) You do
> > however rely on a 3rd party to faithfully provide you
> > correct code based on the version, and host it forever.
> > 
> > > On the contrary, submodules have a couple of disadvantages that I really
> > > dislike:
> > > 
> > > - submodules do not get updated automatically when doing a "git checkout",
> > > we have to update them via a script instead. This causes e.g. trouble if you
> > > rsync your source tree to a machine that has no access to the internet and
> > > you forgot to update the submodule before the sync
> > 
> > how is pip better?
> 
> You don't end up with an inconsistent source tree in that case (which
> happens with submodules quite easily, at least for me it happened a couple
> of times already). Either the machine has an internet connection, so that
> pip can install the missing bits, or it does not and the test has to be
> skipped.

skipped tests are too easy to ignore ...

> But if I copy the wrong state of a submodule around, things get
> messed up quite easily in my experience. Ok, you could say that this is just
> my special setup with rsync, but already given the fact that "git checkout"
> creates an inconsistent state of your source tree until you run the script
> for updating the submodules the next time is an indication that submodules
> are rather a shaky thing (e.g. if you'd create a tarball for somebody else
> from your source tree right after doing a "git checkout").

yea one has to remember to set submodule.recurse = true in .gitconfig

I agree it's annoying, I guess they don't change it for compat reasons.


> > > - the content of submodules is not added to the tarballs that get created on
> > > the git forges automatically. There were lots of requests from users in the
> > > past that tried to download a tarball from github and then wondered why they
> > > couldn't compile QEMU.
> > 
> > how is pip better here?
> 
> You don't get incomplete/non-working tarballs in that case.

So skip the test ;)

> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >   * We already got bug reports for the code in the submodule,
> > >     where people did not understand that they should report that
> > >     rather to the original project instead (i.e. you ship it - you
> > >     own it)
> > >   * People get the impression that QEMU is a huge monster
> > >     application if they count the number of code lines, run
> > >     their code scanner tools on the tarball contents, etc.
> > >     Remember "nemu", for example, where one of the main complaints
> > >     was that QEMU has too many lines of code?
> > 
> > I think we can skip the checkout in the tarball if we like.
> > If people want to run the test they can checkout then.
> 
> Release tarballs don't include the ".git" folder infrastructur, so everybody
> who downloads a tarball will simply never be able to run the test.

I actually think I'm fine with that for this specific case.

> > 
> > > - If programs includes code via submodules, this gets a higher
> > >    burder for distro maintainers, since they have to patch each
> > >    and every package when there is a bug, instead of being able to
> > >    fix it in one central place.
> > 
> > Come on, this is just a test. We *really* don't care if an ISO
> > we use to test ACPI is using an exploitable version of grub.
> 
> Wait, I thought we were only talking about tappy here? The ISO binaries
> should certainly *not* be bundled in the QEMU tarballs (they are too big
> already anyway, we should rather think of moving the firmware binaries out
> of the tarball instead).
> 
>  Thomas

IIUC there are three things we are discussing
- biosbits source
- biosbits image
- tappy
Michael S. Tsirkin June 28, 2022, 11:18 a.m. UTC | #44
On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > > If it is actually booting a real guest image (from biosbits) and interacting
> > > > with it, then it does feel like the scope of this testing is more appropriate
> > > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > > python for it all.
> > > >
> > > > With regards,
> > > > Daniel
> > >
> > > I feel avocado is directed towards booting full fledged guest OS.
> > > It makes it much easier to figure out guest issues but it also
> > > prone to false positives and is harder to debug as a result.
> > > Booting a minimal image like this shouldn't require that.
> > 
> > Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> > image. It runs off grub, that is, directly from bootloader stage. The
> > interraction with the VM is minimal.
> 
> Just because it doesn't run a whole Linux kernel, doesn't make it
> not a guest OS image. It is merely unsual in that it can do everything
> it needs from grub stage, because it is just poking low level BIOS
> stuff and doesn't need a full OS like Linux on top. This is still
> functional integration testing IMHO and relevant to avocado in QEMU.
> 
> With regards,
> Daniel

I wonder why it needs grub BTW. Let's just load all of disk into ram
and be done with it?


> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Michael S. Tsirkin June 28, 2022, 11:28 a.m. UTC | #45
On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > > If it is actually booting a real guest image (from biosbits) and interacting
> > > > with it, then it does feel like the scope of this testing is more appropriate
> > > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > > python for it all.
> > > >
> > > > With regards,
> > > > Daniel
> > >
> > > I feel avocado is directed towards booting full fledged guest OS.
> > > It makes it much easier to figure out guest issues but it also
> > > prone to false positives and is harder to debug as a result.
> > > Booting a minimal image like this shouldn't require that.
> > 
> > Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> > image. It runs off grub, that is, directly from bootloader stage. The
> > interraction with the VM is minimal.
> 
> Just because it doesn't run a whole Linux kernel, doesn't make it
> not a guest OS image. It is merely unsual in that it can do everything
> it needs from grub stage, because it is just poking low level BIOS
> stuff and doesn't need a full OS like Linux on top. This is still
> functional integration testing IMHO and relevant to avocado in QEMU.
> 
> With regards,
> Daniel

I think the main difference is not even in how it works, it's
in what it does. Which is check that ACPI tables are sane.
Who cares about that? Well developers do when they change the
tables. Users really don't because for users we have the expected
tables in tree and we check against these.
Peter Maydell June 28, 2022, 12:10 p.m. UTC | #46
On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> I think the main difference is not even in how it works, it's
> in what it does. Which is check that ACPI tables are sane.
> Who cares about that? Well developers do when they change the
> tables. Users really don't because for users we have the expected
> tables in tree and we check against these.

It wants to build and run a big guest binary blob -- that to me is
the main difference. Users don't much care about any of our tests,
whether they're under 'make check' or 'make check-avocado' or the
iotests framework. The reason to pick one framework or another
is mostly I think whether the properties of the test are such
that one framework works better. Avocado is (for better or worse)
the one we have for dealing with "actually run a guest machine
with a big lump of guest code in it".

-- PMM
Ani Sinha June 28, 2022, 12:36 p.m. UTC | #47
On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > I think the main difference is not even in how it works, it's
> > in what it does. Which is check that ACPI tables are sane.
> > Who cares about that? Well developers do when they change the
> > tables. Users really don't because for users we have the expected
> > tables in tree and we check against these.
>
> It wants to build and run a big guest binary blob -- that to me is
> the main difference. Users don't much care about any of our tests,

perhaps but we do enforce patch submitters to make sure make check
passes before submitting patches. make check-avocado is not run as
part of make check, requires considerable disk space to download all
guest images and hence generally not run by patch submitters. Making
bits parts of avocado tests almost defeats the purpose of having this
test at all.

> whether they're under 'make check' or 'make check-avocado' or the
> iotests framework. The reason to pick one framework or another
> is mostly I think whether the properties of the test are such
> that one framework works better. Avocado is (for better or worse)
> the one we have for dealing with "actually run a guest machine
> with a big lump of guest code in it".
>
> -- PMM
Thomas Huth June 28, 2022, 12:39 p.m. UTC | #48
On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
[...]
>>> Come on, this is just a test. We *really* don't care if an ISO
>>> we use to test ACPI is using an exploitable version of grub.
>>
>> Wait, I thought we were only talking about tappy here? The ISO binaries
>> should certainly *not* be bundled in the QEMU tarballs (they are too big
>> already anyway, we should rather think of moving the firmware binaries out
>> of the tarball instead).
>>
>>   Thomas
> 
> IIUC there are three things we are discussing
> - biosbits source
> - biosbits image
> - tappy

Oh well, I missed that part of the discussion so far since the corresponding 
patches did not make it to the mailing list ¯\_(ツ)_/¯

Anyway, that's just another indication that it might not be the right fit 
for inclusion into the QEMU source tree. So either download it similar to 
(or included in) the avocado tests, or maybe another solution would be to 
have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ?

  Thomas
Thomas Huth June 28, 2022, 12:42 p.m. UTC | #49
On 28/06/2022 14.36, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
>>> I think the main difference is not even in how it works, it's
>>> in what it does. Which is check that ACPI tables are sane.
>>> Who cares about that? Well developers do when they change the
>>> tables. Users really don't because for users we have the expected
>>> tables in tree and we check against these.
>>
>> It wants to build and run a big guest binary blob -- that to me is
>> the main difference. Users don't much care about any of our tests,
> 
> perhaps but we do enforce patch submitters to make sure make check
> passes before submitting patches. make check-avocado is not run as
> part of make check, requires considerable disk space to download all
> guest images and hence generally not run by patch submitters. Making
> bits parts of avocado tests almost defeats the purpose of having this
> test at all.
You can define sane subsets of the avocado tests by using tags. For example, 
as I'm one of the s390x maintainers, I'm also only running the tests that 
are tagged with tags=arch:s390x there. You could simply introduce an "acpi" 
tag for your tests, too.

  Thomas
Daniel P. Berrangé June 28, 2022, 12:55 p.m. UTC | #50
On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > I think the main difference is not even in how it works, it's
> > > in what it does. Which is check that ACPI tables are sane.
> > > Who cares about that? Well developers do when they change the
> > > tables. Users really don't because for users we have the expected
> > > tables in tree and we check against these.
> >
> > It wants to build and run a big guest binary blob -- that to me is
> > the main difference. Users don't much care about any of our tests,
> 
> perhaps but we do enforce patch submitters to make sure make check
> passes before submitting patches. make check-avocado is not run as
> part of make check, requires considerable disk space to download all
> guest images and hence generally not run by patch submitters. Making
> bits parts of avocado tests almost defeats the purpose of having this
> test at all.

This proposed biosbits test also involves a considerable download.
The test is said to be irrelevant for anyone except those working
on a fairly narrow set of QEMU firmware related bits. So by the same
rationale we shouldn't impose that burden on everyone working on
QEMU by having it in qtest. Making it entirely optional, only
downloaded by avocado on demand, for the people who need to run
the tests is best.

With regards,
Daniel
Ani Sinha June 28, 2022, 1:22 p.m. UTC | #51
On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > I think the main difference is not even in how it works, it's
> > > > in what it does. Which is check that ACPI tables are sane.
> > > > Who cares about that? Well developers do when they change the
> > > > tables. Users really don't because for users we have the expected
> > > > tables in tree and we check against these.
> > >
> > > It wants to build and run a big guest binary blob -- that to me is
> > > the main difference. Users don't much care about any of our tests,
> >
> > perhaps but we do enforce patch submitters to make sure make check
> > passes before submitting patches. make check-avocado is not run as
> > part of make check, requires considerable disk space to download all
> > guest images and hence generally not run by patch submitters. Making
> > bits parts of avocado tests almost defeats the purpose of having this
> > test at all.
>
> This proposed biosbits test also involves a considerable download.

I do not think 50 MB is "considerable" . Last time I tried to run
avocado tests, my laptop ran out of disk space!

> The test is said to be irrelevant for anyone except those working
> on a fairly narrow set of QEMU firmware related bits.

Well ok that is just a bad argument. You can say the same thing for
most qtests. In fact, that is why most qtetes can run directly simply
by passing QTEST_QEMU_BINARY in the environment. No need to go through
make check. Same with the bits test. It can be run directly.

So by the same
> rationale we shouldn't impose that burden on everyone working on
> QEMU by having it in qtest.

So why burden everyone by having bios-tables-test when it only affects
acpi/smbios developers?

Making it entirely optional, only
> downloaded by avocado on demand, for the people who need to run
> the tests is best.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
Peter Maydell June 28, 2022, 1:44 p.m. UTC | #52
On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote:
> On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > This proposed biosbits test also involves a considerable download.
>
> I do not think 50 MB is "considerable" . Last time I tried to run
> avocado tests, my laptop ran out of disk space!

I think 50MB is pretty big. It might be smaller than some other
avocado tests, but it's not exactly the "no binary involved"
that most qtests are.

> > The test is said to be irrelevant for anyone except those working
> > on a fairly narrow set of QEMU firmware related bits.
>
> Well ok that is just a bad argument. You can say the same thing for
> most qtests. In fact, that is why most qtetes can run directly simply
> by passing QTEST_QEMU_BINARY in the environment. No need to go through
> make check. Same with the bits test. It can be run directly.

'make check' is generally the small, fast, no-binary-blobs tests.
Very few 'make check' tests even run code in the guest.

> So by the same
> > rationale we shouldn't impose that burden on everyone working on
> > QEMU by having it in qtest.
>
> So why burden everyone by having bios-tables-test when it only affects
> acpi/smbios developers?

Because it's small and fast and doesn't have a binary blob to download.

There are definitely some awkwardnesses with 'check-avocado',
but we should work on fixing those, not use them as a reason
to refuse to put tests into the avocado tests if that's where
they fit best.

-- PMM
Ani Sinha June 28, 2022, 1:53 p.m. UTC | #53
On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote:
> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> > > This proposed biosbits test also involves a considerable download.
> >
> > I do not think 50 MB is "considerable" . Last time I tried to run
> > avocado tests, my laptop ran out of disk space!
>
> I think 50MB is pretty big. It might be smaller than some other
> avocado tests, but it's not exactly the "no binary involved"
> that most qtests are.


Well bios-tables-test uses the binary blobs of the acpi tables. Only
difference is that in this case, we could maintain them within  the qemu
tree. In this case the blob in slightly larger and comes from a third
party. That is the difference.



>
> > > The test is said to be irrelevant for anyone except those working
> > > on a fairly narrow set of QEMU firmware related bits.
> >
> > Well ok that is just a bad argument. You can say the same thing for
> > most qtests. In fact, that is why most qtetes can run directly simply
> > by passing QTEST_QEMU_BINARY in the environment. No need to go through
> > make check. Same with the bits test. It can be run directly.
>
> 'make check' is generally the small, fast, no-binary-blobs tests.


See above.


> Very few 'make check' tests even run code in the guest.


So bits test is similar here. It runs code in the guest vm.


>
> > So by the same
> > > rationale we shouldn't impose that burden on everyone working on
> > > QEMU by having it in qtest.
> >
> > So why burden everyone by having bios-tables-test when it only affects
> > acpi/smbios developers?
>
> Because it's small and fast and doesn't have a binary blob to download.
>
> There are definitely some awkwardnesses with 'check-avocado',
> but we should work on fixing those, not use them as a reason
> to refuse to put tests into the avocado tests if that's where
> they fit best.


I think this test fits best in the qtrst not with the integration test
framework. Very few path developers will ever run it and wrote new tests
for it if we have it there. I would be terribly discouraged if that’s where
this test landed.


>
> -- PMM
>
Peter Maydell June 28, 2022, 1:55 p.m. UTC | #54
On Tue, 28 Jun 2022 at 14:53, Ani Sinha <ani@anisinha.ca> wrote:
>
>
>
> On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote:
>> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>> > > This proposed biosbits test also involves a considerable download.
>> >
>> > I do not think 50 MB is "considerable" . Last time I tried to run
>> > avocado tests, my laptop ran out of disk space!
>>
>> I think 50MB is pretty big. It might be smaller than some other
>> avocado tests, but it's not exactly the "no binary involved"
>> that most qtests are.
>
>
> Well bios-tables-test uses the binary blobs of the acpi tables. Only difference is that in this case, we could maintain them within  the qemu tree. In this case the blob in slightly larger and comes from a third party. That is the difference.

Yes. That is exactly the difference that means it should go in
the avocado tests.

-- PMM
Michael S. Tsirkin June 28, 2022, 2:38 p.m. UTC | #55
On Tue, Jun 28, 2022 at 01:55:48PM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > I think the main difference is not even in how it works, it's
> > > > in what it does. Which is check that ACPI tables are sane.
> > > > Who cares about that? Well developers do when they change the
> > > > tables. Users really don't because for users we have the expected
> > > > tables in tree and we check against these.
> > >
> > > It wants to build and run a big guest binary blob -- that to me is
> > > the main difference. Users don't much care about any of our tests,
> > 
> > perhaps but we do enforce patch submitters to make sure make check
> > passes before submitting patches. make check-avocado is not run as
> > part of make check, requires considerable disk space to download all
> > guest images and hence generally not run by patch submitters. Making
> > bits parts of avocado tests almost defeats the purpose of having this
> > test at all.
> 
> This proposed biosbits test also involves a considerable download.
> The test is said to be irrelevant for anyone except those working
> on a fairly narrow set of QEMU firmware related bits. So by the same
> rationale we shouldn't impose that burden on everyone working on
> QEMU by having it in qtest. Making it entirely optional, only
> downloaded by avocado on demand, for the people who need to run
> the tests is best.
> 
> With regards,
> Daniel

Optional is fine. I would like to make life easy for
developers working on ACPI though, the field is unforgiving
enough as it is.
How about we skip the test if the submodule is not checked out?

> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Michael S. Tsirkin June 28, 2022, 2:41 p.m. UTC | #56
On Tue, Jun 28, 2022 at 06:52:55PM +0530, Ani Sinha wrote:
> So why burden everyone by having bios-tables-test when it only affects
> acpi/smbios developers?

No it doesn't. Any change in qemu can break ACPI - it's enough to
register an MR overlapping the ACPI registers.  So we have a basic test
making sure ACPI is in expected state and have everyone run it.
Michael S. Tsirkin June 28, 2022, 2:45 p.m. UTC | #57
On Tue, Jun 28, 2022 at 02:39:31PM +0200, Thomas Huth wrote:
> On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> [...]
> > > > Come on, this is just a test. We *really* don't care if an ISO
> > > > we use to test ACPI is using an exploitable version of grub.
> > > 
> > > Wait, I thought we were only talking about tappy here? The ISO binaries
> > > should certainly *not* be bundled in the QEMU tarballs (they are too big
> > > already anyway, we should rather think of moving the firmware binaries out
> > > of the tarball instead).
> > > 
> > >   Thomas
> > 
> > IIUC there are three things we are discussing
> > - biosbits source
> > - biosbits image
> > - tappy
> 
> Oh well, I missed that part of the discussion so far since the corresponding
> patches did not make it to the mailing list ¯\_(ツ)_/¯
> 
> Anyway, that's just another indication that it might not be the right fit
> for inclusion into the QEMU source tree. So either download it similar to
> (or included in) the avocado tests, or maybe another solution would be to
> have a separate "qemu-ci" or "qemu-testing" repository for stuff like this
> ... ?
> 
>  Thomas

I don't think anyone suggested sticking all those blobs in qemu.git: I
proposed a submodule, others proposed the web (fetched with avocado or
just wget). If it's a submodule we'll want to skip it in the tarball,
and skip the test if not checked out.  Whoever is trying to do
development out of a tarball should just stop :).
Warner Losh June 28, 2022, 2:54 p.m. UTC | #58
On Tue, Jun 28, 2022 at 5:10 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> git submodules are awkward basically because they are an automated wget.
> I don't think an explicit wget is much better ... but
> looks like I'm alone in this. Oh well.
>

submodules are 90% of the hassles I have  in upstreaming and updating the
bsd-user stuff. They play terribly with branches, rebases, and any number of
other things and are a pain to reset at times. They are easily the most
brittle,
error-prone and difficult to use feature of git.

I hate them with a burning passion. You are not alone.

Warner
Ani Sinha June 28, 2022, 3:54 p.m. UTC | #59
On Tue, Jun 28, 2022 at 6:09 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> [...]
> >>> Come on, this is just a test. We *really* don't care if an ISO
> >>> we use to test ACPI is using an exploitable version of grub.
> >>
> >> Wait, I thought we were only talking about tappy here? The ISO binaries
> >> should certainly *not* be bundled in the QEMU tarballs (they are too big
> >> already anyway, we should rather think of moving the firmware binaries out
> >> of the tarball instead).
> >>
> >>   Thomas
> >
> > IIUC there are three things we are discussing
> > - biosbits source
> > - biosbits image
> > - tappy
>
> Oh well, I missed that part of the discussion so far since the corresponding
> patches did not make it to the mailing list ¯\_(ツ)_/¯
>
> Anyway, that's just another indication that it might not be the right fit
> for inclusion into the QEMU source tree. So either download it similar to
> (or included in) the avocado tests, or maybe another solution would be to
> have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ?

Yes, I think we should have a separate repo for all testing related
stuff like blobs, guest images etc. Curently, some of them are in
personal github accounts which is not idea IMHo.

Bits stuff can reside in the same place.
Daniel P. Berrangé June 28, 2022, 4:15 p.m. UTC | #60
On Tue, Jun 28, 2022 at 09:24:34PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 6:09 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> > [...]
> > >>> Come on, this is just a test. We *really* don't care if an ISO
> > >>> we use to test ACPI is using an exploitable version of grub.
> > >>
> > >> Wait, I thought we were only talking about tappy here? The ISO binaries
> > >> should certainly *not* be bundled in the QEMU tarballs (they are too big
> > >> already anyway, we should rather think of moving the firmware binaries out
> > >> of the tarball instead).
> > >>
> > >>   Thomas
> > >
> > > IIUC there are three things we are discussing
> > > - biosbits source
> > > - biosbits image
> > > - tappy
> >
> > Oh well, I missed that part of the discussion so far since the corresponding
> > patches did not make it to the mailing list ¯\_(ツ)_/¯
> >
> > Anyway, that's just another indication that it might not be the right fit
> > for inclusion into the QEMU source tree. So either download it similar to
> > (or included in) the avocado tests, or maybe another solution would be to
> > have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ?
> 
> Yes, I think we should have a separate repo for all testing related
> stuff like blobs, guest images etc. Curently, some of them are in
> personal github accounts which is not idea IMHo.
> 
> Bits stuff can reside in the same place.

FYI, the reason much of this is intentionally NOT under the /qemu-project
gitlab namespace is that we did not want to be responsible for distributing
arbitrary binary blobs/images. That in turn makes the QEMU project responsible
for license compliance, which is non-trivial todo correctly for much of this
stuff. As such it is highly desirable to delegate both the hosting the
binaries and source to the third party who builds it.

I agree the use of personal github accounts is not nice, but it was the
least worst solution identified.

With regards,
Daniel
Michael S. Tsirkin June 28, 2022, 6 p.m. UTC | #61
On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
> FYI, the reason much of this is intentionally NOT under the /qemu-project
> gitlab namespace is that we did not want to be responsible for distributing
> arbitrary binary blobs/images. That in turn makes the QEMU project responsible
> for license compliance, which is non-trivial todo correctly for much of this
> stuff. As such it is highly desirable to delegate both the hosting the
> binaries and source to the third party who builds it.

This might be understadable for random guest OS images which include tons of stuff
and are thus hard to audit.  But not to biosbits which has its own
license (more or less bsd) + gpl for grub:
https://github.com/biosbits/bits/blob/master/COPYING

> I agree the use of personal github accounts is not nice, but it was the
> least worst solution identified.
Ani Sinha June 29, 2022, 6:28 a.m. UTC | #62
On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
> > FYI, the reason much of this is intentionally NOT under the /qemu-project
> > gitlab namespace is that we did not want to be responsible for distributing
> > arbitrary binary blobs/images. That in turn makes the QEMU project responsible
> > for license compliance, which is non-trivial todo correctly for much of this
> > stuff. As such it is highly desirable to delegate both the hosting the
> > binaries and source to the third party who builds it.
>
> This might be understadable for random guest OS images which include tons of stuff
> and are thus hard to audit.  But not to biosbits which has its own
> license (more or less bsd) + gpl for grub:
> https://github.com/biosbits/bits/blob/master/COPYING

These are all the dependencies:
https://github.com/biosbits/bits/tree/master/deps

We can go through the licenses for each and make a determination. The
audit would be lost easier because there is a bounded number of
dependencies for bits.

>
> > I agree the use of personal github accounts is not nice, but it was the
> > least worst solution identified.
>
>
> --
> MST
>
Thomas Huth July 1, 2022, 3:34 a.m. UTC | #63
On 29/06/2022 08.28, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
>>> FYI, the reason much of this is intentionally NOT under the /qemu-project
>>> gitlab namespace is that we did not want to be responsible for distributing
>>> arbitrary binary blobs/images. That in turn makes the QEMU project responsible
>>> for license compliance, which is non-trivial todo correctly for much of this
>>> stuff. As such it is highly desirable to delegate both the hosting the
>>> binaries and source to the third party who builds it.
>>
>> This might be understadable for random guest OS images which include tons of stuff
>> and are thus hard to audit.  But not to biosbits which has its own
>> license (more or less bsd) + gpl for grub:
>> https://github.com/biosbits/bits/blob/master/COPYING
> 
> These are all the dependencies:
> https://github.com/biosbits/bits/tree/master/deps
> 
> We can go through the licenses for each and make a determination. The
> audit would be lost easier because there is a bounded number of
> dependencies for bits.

That's quite a bit of dependencies already ... I don't think that we should 
put the burden of keeping up with the licenses of those projects to the QEMU 
project. So just make sure that the binaries are available somewhere and 
then include your test into the avocado framework or download via curl/wget 
as suggested in other mails.

  Thomas
Thomas Huth July 1, 2022, 4:12 a.m. UTC | #64
On 28/06/2022 15.53, Ani Sinha wrote:
> 
> 
> On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org 
> <mailto:peter.maydell@linaro.org>> wrote:
> 
>     On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca
>     <mailto:ani@anisinha.ca>> wrote:
>      > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé
>     <berrange@redhat.com <mailto:berrange@redhat.com>> wrote:
>      > > This proposed biosbits test also involves a considerable download.
>      >
>      > I do not think 50 MB is "considerable" . Last time I tried to run
>      > avocado tests, my laptop ran out of disk space!
> 
>     I think 50MB is pretty big. It might be smaller than some other
>     avocado tests, but it's not exactly the "no binary involved"
>     that most qtests are.
> 
> 
> Well bios-tables-test uses the binary blobs of the acpi tables. Only 
> difference is that in this case, we could maintain them within  the qemu 
> tree. In this case the blob in slightly larger and comes from a third party. 
> That is the difference.

"slightly larger" ... it apparently contains already a complete grub and 
python 2.7 interpreter ... I'd consider that as *much* larger than the ~ 2k 
loc bios-tables-test ;-)

>     Very few 'make check' tests even run code in the guest.
> 
> So bits test is similar here. It runs code in the guest vm.

The qtests that run some TCG code only use some few lines of code. The tests 
that run third party firmware images are the avocado tests.

>     There are definitely some awkwardnesses with 'check-avocado',
>     but we should work on fixing those, not use them as a reason
>     to refuse to put tests into the avocado tests if that's where
>     they fit best.
> 
> I think this test fits best in the qtrst not with the integration test 
> framework.

I disagree. Third party binary blob tests are certainly nothing for the 
qtest directory.

> Very few path developers will ever run it and wrote new tests for 
> it if we have it there. I would be terribly discouraged if that’s where this 
> test landed.

I see your point - I'm also hardly ever running whole "make check-avocado" 
since it's too slow/fat/broken. But using this as a reason to stick your 
test into the qtest framework also does not work/scale: Being one of the 
s390x maintainers - What if I now also want my s390x related tests to be 
executed by everybody? The kernel + initrd images from 
tests/avocado/machine_s390_ccw_virtio.py are also not that big, ca. 50 MiB 
each. Should I now move that out of the avocade directory, too, and force 
everybody to consume it via a submodule? Then who's next? QEMU has 21 target 
architectures, if every maintainers adds a 50 MiB test to the tree, we're at 
more than a gigabyte of data already that you'd have to download before you 
were able to run "make check". This simply does not scale.

So the avocado framework is currently still the best place that we have for 
such tests. You just have to get used to consume it via "tags" instead of 
running the whole "make check-avocado" suite. For example, being a s390x 
maintainer, I'm only running:

  make check-venv
  ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/

and that finishes within some few minutes and only downloads some few 
hundreds of megabytes. You could do something similar with acpi. You could 
even wrap it in a "make check-avocado-acpi" target if you don't like to 
execute avocado directly.

I even wouldn't mind if you put your python stuff in a new directory like 
tests/pytests/ for example, as long as it downloads your binaries separately 
- as I wrote in another mail, the avocado framework rather looks like an 
oddball in our test framework nowadays since it uses a separate test runner 
and not the meson test harness, so having a new approach for python-based 
tests is maybe even a good idea. I just really want to avoid that this goes 
into tests/qtest (since it really does not belong there), and please don't 
add more external stuff via git submodules, that's really the wrong approach 
for this.

  Thomas
Michael S. Tsirkin July 1, 2022, 6:53 a.m. UTC | #65
On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> I even wouldn't mind if you put your python stuff in a new directory like
> tests/pytests/ for example, as long as it downloads your binaries separately
> - as I wrote in another mail, the avocado framework rather looks like an
> oddball in our test framework nowadays since it uses a separate test runner
> and not the meson test harness, so having a new approach for python-based
> tests is maybe even a good idea. I just really want to avoid that this goes
> into tests/qtest (since it really does not belong there), and please don't
> add more external stuff via git submodules, that's really the wrong approach
> for this.

I get it, people hate submodules with passion.  I think trying another
approach for testing that is neither avocado nor qtest is
not too bad. As long as this is not user visible, we can
allow ourselves space to experiment.

OK so, how about this:
- put it in a new directory: tests/roms?
- create repo for a fork of biosbits under git.qemu.org
- roll our own analog to git submodules: a script
  that clones the repo
- new target make check-roms, if the clone exists locally -
  run the test, if not - skip it
- as for using pre-generates ISOs as an optimization,
  I'm not sure how important that is, if yes -
  we can add another repo and another make target along the
  same lines
Ani Sinha July 1, 2022, 7:28 a.m. UTC | #66
On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > I even wouldn't mind if you put your python stuff in a new directory like
> > tests/pytests/ for example, as long as it downloads your binaries separately
> > - as I wrote in another mail, the avocado framework rather looks like an
> > oddball in our test framework nowadays since it uses a separate test runner
> > and not the meson test harness, so having a new approach for python-based
> > tests is maybe even a good idea. I just really want to avoid that this goes
> > into tests/qtest (since it really does not belong there), and please don't
> > add more external stuff via git submodules, that's really the wrong approach
> > for this.
>
> I get it, people hate submodules with passion.  I think trying another
> approach for testing that is neither avocado nor qtest is
> not too bad. As long as this is not user visible, we can
> allow ourselves space to experiment.
>
> OK so, how about this:
> - put it in a new directory: tests/roms?
> - create repo for a fork of biosbits under git.qemu.org
> - roll our own analog to git submodules: a script
>   that clones the repo

No need to clone the whole repo. We can simply download the binaries
that the girlab CI job would generate from the bits sources in that
repo.
We need to clone if we are always building bits from source for every
test. That is not necessary IMHO since much of the bits package would
remain as is without modification.

> - new target make check-roms,

I think make pytest or some such is better and more generic if other
such tests in other areas follow suit.

if the clone exists locally -
>   run the test, if not - skip it

if download of the bits binaries fail, skip it.

> - as for using pre-generates ISOs as an optimization,
>   I'm not sure how important that is, if yes -
>   we can add another repo and another make target along the
>   same lines
>
>
>
> --
> MST
>
Michael S. Tsirkin July 1, 2022, 7:38 a.m. UTC | #67
On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > I even wouldn't mind if you put your python stuff in a new directory like
> > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > - as I wrote in another mail, the avocado framework rather looks like an
> > > oddball in our test framework nowadays since it uses a separate test runner
> > > and not the meson test harness, so having a new approach for python-based
> > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > into tests/qtest (since it really does not belong there), and please don't
> > > add more external stuff via git submodules, that's really the wrong approach
> > > for this.
> >
> > I get it, people hate submodules with passion.  I think trying another
> > approach for testing that is neither avocado nor qtest is
> > not too bad. As long as this is not user visible, we can
> > allow ourselves space to experiment.
> >
> > OK so, how about this:
> > - put it in a new directory: tests/roms?
> > - create repo for a fork of biosbits under git.qemu.org
> > - roll our own analog to git submodules: a script
> >   that clones the repo
> 
> No need to clone the whole repo. We can simply download the binaries
> that the girlab CI job would generate from the bits sources in that
> repo.
> We need to clone if we are always building bits from source for every
> test. That is not necessary IMHO since much of the bits package would
> remain as is without modification.

IMHO CI job idea isn't great since isn't versioned at all, is it?
Also as long as test passes, sure. But if it fails one will
need the sources to investigate.

Let's start with building things from source. Add an option
of prebuilt binaries as an optimization once things
stabilize.


> > - new target make check-roms,
> 
> I think make pytest or some such is better and more generic if other
> such tests in other areas follow suit.

The name is not critical in my mind, but I think we need to decide
what exactly differentiates it from other tests.


> 
> if the clone exists locally -
> >   run the test, if not - skip it
> 
> if download of the bits binaries fail, skip it.

You seem to be recreating either git or avocado or both here.

Personally I want something that works offline.



> > - as for using pre-generates ISOs as an optimization,
> >   I'm not sure how important that is, if yes -
> >   we can add another repo and another make target along the
> >   same lines
> >
> >
> >
> > --
> > MST
> >
Ani Sinha July 1, 2022, 7:50 a.m. UTC | #68
On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > and not the meson test harness, so having a new approach for python-based
> > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > into tests/qtest (since it really does not belong there), and please don't
> > > > add more external stuff via git submodules, that's really the wrong approach
> > > > for this.
> > >
> > > I get it, people hate submodules with passion.  I think trying another
> > > approach for testing that is neither avocado nor qtest is
> > > not too bad. As long as this is not user visible, we can
> > > allow ourselves space to experiment.
> > >
> > > OK so, how about this:
> > > - put it in a new directory: tests/roms?
> > > - create repo for a fork of biosbits under git.qemu.org
> > > - roll our own analog to git submodules: a script
> > >   that clones the repo
> >
> > No need to clone the whole repo. We can simply download the binaries
> > that the girlab CI job would generate from the bits sources in that
> > repo.
> > We need to clone if we are always building bits from source for every
> > test. That is not necessary IMHO since much of the bits package would
> > remain as is without modification.
>
> IMHO CI job idea isn't great since isn't versioned at all, is it?

bits is versioned yes, in a crude way. every time you make a commit in
the top level repo, the version would increment by one.

> Also as long as test passes, sure. But if it fails one will
> need the sources to investigate.

sources might also be needed to write the tests.

>
> Let's start with building things from source.

hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
build depending on parallelity and build host.

Add an option
> of prebuilt binaries as an optimization once things
> stabilize.
>
>
> > > - new target make check-roms,
> >
> > I think make pytest or some such is better and more generic if other
> > such tests in other areas follow suit.
>
> The name is not critical in my mind, but I think we need to decide
> what exactly differentiates it from other tests.
>
>
> >
> > if the clone exists locally -
> > >   run the test, if not - skip it
> >
> > if download of the bits binaries fail, skip it.
>
> You seem to be recreating either git or avocado or both here.
>
> Personally I want something that works offline.
>
>
>
> > > - as for using pre-generates ISOs as an optimization,
> > >   I'm not sure how important that is, if yes -
> > >   we can add another repo and another make target along the
> > >   same lines
> > >
> > >
> > >
> > > --
> > > MST
> > >
>
Michael S. Tsirkin July 1, 2022, 9:42 a.m. UTC | #69
On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > > and not the meson test harness, so having a new approach for python-based
> > > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > > into tests/qtest (since it really does not belong there), and please don't
> > > > > add more external stuff via git submodules, that's really the wrong approach
> > > > > for this.
> > > >
> > > > I get it, people hate submodules with passion.  I think trying another
> > > > approach for testing that is neither avocado nor qtest is
> > > > not too bad. As long as this is not user visible, we can
> > > > allow ourselves space to experiment.
> > > >
> > > > OK so, how about this:
> > > > - put it in a new directory: tests/roms?
> > > > - create repo for a fork of biosbits under git.qemu.org
> > > > - roll our own analog to git submodules: a script
> > > >   that clones the repo
> > >
> > > No need to clone the whole repo. We can simply download the binaries
> > > that the girlab CI job would generate from the bits sources in that
> > > repo.
> > > We need to clone if we are always building bits from source for every
> > > test. That is not necessary IMHO since much of the bits package would
> > > remain as is without modification.
> >
> > IMHO CI job idea isn't great since isn't versioned at all, is it?
> 
> bits is versioned yes, in a crude way. every time you make a commit in
> the top level repo, the version would increment by one.

Is it easy to find out which source was this generated from?
And is there a promise to keep these around indefinitely?

> > Also as long as test passes, sure. But if it fails one will
> > need the sources to investigate.
> 
> sources might also be needed to write the tests.
> 
> >
> > Let's start with building things from source.
> 
> hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> build depending on parallelity and build host.

Right. But whoever wants to use these just needs to do it once.


> Add an option
> > of prebuilt binaries as an optimization once things
> > stabilize.
> >
> >
> > > > - new target make check-roms,
> > >
> > > I think make pytest or some such is better and more generic if other
> > > such tests in other areas follow suit.
> >
> > The name is not critical in my mind, but I think we need to decide
> > what exactly differentiates it from other tests.
> >
> >
> > >
> > > if the clone exists locally -
> > > >   run the test, if not - skip it
> > >
> > > if download of the bits binaries fail, skip it.
> >
> > You seem to be recreating either git or avocado or both here.
> >
> > Personally I want something that works offline.
> >
> >
> >
> > > > - as for using pre-generates ISOs as an optimization,
> > > >   I'm not sure how important that is, if yes -
> > > >   we can add another repo and another make target along the
> > > >   same lines
> > > >
> > > >
> > > >
> > > > --
> > > > MST
> > > >
> >
Ani Sinha July 1, 2022, 10:14 a.m. UTC | #70
On Fri, Jul 1, 2022 at 3:12 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > > > and not the meson test harness, so having a new approach for python-based
> > > > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > > > into tests/qtest (since it really does not belong there), and please don't
> > > > > > add more external stuff via git submodules, that's really the wrong approach
> > > > > > for this.
> > > > >
> > > > > I get it, people hate submodules with passion.  I think trying another
> > > > > approach for testing that is neither avocado nor qtest is
> > > > > not too bad. As long as this is not user visible, we can
> > > > > allow ourselves space to experiment.
> > > > >
> > > > > OK so, how about this:
> > > > > - put it in a new directory: tests/roms?
> > > > > - create repo for a fork of biosbits under git.qemu.org
> > > > > - roll our own analog to git submodules: a script
> > > > >   that clones the repo
> > > >
> > > > No need to clone the whole repo. We can simply download the binaries
> > > > that the girlab CI job would generate from the bits sources in that
> > > > repo.
> > > > We need to clone if we are always building bits from source for every
> > > > test. That is not necessary IMHO since much of the bits package would
> > > > remain as is without modification.
> > >
> > > IMHO CI job idea isn't great since isn't versioned at all, is it?
> >
> > bits is versioned yes, in a crude way. every time you make a commit in
> > the top level repo, the version would increment by one.
>
> Is it easy to find out which source was this generated from?

yes because this is how the version number is generated:

expr 2000 + `git rev-list HEAD 2>/dev/null | wc -l`

That means we simply count the number of commits in the top level
directory. If we overwrote history by force pushing, then well yes all
bets are off.

> And is there a promise to keep these around indefinitely?

We do not need to maintain multiple versions of bits or increment the
version number every time we wrote a test. Typically new
commits/changes will be made if we wanted to fix a bug or upgrade
acpica or whatever. Test scripts are python based and can be pushed
into bits while generating a new iso for testing purposes.

>
> > > Also as long as test passes, sure. But if it fails one will
> > > need the sources to investigate.
> >
> > sources might also be needed to write the tests.
> >
> > >
> > > Let's start with building things from source.
> >
> > hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> > build depending on parallelity and build host.
>
> Right. But whoever wants to use these just needs to do it once.

but I thought you were suggesting we built bits every time the test is run?

>
>
> > Add an option
> > > of prebuilt binaries as an optimization once things
> > > stabilize.
> > >
> > >
> > > > > - new target make check-roms,
> > > >
> > > > I think make pytest or some such is better and more generic if other
> > > > such tests in other areas follow suit.
> > >
> > > The name is not critical in my mind, but I think we need to decide
> > > what exactly differentiates it from other tests.
> > >
> > >
> > > >
> > > > if the clone exists locally -
> > > > >   run the test, if not - skip it
> > > >
> > > > if download of the bits binaries fail, skip it.
> > >
> > > You seem to be recreating either git or avocado or both here.
> > >
> > > Personally I want something that works offline.
> > >
> > >
> > >
> > > > > - as for using pre-generates ISOs as an optimization,
> > > > >   I'm not sure how important that is, if yes -
> > > > >   we can add another repo and another make target along the
> > > > >   same lines
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > MST
> > > > >
> > >
>
Michael S. Tsirkin July 1, 2022, 12:54 p.m. UTC | #71
On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> but I thought you were suggesting we built bits every time the test is run?

In my opinion 3 scenarios are worth supporting:
- people not touching ACPI, including users - simply don't run the tests,
	comparing tables with expected output should be enough
- people making changes indirectly affecting ACPI -
 	use tests to validate that tables are still well formed,
	using a pre built binary should be enough IMO
- people working on ACPI - use tests to analyse the tables,
	building from source might be necessary for debugging,
	sources change very rarely
- people developing the tests
	building from source is required

So I would suggest basically two github repos, one with binaries one with
sources. We'll keep relevant hashes to use in the script.
All in all not that different from submodules but I guess
people have submodules and that is that.

And I personally would probably not tie it to CI whoever owns the
repository can worry about the builds, and I think keeping
things distributed is important.

So
- people not touching ACPI - make check should see directory not found
	and skip the test
- people making changes indirectly affecting ACPI -
	check out binaries and use
- people working on ACPI -
	see that source directory is present, go there
	and run make. should not rebuild each time right?
- people developing the tests
	building from source is required
Philippe Mathieu-Daudé July 2, 2022, 12:05 a.m. UTC | #72
On Fri, Jul 1, 2022 at 5:37 AM Thomas Huth <thuth@redhat.com> wrote:
> On 29/06/2022 08.28, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
> >>> FYI, the reason much of this is intentionally NOT under the /qemu-project
> >>> gitlab namespace is that we did not want to be responsible for distributing
> >>> arbitrary binary blobs/images. That in turn makes the QEMU project responsible
> >>> for license compliance, which is non-trivial todo correctly for much of this
> >>> stuff. As such it is highly desirable to delegate both the hosting the
> >>> binaries and source to the third party who builds it.
> >>
> >> This might be understadable for random guest OS images which include tons of stuff
> >> and are thus hard to audit.  But not to biosbits which has its own
> >> license (more or less bsd) + gpl for grub:
> >> https://github.com/biosbits/bits/blob/master/COPYING
> >
> > These are all the dependencies:
> > https://github.com/biosbits/bits/tree/master/deps
> >
> > We can go through the licenses for each and make a determination. The
> > audit would be lost easier because there is a bounded number of
> > dependencies for bits.
>
> That's quite a bit of dependencies already ... I don't think that we should
> put the burden of keeping up with the licenses of those projects to the QEMU
> project. So just make sure that the binaries are available somewhere and
> then include your test into the avocado framework or download via curl/wget
> as suggested in other mails.

"available somewhere" until they get moved or deleted...

This happened a few times with archives/artifacts Avocado tests fetch.

We ended up committing one or a set of URLs pointing to a file, its
usual name and its hash.
If you downloaded the file in time and don't flush the Avocado cache,
you can keep testing.
If it is too late (the file disappeared) and you have time to dig for
such a file online, and are lucky because you found it and its hash
matches, then you can run some tests. As a maintainer I find this
feature useful, and find it worse to lose the ability to test
something that used to work.

My 2¢...

Phil.
Ani Sinha July 4, 2022, 1:32 p.m. UTC | #73
On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> > but I thought you were suggesting we built bits every time the test is run?
>
> In my opinion 3 scenarios are worth supporting:
> - people not touching ACPI, including users - simply don't run the tests,
>         comparing tables with expected output should be enough
> - people making changes indirectly affecting ACPI -
>         use tests to validate that tables are still well formed,
>         using a pre built binary should be enough IMO
> - people working on ACPI - use tests to analyse the tables,
>         building from source might be necessary for debugging,
>         sources change very rarely
> - people developing the tests
>         building from source is required
>
> So I would suggest basically two github repos, one with binaries one with
> sources. We'll keep relevant hashes to use in the script.
> All in all not that different from submodules but I guess
> people have submodules and that is that.
>
> And I personally would probably not tie it to CI whoever owns the
> repository can worry about the builds, and I think keeping
> things distributed is important.
>
> So
> - people not touching ACPI - make check should see directory not found
>         and skip the test
> - people making changes indirectly affecting ACPI -
>         check out binaries and use
> - people working on ACPI -
>         see that source directory is present, go there
>         and run make. should not rebuild each time right?
> - people developing the tests
>         building from source is required

Ok I have now committed a Dockerfile that has all the build time
dependencies and builds bits and generates the tarballs/zip file that
my test requires:
https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile

We just need to fork the repo and generate automated builds with this
Dockerfile and put the binaries somewhere. This should also help
developers requiring to rebuild bits when necessary.
Oh and btw, I also made bits compliant with the latest gcc 11 compiler
that comes with Ubuntu 22.04 LTS :-)
Ani Sinha July 5, 2022, 1:48 p.m. UTC | #74
On Mon, Jul 4, 2022 at 7:02 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> > > but I thought you were suggesting we built bits every time the test is run?
> >
> > In my opinion 3 scenarios are worth supporting:
> > - people not touching ACPI, including users - simply don't run the tests,
> >         comparing tables with expected output should be enough
> > - people making changes indirectly affecting ACPI -
> >         use tests to validate that tables are still well formed,
> >         using a pre built binary should be enough IMO
> > - people working on ACPI - use tests to analyse the tables,
> >         building from source might be necessary for debugging,
> >         sources change very rarely
> > - people developing the tests
> >         building from source is required
> >
> > So I would suggest basically two github repos, one with binaries one with
> > sources. We'll keep relevant hashes to use in the script.
> > All in all not that different from submodules but I guess
> > people have submodules and that is that.
> >
> > And I personally would probably not tie it to CI whoever owns the
> > repository can worry about the builds, and I think keeping
> > things distributed is important.
> >
> > So
> > - people not touching ACPI - make check should see directory not found
> >         and skip the test
> > - people making changes indirectly affecting ACPI -
> >         check out binaries and use
> > - people working on ACPI -
> >         see that source directory is present, go there
> >         and run make. should not rebuild each time right?
> > - people developing the tests
> >         building from source is required
>
> Ok I have now committed a Dockerfile that has all the build time
> dependencies and builds bits and generates the tarballs/zip file that
> my test requires:
> https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
>
> We just need to fork the repo and generate automated builds with this
> Dockerfile and put the binaries somewhere. This should also help
> developers requiring to rebuild bits when necessary.
> Oh and btw, I also made bits compliant with the latest gcc 11 compiler
> that comes with Ubuntu 22.04 LTS :-)

Pushed a build script here:
https://github.com/ani-sinha/bits/commit/90b99ef05d55ead4b33b2fb19ad07dfb9682ec92

and the bios bits binaries are in this branch:
https://github.com/ani-sinha/bits/tree/bits-builds
Ani Sinha July 7, 2022, 12:49 p.m. UTC | #75
On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > and not the meson test harness, so having a new approach for python-based
> > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > into tests/qtest (since it really does not belong there), and please don't
> > > > add more external stuff via git submodules, that's really the wrong approach
> > > > for this.
> > >
> > > I get it, people hate submodules with passion.  I think trying another
> > > approach for testing that is neither avocado nor qtest is
> > > not too bad. As long as this is not user visible, we can
> > > allow ourselves space to experiment.
> > >
> > > OK so, how about this:
> > > - put it in a new directory: tests/roms?
> > > - create repo for a fork of biosbits under git.qemu.org
> > > - roll our own analog to git submodules: a script
> > >   that clones the repo
> >
> > No need to clone the whole repo. We can simply download the binaries
> > that the girlab CI job would generate from the bits sources in that
> > repo.
> > We need to clone if we are always building bits from source for every
> > test. That is not necessary IMHO since much of the bits package would
> > remain as is without modification.
>
> IMHO CI job idea isn't great since isn't versioned at all, is it?
> Also as long as test passes, sure. But if it fails one will
> need the sources to investigate.
>
> Let's start with building things from source. Add an option
> of prebuilt binaries as an optimization once things
> stabilize.
>
>
> > > - new target make check-roms,
> >
> > I think make pytest or some such is better and more generic if other
> > such tests in other areas follow suit.
>
> The name is not critical in my mind, but I think we need to decide
> what exactly differentiates it from other tests.
>
>
> >
> > if the clone exists locally -
> > >   run the test, if not - skip it
> >
> > if download of the bits binaries fail, skip it.
>
> You seem to be recreating either git or avocado or both here.
>
> Personally I want something that works offline.

Well we need to make a call as to how to make things work offline, if
that's what we wanted.  I have made changes in v2 to download the
binaries from my github repo. It works great. If download fails, it
skips the test. If those files are available before, it will not
attempt to download them.
https://pastebin.com/nrcdN8WS and
https://pastebin.com/m7Dx3PZk

The Dockerfile and build script I have in the repo will generate these
artifacts. So if you wanted to run the test offline, either build
those from source and place them where the test expects to find them
or build them from source using Dockerfile/build script and place them
there before running the test.

>
>
>
> > > - as for using pre-generates ISOs as an optimization,
> > >   I'm not sure how important that is, if yes -
> > >   we can add another repo and another make target along the
> > >   same lines
> > >
> > >
> > >
> > > --
> > > MST
> > >
>
Michael S. Tsirkin Sept. 28, 2022, 9:26 a.m. UTC | #76
On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> [...]
> > For biosbits if we are going this route then I feel a submodule is much
> > better.  It records which version exactly each qemu version wants.
> 
> As far as I know, you can also specify the version when using pip, can't
> you? So that's not really an advantage here.
> 
> On the contrary, submodules have a couple of disadvantages that I really
> dislike:
> 
> - submodules do not get updated automatically when doing a "git checkout",
> we have to update them via a script instead. This causes e.g. trouble if you
> rsync your source tree to a machine that has no access to the internet and
> you forgot to update the submodule before the sync
> 
> - the content of submodules is not added to the tarballs that get created on
> the git forges automatically. There were lots of requests from users in the
> past that tried to download a tarball from github and then wondered why they
> couldn't compile QEMU.
> 
> - we include the submodule content in our release tarballs, so people get
> the impression that hte submodule content is part of the QEMU sources. This
> has two disadvantages:
>  * We already got bug reports for the code in the submodule,
>    where people did not understand that they should report that
>    rather to the original project instead (i.e. you ship it - you
>    own it)
>  * People get the impression that QEMU is a huge monster
>    application if they count the number of code lines, run
>    their code scanner tools on the tarball contents, etc.
>    Remember "nemu", for example, where one of the main complaints
>    was that QEMU has too many lines of code?
> 
> - If programs includes code via submodules, this gets a higher
>   burder for distro maintainers, since they have to patch each
>   and every package when there is a bug, instead of being able to
>   fix it in one central place.
> 
> So in my opinion we should avoid new submodules if there is an alternative.
> 
>  Thomas

So looking at the latest proposals downloading files from CI,
checksumming them etc etc. No auto checkout, not added automatically
either, right?

This seems to be the only difference:
- we include the submodule content in our release tarballs

How about we just fix that? Thomas would that address your
concern at least wrt tests?
Thomas Huth Sept. 28, 2022, 9:33 a.m. UTC | #77
On 28/09/2022 11.26, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>> [...]
>>> For biosbits if we are going this route then I feel a submodule is much
>>> better.  It records which version exactly each qemu version wants.
>>
>> As far as I know, you can also specify the version when using pip, can't
>> you? So that's not really an advantage here.
>>
>> On the contrary, submodules have a couple of disadvantages that I really
>> dislike:
>>
>> - submodules do not get updated automatically when doing a "git checkout",
>> we have to update them via a script instead. This causes e.g. trouble if you
>> rsync your source tree to a machine that has no access to the internet and
>> you forgot to update the submodule before the sync
>>
>> - the content of submodules is not added to the tarballs that get created on
>> the git forges automatically. There were lots of requests from users in the
>> past that tried to download a tarball from github and then wondered why they
>> couldn't compile QEMU.
>>
>> - we include the submodule content in our release tarballs, so people get
>> the impression that hte submodule content is part of the QEMU sources. This
>> has two disadvantages:
>>   * We already got bug reports for the code in the submodule,
>>     where people did not understand that they should report that
>>     rather to the original project instead (i.e. you ship it - you
>>     own it)
>>   * People get the impression that QEMU is a huge monster
>>     application if they count the number of code lines, run
>>     their code scanner tools on the tarball contents, etc.
>>     Remember "nemu", for example, where one of the main complaints
>>     was that QEMU has too many lines of code?
>>
>> - If programs includes code via submodules, this gets a higher
>>    burder for distro maintainers, since they have to patch each
>>    and every package when there is a bug, instead of being able to
>>    fix it in one central place.
>>
>> So in my opinion we should avoid new submodules if there is an alternative.
>>
>>   Thomas
> 
> So looking at the latest proposals downloading files from CI,
> checksumming them etc etc. No auto checkout, not added automatically
> either, right?
> 
> This seems to be the only difference:
> - we include the submodule content in our release tarballs
> 
> How about we just fix that? Thomas would that address your
> concern at least wrt tests?

If I'm not forced to checkout that submodule, and if we don't add it to the 
release tarball, I guess there's not much left I can complain about ;-)

  Thomas
Daniel P. Berrangé Sept. 28, 2022, 9:37 a.m. UTC | #78
On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > [...]
> > > For biosbits if we are going this route then I feel a submodule is much
> > > better.  It records which version exactly each qemu version wants.
> > 
> > As far as I know, you can also specify the version when using pip, can't
> > you? So that's not really an advantage here.
> > 
> > On the contrary, submodules have a couple of disadvantages that I really
> > dislike:
> > 
> > - submodules do not get updated automatically when doing a "git checkout",
> > we have to update them via a script instead. This causes e.g. trouble if you
> > rsync your source tree to a machine that has no access to the internet and
> > you forgot to update the submodule before the sync
> > 
> > - the content of submodules is not added to the tarballs that get created on
> > the git forges automatically. There were lots of requests from users in the
> > past that tried to download a tarball from github and then wondered why they
> > couldn't compile QEMU.
> > 
> > - we include the submodule content in our release tarballs, so people get
> > the impression that hte submodule content is part of the QEMU sources. This
> > has two disadvantages:
> >  * We already got bug reports for the code in the submodule,
> >    where people did not understand that they should report that
> >    rather to the original project instead (i.e. you ship it - you
> >    own it)
> >  * People get the impression that QEMU is a huge monster
> >    application if they count the number of code lines, run
> >    their code scanner tools on the tarball contents, etc.
> >    Remember "nemu", for example, where one of the main complaints
> >    was that QEMU has too many lines of code?
> > 
> > - If programs includes code via submodules, this gets a higher
> >   burder for distro maintainers, since they have to patch each
> >   and every package when there is a bug, instead of being able to
> >   fix it in one central place.
> > 
> > So in my opinion we should avoid new submodules if there is an alternative.
> > 
> >  Thomas
> 
> So looking at the latest proposals downloading files from CI,
> checksumming them etc etc. No auto checkout, not added automatically
> either, right?
> 
> This seems to be the only difference:
> - we include the submodule content in our release tarballs

That's just one of the issues with submodules. Working with them in general
is not a pleasant experiance. Thomas pointed out some of the issues, such
as 'git checkout' ignoring submodules, requiring extra steps to sync them.
The flipside on tarballs is that the auto-generated tarballs from gitlab
do not include submodules, so the tests will be missing the files they
wanted. There's also the perenial problem that developers frequently send
patches that mistakenly include submodule changes, which is related to the
way that 'git checkout' doesn't sync submodule state when switching branches.
I'd really like to see us doing more to eliminate as much use of submodules
as is possible over time. 

With regards,
Daniel
Michael S. Tsirkin Sept. 28, 2022, 9:47 a.m. UTC | #79
On Wed, Sep 28, 2022 at 11:33:52AM +0200, Thomas Huth wrote:
> On 28/09/2022 11.26, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > [...]
> > > > For biosbits if we are going this route then I feel a submodule is much
> > > > better.  It records which version exactly each qemu version wants.
> > > 
> > > As far as I know, you can also specify the version when using pip, can't
> > > you? So that's not really an advantage here.
> > > 
> > > On the contrary, submodules have a couple of disadvantages that I really
> > > dislike:
> > > 
> > > - submodules do not get updated automatically when doing a "git checkout",
> > > we have to update them via a script instead. This causes e.g. trouble if you
> > > rsync your source tree to a machine that has no access to the internet and
> > > you forgot to update the submodule before the sync
> > > 
> > > - the content of submodules is not added to the tarballs that get created on
> > > the git forges automatically. There were lots of requests from users in the
> > > past that tried to download a tarball from github and then wondered why they
> > > couldn't compile QEMU.
> > > 
> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >   * We already got bug reports for the code in the submodule,
> > >     where people did not understand that they should report that
> > >     rather to the original project instead (i.e. you ship it - you
> > >     own it)
> > >   * People get the impression that QEMU is a huge monster
> > >     application if they count the number of code lines, run
> > >     their code scanner tools on the tarball contents, etc.
> > >     Remember "nemu", for example, where one of the main complaints
> > >     was that QEMU has too many lines of code?
> > > 
> > > - If programs includes code via submodules, this gets a higher
> > >    burder for distro maintainers, since they have to patch each
> > >    and every package when there is a bug, instead of being able to
> > >    fix it in one central place.
> > > 
> > > So in my opinion we should avoid new submodules if there is an alternative.
> > > 
> > >   Thomas
> > 
> > So looking at the latest proposals downloading files from CI,
> > checksumming them etc etc. No auto checkout, not added automatically
> > either, right?
> > 
> > This seems to be the only difference:
> > - we include the submodule content in our release tarballs
> > 
> > How about we just fix that? Thomas would that address your
> > concern at least wrt tests?
> 
> If I'm not forced to checkout that submodule,

I think the make check script can do that?

> and if we don't add it to the
> release tarball, I guess there's not much left I can complain about ;-)
> 
>  Thomas
Michael S. Tsirkin Sept. 28, 2022, 9:53 a.m. UTC | #80
On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > [...]
> > > > For biosbits if we are going this route then I feel a submodule is much
> > > > better.  It records which version exactly each qemu version wants.
> > > 
> > > As far as I know, you can also specify the version when using pip, can't
> > > you? So that's not really an advantage here.
> > > 
> > > On the contrary, submodules have a couple of disadvantages that I really
> > > dislike:
> > > 
> > > - submodules do not get updated automatically when doing a "git checkout",
> > > we have to update them via a script instead. This causes e.g. trouble if you
> > > rsync your source tree to a machine that has no access to the internet and
> > > you forgot to update the submodule before the sync
> > > 
> > > - the content of submodules is not added to the tarballs that get created on
> > > the git forges automatically. There were lots of requests from users in the
> > > past that tried to download a tarball from github and then wondered why they
> > > couldn't compile QEMU.
> > > 
> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >  * We already got bug reports for the code in the submodule,
> > >    where people did not understand that they should report that
> > >    rather to the original project instead (i.e. you ship it - you
> > >    own it)
> > >  * People get the impression that QEMU is a huge monster
> > >    application if they count the number of code lines, run
> > >    their code scanner tools on the tarball contents, etc.
> > >    Remember "nemu", for example, where one of the main complaints
> > >    was that QEMU has too many lines of code?
> > > 
> > > - If programs includes code via submodules, this gets a higher
> > >   burder for distro maintainers, since they have to patch each
> > >   and every package when there is a bug, instead of being able to
> > >   fix it in one central place.
> > > 
> > > So in my opinion we should avoid new submodules if there is an alternative.
> > > 
> > >  Thomas
> > 
> > So looking at the latest proposals downloading files from CI,
> > checksumming them etc etc. No auto checkout, not added automatically
> > either, right?
> > 
> > This seems to be the only difference:
> > - we include the submodule content in our release tarballs
> 
> That's just one of the issues with submodules. Working with them in general
> is not a pleasant experiance.

This is what I asked about at the maintainers summit.
I'd like to map the issues and see if there's anything
we can do to solve them. In particular we will likely
keep using submodules where we historically did
so it's time well spent.

I agree generally but the big question is what to replace these with.
Below I assume the replacement is a script such as avocado or pytest
with its own hashing, calling wget internally etc etc.


> Thomas pointed out some of the issues, such
> as 'git checkout' ignoring submodules, requiring extra steps to sync them.


Not different from a home grown SCM as part of test script, right?

> The flipside on tarballs is that the auto-generated tarballs from gitlab
> do not include submodules, so the tests will be missing the files they
> wanted.

Not different from a home grown SCM as part of test script, right?

> There's also the perenial problem that developers frequently send
> patches that mistakenly include submodule changes,

OK, so the thing to do would be to look for ways to exclude submodule changes
from git commits.

> which is related to the
> way that 'git checkout' doesn't sync submodule state when switching branches.

Not different from a home grown SCM as part of test script, right?

> I'd really like to see us doing more to eliminate as much use of submodules
> as is possible over time. 

Or try to fix the problems, right?


> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Thomas Huth Sept. 28, 2022, 9:55 a.m. UTC | #81
On 28/09/2022 11.47, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 11:33:52AM +0200, Thomas Huth wrote:
>> On 28/09/2022 11.26, Michael S. Tsirkin wrote:
>>> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>>>> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>>>> [...]
>>>>> For biosbits if we are going this route then I feel a submodule is much
>>>>> better.  It records which version exactly each qemu version wants.
>>>>
>>>> As far as I know, you can also specify the version when using pip, can't
>>>> you? So that's not really an advantage here.
>>>>
>>>> On the contrary, submodules have a couple of disadvantages that I really
>>>> dislike:
>>>>
>>>> - submodules do not get updated automatically when doing a "git checkout",
>>>> we have to update them via a script instead. This causes e.g. trouble if you
>>>> rsync your source tree to a machine that has no access to the internet and
>>>> you forgot to update the submodule before the sync
>>>>
>>>> - the content of submodules is not added to the tarballs that get created on
>>>> the git forges automatically. There were lots of requests from users in the
>>>> past that tried to download a tarball from github and then wondered why they
>>>> couldn't compile QEMU.
>>>>
>>>> - we include the submodule content in our release tarballs, so people get
>>>> the impression that hte submodule content is part of the QEMU sources. This
>>>> has two disadvantages:
>>>>    * We already got bug reports for the code in the submodule,
>>>>      where people did not understand that they should report that
>>>>      rather to the original project instead (i.e. you ship it - you
>>>>      own it)
>>>>    * People get the impression that QEMU is a huge monster
>>>>      application if they count the number of code lines, run
>>>>      their code scanner tools on the tarball contents, etc.
>>>>      Remember "nemu", for example, where one of the main complaints
>>>>      was that QEMU has too many lines of code?
>>>>
>>>> - If programs includes code via submodules, this gets a higher
>>>>     burder for distro maintainers, since they have to patch each
>>>>     and every package when there is a bug, instead of being able to
>>>>     fix it in one central place.
>>>>
>>>> So in my opinion we should avoid new submodules if there is an alternative.
>>>>
>>>>    Thomas
>>>
>>> So looking at the latest proposals downloading files from CI,
>>> checksumming them etc etc. No auto checkout, not added automatically
>>> either, right?
>>>
>>> This seems to be the only difference:
>>> - we include the submodule content in our release tarballs
>>>
>>> How about we just fix that? Thomas would that address your
>>> concern at least wrt tests?
>>
>> If I'm not forced to checkout that submodule,
> 
> I think the make check script can do that?

Yes, currently there are only some submodules that are always checked out: 
dtc, meson, tests/fp/berkeley-softfloat-3, tests/fp/berkeley-testfloat-3 and 
ui/keycodemapdb.
And for example the ones in the roms/ directory are not checked out by default.

  Thomas
Daniel P. Berrangé Sept. 28, 2022, 9:57 a.m. UTC | #82
On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > > [...]
> > > > > For biosbits if we are going this route then I feel a submodule is much
> > > > > better.  It records which version exactly each qemu version wants.
> > > > 
> > > > As far as I know, you can also specify the version when using pip, can't
> > > > you? So that's not really an advantage here.
> > > > 
> > > > On the contrary, submodules have a couple of disadvantages that I really
> > > > dislike:
> > > > 
> > > > - submodules do not get updated automatically when doing a "git checkout",
> > > > we have to update them via a script instead. This causes e.g. trouble if you
> > > > rsync your source tree to a machine that has no access to the internet and
> > > > you forgot to update the submodule before the sync
> > > > 
> > > > - the content of submodules is not added to the tarballs that get created on
> > > > the git forges automatically. There were lots of requests from users in the
> > > > past that tried to download a tarball from github and then wondered why they
> > > > couldn't compile QEMU.
> > > > 
> > > > - we include the submodule content in our release tarballs, so people get
> > > > the impression that hte submodule content is part of the QEMU sources. This
> > > > has two disadvantages:
> > > >  * We already got bug reports for the code in the submodule,
> > > >    where people did not understand that they should report that
> > > >    rather to the original project instead (i.e. you ship it - you
> > > >    own it)
> > > >  * People get the impression that QEMU is a huge monster
> > > >    application if they count the number of code lines, run
> > > >    their code scanner tools on the tarball contents, etc.
> > > >    Remember "nemu", for example, where one of the main complaints
> > > >    was that QEMU has too many lines of code?
> > > > 
> > > > - If programs includes code via submodules, this gets a higher
> > > >   burder for distro maintainers, since they have to patch each
> > > >   and every package when there is a bug, instead of being able to
> > > >   fix it in one central place.
> > > > 
> > > > So in my opinion we should avoid new submodules if there is an alternative.
> > > > 
> > > >  Thomas
> > > 
> > > So looking at the latest proposals downloading files from CI,
> > > checksumming them etc etc. No auto checkout, not added automatically
> > > either, right?
> > > 
> > > This seems to be the only difference:
> > > - we include the submodule content in our release tarballs
> > 
> > That's just one of the issues with submodules. Working with them in general
> > is not a pleasant experiance.
> 
> This is what I asked about at the maintainers summit.
> I'd like to map the issues and see if there's anything
> we can do to solve them. In particular we will likely
> keep using submodules where we historically did
> so it's time well spent.
> 
> I agree generally but the big question is what to replace these with.
> Below I assume the replacement is a script such as avocado or pytest
> with its own hashing, calling wget internally etc etc.
> 
> 
> > Thomas pointed out some of the issues, such
> > as 'git checkout' ignoring submodules, requiring extra steps to sync them.
> 
> 
> Not different from a home grown SCM as part of test script, right?

We're not building a home grown SCM as part of a test script, so
this answer is irrelevant.

> > There's also the perenial problem that developers frequently send
> > patches that mistakenly include submodule changes,
> 
> OK, so the thing to do would be to look for ways to exclude submodule changes
> from git commits.

If someone wants to make git suck less with submodules great, but needs
someone to actually do the work.

> > I'd really like to see us doing more to eliminate as much use of submodules
> > as is possible over time. 
> 
> Or try to fix the problems, right?

Again needs someone to actually make it happen.

Meanwhile  QEMU already has an integrated test harness in the form
of Avocado that does everything needed. If Avocado had just been
used for this biosbits test in the first place, the test would
likely have already been merged to QEMU, instead of us having this
never ending debate on how to re-invent an alternative to what
already avocado does.

With regards,
Daniel
Michael S. Tsirkin Sept. 28, 2022, 10:07 a.m. UTC | #83
On Wed, Sep 28, 2022 at 10:57:47AM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > > > [...]
> > > > > > For biosbits if we are going this route then I feel a submodule is much
> > > > > > better.  It records which version exactly each qemu version wants.
> > > > > 
> > > > > As far as I know, you can also specify the version when using pip, can't
> > > > > you? So that's not really an advantage here.
> > > > > 
> > > > > On the contrary, submodules have a couple of disadvantages that I really
> > > > > dislike:
> > > > > 
> > > > > - submodules do not get updated automatically when doing a "git checkout",
> > > > > we have to update them via a script instead. This causes e.g. trouble if you
> > > > > rsync your source tree to a machine that has no access to the internet and
> > > > > you forgot to update the submodule before the sync
> > > > > 
> > > > > - the content of submodules is not added to the tarballs that get created on
> > > > > the git forges automatically. There were lots of requests from users in the
> > > > > past that tried to download a tarball from github and then wondered why they
> > > > > couldn't compile QEMU.
> > > > > 
> > > > > - we include the submodule content in our release tarballs, so people get
> > > > > the impression that hte submodule content is part of the QEMU sources. This
> > > > > has two disadvantages:
> > > > >  * We already got bug reports for the code in the submodule,
> > > > >    where people did not understand that they should report that
> > > > >    rather to the original project instead (i.e. you ship it - you
> > > > >    own it)
> > > > >  * People get the impression that QEMU is a huge monster
> > > > >    application if they count the number of code lines, run
> > > > >    their code scanner tools on the tarball contents, etc.
> > > > >    Remember "nemu", for example, where one of the main complaints
> > > > >    was that QEMU has too many lines of code?
> > > > > 
> > > > > - If programs includes code via submodules, this gets a higher
> > > > >   burder for distro maintainers, since they have to patch each
> > > > >   and every package when there is a bug, instead of being able to
> > > > >   fix it in one central place.
> > > > > 
> > > > > So in my opinion we should avoid new submodules if there is an alternative.
> > > > > 
> > > > >  Thomas
> > > > 
> > > > So looking at the latest proposals downloading files from CI,
> > > > checksumming them etc etc. No auto checkout, not added automatically
> > > > either, right?
> > > > 
> > > > This seems to be the only difference:
> > > > - we include the submodule content in our release tarballs
> > > 
> > > That's just one of the issues with submodules. Working with them in general
> > > is not a pleasant experiance.
> > 
> > This is what I asked about at the maintainers summit.
> > I'd like to map the issues and see if there's anything
> > we can do to solve them. In particular we will likely
> > keep using submodules where we historically did
> > so it's time well spent.
> > 
> > I agree generally but the big question is what to replace these with.
> > Below I assume the replacement is a script such as avocado or pytest
> > with its own hashing, calling wget internally etc etc.
> > 
> > 
> > > Thomas pointed out some of the issues, such
> > > as 'git checkout' ignoring submodules, requiring extra steps to sync them.
> > 
> > 
> > Not different from a home grown SCM as part of test script, right?
> 
> We're not building a home grown SCM as part of a test script, so
> this answer is irrelevant.

If you are managing contents of files using code then you have
a content managing system :)

> > > There's also the perenial problem that developers frequently send
> > > patches that mistakenly include submodule changes,
> > 
> > OK, so the thing to do would be to look for ways to exclude submodule changes
> > from git commits.
> 
> If someone wants to make git suck less with submodules great, but needs
> someone to actually do the work.

I have some experience with git, might be able to if I can figure out
what the issues are. Or maybe there's already a flag to do what we want
git is pretty configurable.

> > > I'd really like to see us doing more to eliminate as much use of submodules
> > > as is possible over time. 
> > 
> > Or try to fix the problems, right?
> 
> Again needs someone to actually make it happen.
> 
> Meanwhile  QEMU already has an integrated test harness in the form
> of Avocado that does everything needed. If Avocado had just been
> used for this biosbits test in the first place, the test would
> likely have already been merged to QEMU, instead of us having this
> never ending debate on how to re-invent an alternative to what
> already avocado does.
> 
> With regards,
> Daniel

I just think there's a ton of difference between linux guest images
which we want to keep running indefinitely and a unit test firmware
which we want to version with qemu.


> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Michael S. Tsirkin Sept. 28, 2022, 10:13 a.m. UTC | #84
On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> There's also the perenial problem that developers frequently send
> patches that mistakenly include submodule changes, which is related to the
> way that 'git checkout' doesn't sync submodule state when switching branches.

Do you happen to know how exactly that happens?
Daniel P. Berrangé Sept. 28, 2022, 10:18 a.m. UTC | #85
On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > There's also the perenial problem that developers frequently send
> > patches that mistakenly include submodule changes, which is related to the
> > way that 'git checkout' doesn't sync submodule state when switching branches.
> 
> Do you happen to know how exactly that happens?

For any given branch the submodule is synced to a given git commit hash.
If the submodule checkout is not synced to the same commit hash it will
show as dirty, and if you git add this pending change, it'll record that
new submodule commit hash.  Seeing dirty state is common when you switch
between branches, either side of a git master change that updated a
submodule.

With regards,
Daniel
Warner Losh Sept. 28, 2022, 1:06 p.m. UTC | #86
On Wed, Sep 28, 2022 at 6:44 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > There's also the perenial problem that developers frequently send
> > patches that mistakenly include submodule changes, which is related to
> the
> > way that 'git checkout' doesn't sync submodule state when switching
> branches.
>
> Do you happen to know how exactly that happens?
>

Here's how it happens to me.

% git checkout master
% git checkout -b bsd-user-2022q4
# bring changes in, doesn't matter
# Build a few times
# time passes, master upstream evolves, submodules move
% git checkout master
% git pull # master fast forwards
% git rebase -i master bsd-user-2022q4

at this point I have to do unnatural git submodule recursive things to make
it stop
complaining about how all the submodules are now 'modified' because they
have
moved on the master branch.

It is real, lasting, lingering pain and a large part of the reason that I
refuse to allow FreeBSD
to use submodules[*]. It's just too much extra hassle for no gain at all
(at least for me, I
see only the pain and no benefits to me or anybody else). I've sent at
least three review
requests that have this pollution in it since it's hard to avoid if you
press forward after the
rebase not to have something swept up into an innocuous change.

Warner

[*] They are also generally a poor match for FreeBSD since we rarely import
things
verbatim, unchanged from upstreams that have well-oiled CI to give us
assurance
that things are good...
Michael S. Tsirkin Sept. 28, 2022, 1:12 p.m. UTC | #87
On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > There's also the perenial problem that developers frequently send
> > > patches that mistakenly include submodule changes, which is related to the
> > > way that 'git checkout' doesn't sync submodule state when switching branches.
> > 
> > Do you happen to know how exactly that happens?
> 
> For any given branch the submodule is synced to a given git commit hash.
> If the submodule checkout is not synced to the same commit hash it will
> show as dirty, and if you git add this pending change, it'll record that
> new submodule commit hash.  Seeing dirty state is common when you switch
> between branches, either side of a git master change that updated a
> submodule.
> 
> With regards,
> Daniel

I see. It is interesting.

So apparently what you want is ignore submodule
changes, right? If yes this is close to what we want:

git submodule update --checkout testsub
git update-index --skip-worktree testsub

A script checking out the submodule can just run this
command.



> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Warner Losh Sept. 28, 2022, 1:15 p.m. UTC | #88
On Wed, Sep 28, 2022 at 7:09 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > > > [...]
> > > > > > For biosbits if we are going this route then I feel a submodule
> is much
> > > > > > better.  It records which version exactly each qemu version
> wants.
> > > > >
> > > > > As far as I know, you can also specify the version when using pip,
> can't
> > > > > you? So that's not really an advantage here.
> > > > >
> > > > > On the contrary, submodules have a couple of disadvantages that I
> really
> > > > > dislike:
> > > > >
> > > > > - submodules do not get updated automatically when doing a "git
> checkout",
> > > > > we have to update them via a script instead. This causes e.g.
> trouble if you
> > > > > rsync your source tree to a machine that has no access to the
> internet and
> > > > > you forgot to update the submodule before the sync
> > > > >
> > > > > - the content of submodules is not added to the tarballs that get
> created on
> > > > > the git forges automatically. There were lots of requests from
> users in the
> > > > > past that tried to download a tarball from github and then
> wondered why they
> > > > > couldn't compile QEMU.
> > > > >
> > > > > - we include the submodule content in our release tarballs, so
> people get
> > > > > the impression that hte submodule content is part of the QEMU
> sources. This
> > > > > has two disadvantages:
> > > > >  * We already got bug reports for the code in the submodule,
> > > > >    where people did not understand that they should report that
> > > > >    rather to the original project instead (i.e. you ship it - you
> > > > >    own it)
> > > > >  * People get the impression that QEMU is a huge monster
> > > > >    application if they count the number of code lines, run
> > > > >    their code scanner tools on the tarball contents, etc.
> > > > >    Remember "nemu", for example, where one of the main complaints
> > > > >    was that QEMU has too many lines of code?
> > > > >
> > > > > - If programs includes code via submodules, this gets a higher
> > > > >   burder for distro maintainers, since they have to patch each
> > > > >   and every package when there is a bug, instead of being able to
> > > > >   fix it in one central place.
> > > > >
> > > > > So in my opinion we should avoid new submodules if there is an
> alternative.
> > > > >
> > > > >  Thomas
> > > >
> > > > So looking at the latest proposals downloading files from CI,
> > > > checksumming them etc etc. No auto checkout, not added automatically
> > > > either, right?
> > > >
> > > > This seems to be the only difference:
> > > > - we include the submodule content in our release tarballs
> > >
> > > That's just one of the issues with submodules. Working with them in
> general
> > > is not a pleasant experiance.
> >
> > This is what I asked about at the maintainers summit.
> > I'd like to map the issues and see if there's anything
> > we can do to solve them. In particular we will likely
> > keep using submodules where we historically did
> > so it's time well spent.
> >
> > I agree generally but the big question is what to replace these with.
> > Below I assume the replacement is a script such as avocado or pytest
> > with its own hashing, calling wget internally etc etc.
> >
> >
> > > Thomas pointed out some of the issues, such
> > > as 'git checkout' ignoring submodules, requiring extra steps to sync
> them.
> >
> >
> > Not different from a home grown SCM as part of test script, right?
>
> We're not building a home grown SCM as part of a test script, so
> this answer is irrelevant.
>
> > > There's also the perenial problem that developers frequently send
> > > patches that mistakenly include submodule changes,
> >
> > OK, so the thing to do would be to look for ways to exclude submodule
> changes
> > from git commits.
>
> If someone wants to make git suck less with submodules great, but needs
> someone to actually do the work.
>

A big part of the problem is knowing which of the following commands I have
to
do to undo the uncommitted changes, the committed changes, the staged
changes,
etc:

git submodule update --init --recursive
git submodule update --init
git submodule foreach --recursive git reset --hard
git submodule foreach --recursive git clean -xfd
git submodule foreach --recursive git clean -xfg

(all of these are in my history, I honestly don't know the difference
between the last two).
And each 'oops' takes time away from upstreaming bsd-user I don't really
have that
much of. I've wasted hours on this over the past year between all the
different ways
it can screw up.

To be fair, it is a relatively small fraction of the time, but as you can
tell from the
animation in my email it inspires much passion.

Warner


> > > I'd really like to see us doing more to eliminate as much use of
> submodules
> > > as is possible over time.p
> >
> > Or try to fix the problems, right?
>
> Again needs someone to actually make it happen.
>
> Meanwhile  QEMU already has an integrated test harness in the form
> of Avocado that does everything needed. If Avocado had just been
> used for this biosbits test in the first place, the test would
> likely have already been merged to QEMU, instead of us having this
> never ending debate on how to re-invent an alternative to what
> already avocado does.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>
>
Michael S. Tsirkin Sept. 28, 2022, 1:22 p.m. UTC | #89
On Wed, Sep 28, 2022 at 07:15:53AM -0600, Warner Losh wrote:
> 
> 
> On Wed, Sep 28, 2022 at 7:09 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
>     On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
>     > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
>     > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
>     > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>     > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>     > > > > [...]
>     > > > > > For biosbits if we are going this route then I feel a submodule
>     is much
>     > > > > > better.  It records which version exactly each qemu version
>     wants.
>     > > > >
>     > > > > As far as I know, you can also specify the version when using pip,
>     can't
>     > > > > you? So that's not really an advantage here.
>     > > > >
>     > > > > On the contrary, submodules have a couple of disadvantages that I
>     really
>     > > > > dislike:
>     > > > >
>     > > > > - submodules do not get updated automatically when doing a "git
>     checkout",
>     > > > > we have to update them via a script instead. This causes e.g.
>     trouble if you
>     > > > > rsync your source tree to a machine that has no access to the
>     internet and
>     > > > > you forgot to update the submodule before the sync
>     > > > >
>     > > > > - the content of submodules is not added to the tarballs that get
>     created on
>     > > > > the git forges automatically. There were lots of requests from
>     users in the
>     > > > > past that tried to download a tarball from github and then wondered
>     why they
>     > > > > couldn't compile QEMU.
>     > > > >
>     > > > > - we include the submodule content in our release tarballs, so
>     people get
>     > > > > the impression that hte submodule content is part of the QEMU
>     sources. This
>     > > > > has two disadvantages:
>     > > > >  * We already got bug reports for the code in the submodule,
>     > > > >    where people did not understand that they should report that
>     > > > >    rather to the original project instead (i.e. you ship it - you
>     > > > >    own it)
>     > > > >  * People get the impression that QEMU is a huge monster
>     > > > >    application if they count the number of code lines, run
>     > > > >    their code scanner tools on the tarball contents, etc.
>     > > > >    Remember "nemu", for example, where one of the main complaints
>     > > > >    was that QEMU has too many lines of code?
>     > > > >
>     > > > > - If programs includes code via submodules, this gets a higher
>     > > > >   burder for distro maintainers, since they have to patch each
>     > > > >   and every package when there is a bug, instead of being able to
>     > > > >   fix it in one central place.
>     > > > >
>     > > > > So in my opinion we should avoid new submodules if there is an
>     alternative.
>     > > > >
>     > > > >  Thomas
>     > > >
>     > > > So looking at the latest proposals downloading files from CI,
>     > > > checksumming them etc etc. No auto checkout, not added automatically
>     > > > either, right?
>     > > >
>     > > > This seems to be the only difference:
>     > > > - we include the submodule content in our release tarballs
>     > >
>     > > That's just one of the issues with submodules. Working with them in
>     general
>     > > is not a pleasant experiance.
>     >
>     > This is what I asked about at the maintainers summit.
>     > I'd like to map the issues and see if there's anything
>     > we can do to solve them. In particular we will likely
>     > keep using submodules where we historically did
>     > so it's time well spent.
>     >
>     > I agree generally but the big question is what to replace these with.
>     > Below I assume the replacement is a script such as avocado or pytest
>     > with its own hashing, calling wget internally etc etc.
>     >
>     >
>     > > Thomas pointed out some of the issues, such
>     > > as 'git checkout' ignoring submodules, requiring extra steps to sync
>     them.
>     >
>     >
>     > Not different from a home grown SCM as part of test script, right?
> 
>     We're not building a home grown SCM as part of a test script, so
>     this answer is irrelevant.
> 
>     > > There's also the perenial problem that developers frequently send
>     > > patches that mistakenly include submodule changes,
>     >
>     > OK, so the thing to do would be to look for ways to exclude submodule
>     changes
>     > from git commits.
> 
>     If someone wants to make git suck less with submodules great, but needs
>     someone to actually do the work.
> 
> 
> A big part of the problem is knowing which of the following commands I have to
> do to undo the uncommitted changes, the committed changes, the staged changes,
> etc:
> 
> git submodule update --init --recursive
> git submodule update --init
> git submodule foreach --recursive git reset --hard
> git submodule foreach --recursive git clean -xfd
> git submodule foreach --recursive git clean -xfg
> 
> (all of these are in my history, I honestly don't know the difference between
> the last two).
> And each 'oops' takes time away from upstreaming bsd-user I don't really have
> that
> much of. I've wasted hours on this over the past year between all the different
> ways
> it can screw up.
> 
> To be fair, it is a relatively small fraction of the time, but as you can tell
> from the
> animation in my email it inspires much passion.
> 
> Warner
>  

OK this is understandable.

I think for what we do in qemu (never change upstream really)
	git submodule update --checkout testsub
	git update-index --skip-worktree testsub
is more or less what most people want.

This assumes you have a script that basically does
	git submodule deinit XYZ
	git submodule update --init XYZ
	git update-index --skip-worktree XYZ
each time, and do not need to work on the submodule proper.




>     > > I'd really like to see us doing more to eliminate as much use of
>     submodules
>     > > as is possible over time.p
>     >
>     > Or try to fix the problems, right?
> 
>     Again needs someone to actually make it happen.
> 
>     Meanwhile  QEMU already has an integrated test harness in the form
>     of Avocado that does everything needed. If Avocado had just been
>     used for this biosbits test in the first place, the test would
>     likely have already been merged to QEMU, instead of us having this
>     never ending debate on how to re-invent an alternative to what
>     already avocado does.
> 
>     With regards,
>     Daniel
>     --
>     |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
>     :|
>     |: https://libvirt.org         -o-            https://fstop138.berrange.com
>     :|
>     |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
>     :|
> 
> 
>
Peter Maydell Sept. 28, 2022, 3:07 p.m. UTC | #90
On Wed, 28 Sept 2022 at 15:29, Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > > There's also the perenial problem that developers frequently send
> > > > patches that mistakenly include submodule changes, which is related to the
> > > > way that 'git checkout' doesn't sync submodule state when switching branches.
> > >
> > > Do you happen to know how exactly that happens?
> >
> > For any given branch the submodule is synced to a given git commit hash.
> > If the submodule checkout is not synced to the same commit hash it will
> > show as dirty, and if you git add this pending change, it'll record that
> > new submodule commit hash.  Seeing dirty state is common when you switch
> > between branches, either side of a git master change that updated a
> > submodule.
> I see. It is interesting.
>
> So apparently what you want is ignore submodule
> changes, right? If yes this is close to what we want:
>
> git submodule update --checkout testsub
> git update-index --skip-worktree testsub
>
> A script checking out the submodule can just run this
> command.

The problem happens not when you check out the submodule, but when
you do basic not-submodule-related git operations like
"git checkout my-working-branch". The fix would be if git itself
automatically kept submodule state in sync with the main repo's
branch. But it doesn't, so the UI is a massive beartrap.

-- PMM
Michael S. Tsirkin Sept. 28, 2022, 7:59 p.m. UTC | #91
On Wed, Sep 28, 2022 at 04:07:40PM +0100, Peter Maydell wrote:
> On Wed, 28 Sept 2022 at 15:29, Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> > > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > > > There's also the perenial problem that developers frequently send
> > > > > patches that mistakenly include submodule changes, which is related to the
> > > > > way that 'git checkout' doesn't sync submodule state when switching branches.
> > > >
> > > > Do you happen to know how exactly that happens?
> > >
> > > For any given branch the submodule is synced to a given git commit hash.
> > > If the submodule checkout is not synced to the same commit hash it will
> > > show as dirty, and if you git add this pending change, it'll record that
> > > new submodule commit hash.  Seeing dirty state is common when you switch
> > > between branches, either side of a git master change that updated a
> > > submodule.
> > I see. It is interesting.
> >
> > So apparently what you want is ignore submodule
> > changes, right? If yes this is close to what we want:
> >
> > git submodule update --checkout testsub
> > git update-index --skip-worktree testsub
> >
> > A script checking out the submodule can just run this
> > command.
> 
> The problem happens not when you check out the submodule, but when
> you do basic not-submodule-related git operations like
> "git checkout my-working-branch". The fix would be if git itself
> automatically kept submodule state in sync with the main repo's
> branch. But it doesn't, so the UI is a massive beartrap.
> 
> -- PMM

If this is what you want, you can make it do that too:

$ cat .gitmodules 
[submodule "testsub"]
        path = testsub
        recurse = true
        url = /tmp/testsub/


git checkout (and a bunch of other commands) will now recurse into this
submodule.
Michal Suchánek Sept. 28, 2022, 8:48 p.m. UTC | #92
Hello,

On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote:

> 
> git submodules are awkward basically because they are an automated wget.
> I don't think an explicit wget is much better ... but
> looks like I'm alone in this. Oh well.
> So it will be a weird dance of wget a tarball, unpack, generate
> ISO and run. God help you if you need to patch the test - it's
> wget all the way down.

That's the problem - the submodules are not automated. They are
half-automated, and the rules for when the automation works and for when
the automation falls apart are not intellibible for the general Joe
Developer.

You might spend a few days studying how they behave exactly, and then you
will know. But unless you will use them every day you will forget again,
because the rules do not lend themselves to some abstraction easily
understood by humans.

Thanks

Michal
Michael S. Tsirkin Sept. 28, 2022, 9:07 p.m. UTC | #93
On Wed, Sep 28, 2022 at 10:48:03PM +0200, Michal Suchánek wrote:
> Hello,
> 
> On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote:
> 
> > 
> > git submodules are awkward basically because they are an automated wget.
> > I don't think an explicit wget is much better ... but
> > looks like I'm alone in this. Oh well.
> > So it will be a weird dance of wget a tarball, unpack, generate
> > ISO and run. God help you if you need to patch the test - it's
> > wget all the way down.
> 
> That's the problem - the submodules are not automated. They are
> half-automated, and the rules for when the automation works and for when
> the automation falls apart are not intellibible for the general Joe
> Developer.
> 
> You might spend a few days studying how they behave exactly, and then you
> will know. But unless you will use them every day you will forget again,
> because the rules do not lend themselves to some abstraction easily
> understood by humans.
> 
> Thanks
> 
> Michal

But why would learning a different tool be easier?
Michal Suchánek Sept. 28, 2022, 9:43 p.m. UTC | #94
On Wed, Sep 28, 2022 at 05:07:48PM -0400, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 10:48:03PM +0200, Michal Suchánek wrote:
> > Hello,
> > 
> > On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote:
> > 
> > > 
> > > git submodules are awkward basically because they are an automated wget.
> > > I don't think an explicit wget is much better ... but
> > > looks like I'm alone in this. Oh well.
> > > So it will be a weird dance of wget a tarball, unpack, generate
> > > ISO and run. God help you if you need to patch the test - it's
> > > wget all the way down.
> > 
> > That's the problem - the submodules are not automated. They are
> > half-automated, and the rules for when the automation works and for when
> > the automation falls apart are not intellibible for the general Joe
> > Developer.
> > 
> > You might spend a few days studying how they behave exactly, and then you
> > will know. But unless you will use them every day you will forget again,
> > because the rules do not lend themselves to some abstraction easily
> > understood by humans.
> > 
> > Thanks
> > 
> > Michal
> 
> But why would learning a different tool be easier?

a) it's working lends itself to explaining in human-intelligible
concepts

b) it's used elsewhere as well

Thanks

Michal
diff mbox series

Patch

diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
new file mode 100644
index 0000000000..97b15f1665
--- /dev/null
+++ b/tests/qtest/acpi-bits/README
@@ -0,0 +1,168 @@ 
+=============================================================================
+ACPI/SMBIOS QTESTS USING BIOSBITS
+=============================================================================
+
+Biosbits is a software written by Josh Triplett that can be downloaded by
+visiting https://biosbits.org/. The github codebase can be found here:
+https://github.com/biosbits/bits/tree/master. It is a software that exercizes
+the bios components such as acpi and smbios tables directly through acpica
+bios interpreter (a freely available C based library written by Intel,
+downloadable from https://acpica.org/ and is included with biosbits) without an
+operating system getting involved in between.
+There are several advantages to directly testing the bios in a real physical
+machine or VM as opposed to indirectly discovering bios issues through the
+operating system. For one thing, the OSes tend to hide bios problems from the
+end user. The other is that we have more control of what we wanted to test
+and how by directly using acpica interpreter on top of the bios on a running
+system. More details on the inspiration for developing biosbits and its real
+life uses can be found in (a) and (b).
+This directory contains QEMU qtests written in python that exercizes the QEMU
+bios components using biosbits and reports test failures.
+
+These tests use python virtual environment. In debian/ubuntu system, the tests
+would require python3.8-venv and python3-pip packages to be installed.
+
+A brief description of the contents of this directory follows:
+
+├── acpi-bits-test.py
+├── acpi-bits-test-venv.sh
+├── bits-config
+│   ├── bits-cfg.txt
+│   └── meson.build
+├── bits-tests
+│   ├── meson.build
+│   ├── smbios.py
+│   ├── smilatency.py
+│   ├── testacpi.py
+│   └── testcpuid.py
+├── meson.build
+├── prebuilt
+│   ├── bits-2095-grub.tar.gz
+│   ├── bits-2095.zip
+│   └── meson.build
+├── README
+└── requirements.txt
+
+acpi-bits:
+ - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
+   environment necessary for the actual python test script to run. Amongst
+   other things, it makes sure that QEMU python library is available within
+   that venv so that QEMU machines can be forked. QEMU python library can be
+   found within python/ directory in QEMU source.
+   After setting up the virtual environment, it runs the python test script
+   from within that environment.
+   If you want to enable verbose mode only for bits test and run make check,
+   one trick is to add V=1 before the call to execute the python script in
+   this file.
+ - acpi-bits-test.py: This is the main python test script that generates a
+   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
+   test failures. This is the script one would be interested in if they wanted
+   to add or change some component of the log parsing, add a new commandline to
+   how QEMU is spawned etc. Test writers typically would not need to modify
+   this script unless they wanted to enhance or change the log parsing for
+   their tests.
+ - requirements.txt: This text file contains the dependent python libraries
+   needed for the tests to run. If a new dependent library is needed, it would
+   be added here as a new entry and then acpi-bits-test-venv.sh would install
+   it when setting up the python virtual environment using pip.
+ - README: This text file.
+
+acpi-bits/bits-config:
+   This location contains biosbits config files that determine how the software
+   runs the tests.
+ - bits-config.txt: this is the biosbits config file that determines what tests
+   or actions are performed by bits. The description of the config options are
+   provided in the file itself.
+
+acpi-bits/prebuilt:
+   This location contains prebuilt biosbits binaries that are used to generate
+   the bits iso file for testing.
+ - bits-2095.zip: The contents from this zip archive are the main contents of
+    the iso file that are used for testing. This binary zip archive also
+    contains the full source of the bits software including the full text of
+    the license agreement listed here:
+    https://github.com/biosbits/bits/blob/master/COPYING
+    The source tarball can be found in this location in the zip file:
+    boot/src/bits-2095.tar.gz
+    The additional changes beyond those that are present in the official
+    biosbits github repository can be found here:
+    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
+
+    Basically these changes can be categorized into the following:
+    (a) changes to make sure biosbits builds with the latest gcc compiler
+    (gcc 9.4).
+    (b) upgrade of acpica to the latest version (march 2022).
+    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
+    debugcon can be used to collect the logs.
+    (d) send a gub halt command to shutdown the VM once all the tests have been
+    executed.
+
+    This zip file is automatically generated by the bits build system. It can
+    be found in the bits build directory and it is suffixed by the bits build
+    number.
+    Normally, there is no need to make any alteration to this zip archive
+    unless one wanted to change the bits software itself (for example to add
+    a new config option or change actions of existing options or change the
+    debug IO port etc). When such changes are made and a new biosbits software
+    is needed to be generated, please refresh this zip archive as well as the
+    grub tarball at the same time. Tests will break if changes are made to bits
+    that are incompatible with existing behavior. So care must be taken to make
+    sure that the change is compatible with existing bits software as much as
+    possible. When a new config option is introduced for example, bits must
+    be upadated here first before introducing the new config option through
+    a later commit.
+ - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
+    scripts and modules to the prebuilt directory. These prebuilt grub
+    artifacts are required in order to generate a bootable bits iso file that
+    can run tests.
+    In order to generate this tar archive, please put the following two
+    directories that can be found in the bits build directory in a single
+    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
+    bits that generated the archive:
+
+    grub-inst-x86_64-efi
+    grub-inst
+
+    This version should be the same as the version number of bits that generated
+    bits-<n>.zip file. In other words, the two files must be in sync and should
+    come from the same build of biosbits of the same version. Things may still
+    work if they come from different versions but mixing version numbers are
+    not recommended.
+    There is normally no need to regenerate this archive unless some fixes or
+    changes have gone into the grub that are part of biosbits.
+
+acpi-bits/bits-tests:
+   This directory contains biosbits python based tests that are run from within
+   the biosbits environment in the spawned VM. New additions of test cases can
+   be made in the appropriate test file. For example, new acpi tests can go
+   into testacpi.py and one would call testsuite.add_test() to register the new
+   test so that it gets executed as a part of the ACPI tests.
+   It might be occassionally necessary to disable some subtests or add a new
+   test that belongs to a test suite not already present in this directory. To
+   do this, please extract the bits source from the zip file mentioned above.
+   Copy the test suite/script that needs modification (addition of new tests
+   or disabling them) from boot/python location of the extracted directory
+   into this directory.
+
+   step (a): copy unmodified test script to this directory.
+   step (b): update meson.build and add this file to the list.
+   Commit (a) and (b) together in the same commit.
+
+   step (c): perform modifications to the test.
+   Commit (c) in a separate commit.
+
+   The test framework will then use your modified test script to run the test.
+   No further changes would be needed. Please check the logs to make sure that
+   appropriate changes have taken effect.
+
+meson.build files makes sure that the bits qtests are appropriately integrated
+into the QEMU qtest framework and are run with "make check-qtest".
+Currently, the bits test is configured to run only for x86_64 architectures. Bits
+has been built only for this architecture.
+
+
+Author: Ani Sinha <ani@anisinha.ca>
+
+References:
+(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
+(b) https://www.youtube.com/watch?v=36QIepyUuhg