| Message ID | 20220627072856.1529357-12-ani@anisinha.ca (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Introduce new acpi/smbios qtests using biosbits | expand |
On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > The README file is added describing the directory structure and the purpose > of every file it contains. It also describes how to add new tests, make changes > to existing tests or bits config files or regenerate the bits software. > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > 1 file changed, 168 insertions(+) > create mode 100644 tests/qtest/acpi-bits/README > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > new file mode 100644 > index 0000000000..97b15f1665 > --- /dev/null > +++ b/tests/qtest/acpi-bits/README > @@ -0,0 +1,168 @@ > +============================================================================= > +ACPI/SMBIOS QTESTS USING BIOSBITS > +============================================================================= > + > +Biosbits is a software written by Josh Triplett that can be downloaded by > +visiting https://biosbits.org/. The github codebase can be found here: > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > +the bios components such as acpi and smbios tables directly through acpica > +bios interpreter (a freely available C based library written by Intel, > +downloadable from https://acpica.org/ and is included with biosbits) without an > +operating system getting involved in between. > +There are several advantages to directly testing the bios in a real physical > +machine or VM as opposed to indirectly discovering bios issues through the > +operating system. For one thing, the OSes tend to hide bios problems from the > +end user. The other is that we have more control of what we wanted to test > +and how by directly using acpica interpreter on top of the bios on a running > +system. More details on the inspiration for developing biosbits and its real > +life uses can be found in (a) and (b). > +This directory contains QEMU qtests written in python that exercizes the QEMU > +bios components using biosbits and reports test failures. > + > +These tests use python virtual environment. In debian/ubuntu system, the tests > +would require python3.8-venv and python3-pip packages to be installed. Why do we mess with venv and pip? Certainly possible but what's wrong with using distro provided packages? > + > +A brief description of the contents of this directory follows: > + > +├── acpi-bits-test.py > +├── acpi-bits-test-venv.sh > +├── bits-config > +│ ├── bits-cfg.txt > +│ └── meson.build > +├── bits-tests > +│ ├── meson.build > +│ ├── smbios.py > +│ ├── smilatency.py > +│ ├── testacpi.py > +│ └── testcpuid.py > +├── meson.build > +├── prebuilt > +│ ├── bits-2095-grub.tar.gz > +│ ├── bits-2095.zip > +│ └── meson.build > +├── README > +└── requirements.txt > + > +acpi-bits: > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > + environment necessary for the actual python test script to run. Amongst > + other things, it makes sure that QEMU python library is available within > + that venv so that QEMU machines can be forked. QEMU python library can be > + found within python/ directory in QEMU source. > + After setting up the virtual environment, it runs the python test script > + from within that environment. > + If you want to enable verbose mode only for bits test and run make check, > + one trick is to add V=1 before the call to execute the python script in > + this file. > + - acpi-bits-test.py: This is the main python test script that generates a > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > + test failures. This is the script one would be interested in if they wanted > + to add or change some component of the log parsing, add a new commandline to > + how QEMU is spawned etc. Test writers typically would not need to modify > + this script unless they wanted to enhance or change the log parsing for > + their tests. > + - requirements.txt: This text file contains the dependent python libraries > + needed for the tests to run. If a new dependent library is needed, it would > + be added here as a new entry and then acpi-bits-test-venv.sh would install > + it when setting up the python virtual environment using pip. > + - README: This text file. > + > +acpi-bits/bits-config: > + This location contains biosbits config files that determine how the software > + runs the tests. > + - bits-config.txt: this is the biosbits config file that determines what tests > + or actions are performed by bits. The description of the config options are > + provided in the file itself. > + > +acpi-bits/prebuilt: > + This location contains prebuilt biosbits binaries that are used to generate > + the bits iso file for testing. > + - bits-2095.zip: The contents from this zip archive are the main contents of > + the iso file that are used for testing. This binary zip archive also > + contains the full source of the bits software including the full text of > + the license agreement listed here: > + https://github.com/biosbits/bits/blob/master/COPYING > + The source tarball can be found in this location in the zip file: > + boot/src/bits-2095.tar.gz > + The additional changes beyond those that are present in the official > + biosbits github repository can be found here: > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > + > + Basically these changes can be categorized into the following: > + (a) changes to make sure biosbits builds with the latest gcc compiler > + (gcc 9.4). > + (b) upgrade of acpica to the latest version (march 2022). > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > + debugcon can be used to collect the logs. > + (d) send a gub halt command to shutdown the VM once all the tests have been > + executed. > + > + This zip file is automatically generated by the bits build system. It can > + be found in the bits build directory and it is suffixed by the bits build > + number. > + Normally, there is no need to make any alteration to this zip archive > + unless one wanted to change the bits software itself (for example to add > + a new config option or change actions of existing options or change the > + debug IO port etc). When such changes are made and a new biosbits software > + is needed to be generated, please refresh this zip archive as well as the > + grub tarball at the same time. Tests will break if changes are made to bits > + that are incompatible with existing behavior. So care must be taken to make > + sure that the change is compatible with existing bits software as much as > + possible. When a new config option is introduced for example, bits must > + be upadated here first before introducing the new config option through > + a later commit. I don't think playing with source tarballs is a reasonable work environment. Let's use submodules just like e.g. firmware does? > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > + scripts and modules to the prebuilt directory. These prebuilt grub > + artifacts are required in order to generate a bootable bits iso file that > + can run tests. > + In order to generate this tar archive, please put the following two > + directories that can be found in the bits build directory in a single > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > + bits that generated the archive: > + > + grub-inst-x86_64-efi > + grub-inst > + > + This version should be the same as the version number of bits that generated > + bits-<n>.zip file. In other words, the two files must be in sync and should > + come from the same build of biosbits of the same version. Things may still > + work if they come from different versions but mixing version numbers are > + not recommended. > + There is normally no need to regenerate this archive unless some fixes or > + changes have gone into the grub that are part of biosbits. > + > +acpi-bits/bits-tests: > + This directory contains biosbits python based tests that are run from within > + the biosbits environment in the spawned VM. New additions of test cases can > + be made in the appropriate test file. For example, new acpi tests can go > + into testacpi.py and one would call testsuite.add_test() to register the new > + test so that it gets executed as a part of the ACPI tests. > + It might be occassionally necessary to disable some subtests or add a new > + test that belongs to a test suite not already present in this directory. To > + do this, please extract the bits source from the zip file mentioned above. > + Copy the test suite/script that needs modification (addition of new tests > + or disabling them) from boot/python location of the extracted directory > + into this directory. > + > + step (a): copy unmodified test script to this directory. > + step (b): update meson.build and add this file to the list. > + Commit (a) and (b) together in the same commit. > + > + step (c): perform modifications to the test. > + Commit (c) in a separate commit. > + > + The test framework will then use your modified test script to run the test. > + No further changes would be needed. Please check the logs to make sure that > + appropriate changes have taken effect. > + > +meson.build files makes sure that the bits qtests are appropriately integrated > +into the QEMU qtest framework and are run with "make check-qtest". > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > +has been built only for this architecture. > + > + > +Author: Ani Sinha <ani@anisinha.ca> > + > +References: > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > -- > 2.25.1
On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > The README file is added describing the directory structure and the purpose > > of every file it contains. It also describes how to add new tests, make changes > > to existing tests or bits config files or regenerate the bits software. > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > 1 file changed, 168 insertions(+) > > create mode 100644 tests/qtest/acpi-bits/README > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > new file mode 100644 > > index 0000000000..97b15f1665 > > --- /dev/null > > +++ b/tests/qtest/acpi-bits/README > > @@ -0,0 +1,168 @@ > > +============================================================================= > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > +============================================================================= > > + > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > +visiting https://biosbits.org/. The github codebase can be found here: > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > +the bios components such as acpi and smbios tables directly through acpica > > +bios interpreter (a freely available C based library written by Intel, > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > +operating system getting involved in between. > > +There are several advantages to directly testing the bios in a real physical > > +machine or VM as opposed to indirectly discovering bios issues through the > > +operating system. For one thing, the OSes tend to hide bios problems from the > > +end user. The other is that we have more control of what we wanted to test > > +and how by directly using acpica interpreter on top of the bios on a running > > +system. More details on the inspiration for developing biosbits and its real > > +life uses can be found in (a) and (b). > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > +bios components using biosbits and reports test failures. > > + > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > +would require python3.8-venv and python3-pip packages to be installed. > > Why do we mess with venv and pip? Certainly possible but > what's wrong with using distro provided packages? There are two things: (a) We are already using pip and venv for our avocado based integration tests. Look for TESTS_VENV_DIR in Makefile.include under tests. (b) the venv is primarily needed because I wanted to take advantage of our rich python library that handles QEMU based machines. There are python qtest libraries as well. These are well tested and used with integration tests and I wanted to keep the test part of the code simple by simply reusing them. however, in order to use them, we need a venv environment within which these qemu python libraries are installed. Integration tests does the same thing. A note about my language of choice - python. I gave a lot of thoughts on this. We do not do a lot of stuff here. All we do is: (a) generate bits iso. (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. (c) collect and analyze logs. We are not inspecting guest memory or manipulating devices or pci buses. We do not need the power of C here. We need something that is simple to write, easy to maintain and understand and can deal with things like manipulating text files and configs easily. Python seems a better fit for the role. > > > + > > +A brief description of the contents of this directory follows: > > + > > +├── acpi-bits-test.py > > +├── acpi-bits-test-venv.sh > > +├── bits-config > > +│ ├── bits-cfg.txt > > +│ └── meson.build > > +├── bits-tests > > +│ ├── meson.build > > +│ ├── smbios.py > > +│ ├── smilatency.py > > +│ ├── testacpi.py > > +│ └── testcpuid.py > > +├── meson.build > > +├── prebuilt > > +│ ├── bits-2095-grub.tar.gz > > +│ ├── bits-2095.zip > > +│ └── meson.build > > +├── README > > +└── requirements.txt > > + > > +acpi-bits: > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > + environment necessary for the actual python test script to run. Amongst > > + other things, it makes sure that QEMU python library is available within > > + that venv so that QEMU machines can be forked. QEMU python library can be > > + found within python/ directory in QEMU source. > > + After setting up the virtual environment, it runs the python test script > > + from within that environment. > > + If you want to enable verbose mode only for bits test and run make check, > > + one trick is to add V=1 before the call to execute the python script in > > + this file. > > + - acpi-bits-test.py: This is the main python test script that generates a > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > + test failures. This is the script one would be interested in if they wanted > > + to add or change some component of the log parsing, add a new commandline to > > + how QEMU is spawned etc. Test writers typically would not need to modify > > + this script unless they wanted to enhance or change the log parsing for > > + their tests. > > + - requirements.txt: This text file contains the dependent python libraries > > + needed for the tests to run. If a new dependent library is needed, it would > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > + it when setting up the python virtual environment using pip. > > + - README: This text file. > > + > > +acpi-bits/bits-config: > > + This location contains biosbits config files that determine how the software > > + runs the tests. > > + - bits-config.txt: this is the biosbits config file that determines what tests > > + or actions are performed by bits. The description of the config options are > > + provided in the file itself. > > + > > +acpi-bits/prebuilt: > > + This location contains prebuilt biosbits binaries that are used to generate > > + the bits iso file for testing. > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > + the iso file that are used for testing. This binary zip archive also > > + contains the full source of the bits software including the full text of > > + the license agreement listed here: > > + https://github.com/biosbits/bits/blob/master/COPYING > > + The source tarball can be found in this location in the zip file: > > + boot/src/bits-2095.tar.gz > > + The additional changes beyond those that are present in the official > > + biosbits github repository can be found here: > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > + > > + Basically these changes can be categorized into the following: > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > + (gcc 9.4). > > + (b) upgrade of acpica to the latest version (march 2022). > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > + debugcon can be used to collect the logs. > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > + executed. > > + > > + This zip file is automatically generated by the bits build system. It can > > + be found in the bits build directory and it is suffixed by the bits build > > + number. > > + Normally, there is no need to make any alteration to this zip archive > > + unless one wanted to change the bits software itself (for example to add > > + a new config option or change actions of existing options or change the > > + debug IO port etc). When such changes are made and a new biosbits software > > + is needed to be generated, please refresh this zip archive as well as the > > + grub tarball at the same time. Tests will break if changes are made to bits > > + that are incompatible with existing behavior. So care must be taken to make > > + sure that the change is compatible with existing bits software as much as > > + possible. When a new config option is introduced for example, bits must > > + be upadated here first before introducing the new config option through > > + a later commit. > > I don't think playing with source tarballs is a reasonable work environment. I agree. However, we do not do much with the source tarball. It is there as part of bits to satisfy the license requirement. If we need to manipulate any test script that is in the source file, we would copy it over and keep it in the bits-test directory and change it there. > Let's use submodules just like e.g. firmware does? Yes I also proposed this to Igor on IRC. We can/maybe need to figure out a place to store the bits source if we think my github is not the best place. However, we need the source only if/when we need to rebuild bits. I suspect it won't be too often if at all. > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > + scripts and modules to the prebuilt directory. These prebuilt grub > > + artifacts are required in order to generate a bootable bits iso file that > > + can run tests. > > + In order to generate this tar archive, please put the following two > > + directories that can be found in the bits build directory in a single > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > + bits that generated the archive: > > + > > + grub-inst-x86_64-efi > > + grub-inst > > + > > + This version should be the same as the version number of bits that generated > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > + come from the same build of biosbits of the same version. Things may still > > + work if they come from different versions but mixing version numbers are > > + not recommended. > > + There is normally no need to regenerate this archive unless some fixes or > > + changes have gone into the grub that are part of biosbits. > > + > > +acpi-bits/bits-tests: > > + This directory contains biosbits python based tests that are run from within > > + the biosbits environment in the spawned VM. New additions of test cases can > > + be made in the appropriate test file. For example, new acpi tests can go > > + into testacpi.py and one would call testsuite.add_test() to register the new > > + test so that it gets executed as a part of the ACPI tests. > > + It might be occassionally necessary to disable some subtests or add a new > > + test that belongs to a test suite not already present in this directory. To > > + do this, please extract the bits source from the zip file mentioned above. > > + Copy the test suite/script that needs modification (addition of new tests > > + or disabling them) from boot/python location of the extracted directory > > + into this directory. > > + > > + step (a): copy unmodified test script to this directory. > > + step (b): update meson.build and add this file to the list. > > + Commit (a) and (b) together in the same commit. > > + > > + step (c): perform modifications to the test. > > + Commit (c) in a separate commit. > > + > > + The test framework will then use your modified test script to run the test. > > + No further changes would be needed. Please check the logs to make sure that > > + appropriate changes have taken effect. > > + > > +meson.build files makes sure that the bits qtests are appropriately integrated > > +into the QEMU qtest framework and are run with "make check-qtest". > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > +has been built only for this architecture. > > + > > + > > +Author: Ani Sinha <ani@anisinha.ca> > > + > > +References: > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > -- > > 2.25.1 >
On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > The README file is added describing the directory structure and the purpose > > > of every file it contains. It also describes how to add new tests, make changes > > > to existing tests or bits config files or regenerate the bits software. > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > --- > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > 1 file changed, 168 insertions(+) > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > new file mode 100644 > > > index 0000000000..97b15f1665 > > > --- /dev/null > > > +++ b/tests/qtest/acpi-bits/README > > > @@ -0,0 +1,168 @@ > > > +============================================================================= > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > +============================================================================= > > > + > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > +the bios components such as acpi and smbios tables directly through acpica > > > +bios interpreter (a freely available C based library written by Intel, > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > +operating system getting involved in between. > > > +There are several advantages to directly testing the bios in a real physical > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > +end user. The other is that we have more control of what we wanted to test > > > +and how by directly using acpica interpreter on top of the bios on a running > > > +system. More details on the inspiration for developing biosbits and its real > > > +life uses can be found in (a) and (b). > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > +bios components using biosbits and reports test failures. > > > + > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > Why do we mess with venv and pip? Certainly possible but > > what's wrong with using distro provided packages? > > There are two things: > (a) We are already using pip and venv for our avocado based > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > tests. > (b) the venv is primarily needed because I wanted to take advantage of > our rich python library that handles QEMU based machines. There are > python qtest libraries as well. These are well tested and used with > integration tests and I wanted to keep the test part of the code > simple by simply reusing them. however, in order to use them, we need > a venv environment within which these qemu python libraries are > installed. Integration tests does the same thing. > > A note about my language of choice - python. I gave a lot of thoughts > on this. We do not do a lot of stuff here. All we do is: > (a) generate bits iso. > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > (c) collect and analyze logs. > > We are not inspecting guest memory or manipulating devices or pci > buses. We do not need the power of C here. We need something that is > simple to write, easy to maintain and understand and can deal with > things like manipulating text files and configs easily. Python seems a > better fit for the role. No problem with that. So that's venv. But do we need pip and pulling packages from the net during testing? > > > > > + > > > +A brief description of the contents of this directory follows: > > > + > > > +├── acpi-bits-test.py > > > +├── acpi-bits-test-venv.sh > > > +├── bits-config > > > +│ ├── bits-cfg.txt > > > +│ └── meson.build > > > +├── bits-tests > > > +│ ├── meson.build > > > +│ ├── smbios.py > > > +│ ├── smilatency.py > > > +│ ├── testacpi.py > > > +│ └── testcpuid.py > > > +├── meson.build > > > +├── prebuilt > > > +│ ├── bits-2095-grub.tar.gz > > > +│ ├── bits-2095.zip > > > +│ └── meson.build > > > +├── README > > > +└── requirements.txt > > > + > > > +acpi-bits: > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > + environment necessary for the actual python test script to run. Amongst > > > + other things, it makes sure that QEMU python library is available within > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > + found within python/ directory in QEMU source. > > > + After setting up the virtual environment, it runs the python test script > > > + from within that environment. > > > + If you want to enable verbose mode only for bits test and run make check, > > > + one trick is to add V=1 before the call to execute the python script in > > > + this file. > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > + test failures. This is the script one would be interested in if they wanted > > > + to add or change some component of the log parsing, add a new commandline to > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > + this script unless they wanted to enhance or change the log parsing for > > > + their tests. > > > + - requirements.txt: This text file contains the dependent python libraries > > > + needed for the tests to run. If a new dependent library is needed, it would > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > + it when setting up the python virtual environment using pip. > > > + - README: This text file. > > > + > > > +acpi-bits/bits-config: > > > + This location contains biosbits config files that determine how the software > > > + runs the tests. > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > + or actions are performed by bits. The description of the config options are > > > + provided in the file itself. > > > + > > > +acpi-bits/prebuilt: > > > + This location contains prebuilt biosbits binaries that are used to generate > > > + the bits iso file for testing. > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > + the iso file that are used for testing. This binary zip archive also > > > + contains the full source of the bits software including the full text of > > > + the license agreement listed here: > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > + The source tarball can be found in this location in the zip file: > > > + boot/src/bits-2095.tar.gz > > > + The additional changes beyond those that are present in the official > > > + biosbits github repository can be found here: > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > + > > > + Basically these changes can be categorized into the following: > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > + (gcc 9.4). > > > + (b) upgrade of acpica to the latest version (march 2022). > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > + debugcon can be used to collect the logs. > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > + executed. > > > + > > > + This zip file is automatically generated by the bits build system. It can > > > + be found in the bits build directory and it is suffixed by the bits build > > > + number. > > > + Normally, there is no need to make any alteration to this zip archive > > > + unless one wanted to change the bits software itself (for example to add > > > + a new config option or change actions of existing options or change the > > > + debug IO port etc). When such changes are made and a new biosbits software > > > + is needed to be generated, please refresh this zip archive as well as the > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > + that are incompatible with existing behavior. So care must be taken to make > > > + sure that the change is compatible with existing bits software as much as > > > + possible. When a new config option is introduced for example, bits must > > > + be upadated here first before introducing the new config option through > > > + a later commit. > > > > I don't think playing with source tarballs is a reasonable work environment. > > I agree. However, we do not do much with the source tarball. It is > there as part of bits to satisfy the license requirement. If we need > to manipulate any test script that is in the source file, we would > copy it over and keep it in the bits-test directory and change it > there. > > > Let's use submodules just like e.g. firmware does? > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > out a place to store the bits source if we think my github is not the > best place. However, we need the source only if/when we need to > rebuild bits. I suspect it won't be too often if at all. Why not on git.qemu.org ? > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > + artifacts are required in order to generate a bootable bits iso file that > > > + can run tests. > > > + In order to generate this tar archive, please put the following two > > > + directories that can be found in the bits build directory in a single > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > + bits that generated the archive: > > > + > > > + grub-inst-x86_64-efi > > > + grub-inst > > > + > > > + This version should be the same as the version number of bits that generated > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > + come from the same build of biosbits of the same version. Things may still > > > + work if they come from different versions but mixing version numbers are > > > + not recommended. > > > + There is normally no need to regenerate this archive unless some fixes or > > > + changes have gone into the grub that are part of biosbits. > > > + > > > +acpi-bits/bits-tests: > > > + This directory contains biosbits python based tests that are run from within > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > + be made in the appropriate test file. For example, new acpi tests can go > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > + test so that it gets executed as a part of the ACPI tests. > > > + It might be occassionally necessary to disable some subtests or add a new > > > + test that belongs to a test suite not already present in this directory. To > > > + do this, please extract the bits source from the zip file mentioned above. > > > + Copy the test suite/script that needs modification (addition of new tests > > > + or disabling them) from boot/python location of the extracted directory > > > + into this directory. > > > + > > > + step (a): copy unmodified test script to this directory. > > > + step (b): update meson.build and add this file to the list. > > > + Commit (a) and (b) together in the same commit. > > > + > > > + step (c): perform modifications to the test. > > > + Commit (c) in a separate commit. > > > + > > > + The test framework will then use your modified test script to run the test. > > > + No further changes would be needed. Please check the logs to make sure that > > > + appropriate changes have taken effect. > > > + > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > +has been built only for this architecture. > > > + > > > + > > > +Author: Ani Sinha <ani@anisinha.ca> > > > + > > > +References: > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > -- > > > 2.25.1 > >
On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > > The README file is added describing the directory structure and the purpose > > > > of every file it contains. It also describes how to add new tests, make changes > > > > to existing tests or bits config files or regenerate the bits software. > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > --- > > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > > 1 file changed, 168 insertions(+) > > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > > new file mode 100644 > > > > index 0000000000..97b15f1665 > > > > --- /dev/null > > > > +++ b/tests/qtest/acpi-bits/README > > > > @@ -0,0 +1,168 @@ > > > > +============================================================================= > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > > +============================================================================= > > > > + > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > > +the bios components such as acpi and smbios tables directly through acpica > > > > +bios interpreter (a freely available C based library written by Intel, > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > > +operating system getting involved in between. > > > > +There are several advantages to directly testing the bios in a real physical > > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > > +end user. The other is that we have more control of what we wanted to test > > > > +and how by directly using acpica interpreter on top of the bios on a running > > > > +system. More details on the inspiration for developing biosbits and its real > > > > +life uses can be found in (a) and (b). > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > > +bios components using biosbits and reports test failures. > > > > + > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > > > Why do we mess with venv and pip? Certainly possible but > > > what's wrong with using distro provided packages? > > > > There are two things: > > (a) We are already using pip and venv for our avocado based > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > > tests. > > (b) the venv is primarily needed because I wanted to take advantage of > > our rich python library that handles QEMU based machines. There are > > python qtest libraries as well. These are well tested and used with > > integration tests and I wanted to keep the test part of the code > > simple by simply reusing them. however, in order to use them, we need > > a venv environment within which these qemu python libraries are > > installed. Integration tests does the same thing. > > > > A note about my language of choice - python. I gave a lot of thoughts > > on this. We do not do a lot of stuff here. All we do is: > > (a) generate bits iso. > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > > (c) collect and analyze logs. > > > > We are not inspecting guest memory or manipulating devices or pci > > buses. We do not need the power of C here. We need something that is > > simple to write, easy to maintain and understand and can deal with > > things like manipulating text files and configs easily. Python seems a > > better fit for the role. > > No problem with that. So that's venv. But do we need pip and pulling > packages from the net during testing? We do that too. See requirements.txt in tests/ Following two are downloaded: avocado-framework==88.1 pycdlib==1.11.0 Also see this line in Makefie.include: $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > + > > > > +A brief description of the contents of this directory follows: > > > > + > > > > +├── acpi-bits-test.py > > > > +├── acpi-bits-test-venv.sh > > > > +├── bits-config > > > > +│ ├── bits-cfg.txt > > > > +│ └── meson.build > > > > +├── bits-tests > > > > +│ ├── meson.build > > > > +│ ├── smbios.py > > > > +│ ├── smilatency.py > > > > +│ ├── testacpi.py > > > > +│ └── testcpuid.py > > > > +├── meson.build > > > > +├── prebuilt > > > > +│ ├── bits-2095-grub.tar.gz > > > > +│ ├── bits-2095.zip > > > > +│ └── meson.build > > > > +├── README > > > > +└── requirements.txt > > > > + > > > > +acpi-bits: > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > > + environment necessary for the actual python test script to run. Amongst > > > > + other things, it makes sure that QEMU python library is available within > > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > > + found within python/ directory in QEMU source. > > > > + After setting up the virtual environment, it runs the python test script > > > > + from within that environment. > > > > + If you want to enable verbose mode only for bits test and run make check, > > > > + one trick is to add V=1 before the call to execute the python script in > > > > + this file. > > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > > + test failures. This is the script one would be interested in if they wanted > > > > + to add or change some component of the log parsing, add a new commandline to > > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > > + this script unless they wanted to enhance or change the log parsing for > > > > + their tests. > > > > + - requirements.txt: This text file contains the dependent python libraries > > > > + needed for the tests to run. If a new dependent library is needed, it would > > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > > + it when setting up the python virtual environment using pip. > > > > + - README: This text file. > > > > + > > > > +acpi-bits/bits-config: > > > > + This location contains biosbits config files that determine how the software > > > > + runs the tests. > > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > > + or actions are performed by bits. The description of the config options are > > > > + provided in the file itself. > > > > + > > > > +acpi-bits/prebuilt: > > > > + This location contains prebuilt biosbits binaries that are used to generate > > > > + the bits iso file for testing. > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > > + the iso file that are used for testing. This binary zip archive also > > > > + contains the full source of the bits software including the full text of > > > > + the license agreement listed here: > > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > > + The source tarball can be found in this location in the zip file: > > > > + boot/src/bits-2095.tar.gz > > > > + The additional changes beyond those that are present in the official > > > > + biosbits github repository can be found here: > > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > > + > > > > + Basically these changes can be categorized into the following: > > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > > + (gcc 9.4). > > > > + (b) upgrade of acpica to the latest version (march 2022). > > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > > + debugcon can be used to collect the logs. > > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > > + executed. > > > > + > > > > + This zip file is automatically generated by the bits build system. It can > > > > + be found in the bits build directory and it is suffixed by the bits build > > > > + number. > > > > + Normally, there is no need to make any alteration to this zip archive > > > > + unless one wanted to change the bits software itself (for example to add > > > > + a new config option or change actions of existing options or change the > > > > + debug IO port etc). When such changes are made and a new biosbits software > > > > + is needed to be generated, please refresh this zip archive as well as the > > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > > + that are incompatible with existing behavior. So care must be taken to make > > > > + sure that the change is compatible with existing bits software as much as > > > > + possible. When a new config option is introduced for example, bits must > > > > + be upadated here first before introducing the new config option through > > > > + a later commit. > > > > > > I don't think playing with source tarballs is a reasonable work environment. > > > > I agree. However, we do not do much with the source tarball. It is > > there as part of bits to satisfy the license requirement. If we need > > to manipulate any test script that is in the source file, we would > > copy it over and keep it in the bits-test directory and change it > > there. > > > > > Let's use submodules just like e.g. firmware does? > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > > out a place to store the bits source if we think my github is not the > > best place. However, we need the source only if/when we need to > > rebuild bits. I suspect it won't be too often if at all. > > Why not on git.qemu.org ? Sounds good to me. Do I get push access? :-) > > > > > > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > > + artifacts are required in order to generate a bootable bits iso file that > > > > + can run tests. > > > > + In order to generate this tar archive, please put the following two > > > > + directories that can be found in the bits build directory in a single > > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > > + bits that generated the archive: > > > > + > > > > + grub-inst-x86_64-efi > > > > + grub-inst > > > > + > > > > + This version should be the same as the version number of bits that generated > > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > > + come from the same build of biosbits of the same version. Things may still > > > > + work if they come from different versions but mixing version numbers are > > > > + not recommended. > > > > + There is normally no need to regenerate this archive unless some fixes or > > > > + changes have gone into the grub that are part of biosbits. > > > > + > > > > +acpi-bits/bits-tests: > > > > + This directory contains biosbits python based tests that are run from within > > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > > + be made in the appropriate test file. For example, new acpi tests can go > > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > > + test so that it gets executed as a part of the ACPI tests. > > > > + It might be occassionally necessary to disable some subtests or add a new > > > > + test that belongs to a test suite not already present in this directory. To > > > > + do this, please extract the bits source from the zip file mentioned above. > > > > + Copy the test suite/script that needs modification (addition of new tests > > > > + or disabling them) from boot/python location of the extracted directory > > > > + into this directory. > > > > + > > > > + step (a): copy unmodified test script to this directory. > > > > + step (b): update meson.build and add this file to the list. > > > > + Commit (a) and (b) together in the same commit. > > > > + > > > > + step (c): perform modifications to the test. > > > > + Commit (c) in a separate commit. > > > > + > > > > + The test framework will then use your modified test script to run the test. > > > > + No further changes would be needed. Please check the logs to make sure that > > > > + appropriate changes have taken effect. > > > > + > > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > > +has been built only for this architecture. > > > > + > > > > + > > > > +Author: Ani Sinha <ani@anisinha.ca> > > > > + > > > > +References: > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > > -- > > > > 2.25.1 > > > >
On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > > > The README file is added describing the directory structure and the purpose > > > > > of every file it contains. It also describes how to add new tests, make changes > > > > > to existing tests or bits config files or regenerate the bits software. > > > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > > --- > > > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > > > 1 file changed, 168 insertions(+) > > > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > > > new file mode 100644 > > > > > index 0000000000..97b15f1665 > > > > > --- /dev/null > > > > > +++ b/tests/qtest/acpi-bits/README > > > > > @@ -0,0 +1,168 @@ > > > > > +============================================================================= > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > > > +============================================================================= > > > > > + > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > > > +the bios components such as acpi and smbios tables directly through acpica > > > > > +bios interpreter (a freely available C based library written by Intel, > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > > > +operating system getting involved in between. > > > > > +There are several advantages to directly testing the bios in a real physical > > > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > > > +end user. The other is that we have more control of what we wanted to test > > > > > +and how by directly using acpica interpreter on top of the bios on a running > > > > > +system. More details on the inspiration for developing biosbits and its real > > > > > +life uses can be found in (a) and (b). > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > > > +bios components using biosbits and reports test failures. > > > > > + > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > > > > > Why do we mess with venv and pip? Certainly possible but > > > > what's wrong with using distro provided packages? > > > > > > There are two things: > > > (a) We are already using pip and venv for our avocado based > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > > > tests. > > > (b) the venv is primarily needed because I wanted to take advantage of > > > our rich python library that handles QEMU based machines. There are > > > python qtest libraries as well. These are well tested and used with > > > integration tests and I wanted to keep the test part of the code > > > simple by simply reusing them. however, in order to use them, we need > > > a venv environment within which these qemu python libraries are > > > installed. Integration tests does the same thing. > > > > > > A note about my language of choice - python. I gave a lot of thoughts > > > on this. We do not do a lot of stuff here. All we do is: > > > (a) generate bits iso. > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > > > (c) collect and analyze logs. > > > > > > We are not inspecting guest memory or manipulating devices or pci > > > buses. We do not need the power of C here. We need something that is > > > simple to write, easy to maintain and understand and can deal with > > > things like manipulating text files and configs easily. Python seems a > > > better fit for the role. > > > > No problem with that. So that's venv. But do we need pip and pulling > > packages from the net during testing? > > We do that too. See requirements.txt in tests/ > Following two are downloaded: > avocado-framework==88.1 > pycdlib==1.11.0 > > Also see this line in Makefie.include: > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) Right but that's avocado since it pulls lots of stuff from the net anyway. Are the libraries in question not packaged on major distros? > > > > > > > > > > > + > > > > > +A brief description of the contents of this directory follows: > > > > > + > > > > > +├── acpi-bits-test.py > > > > > +├── acpi-bits-test-venv.sh > > > > > +├── bits-config > > > > > +│ ├── bits-cfg.txt > > > > > +│ └── meson.build > > > > > +├── bits-tests > > > > > +│ ├── meson.build > > > > > +│ ├── smbios.py > > > > > +│ ├── smilatency.py > > > > > +│ ├── testacpi.py > > > > > +│ └── testcpuid.py > > > > > +├── meson.build > > > > > +├── prebuilt > > > > > +│ ├── bits-2095-grub.tar.gz > > > > > +│ ├── bits-2095.zip > > > > > +│ └── meson.build > > > > > +├── README > > > > > +└── requirements.txt > > > > > + > > > > > +acpi-bits: > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > > > + environment necessary for the actual python test script to run. Amongst > > > > > + other things, it makes sure that QEMU python library is available within > > > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > > > + found within python/ directory in QEMU source. > > > > > + After setting up the virtual environment, it runs the python test script > > > > > + from within that environment. > > > > > + If you want to enable verbose mode only for bits test and run make check, > > > > > + one trick is to add V=1 before the call to execute the python script in > > > > > + this file. > > > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > > > + test failures. This is the script one would be interested in if they wanted > > > > > + to add or change some component of the log parsing, add a new commandline to > > > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > > > + this script unless they wanted to enhance or change the log parsing for > > > > > + their tests. > > > > > + - requirements.txt: This text file contains the dependent python libraries > > > > > + needed for the tests to run. If a new dependent library is needed, it would > > > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > > > + it when setting up the python virtual environment using pip. > > > > > + - README: This text file. > > > > > + > > > > > +acpi-bits/bits-config: > > > > > + This location contains biosbits config files that determine how the software > > > > > + runs the tests. > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > > > + or actions are performed by bits. The description of the config options are > > > > > + provided in the file itself. > > > > > + > > > > > +acpi-bits/prebuilt: > > > > > + This location contains prebuilt biosbits binaries that are used to generate > > > > > + the bits iso file for testing. > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > > > + the iso file that are used for testing. This binary zip archive also > > > > > + contains the full source of the bits software including the full text of > > > > > + the license agreement listed here: > > > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > > > + The source tarball can be found in this location in the zip file: > > > > > + boot/src/bits-2095.tar.gz > > > > > + The additional changes beyond those that are present in the official > > > > > + biosbits github repository can be found here: > > > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > > > + > > > > > + Basically these changes can be categorized into the following: > > > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > > > + (gcc 9.4). > > > > > + (b) upgrade of acpica to the latest version (march 2022). > > > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > > > + debugcon can be used to collect the logs. > > > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > > > + executed. > > > > > + > > > > > + This zip file is automatically generated by the bits build system. It can > > > > > + be found in the bits build directory and it is suffixed by the bits build > > > > > + number. > > > > > + Normally, there is no need to make any alteration to this zip archive > > > > > + unless one wanted to change the bits software itself (for example to add > > > > > + a new config option or change actions of existing options or change the > > > > > + debug IO port etc). When such changes are made and a new biosbits software > > > > > + is needed to be generated, please refresh this zip archive as well as the > > > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > > > + that are incompatible with existing behavior. So care must be taken to make > > > > > + sure that the change is compatible with existing bits software as much as > > > > > + possible. When a new config option is introduced for example, bits must > > > > > + be upadated here first before introducing the new config option through > > > > > + a later commit. > > > > > > > > I don't think playing with source tarballs is a reasonable work environment. > > > > > > I agree. However, we do not do much with the source tarball. It is > > > there as part of bits to satisfy the license requirement. If we need > > > to manipulate any test script that is in the source file, we would > > > copy it over and keep it in the bits-test directory and change it > > > there. > > > > > > > Let's use submodules just like e.g. firmware does? > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > > > out a place to store the bits source if we think my github is not the > > > best place. However, we need the source only if/when we need to > > > rebuild bits. I suspect it won't be too often if at all. > > > > Why not on git.qemu.org ? > > Sounds good to me. Do I get push access? :-) > > > > > > > > > > > > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > > > + artifacts are required in order to generate a bootable bits iso file that > > > > > + can run tests. > > > > > + In order to generate this tar archive, please put the following two > > > > > + directories that can be found in the bits build directory in a single > > > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > > > + bits that generated the archive: > > > > > + > > > > > + grub-inst-x86_64-efi > > > > > + grub-inst > > > > > + > > > > > + This version should be the same as the version number of bits that generated > > > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > > > + come from the same build of biosbits of the same version. Things may still > > > > > + work if they come from different versions but mixing version numbers are > > > > > + not recommended. > > > > > + There is normally no need to regenerate this archive unless some fixes or > > > > > + changes have gone into the grub that are part of biosbits. > > > > > + > > > > > +acpi-bits/bits-tests: > > > > > + This directory contains biosbits python based tests that are run from within > > > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > > > + be made in the appropriate test file. For example, new acpi tests can go > > > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > > > + test so that it gets executed as a part of the ACPI tests. > > > > > + It might be occassionally necessary to disable some subtests or add a new > > > > > + test that belongs to a test suite not already present in this directory. To > > > > > + do this, please extract the bits source from the zip file mentioned above. > > > > > + Copy the test suite/script that needs modification (addition of new tests > > > > > + or disabling them) from boot/python location of the extracted directory > > > > > + into this directory. > > > > > + > > > > > + step (a): copy unmodified test script to this directory. > > > > > + step (b): update meson.build and add this file to the list. > > > > > + Commit (a) and (b) together in the same commit. > > > > > + > > > > > + step (c): perform modifications to the test. > > > > > + Commit (c) in a separate commit. > > > > > + > > > > > + The test framework will then use your modified test script to run the test. > > > > > + No further changes would be needed. Please check the logs to make sure that > > > > > + appropriate changes have taken effect. > > > > > + > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > > > +has been built only for this architecture. > > > > > + > > > > > + > > > > > +Author: Ani Sinha <ani@anisinha.ca> > > > > > + > > > > > +References: > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > > > -- > > > > > 2.25.1 > > > > > >
On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > > > > The README file is added describing the directory structure and the purpose > > > > > > of every file it contains. It also describes how to add new tests, make changes > > > > > > to existing tests or bits config files or regenerate the bits software. > > > > > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > > > --- > > > > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > > > > 1 file changed, 168 insertions(+) > > > > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > > > > new file mode 100644 > > > > > > index 0000000000..97b15f1665 > > > > > > --- /dev/null > > > > > > +++ b/tests/qtest/acpi-bits/README > > > > > > @@ -0,0 +1,168 @@ > > > > > > +============================================================================= > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > > > > +============================================================================= > > > > > > + > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > > > > +the bios components such as acpi and smbios tables directly through acpica > > > > > > +bios interpreter (a freely available C based library written by Intel, > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > > > > +operating system getting involved in between. > > > > > > +There are several advantages to directly testing the bios in a real physical > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > > > > +end user. The other is that we have more control of what we wanted to test > > > > > > +and how by directly using acpica interpreter on top of the bios on a running > > > > > > +system. More details on the inspiration for developing biosbits and its real > > > > > > +life uses can be found in (a) and (b). > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > > > > +bios components using biosbits and reports test failures. > > > > > > + > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > > > > > > > Why do we mess with venv and pip? Certainly possible but > > > > > what's wrong with using distro provided packages? > > > > > > > > There are two things: > > > > (a) We are already using pip and venv for our avocado based > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > > > > tests. > > > > (b) the venv is primarily needed because I wanted to take advantage of > > > > our rich python library that handles QEMU based machines. There are > > > > python qtest libraries as well. These are well tested and used with > > > > integration tests and I wanted to keep the test part of the code > > > > simple by simply reusing them. however, in order to use them, we need > > > > a venv environment within which these qemu python libraries are > > > > installed. Integration tests does the same thing. > > > > > > > > A note about my language of choice - python. I gave a lot of thoughts > > > > on this. We do not do a lot of stuff here. All we do is: > > > > (a) generate bits iso. > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > > > > (c) collect and analyze logs. > > > > > > > > We are not inspecting guest memory or manipulating devices or pci > > > > buses. We do not need the power of C here. We need something that is > > > > simple to write, easy to maintain and understand and can deal with > > > > things like manipulating text files and configs easily. Python seems a > > > > better fit for the role. > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > packages from the net during testing? > > > > We do that too. See requirements.txt in tests/ > > Following two are downloaded: > > avocado-framework==88.1 > > pycdlib==1.11.0 > > > > Also see this line in Makefie.include: > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > Right but that's avocado since it pulls lots of stuff from > the net anyway. > Are the libraries in question not packaged on major distros? Currently I only need this: https://github.com/python-tap/tappy which is the basic TAP processing library for python. It seems its only installed through pip: https://tappy.readthedocs.io/en/latest/ I do not think this is packaged by default. It's such a basic library for parsing test output that maybe we can keep this somewhere within the python src tree? Not sure ... > > > > > > > > > > > > > > > + > > > > > > +A brief description of the contents of this directory follows: > > > > > > + > > > > > > +├── acpi-bits-test.py > > > > > > +├── acpi-bits-test-venv.sh > > > > > > +├── bits-config > > > > > > +│ ├── bits-cfg.txt > > > > > > +│ └── meson.build > > > > > > +├── bits-tests > > > > > > +│ ├── meson.build > > > > > > +│ ├── smbios.py > > > > > > +│ ├── smilatency.py > > > > > > +│ ├── testacpi.py > > > > > > +│ └── testcpuid.py > > > > > > +├── meson.build > > > > > > +├── prebuilt > > > > > > +│ ├── bits-2095-grub.tar.gz > > > > > > +│ ├── bits-2095.zip > > > > > > +│ └── meson.build > > > > > > +├── README > > > > > > +└── requirements.txt > > > > > > + > > > > > > +acpi-bits: > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > > > > + environment necessary for the actual python test script to run. Amongst > > > > > > + other things, it makes sure that QEMU python library is available within > > > > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > > > > + found within python/ directory in QEMU source. > > > > > > + After setting up the virtual environment, it runs the python test script > > > > > > + from within that environment. > > > > > > + If you want to enable verbose mode only for bits test and run make check, > > > > > > + one trick is to add V=1 before the call to execute the python script in > > > > > > + this file. > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > > > > + test failures. This is the script one would be interested in if they wanted > > > > > > + to add or change some component of the log parsing, add a new commandline to > > > > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > > > > + this script unless they wanted to enhance or change the log parsing for > > > > > > + their tests. > > > > > > + - requirements.txt: This text file contains the dependent python libraries > > > > > > + needed for the tests to run. If a new dependent library is needed, it would > > > > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > > > > + it when setting up the python virtual environment using pip. > > > > > > + - README: This text file. > > > > > > + > > > > > > +acpi-bits/bits-config: > > > > > > + This location contains biosbits config files that determine how the software > > > > > > + runs the tests. > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > > > > + or actions are performed by bits. The description of the config options are > > > > > > + provided in the file itself. > > > > > > + > > > > > > +acpi-bits/prebuilt: > > > > > > + This location contains prebuilt biosbits binaries that are used to generate > > > > > > + the bits iso file for testing. > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > > > > + the iso file that are used for testing. This binary zip archive also > > > > > > + contains the full source of the bits software including the full text of > > > > > > + the license agreement listed here: > > > > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > > > > + The source tarball can be found in this location in the zip file: > > > > > > + boot/src/bits-2095.tar.gz > > > > > > + The additional changes beyond those that are present in the official > > > > > > + biosbits github repository can be found here: > > > > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > > > > + > > > > > > + Basically these changes can be categorized into the following: > > > > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > > > > + (gcc 9.4). > > > > > > + (b) upgrade of acpica to the latest version (march 2022). > > > > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > > > > + debugcon can be used to collect the logs. > > > > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > > > > + executed. > > > > > > + > > > > > > + This zip file is automatically generated by the bits build system. It can > > > > > > + be found in the bits build directory and it is suffixed by the bits build > > > > > > + number. > > > > > > + Normally, there is no need to make any alteration to this zip archive > > > > > > + unless one wanted to change the bits software itself (for example to add > > > > > > + a new config option or change actions of existing options or change the > > > > > > + debug IO port etc). When such changes are made and a new biosbits software > > > > > > + is needed to be generated, please refresh this zip archive as well as the > > > > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > > > > + that are incompatible with existing behavior. So care must be taken to make > > > > > > + sure that the change is compatible with existing bits software as much as > > > > > > + possible. When a new config option is introduced for example, bits must > > > > > > + be upadated here first before introducing the new config option through > > > > > > + a later commit. > > > > > > > > > > I don't think playing with source tarballs is a reasonable work environment. > > > > > > > > I agree. However, we do not do much with the source tarball. It is > > > > there as part of bits to satisfy the license requirement. If we need > > > > to manipulate any test script that is in the source file, we would > > > > copy it over and keep it in the bits-test directory and change it > > > > there. > > > > > > > > > Let's use submodules just like e.g. firmware does? > > > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > > > > out a place to store the bits source if we think my github is not the > > > > best place. However, we need the source only if/when we need to > > > > rebuild bits. I suspect it won't be too often if at all. > > > > > > Why not on git.qemu.org ? > > > > Sounds good to me. Do I get push access? :-) > > > > > > > > > > > > > > > > > > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > > > > + artifacts are required in order to generate a bootable bits iso file that > > > > > > + can run tests. > > > > > > + In order to generate this tar archive, please put the following two > > > > > > + directories that can be found in the bits build directory in a single > > > > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > > > > + bits that generated the archive: > > > > > > + > > > > > > + grub-inst-x86_64-efi > > > > > > + grub-inst > > > > > > + > > > > > > + This version should be the same as the version number of bits that generated > > > > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > > > > + come from the same build of biosbits of the same version. Things may still > > > > > > + work if they come from different versions but mixing version numbers are > > > > > > + not recommended. > > > > > > + There is normally no need to regenerate this archive unless some fixes or > > > > > > + changes have gone into the grub that are part of biosbits. > > > > > > + > > > > > > +acpi-bits/bits-tests: > > > > > > + This directory contains biosbits python based tests that are run from within > > > > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > > > > + be made in the appropriate test file. For example, new acpi tests can go > > > > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > > > > + test so that it gets executed as a part of the ACPI tests. > > > > > > + It might be occassionally necessary to disable some subtests or add a new > > > > > > + test that belongs to a test suite not already present in this directory. To > > > > > > + do this, please extract the bits source from the zip file mentioned above. > > > > > > + Copy the test suite/script that needs modification (addition of new tests > > > > > > + or disabling them) from boot/python location of the extracted directory > > > > > > + into this directory. > > > > > > + > > > > > > + step (a): copy unmodified test script to this directory. > > > > > > + step (b): update meson.build and add this file to the list. > > > > > > + Commit (a) and (b) together in the same commit. > > > > > > + > > > > > > + step (c): perform modifications to the test. > > > > > > + Commit (c) in a separate commit. > > > > > > + > > > > > > + The test framework will then use your modified test script to run the test. > > > > > > + No further changes would be needed. Please check the logs to make sure that > > > > > > + appropriate changes have taken effect. > > > > > > + > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > > > > +has been built only for this architecture. > > > > > > + > > > > > > + > > > > > > +Author: Ani Sinha <ani@anisinha.ca> > > > > > > + > > > > > > +References: > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > > > > -- > > > > > > 2.25.1 > > > > > > > > >
On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote: > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > > > > > The README file is added describing the directory structure and the purpose > > > > > > > of every file it contains. It also describes how to add new tests, make changes > > > > > > > to existing tests or bits config files or regenerate the bits software. > > > > > > > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > > > > --- > > > > > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > > > > > 1 file changed, 168 insertions(+) > > > > > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > > > > > new file mode 100644 > > > > > > > index 0000000000..97b15f1665 > > > > > > > --- /dev/null > > > > > > > +++ b/tests/qtest/acpi-bits/README > > > > > > > @@ -0,0 +1,168 @@ > > > > > > > +============================================================================= > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > > > > > +============================================================================= > > > > > > > + > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > > > > > +the bios components such as acpi and smbios tables directly through acpica > > > > > > > +bios interpreter (a freely available C based library written by Intel, > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > > > > > +operating system getting involved in between. > > > > > > > +There are several advantages to directly testing the bios in a real physical > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > > > > > +end user. The other is that we have more control of what we wanted to test > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running > > > > > > > +system. More details on the inspiration for developing biosbits and its real > > > > > > > +life uses can be found in (a) and (b). > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > > > > > +bios components using biosbits and reports test failures. > > > > > > > + > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > > > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > > > > > > > > > Why do we mess with venv and pip? Certainly possible but > > > > > > what's wrong with using distro provided packages? > > > > > > > > > > There are two things: > > > > > (a) We are already using pip and venv for our avocado based > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > > > > > tests. > > > > > (b) the venv is primarily needed because I wanted to take advantage of > > > > > our rich python library that handles QEMU based machines. There are > > > > > python qtest libraries as well. These are well tested and used with > > > > > integration tests and I wanted to keep the test part of the code > > > > > simple by simply reusing them. however, in order to use them, we need > > > > > a venv environment within which these qemu python libraries are > > > > > installed. Integration tests does the same thing. > > > > > > > > > > A note about my language of choice - python. I gave a lot of thoughts > > > > > on this. We do not do a lot of stuff here. All we do is: > > > > > (a) generate bits iso. > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > > > > > (c) collect and analyze logs. > > > > > > > > > > We are not inspecting guest memory or manipulating devices or pci > > > > > buses. We do not need the power of C here. We need something that is > > > > > simple to write, easy to maintain and understand and can deal with > > > > > things like manipulating text files and configs easily. Python seems a > > > > > better fit for the role. > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > packages from the net during testing? > > > > > > We do that too. See requirements.txt in tests/ > > > Following two are downloaded: > > > avocado-framework==88.1 > > > pycdlib==1.11.0 > > > > > > Also see this line in Makefie.include: > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > Right but that's avocado since it pulls lots of stuff from > > the net anyway. > > Are the libraries in question not packaged on major distros? > > Currently I only need this: > https://github.com/python-tap/tappy > which is the basic TAP processing library for python. > > It seems its only installed through pip: > https://tappy.readthedocs.io/en/latest/ > > I do not think this is packaged by default. It's such a basic library > for parsing test output that maybe we can keep this somewhere within > the python src tree? Not sure ... It's pretty small for sure. Another submodule? > > > > > > > > > > > > > > > > > > > + > > > > > > > +A brief description of the contents of this directory follows: > > > > > > > + > > > > > > > +├── acpi-bits-test.py > > > > > > > +├── acpi-bits-test-venv.sh > > > > > > > +├── bits-config > > > > > > > +│ ├── bits-cfg.txt > > > > > > > +│ └── meson.build > > > > > > > +├── bits-tests > > > > > > > +│ ├── meson.build > > > > > > > +│ ├── smbios.py > > > > > > > +│ ├── smilatency.py > > > > > > > +│ ├── testacpi.py > > > > > > > +│ └── testcpuid.py > > > > > > > +├── meson.build > > > > > > > +├── prebuilt > > > > > > > +│ ├── bits-2095-grub.tar.gz > > > > > > > +│ ├── bits-2095.zip > > > > > > > +│ └── meson.build > > > > > > > +├── README > > > > > > > +└── requirements.txt > > > > > > > + > > > > > > > +acpi-bits: > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > > > > > + environment necessary for the actual python test script to run. Amongst > > > > > > > + other things, it makes sure that QEMU python library is available within > > > > > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > > > > > + found within python/ directory in QEMU source. > > > > > > > + After setting up the virtual environment, it runs the python test script > > > > > > > + from within that environment. > > > > > > > + If you want to enable verbose mode only for bits test and run make check, > > > > > > > + one trick is to add V=1 before the call to execute the python script in > > > > > > > + this file. > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > > > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > > > > > + test failures. This is the script one would be interested in if they wanted > > > > > > > + to add or change some component of the log parsing, add a new commandline to > > > > > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > > > > > + this script unless they wanted to enhance or change the log parsing for > > > > > > > + their tests. > > > > > > > + - requirements.txt: This text file contains the dependent python libraries > > > > > > > + needed for the tests to run. If a new dependent library is needed, it would > > > > > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > > > > > + it when setting up the python virtual environment using pip. > > > > > > > + - README: This text file. > > > > > > > + > > > > > > > +acpi-bits/bits-config: > > > > > > > + This location contains biosbits config files that determine how the software > > > > > > > + runs the tests. > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > > > > > + or actions are performed by bits. The description of the config options are > > > > > > > + provided in the file itself. > > > > > > > + > > > > > > > +acpi-bits/prebuilt: > > > > > > > + This location contains prebuilt biosbits binaries that are used to generate > > > > > > > + the bits iso file for testing. > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > > > > > + the iso file that are used for testing. This binary zip archive also > > > > > > > + contains the full source of the bits software including the full text of > > > > > > > + the license agreement listed here: > > > > > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > > > > > + The source tarball can be found in this location in the zip file: > > > > > > > + boot/src/bits-2095.tar.gz > > > > > > > + The additional changes beyond those that are present in the official > > > > > > > + biosbits github repository can be found here: > > > > > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > > > > > + > > > > > > > + Basically these changes can be categorized into the following: > > > > > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > > > > > + (gcc 9.4). > > > > > > > + (b) upgrade of acpica to the latest version (march 2022). > > > > > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > > > > > + debugcon can be used to collect the logs. > > > > > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > > > > > + executed. > > > > > > > + > > > > > > > + This zip file is automatically generated by the bits build system. It can > > > > > > > + be found in the bits build directory and it is suffixed by the bits build > > > > > > > + number. > > > > > > > + Normally, there is no need to make any alteration to this zip archive > > > > > > > + unless one wanted to change the bits software itself (for example to add > > > > > > > + a new config option or change actions of existing options or change the > > > > > > > + debug IO port etc). When such changes are made and a new biosbits software > > > > > > > + is needed to be generated, please refresh this zip archive as well as the > > > > > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > > > > > + that are incompatible with existing behavior. So care must be taken to make > > > > > > > + sure that the change is compatible with existing bits software as much as > > > > > > > + possible. When a new config option is introduced for example, bits must > > > > > > > + be upadated here first before introducing the new config option through > > > > > > > + a later commit. > > > > > > > > > > > > I don't think playing with source tarballs is a reasonable work environment. > > > > > > > > > > I agree. However, we do not do much with the source tarball. It is > > > > > there as part of bits to satisfy the license requirement. If we need > > > > > to manipulate any test script that is in the source file, we would > > > > > copy it over and keep it in the bits-test directory and change it > > > > > there. > > > > > > > > > > > Let's use submodules just like e.g. firmware does? > > > > > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > > > > > out a place to store the bits source if we think my github is not the > > > > > best place. However, we need the source only if/when we need to > > > > > rebuild bits. I suspect it won't be too often if at all. > > > > > > > > Why not on git.qemu.org ? > > > > > > Sounds good to me. Do I get push access? :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > > > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > > > > > + artifacts are required in order to generate a bootable bits iso file that > > > > > > > + can run tests. > > > > > > > + In order to generate this tar archive, please put the following two > > > > > > > + directories that can be found in the bits build directory in a single > > > > > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > > > > > + bits that generated the archive: > > > > > > > + > > > > > > > + grub-inst-x86_64-efi > > > > > > > + grub-inst > > > > > > > + > > > > > > > + This version should be the same as the version number of bits that generated > > > > > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > > > > > + come from the same build of biosbits of the same version. Things may still > > > > > > > + work if they come from different versions but mixing version numbers are > > > > > > > + not recommended. > > > > > > > + There is normally no need to regenerate this archive unless some fixes or > > > > > > > + changes have gone into the grub that are part of biosbits. > > > > > > > + > > > > > > > +acpi-bits/bits-tests: > > > > > > > + This directory contains biosbits python based tests that are run from within > > > > > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > > > > > + be made in the appropriate test file. For example, new acpi tests can go > > > > > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > > > > > + test so that it gets executed as a part of the ACPI tests. > > > > > > > + It might be occassionally necessary to disable some subtests or add a new > > > > > > > + test that belongs to a test suite not already present in this directory. To > > > > > > > + do this, please extract the bits source from the zip file mentioned above. > > > > > > > + Copy the test suite/script that needs modification (addition of new tests > > > > > > > + or disabling them) from boot/python location of the extracted directory > > > > > > > + into this directory. > > > > > > > + > > > > > > > + step (a): copy unmodified test script to this directory. > > > > > > > + step (b): update meson.build and add this file to the list. > > > > > > > + Commit (a) and (b) together in the same commit. > > > > > > > + > > > > > > > + step (c): perform modifications to the test. > > > > > > > + Commit (c) in a separate commit. > > > > > > > + > > > > > > > + The test framework will then use your modified test script to run the test. > > > > > > > + No further changes would be needed. Please check the logs to make sure that > > > > > > > + appropriate changes have taken effect. > > > > > > > + > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > > > > > +has been built only for this architecture. > > > > > > > + > > > > > > > + > > > > > > > +Author: Ani Sinha <ani@anisinha.ca> > > > > > > > + > > > > > > > +References: > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > > > > > -- > > > > > > > 2.25.1 > > > > > > > > > > > >
On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote: > > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > > > > > > The README file is added describing the directory structure and the purpose > > > > > > > > of every file it contains. It also describes how to add new tests, make changes > > > > > > > > to existing tests or bits config files or regenerate the bits software. > > > > > > > > > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > > > > > --- > > > > > > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > > > > > > 1 file changed, 168 insertions(+) > > > > > > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > > > > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > > > > > > new file mode 100644 > > > > > > > > index 0000000000..97b15f1665 > > > > > > > > --- /dev/null > > > > > > > > +++ b/tests/qtest/acpi-bits/README > > > > > > > > @@ -0,0 +1,168 @@ > > > > > > > > +============================================================================= > > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > > > > > > +============================================================================= > > > > > > > > + > > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > > > > > > +the bios components such as acpi and smbios tables directly through acpica > > > > > > > > +bios interpreter (a freely available C based library written by Intel, > > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > > > > > > +operating system getting involved in between. > > > > > > > > +There are several advantages to directly testing the bios in a real physical > > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > > > > > > +end user. The other is that we have more control of what we wanted to test > > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running > > > > > > > > +system. More details on the inspiration for developing biosbits and its real > > > > > > > > +life uses can be found in (a) and (b). > > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > > > > > > +bios components using biosbits and reports test failures. > > > > > > > > + > > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > > > > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > > > > > > > > > > > Why do we mess with venv and pip? Certainly possible but > > > > > > > what's wrong with using distro provided packages? > > > > > > > > > > > > There are two things: > > > > > > (a) We are already using pip and venv for our avocado based > > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > > > > > > tests. > > > > > > (b) the venv is primarily needed because I wanted to take advantage of > > > > > > our rich python library that handles QEMU based machines. There are > > > > > > python qtest libraries as well. These are well tested and used with > > > > > > integration tests and I wanted to keep the test part of the code > > > > > > simple by simply reusing them. however, in order to use them, we need > > > > > > a venv environment within which these qemu python libraries are > > > > > > installed. Integration tests does the same thing. > > > > > > > > > > > > A note about my language of choice - python. I gave a lot of thoughts > > > > > > on this. We do not do a lot of stuff here. All we do is: > > > > > > (a) generate bits iso. > > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > > > > > > (c) collect and analyze logs. > > > > > > > > > > > > We are not inspecting guest memory or manipulating devices or pci > > > > > > buses. We do not need the power of C here. We need something that is > > > > > > simple to write, easy to maintain and understand and can deal with > > > > > > things like manipulating text files and configs easily. Python seems a > > > > > > better fit for the role. > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > packages from the net during testing? > > > > > > > > We do that too. See requirements.txt in tests/ > > > > Following two are downloaded: > > > > avocado-framework==88.1 > > > > pycdlib==1.11.0 > > > > > > > > Also see this line in Makefie.include: > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > Right but that's avocado since it pulls lots of stuff from > > > the net anyway. > > > Are the libraries in question not packaged on major distros? > > > > Currently I only need this: > > https://github.com/python-tap/tappy > > which is the basic TAP processing library for python. > > > > It seems its only installed through pip: > > https://tappy.readthedocs.io/en/latest/ > > > > I do not think this is packaged by default. It's such a basic library > > for parsing test output that maybe we can keep this somewhere within > > the python src tree? Not sure ... > > It's pretty small for sure. Another submodule? Unlike BITS, this one is likely going to be maintained for a while and will receive new releases through https://pypi.org/project/tap.py/ so forking is OK but someone has to keep this updated. I am open to anything. Whatever feels right is fine to me. > > > > > > > > > > > > > > > > > > > > > > > > + > > > > > > > > +A brief description of the contents of this directory follows: > > > > > > > > + > > > > > > > > +├── acpi-bits-test.py > > > > > > > > +├── acpi-bits-test-venv.sh > > > > > > > > +├── bits-config > > > > > > > > +│ ├── bits-cfg.txt > > > > > > > > +│ └── meson.build > > > > > > > > +├── bits-tests > > > > > > > > +│ ├── meson.build > > > > > > > > +│ ├── smbios.py > > > > > > > > +│ ├── smilatency.py > > > > > > > > +│ ├── testacpi.py > > > > > > > > +│ └── testcpuid.py > > > > > > > > +├── meson.build > > > > > > > > +├── prebuilt > > > > > > > > +│ ├── bits-2095-grub.tar.gz > > > > > > > > +│ ├── bits-2095.zip > > > > > > > > +│ └── meson.build > > > > > > > > +├── README > > > > > > > > +└── requirements.txt > > > > > > > > + > > > > > > > > +acpi-bits: > > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > > > > > > + environment necessary for the actual python test script to run. Amongst > > > > > > > > + other things, it makes sure that QEMU python library is available within > > > > > > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > > > > > > + found within python/ directory in QEMU source. > > > > > > > > + After setting up the virtual environment, it runs the python test script > > > > > > > > + from within that environment. > > > > > > > > + If you want to enable verbose mode only for bits test and run make check, > > > > > > > > + one trick is to add V=1 before the call to execute the python script in > > > > > > > > + this file. > > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > > > > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > > > > > > + test failures. This is the script one would be interested in if they wanted > > > > > > > > + to add or change some component of the log parsing, add a new commandline to > > > > > > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > > > > > > + this script unless they wanted to enhance or change the log parsing for > > > > > > > > + their tests. > > > > > > > > + - requirements.txt: This text file contains the dependent python libraries > > > > > > > > + needed for the tests to run. If a new dependent library is needed, it would > > > > > > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > > > > > > + it when setting up the python virtual environment using pip. > > > > > > > > + - README: This text file. > > > > > > > > + > > > > > > > > +acpi-bits/bits-config: > > > > > > > > + This location contains biosbits config files that determine how the software > > > > > > > > + runs the tests. > > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > > > > > > + or actions are performed by bits. The description of the config options are > > > > > > > > + provided in the file itself. > > > > > > > > + > > > > > > > > +acpi-bits/prebuilt: > > > > > > > > + This location contains prebuilt biosbits binaries that are used to generate > > > > > > > > + the bits iso file for testing. > > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > > > > > > + the iso file that are used for testing. This binary zip archive also > > > > > > > > + contains the full source of the bits software including the full text of > > > > > > > > + the license agreement listed here: > > > > > > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > > > > > > + The source tarball can be found in this location in the zip file: > > > > > > > > + boot/src/bits-2095.tar.gz > > > > > > > > + The additional changes beyond those that are present in the official > > > > > > > > + biosbits github repository can be found here: > > > > > > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > > > > > > + > > > > > > > > + Basically these changes can be categorized into the following: > > > > > > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > > > > > > + (gcc 9.4). > > > > > > > > + (b) upgrade of acpica to the latest version (march 2022). > > > > > > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > > > > > > + debugcon can be used to collect the logs. > > > > > > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > > > > > > + executed. > > > > > > > > + > > > > > > > > + This zip file is automatically generated by the bits build system. It can > > > > > > > > + be found in the bits build directory and it is suffixed by the bits build > > > > > > > > + number. > > > > > > > > + Normally, there is no need to make any alteration to this zip archive > > > > > > > > + unless one wanted to change the bits software itself (for example to add > > > > > > > > + a new config option or change actions of existing options or change the > > > > > > > > + debug IO port etc). When such changes are made and a new biosbits software > > > > > > > > + is needed to be generated, please refresh this zip archive as well as the > > > > > > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > > > > > > + that are incompatible with existing behavior. So care must be taken to make > > > > > > > > + sure that the change is compatible with existing bits software as much as > > > > > > > > + possible. When a new config option is introduced for example, bits must > > > > > > > > + be upadated here first before introducing the new config option through > > > > > > > > + a later commit. > > > > > > > > > > > > > > I don't think playing with source tarballs is a reasonable work environment. > > > > > > > > > > > > I agree. However, we do not do much with the source tarball. It is > > > > > > there as part of bits to satisfy the license requirement. If we need > > > > > > to manipulate any test script that is in the source file, we would > > > > > > copy it over and keep it in the bits-test directory and change it > > > > > > there. > > > > > > > > > > > > > Let's use submodules just like e.g. firmware does? > > > > > > > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > > > > > > out a place to store the bits source if we think my github is not the > > > > > > best place. However, we need the source only if/when we need to > > > > > > rebuild bits. I suspect it won't be too often if at all. > > > > > > > > > > Why not on git.qemu.org ? > > > > > > > > Sounds good to me. Do I get push access? :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > > > > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > > > > > > + artifacts are required in order to generate a bootable bits iso file that > > > > > > > > + can run tests. > > > > > > > > + In order to generate this tar archive, please put the following two > > > > > > > > + directories that can be found in the bits build directory in a single > > > > > > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > > > > > > + bits that generated the archive: > > > > > > > > + > > > > > > > > + grub-inst-x86_64-efi > > > > > > > > + grub-inst > > > > > > > > + > > > > > > > > + This version should be the same as the version number of bits that generated > > > > > > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > > > > > > + come from the same build of biosbits of the same version. Things may still > > > > > > > > + work if they come from different versions but mixing version numbers are > > > > > > > > + not recommended. > > > > > > > > + There is normally no need to regenerate this archive unless some fixes or > > > > > > > > + changes have gone into the grub that are part of biosbits. > > > > > > > > + > > > > > > > > +acpi-bits/bits-tests: > > > > > > > > + This directory contains biosbits python based tests that are run from within > > > > > > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > > > > > > + be made in the appropriate test file. For example, new acpi tests can go > > > > > > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > > > > > > + test so that it gets executed as a part of the ACPI tests. > > > > > > > > + It might be occassionally necessary to disable some subtests or add a new > > > > > > > > + test that belongs to a test suite not already present in this directory. To > > > > > > > > + do this, please extract the bits source from the zip file mentioned above. > > > > > > > > + Copy the test suite/script that needs modification (addition of new tests > > > > > > > > + or disabling them) from boot/python location of the extracted directory > > > > > > > > + into this directory. > > > > > > > > + > > > > > > > > + step (a): copy unmodified test script to this directory. > > > > > > > > + step (b): update meson.build and add this file to the list. > > > > > > > > + Commit (a) and (b) together in the same commit. > > > > > > > > + > > > > > > > > + step (c): perform modifications to the test. > > > > > > > > + Commit (c) in a separate commit. > > > > > > > > + > > > > > > > > + The test framework will then use your modified test script to run the test. > > > > > > > > + No further changes would be needed. Please check the logs to make sure that > > > > > > > > + appropriate changes have taken effect. > > > > > > > > + > > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > > > > > > +has been built only for this architecture. > > > > > > > > + > > > > > > > > + > > > > > > > > +Author: Ani Sinha <ani@anisinha.ca> > > > > > > > > + > > > > > > > > +References: > > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > > > > > > -- > > > > > > > > 2.25.1 > > > > > > > > > > > > > > > >
On 28/06/2022 08.57, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote: >>> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>>> >>>> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote: >>>>> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>>>>> >>>>>> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: >>>>>>> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>>>>>>> >>>>>>>> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: >>>>>>>>> The README file is added describing the directory structure and the purpose >>>>>>>>> of every file it contains. It also describes how to add new tests, make changes >>>>>>>>> to existing tests or bits config files or regenerate the bits software. >>>>>>>>> >>>>>>>>> Signed-off-by: Ani Sinha <ani@anisinha.ca> >>>>>>>>> --- >>>>>>>>> tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ >>>>>>>>> 1 file changed, 168 insertions(+) >>>>>>>>> create mode 100644 tests/qtest/acpi-bits/README >>>>>>>>> >>>>>>>>> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README >>>>>>>>> new file mode 100644 >>>>>>>>> index 0000000000..97b15f1665 >>>>>>>>> --- /dev/null >>>>>>>>> +++ b/tests/qtest/acpi-bits/README >>>>>>>>> @@ -0,0 +1,168 @@ >>>>>>>>> +============================================================================= >>>>>>>>> +ACPI/SMBIOS QTESTS USING BIOSBITS >>>>>>>>> +============================================================================= >>>>>>>>> + >>>>>>>>> +Biosbits is a software written by Josh Triplett that can be downloaded by >>>>>>>>> +visiting https://biosbits.org/. The github codebase can be found here: >>>>>>>>> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes >>>>>>>>> +the bios components such as acpi and smbios tables directly through acpica >>>>>>>>> +bios interpreter (a freely available C based library written by Intel, >>>>>>>>> +downloadable from https://acpica.org/ and is included with biosbits) without an >>>>>>>>> +operating system getting involved in between. >>>>>>>>> +There are several advantages to directly testing the bios in a real physical >>>>>>>>> +machine or VM as opposed to indirectly discovering bios issues through the >>>>>>>>> +operating system. For one thing, the OSes tend to hide bios problems from the >>>>>>>>> +end user. The other is that we have more control of what we wanted to test >>>>>>>>> +and how by directly using acpica interpreter on top of the bios on a running >>>>>>>>> +system. More details on the inspiration for developing biosbits and its real >>>>>>>>> +life uses can be found in (a) and (b). >>>>>>>>> +This directory contains QEMU qtests written in python that exercizes the QEMU >>>>>>>>> +bios components using biosbits and reports test failures. >>>>>>>>> + >>>>>>>>> +These tests use python virtual environment. In debian/ubuntu system, the tests >>>>>>>>> +would require python3.8-venv and python3-pip packages to be installed. >>>>>>>> >>>>>>>> Why do we mess with venv and pip? Certainly possible but >>>>>>>> what's wrong with using distro provided packages? >>>>>>> >>>>>>> There are two things: >>>>>>> (a) We are already using pip and venv for our avocado based >>>>>>> integration tests. Look for TESTS_VENV_DIR in Makefile.include under >>>>>>> tests. >>>>>>> (b) the venv is primarily needed because I wanted to take advantage of >>>>>>> our rich python library that handles QEMU based machines. There are >>>>>>> python qtest libraries as well. These are well tested and used with >>>>>>> integration tests and I wanted to keep the test part of the code >>>>>>> simple by simply reusing them. however, in order to use them, we need >>>>>>> a venv environment within which these qemu python libraries are >>>>>>> installed. Integration tests does the same thing. >>>>>>> >>>>>>> A note about my language of choice - python. I gave a lot of thoughts >>>>>>> on this. We do not do a lot of stuff here. All we do is: >>>>>>> (a) generate bits iso. >>>>>>> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. >>>>>>> (c) collect and analyze logs. >>>>>>> >>>>>>> We are not inspecting guest memory or manipulating devices or pci >>>>>>> buses. We do not need the power of C here. We need something that is >>>>>>> simple to write, easy to maintain and understand and can deal with >>>>>>> things like manipulating text files and configs easily. Python seems a >>>>>>> better fit for the role. >>>>>> >>>>>> No problem with that. So that's venv. But do we need pip and pulling >>>>>> packages from the net during testing? >>>>> >>>>> We do that too. See requirements.txt in tests/ >>>>> Following two are downloaded: >>>>> avocado-framework==88.1 >>>>> pycdlib==1.11.0 >>>>> >>>>> Also see this line in Makefie.include: >>>>> >>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) >>>> >>>> Right but that's avocado since it pulls lots of stuff from >>>> the net anyway. >>>> Are the libraries in question not packaged on major distros? >>> >>> Currently I only need this: >>> https://github.com/python-tap/tappy >>> which is the basic TAP processing library for python. >>> >>> It seems its only installed through pip: >>> https://tappy.readthedocs.io/en/latest/ >>> >>> I do not think this is packaged by default. It's such a basic library >>> for parsing test output that maybe we can keep this somewhere within >>> the python src tree? Not sure ... >> >> It's pretty small for sure. Another submodule? > > Unlike BITS, this one is likely going to be maintained for a while and > will receive new releases through > https://pypi.org/project/tap.py/ > so forking is OK but someone has to keep this updated. > > I am open to anything. Whatever feels right is fine to me. John Snow is currently working on the "Pythonification" of various QEMU bits, I think you should loop him into this discussion, too. Thomas
+danPb who pointed me to the TAP stuff. Maybe he has some good ideas on this. On Tue, Jun 28, 2022 at 12:27 PM Ani Sinha <ani@anisinha.ca> wrote: > > On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote: > > > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote: > > > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote: > > > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote: > > > > > > > > > The README file is added describing the directory structure and the purpose > > > > > > > > > of every file it contains. It also describes how to add new tests, make changes > > > > > > > > > to existing tests or bits config files or regenerate the bits software. > > > > > > > > > > > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > > > > > > --- > > > > > > > > > tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ > > > > > > > > > 1 file changed, 168 insertions(+) > > > > > > > > > create mode 100644 tests/qtest/acpi-bits/README > > > > > > > > > > > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README > > > > > > > > > new file mode 100644 > > > > > > > > > index 0000000000..97b15f1665 > > > > > > > > > --- /dev/null > > > > > > > > > +++ b/tests/qtest/acpi-bits/README > > > > > > > > > @@ -0,0 +1,168 @@ > > > > > > > > > +============================================================================= > > > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS > > > > > > > > > +============================================================================= > > > > > > > > > + > > > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by > > > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here: > > > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes > > > > > > > > > +the bios components such as acpi and smbios tables directly through acpica > > > > > > > > > +bios interpreter (a freely available C based library written by Intel, > > > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > > > > > > > > +operating system getting involved in between. > > > > > > > > > +There are several advantages to directly testing the bios in a real physical > > > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the > > > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the > > > > > > > > > +end user. The other is that we have more control of what we wanted to test > > > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running > > > > > > > > > +system. More details on the inspiration for developing biosbits and its real > > > > > > > > > +life uses can be found in (a) and (b). > > > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU > > > > > > > > > +bios components using biosbits and reports test failures. > > > > > > > > > + > > > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests > > > > > > > > > +would require python3.8-venv and python3-pip packages to be installed. > > > > > > > > > > > > > > > > Why do we mess with venv and pip? Certainly possible but > > > > > > > > what's wrong with using distro provided packages? > > > > > > > > > > > > > > There are two things: > > > > > > > (a) We are already using pip and venv for our avocado based > > > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under > > > > > > > tests. > > > > > > > (b) the venv is primarily needed because I wanted to take advantage of > > > > > > > our rich python library that handles QEMU based machines. There are > > > > > > > python qtest libraries as well. These are well tested and used with > > > > > > > integration tests and I wanted to keep the test part of the code > > > > > > > simple by simply reusing them. however, in order to use them, we need > > > > > > > a venv environment within which these qemu python libraries are > > > > > > > installed. Integration tests does the same thing. > > > > > > > > > > > > > > A note about my language of choice - python. I gave a lot of thoughts > > > > > > > on this. We do not do a lot of stuff here. All we do is: > > > > > > > (a) generate bits iso. > > > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm. > > > > > > > (c) collect and analyze logs. > > > > > > > > > > > > > > We are not inspecting guest memory or manipulating devices or pci > > > > > > > buses. We do not need the power of C here. We need something that is > > > > > > > simple to write, easy to maintain and understand and can deal with > > > > > > > things like manipulating text files and configs easily. Python seems a > > > > > > > better fit for the role. > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > packages from the net during testing? > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > Following two are downloaded: > > > > > avocado-framework==88.1 > > > > > pycdlib==1.11.0 > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > the net anyway. > > > > Are the libraries in question not packaged on major distros? > > > > > > Currently I only need this: > > > https://github.com/python-tap/tappy > > > which is the basic TAP processing library for python. > > > > > > It seems its only installed through pip: > > > https://tappy.readthedocs.io/en/latest/ > > > > > > I do not think this is packaged by default. It's such a basic library > > > for parsing test output that maybe we can keep this somewhere within > > > the python src tree? Not sure ... > > > > It's pretty small for sure. Another submodule? > > Unlike BITS, this one is likely going to be maintained for a while and > will receive new releases through > https://pypi.org/project/tap.py/ > so forking is OK but someone has to keep this updated. > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > + > > > > > > > > > +A brief description of the contents of this directory follows: > > > > > > > > > + > > > > > > > > > +├── acpi-bits-test.py > > > > > > > > > +├── acpi-bits-test-venv.sh > > > > > > > > > +├── bits-config > > > > > > > > > +│ ├── bits-cfg.txt > > > > > > > > > +│ └── meson.build > > > > > > > > > +├── bits-tests > > > > > > > > > +│ ├── meson.build > > > > > > > > > +│ ├── smbios.py > > > > > > > > > +│ ├── smilatency.py > > > > > > > > > +│ ├── testacpi.py > > > > > > > > > +│ └── testcpuid.py > > > > > > > > > +├── meson.build > > > > > > > > > +├── prebuilt > > > > > > > > > +│ ├── bits-2095-grub.tar.gz > > > > > > > > > +│ ├── bits-2095.zip > > > > > > > > > +│ └── meson.build > > > > > > > > > +├── README > > > > > > > > > +└── requirements.txt > > > > > > > > > + > > > > > > > > > +acpi-bits: > > > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual > > > > > > > > > + environment necessary for the actual python test script to run. Amongst > > > > > > > > > + other things, it makes sure that QEMU python library is available within > > > > > > > > > + that venv so that QEMU machines can be forked. QEMU python library can be > > > > > > > > > + found within python/ directory in QEMU source. > > > > > > > > > + After setting up the virtual environment, it runs the python test script > > > > > > > > > + from within that environment. > > > > > > > > > + If you want to enable verbose mode only for bits test and run make check, > > > > > > > > > + one trick is to add V=1 before the call to execute the python script in > > > > > > > > > + this file. > > > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a > > > > > > > > > + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports > > > > > > > > > + test failures. This is the script one would be interested in if they wanted > > > > > > > > > + to add or change some component of the log parsing, add a new commandline to > > > > > > > > > + how QEMU is spawned etc. Test writers typically would not need to modify > > > > > > > > > + this script unless they wanted to enhance or change the log parsing for > > > > > > > > > + their tests. > > > > > > > > > + - requirements.txt: This text file contains the dependent python libraries > > > > > > > > > + needed for the tests to run. If a new dependent library is needed, it would > > > > > > > > > + be added here as a new entry and then acpi-bits-test-venv.sh would install > > > > > > > > > + it when setting up the python virtual environment using pip. > > > > > > > > > + - README: This text file. > > > > > > > > > + > > > > > > > > > +acpi-bits/bits-config: > > > > > > > > > + This location contains biosbits config files that determine how the software > > > > > > > > > + runs the tests. > > > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests > > > > > > > > > + or actions are performed by bits. The description of the config options are > > > > > > > > > + provided in the file itself. > > > > > > > > > + > > > > > > > > > +acpi-bits/prebuilt: > > > > > > > > > + This location contains prebuilt biosbits binaries that are used to generate > > > > > > > > > + the bits iso file for testing. > > > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of > > > > > > > > > + the iso file that are used for testing. This binary zip archive also > > > > > > > > > + contains the full source of the bits software including the full text of > > > > > > > > > + the license agreement listed here: > > > > > > > > > + https://github.com/biosbits/bits/blob/master/COPYING > > > > > > > > > + The source tarball can be found in this location in the zip file: > > > > > > > > > + boot/src/bits-2095.tar.gz > > > > > > > > > + The additional changes beyond those that are present in the official > > > > > > > > > + biosbits github repository can be found here: > > > > > > > > > + https://github.com/ani-sinha/bits/tree/bits-qemu-logging > > > > > > > > > + > > > > > > > > > + Basically these changes can be categorized into the following: > > > > > > > > > + (a) changes to make sure biosbits builds with the latest gcc compiler > > > > > > > > > + (gcc 9.4). > > > > > > > > > + (b) upgrade of acpica to the latest version (march 2022). > > > > > > > > > + (c) send bits logs to the debug IO port at addresss 0x403 so that isa > > > > > > > > > + debugcon can be used to collect the logs. > > > > > > > > > + (d) send a gub halt command to shutdown the VM once all the tests have been > > > > > > > > > + executed. > > > > > > > > > + > > > > > > > > > + This zip file is automatically generated by the bits build system. It can > > > > > > > > > + be found in the bits build directory and it is suffixed by the bits build > > > > > > > > > + number. > > > > > > > > > + Normally, there is no need to make any alteration to this zip archive > > > > > > > > > + unless one wanted to change the bits software itself (for example to add > > > > > > > > > + a new config option or change actions of existing options or change the > > > > > > > > > + debug IO port etc). When such changes are made and a new biosbits software > > > > > > > > > + is needed to be generated, please refresh this zip archive as well as the > > > > > > > > > + grub tarball at the same time. Tests will break if changes are made to bits > > > > > > > > > + that are incompatible with existing behavior. So care must be taken to make > > > > > > > > > + sure that the change is compatible with existing bits software as much as > > > > > > > > > + possible. When a new config option is introduced for example, bits must > > > > > > > > > + be upadated here first before introducing the new config option through > > > > > > > > > + a later commit. > > > > > > > > > > > > > > > > I don't think playing with source tarballs is a reasonable work environment. > > > > > > > > > > > > > > I agree. However, we do not do much with the source tarball. It is > > > > > > > there as part of bits to satisfy the license requirement. If we need > > > > > > > to manipulate any test script that is in the source file, we would > > > > > > > copy it over and keep it in the bits-test directory and change it > > > > > > > there. > > > > > > > > > > > > > > > Let's use submodules just like e.g. firmware does? > > > > > > > > > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure > > > > > > > out a place to store the bits source if we think my github is not the > > > > > > > best place. However, we need the source only if/when we need to > > > > > > > rebuild bits. I suspect it won't be too often if at all. > > > > > > > > > > > > Why not on git.qemu.org ? > > > > > > > > > > Sounds good to me. Do I get push access? :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub > > > > > > > > > + scripts and modules to the prebuilt directory. These prebuilt grub > > > > > > > > > + artifacts are required in order to generate a bootable bits iso file that > > > > > > > > > + can run tests. > > > > > > > > > + In order to generate this tar archive, please put the following two > > > > > > > > > + directories that can be found in the bits build directory in a single > > > > > > > > > + tar archive file named as bits-<n>-grub.tar.gz where n is the version of > > > > > > > > > + bits that generated the archive: > > > > > > > > > + > > > > > > > > > + grub-inst-x86_64-efi > > > > > > > > > + grub-inst > > > > > > > > > + > > > > > > > > > + This version should be the same as the version number of bits that generated > > > > > > > > > + bits-<n>.zip file. In other words, the two files must be in sync and should > > > > > > > > > + come from the same build of biosbits of the same version. Things may still > > > > > > > > > + work if they come from different versions but mixing version numbers are > > > > > > > > > + not recommended. > > > > > > > > > + There is normally no need to regenerate this archive unless some fixes or > > > > > > > > > + changes have gone into the grub that are part of biosbits. > > > > > > > > > + > > > > > > > > > +acpi-bits/bits-tests: > > > > > > > > > + This directory contains biosbits python based tests that are run from within > > > > > > > > > + the biosbits environment in the spawned VM. New additions of test cases can > > > > > > > > > + be made in the appropriate test file. For example, new acpi tests can go > > > > > > > > > + into testacpi.py and one would call testsuite.add_test() to register the new > > > > > > > > > + test so that it gets executed as a part of the ACPI tests. > > > > > > > > > + It might be occassionally necessary to disable some subtests or add a new > > > > > > > > > + test that belongs to a test suite not already present in this directory. To > > > > > > > > > + do this, please extract the bits source from the zip file mentioned above. > > > > > > > > > + Copy the test suite/script that needs modification (addition of new tests > > > > > > > > > + or disabling them) from boot/python location of the extracted directory > > > > > > > > > + into this directory. > > > > > > > > > + > > > > > > > > > + step (a): copy unmodified test script to this directory. > > > > > > > > > + step (b): update meson.build and add this file to the list. > > > > > > > > > + Commit (a) and (b) together in the same commit. > > > > > > > > > + > > > > > > > > > + step (c): perform modifications to the test. > > > > > > > > > + Commit (c) in a separate commit. > > > > > > > > > + > > > > > > > > > + The test framework will then use your modified test script to run the test. > > > > > > > > > + No further changes would be needed. Please check the logs to make sure that > > > > > > > > > + appropriate changes have taken effect. > > > > > > > > > + > > > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated > > > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest". > > > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits > > > > > > > > > +has been built only for this architecture. > > > > > > > > > + > > > > > > > > > + > > > > > > > > > +Author: Ani Sinha <ani@anisinha.ca> > > > > > > > > > + > > > > > > > > > +References: > > > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > > > > > > > > -- > > > > > > > > > 2.25.1 > > > > > > > > > > > > > > > > > > > >
On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > packages from the net during testing? > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > Following two are downloaded: > > > > > > avocado-framework==88.1 > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > the net anyway. > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > Currently I only need this: > > > > https://github.com/python-tap/tappy > > > > which is the basic TAP processing library for python. > > > > > > > > It seems its only installed through pip: > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > for parsing test output that maybe we can keep this somewhere within > > > > the python src tree? Not sure ... > > > > > > It's pretty small for sure. Another submodule? > > > > Unlike BITS, this one is likely going to be maintained for a while and > > will receive new releases through > > https://pypi.org/project/tap.py/ > > so forking is OK but someone has to keep this updated. > > > > I am open to anything. Whatever feels right is fine to me. > > John Snow is currently working on the "Pythonification" of various QEMU > bits, I think you should loop him into this discussion, too. > > Thomas submodule does not mean we fork necessarily. We could have all options: check for the module and use it if there, if not use one from system if not there install with pip .. But yea, I'm not sure what's best either.
On 28/06/2022 09.10, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: >>>>>>>> No problem with that. So that's venv. But do we need pip and pulling >>>>>>>> packages from the net during testing? >>>>>>> >>>>>>> We do that too. See requirements.txt in tests/ >>>>>>> Following two are downloaded: >>>>>>> avocado-framework==88.1 >>>>>>> pycdlib==1.11.0 >>>>>>> >>>>>>> Also see this line in Makefie.include: >>>>>>> >>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) >>>>>> >>>>>> Right but that's avocado since it pulls lots of stuff from >>>>>> the net anyway. >>>>>> Are the libraries in question not packaged on major distros? >>>>> >>>>> Currently I only need this: >>>>> https://github.com/python-tap/tappy >>>>> which is the basic TAP processing library for python. >>>>> >>>>> It seems its only installed through pip: >>>>> https://tappy.readthedocs.io/en/latest/ >>>>> >>>>> I do not think this is packaged by default. It's such a basic library >>>>> for parsing test output that maybe we can keep this somewhere within >>>>> the python src tree? Not sure ... >>>> >>>> It's pretty small for sure. Another submodule? >>> >>> Unlike BITS, this one is likely going to be maintained for a while and >>> will receive new releases through >>> https://pypi.org/project/tap.py/ >>> so forking is OK but someone has to keep this updated. >>> >>> I am open to anything. Whatever feels right is fine to me. >> >> John Snow is currently working on the "Pythonification" of various QEMU >> bits, I think you should loop him into this discussion, too. >> >> Thomas > > submodule does not mean we fork necessarily. We could have > all options: check for the module and use it if there, if not > use one from system if not there install with pip .. > But yea, I'm not sure what's best either. submodules create a dependency on an internet connection, too. So before you add yet another submodule (which have a couple of other disadvantages), I think you could also directly use the venv here. Thomas
On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > Following two are downloaded: > > > > > > > > avocado-framework==88.1 > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > the net anyway. > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > Currently I only need this: > > > > > > https://github.com/python-tap/tappy > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > It seems its only installed through pip: > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > the python src tree? Not sure ... > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > will receive new releases through > > > > https://pypi.org/project/tap.py/ > > > > so forking is OK but someone has to keep this updated. > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > bits, I think you should loop him into this discussion, too. > > > > > > Thomas > > > > submodule does not mean we fork necessarily. We could have > > all options: check for the module and use it if there, if not > > use one from system if not there install with pip .. > > But yea, I'm not sure what's best either. > > submodules create a dependency on an internet connection, too. So before you > add yet another submodule (which have a couple of other disadvantages), I > think you could also directly use the venv here. Definitely not submodules. We need to get out of the mindset that submodules are needed for every new dependancy we add. Submodules are only appropriate if the external project is designed to be used as a copylib (eg the keycodemapdb tool), or if we need to bundle in order to prevent a regression for previously deployed QEMU installs where the dependancy is known not to exist on all our supported platforms. This does not apply in this case, because the proposed use of tappy is merely for a test case. Meson just needs to check if tappy exists and if it does, then use it, otherwise skip the tests that need it. The user can arrange to install tappy, as they do with the majority of other deps. If John's venv stuff is relevant, then we don't even need the meson checks, just delegate to the venv setup. Regardless, no submodules are needed or desirable. With regards, Daniel
On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth <thuth@redhat.com> wrote: > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > >>>>>>>> No problem with that. So that's venv. But do we need pip and pulling > >>>>>>>> packages from the net during testing? > >>>>>>> > >>>>>>> We do that too. See requirements.txt in tests/ > >>>>>>> Following two are downloaded: > >>>>>>> avocado-framework==88.1 > >>>>>>> pycdlib==1.11.0 > >>>>>>> > >>>>>>> Also see this line in Makefie.include: > >>>>>>> > >>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > >>>>>> > >>>>>> Right but that's avocado since it pulls lots of stuff from > >>>>>> the net anyway. > >>>>>> Are the libraries in question not packaged on major distros? > >>>>> > >>>>> Currently I only need this: > >>>>> https://github.com/python-tap/tappy > >>>>> which is the basic TAP processing library for python. > >>>>> > >>>>> It seems its only installed through pip: > >>>>> https://tappy.readthedocs.io/en/latest/ > >>>>> > >>>>> I do not think this is packaged by default. It's such a basic library > >>>>> for parsing test output that maybe we can keep this somewhere within > >>>>> the python src tree? Not sure ... > >>>> > >>>> It's pretty small for sure. Another submodule? > >>> > >>> Unlike BITS, this one is likely going to be maintained for a while and > >>> will receive new releases through > >>> https://pypi.org/project/tap.py/ > >>> so forking is OK but someone has to keep this updated. > >>> > >>> I am open to anything. Whatever feels right is fine to me. > >> > >> John Snow is currently working on the "Pythonification" of various QEMU > >> bits, I think you should loop him into this discussion, too. > >> > >> Thomas > > > > submodule does not mean we fork necessarily. We could have > > all options: check for the module and use it if there, if not > > use one from system if not there install with pip .. > > But yea, I'm not sure what's best either. > > submodules create a dependency on an internet connection, too. So before you > add yet another submodule (which have a couple of other disadvantages), I > think you could also directly use the venv here. Not sure what you mean by directly using the venv. > > Thomas >
On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > > Following two are downloaded: > > > > > > > > > avocado-framework==88.1 > > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > > the net anyway. > > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > > > Currently I only need this: > > > > > > > https://github.com/python-tap/tappy > > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > > > It seems its only installed through pip: > > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > > the python src tree? Not sure ... > > > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > > will receive new releases through > > > > > https://pypi.org/project/tap.py/ > > > > > so forking is OK but someone has to keep this updated. > > > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > > bits, I think you should loop him into this discussion, too. > > > > > > > > Thomas > > > > > > submodule does not mean we fork necessarily. We could have > > > all options: check for the module and use it if there, if not > > > use one from system if not there install with pip .. > > > But yea, I'm not sure what's best either. > > > > submodules create a dependency on an internet connection, too. So before you > > add yet another submodule (which have a couple of other disadvantages), I > > think you could also directly use the venv here. > > Definitely not submodules. > > We need to get out of the mindset that submodules are needed for every new > dependancy we add. Submodules are only appropriate if the external project > is designed to be used as a copylib (eg the keycodemapdb tool), or if we > need to bundle in order to prevent a regression for previously deployed > QEMU installs where the dependancy is known not to exist on all our > supported platforms. > > This does not apply in this case, because the proposed use of tappy is > merely for a test case. Meson just needs to check if tappy exists and if > it does, then use it, otherwise skip the tests that need it. The user can > arrange to install tappy, as they do with the majority of other deps. > > If John's venv stuff is relevant, then we don't even need the meson checks, > just delegate to the venv setup. > > Regardless, no submodules are needed or desirable. What about keeping biosbits stuff? Source or pre-built. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On 28/06/2022 09.49, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth <thuth@redhat.com> wrote: >> >> On 28/06/2022 09.10, Michael S. Tsirkin wrote: >>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: >>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling >>>>>>>>>> packages from the net during testing? >>>>>>>>> >>>>>>>>> We do that too. See requirements.txt in tests/ >>>>>>>>> Following two are downloaded: >>>>>>>>> avocado-framework==88.1 >>>>>>>>> pycdlib==1.11.0 >>>>>>>>> >>>>>>>>> Also see this line in Makefie.include: >>>>>>>>> >>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) >>>>>>>> >>>>>>>> Right but that's avocado since it pulls lots of stuff from >>>>>>>> the net anyway. >>>>>>>> Are the libraries in question not packaged on major distros? >>>>>>> >>>>>>> Currently I only need this: >>>>>>> https://github.com/python-tap/tappy >>>>>>> which is the basic TAP processing library for python. >>>>>>> >>>>>>> It seems its only installed through pip: >>>>>>> https://tappy.readthedocs.io/en/latest/ >>>>>>> >>>>>>> I do not think this is packaged by default. It's such a basic library >>>>>>> for parsing test output that maybe we can keep this somewhere within >>>>>>> the python src tree? Not sure ... >>>>>> >>>>>> It's pretty small for sure. Another submodule? >>>>> >>>>> Unlike BITS, this one is likely going to be maintained for a while and >>>>> will receive new releases through >>>>> https://pypi.org/project/tap.py/ >>>>> so forking is OK but someone has to keep this updated. >>>>> >>>>> I am open to anything. Whatever feels right is fine to me. >>>> >>>> John Snow is currently working on the "Pythonification" of various QEMU >>>> bits, I think you should loop him into this discussion, too. >>>> >>>> Thomas >>> >>> submodule does not mean we fork necessarily. We could have >>> all options: check for the module and use it if there, if not >>> use one from system if not there install with pip .. >>> But yea, I'm not sure what's best either. >> >> submodules create a dependency on an internet connection, too. So before you >> add yet another submodule (which have a couple of other disadvantages), I >> think you could also directly use the venv here. > > Not sure what you mean by directly using the venv. I meant installing the dependency via pip in a venv, as you are currently doing it via the acpi-bits-test-venv.sh script. Thomas
On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > > > Following two are downloaded: > > > > > > > > > > avocado-framework==88.1 > > > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > > > the net anyway. > > > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > > > > > Currently I only need this: > > > > > > > > https://github.com/python-tap/tappy > > > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > > > > > It seems its only installed through pip: > > > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > > > the python src tree? Not sure ... > > > > > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > > > will receive new releases through > > > > > > https://pypi.org/project/tap.py/ > > > > > > so forking is OK but someone has to keep this updated. > > > > > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > > > bits, I think you should loop him into this discussion, too. > > > > > > > > > > Thomas > > > > > > > > submodule does not mean we fork necessarily. We could have > > > > all options: check for the module and use it if there, if not > > > > use one from system if not there install with pip .. > > > > But yea, I'm not sure what's best either. > > > > > > submodules create a dependency on an internet connection, too. So before you > > > add yet another submodule (which have a couple of other disadvantages), I > > > think you could also directly use the venv here. > > > > Definitely not submodules. > > > > We need to get out of the mindset that submodules are needed for every new > > dependancy we add. Submodules are only appropriate if the external project > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we > > need to bundle in order to prevent a regression for previously deployed > > QEMU installs where the dependancy is known not to exist on all our > > supported platforms. > > > > This does not apply in this case, because the proposed use of tappy is > > merely for a test case. Meson just needs to check if tappy exists and if > > it does, then use it, otherwise skip the tests that need it. The user can > > arrange to install tappy, as they do with the majority of other deps. > > > > If John's venv stuff is relevant, then we don't even need the meson checks, > > just delegate to the venv setup. > > > > Regardless, no submodules are needed or desirable. > > What about keeping biosbits stuff? Source or pre-built. Shipping them as pre-built binaries in QEMU is not a viable option IMHO, especially for grub as a GPL'd project we need to be extremely clear about the exact corresponding source and build process for any binary. For this kind of thing I would generally expect the distro to provide packages that we consume. Looking at biosbits I see it is itself bundling a bunch more 3rd party projects, libffi, grub2, and including even an ancient version of python as a submodule. So bundling a pre-built biosbits in QEMU appears to mean that we're in turn going to unexpectedly bundle a bunch of other 3rd party projects too, all with dubious license compliance. I don't think this looks like something we should have in qemu.git or qemu tarballs. It will also make it challenging for the distro to take biosbits at all, unless those 3rd party bundles can be eliminated in favour of using existing builds their have packaged for grub, python, libffi, etc. With regards, Daniel
On 28/06/2022 10.23, Daniel P. Berrangé wrote: > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: >>> >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote: >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling >>>>>>>>>>>> packages from the net during testing? >>>>>>>>>>> >>>>>>>>>>> We do that too. See requirements.txt in tests/ >>>>>>>>>>> Following two are downloaded: >>>>>>>>>>> avocado-framework==88.1 >>>>>>>>>>> pycdlib==1.11.0 >>>>>>>>>>> >>>>>>>>>>> Also see this line in Makefie.include: >>>>>>>>>>> >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) >>>>>>>>>> >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from >>>>>>>>>> the net anyway. >>>>>>>>>> Are the libraries in question not packaged on major distros? >>>>>>>>> >>>>>>>>> Currently I only need this: >>>>>>>>> https://github.com/python-tap/tappy >>>>>>>>> which is the basic TAP processing library for python. >>>>>>>>> >>>>>>>>> It seems its only installed through pip: >>>>>>>>> https://tappy.readthedocs.io/en/latest/ >>>>>>>>> >>>>>>>>> I do not think this is packaged by default. It's such a basic library >>>>>>>>> for parsing test output that maybe we can keep this somewhere within >>>>>>>>> the python src tree? Not sure ... >>>>>>>> >>>>>>>> It's pretty small for sure. Another submodule? >>>>>>> >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and >>>>>>> will receive new releases through >>>>>>> https://pypi.org/project/tap.py/ >>>>>>> so forking is OK but someone has to keep this updated. >>>>>>> >>>>>>> I am open to anything. Whatever feels right is fine to me. >>>>>> >>>>>> John Snow is currently working on the "Pythonification" of various QEMU >>>>>> bits, I think you should loop him into this discussion, too. >>>>>> >>>>>> Thomas >>>>> >>>>> submodule does not mean we fork necessarily. We could have >>>>> all options: check for the module and use it if there, if not >>>>> use one from system if not there install with pip .. >>>>> But yea, I'm not sure what's best either. >>>> >>>> submodules create a dependency on an internet connection, too. So before you >>>> add yet another submodule (which have a couple of other disadvantages), I >>>> think you could also directly use the venv here. >>> >>> Definitely not submodules. >>> >>> We need to get out of the mindset that submodules are needed for every new >>> dependancy we add. Submodules are only appropriate if the external project >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we >>> need to bundle in order to prevent a regression for previously deployed >>> QEMU installs where the dependancy is known not to exist on all our >>> supported platforms. >>> >>> This does not apply in this case, because the proposed use of tappy is >>> merely for a test case. Meson just needs to check if tappy exists and if >>> it does, then use it, otherwise skip the tests that need it. The user can >>> arrange to install tappy, as they do with the majority of other deps. >>> >>> If John's venv stuff is relevant, then we don't even need the meson checks, >>> just delegate to the venv setup. >>> >>> Regardless, no submodules are needed or desirable. >> >> What about keeping biosbits stuff? Source or pre-built. > > Shipping them as pre-built binaries in QEMU is not a viable option > IMHO, especially for grub as a GPL'd project we need to be extremely > clear about the exact corresponding source and build process for any > binary. > > For this kind of thing I would generally expect the distro to provide > packages that we consume. Looking at biosbits I see it is itself > bundling a bunch more 3rd party projects, libffi, grub2, and including > even an ancient version of python as a submodule. > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > turn going to unexpectedly bundle a bunch of other 3rd party projects > too, all with dubious license compliance. I don't think this looks like > something we should have in qemu.git or qemu tarballs. It will also > make it challenging for the distro to take biosbits at all, unless > those 3rd party bundles can be eliminated in favour of using existing > builds their have packaged for grub, python, libffi, etc. So if this depends on some third party binary bits, I think this is pretty similar to the tests in the avocado directory ... there we download third party binaries, too... Wouldn't it make sense to adapt your tests to that framework? Thomas
On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote: > > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > >>> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote: > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling > >>>>>>>>>>>> packages from the net during testing? > >>>>>>>>>>> > >>>>>>>>>>> We do that too. See requirements.txt in tests/ > >>>>>>>>>>> Following two are downloaded: > >>>>>>>>>>> avocado-framework==88.1 > >>>>>>>>>>> pycdlib==1.11.0 > >>>>>>>>>>> > >>>>>>>>>>> Also see this line in Makefie.include: > >>>>>>>>>>> > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > >>>>>>>>>> > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from > >>>>>>>>>> the net anyway. > >>>>>>>>>> Are the libraries in question not packaged on major distros? > >>>>>>>>> > >>>>>>>>> Currently I only need this: > >>>>>>>>> https://github.com/python-tap/tappy > >>>>>>>>> which is the basic TAP processing library for python. > >>>>>>>>> > >>>>>>>>> It seems its only installed through pip: > >>>>>>>>> https://tappy.readthedocs.io/en/latest/ > >>>>>>>>> > >>>>>>>>> I do not think this is packaged by default. It's such a basic library > >>>>>>>>> for parsing test output that maybe we can keep this somewhere within > >>>>>>>>> the python src tree? Not sure ... > >>>>>>>> > >>>>>>>> It's pretty small for sure. Another submodule? > >>>>>>> > >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and > >>>>>>> will receive new releases through > >>>>>>> https://pypi.org/project/tap.py/ > >>>>>>> so forking is OK but someone has to keep this updated. > >>>>>>> > >>>>>>> I am open to anything. Whatever feels right is fine to me. > >>>>>> > >>>>>> John Snow is currently working on the "Pythonification" of various QEMU > >>>>>> bits, I think you should loop him into this discussion, too. > >>>>>> > >>>>>> Thomas > >>>>> > >>>>> submodule does not mean we fork necessarily. We could have > >>>>> all options: check for the module and use it if there, if not > >>>>> use one from system if not there install with pip .. > >>>>> But yea, I'm not sure what's best either. > >>>> > >>>> submodules create a dependency on an internet connection, too. So before you > >>>> add yet another submodule (which have a couple of other disadvantages), I > >>>> think you could also directly use the venv here. > >>> > >>> Definitely not submodules. > >>> > >>> We need to get out of the mindset that submodules are needed for every new > >>> dependancy we add. Submodules are only appropriate if the external project > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we > >>> need to bundle in order to prevent a regression for previously deployed > >>> QEMU installs where the dependancy is known not to exist on all our > >>> supported platforms. > >>> > >>> This does not apply in this case, because the proposed use of tappy is > >>> merely for a test case. Meson just needs to check if tappy exists and if > >>> it does, then use it, otherwise skip the tests that need it. The user can > >>> arrange to install tappy, as they do with the majority of other deps. > >>> > >>> If John's venv stuff is relevant, then we don't even need the meson checks, > >>> just delegate to the venv setup. > >>> > >>> Regardless, no submodules are needed or desirable. > >> > >> What about keeping biosbits stuff? Source or pre-built. > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > IMHO, especially for grub as a GPL'd project we need to be extremely > > clear about the exact corresponding source and build process for any > > binary. > > > > For this kind of thing I would generally expect the distro to provide > > packages that we consume. Looking at biosbits I see it is itself > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > even an ancient version of python as a submodule. > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > too, all with dubious license compliance. I don't think this looks like > > something we should have in qemu.git or qemu tarballs. It will also > > make it challenging for the distro to take biosbits at all, unless > > those 3rd party bundles can be eliminated in favour of using existing > > builds their have packaged for grub, python, libffi, etc. > > So if this depends on some third party binary bits, I think this is pretty > similar to the tests in the avocado directory ... there we download third > party binaries, too... Wouldn't it make sense to adapt your tests to that > framework? I do not want to bring in the whole avocado framework because it would unnecessarily make things complicated. I just need the qemu machine python library and that is enough. For downloading third party stuff, we can simply wget things from somewhere. > > Thomas >
On Tue, Jun 28, 2022 at 14:05 Ani Sinha <ani@anisinha.ca> wrote: > On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote: > > > > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé < > berrange@redhat.com> wrote: > > >>> > > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip > and pulling > > >>>>>>>>>>>> packages from the net during testing? > > >>>>>>>>>>> > > >>>>>>>>>>> We do that too. See requirements.txt in tests/ > > >>>>>>>>>>> Following two are downloaded: > > >>>>>>>>>>> avocado-framework==88.1 > > >>>>>>>>>>> pycdlib==1.11.0 > > >>>>>>>>>>> > > >>>>>>>>>>> Also see this line in Makefie.include: > > >>>>>>>>>>> > > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > >>>>>>>>>> > > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from > > >>>>>>>>>> the net anyway. > > >>>>>>>>>> Are the libraries in question not packaged on major distros? > > >>>>>>>>> > > >>>>>>>>> Currently I only need this: > > >>>>>>>>> https://github.com/python-tap/tappy > > >>>>>>>>> which is the basic TAP processing library for python. > > >>>>>>>>> > > >>>>>>>>> It seems its only installed through pip: > > >>>>>>>>> https://tappy.readthedocs.io/en/latest/ > > >>>>>>>>> > > >>>>>>>>> I do not think this is packaged by default. It's such a basic > library > > >>>>>>>>> for parsing test output that maybe we can keep this somewhere > within > > >>>>>>>>> the python src tree? Not sure ... > > >>>>>>>> > > >>>>>>>> It's pretty small for sure. Another submodule? > > >>>>>>> > > >>>>>>> Unlike BITS, this one is likely going to be maintained for a > while and > > >>>>>>> will receive new releases through > > >>>>>>> https://pypi.org/project/tap.py/ > > >>>>>>> so forking is OK but someone has to keep this updated. > > >>>>>>> > > >>>>>>> I am open to anything. Whatever feels right is fine to me. > > >>>>>> > > >>>>>> John Snow is currently working on the "Pythonification" of > various QEMU > > >>>>>> bits, I think you should loop him into this discussion, too. > > >>>>>> > > >>>>>> Thomas > > >>>>> > > >>>>> submodule does not mean we fork necessarily. We could have > > >>>>> all options: check for the module and use it if there, if not > > >>>>> use one from system if not there install with pip .. > > >>>>> But yea, I'm not sure what's best either. > > >>>> > > >>>> submodules create a dependency on an internet connection, too. So > before you > > >>>> add yet another submodule (which have a couple of other > disadvantages), I > > >>>> think you could also directly use the venv here. > > >>> > > >>> Definitely not submodules. > > >>> > > >>> We need to get out of the mindset that submodules are needed for > every new > > >>> dependancy we add. Submodules are only appropriate if the external > project > > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or > if we > > >>> need to bundle in order to prevent a regression for previously > deployed > > >>> QEMU installs where the dependancy is known not to exist on all our > > >>> supported platforms. > > >>> > > >>> This does not apply in this case, because the proposed use of tappy > is > > >>> merely for a test case. Meson just needs to check if tappy exists > and if > > >>> it does, then use it, otherwise skip the tests that need it. The > user can > > >>> arrange to install tappy, as they do with the majority of other deps. > > >>> > > >>> If John's venv stuff is relevant, then we don't even need the meson > checks, > > >>> just delegate to the venv setup. > > >>> > > >>> Regardless, no submodules are needed or desirable. > > >> > > >> What about keeping biosbits stuff? Source or pre-built. > > > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > > IMHO, especially for grub as a GPL'd project we need to be extremely > > > clear about the exact corresponding source and build process for any > > > binary. > > > > > > For this kind of thing I would generally expect the distro to provide > > > packages that we consume. Looking at biosbits I see it is itself > > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > > even an ancient version of python as a submodule. > > > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > > too, all with dubious license compliance. I don't think this looks like > > > something we should have in qemu.git or qemu tarballs. It will also > > > make it challenging for the distro to take biosbits at all, unless > > > those 3rd party bundles can be eliminated in favour of using existing > > > builds their have packaged for grub, python, libffi, etc. > > > > So if this depends on some third party binary bits, I think this is > pretty > > similar to the tests in the avocado directory ... there we download third > > party binaries, too... Wouldn't it make sense to adapt your tests to that > > framework? > > I do not want to bring in the whole avocado framework because it would > unnecessarily make things complicated. I just need the qemu machine > python library and that is enough. For downloading third party stuff, we can simply wget things from somewhere. https://pypi.org/project/wget/ That get_asset() call is an overkill for downloading two archives. > > > > > Thomas > > >
On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > Following two are downloaded: > > > > > > > > avocado-framework==88.1 > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > the net anyway. > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > Currently I only need this: > > > > > > https://github.com/python-tap/tappy > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > It seems its only installed through pip: > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > the python src tree? Not sure ... > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > will receive new releases through > > > > https://pypi.org/project/tap.py/ > > > > so forking is OK but someone has to keep this updated. > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > bits, I think you should loop him into this discussion, too. > > > > > > Thomas > > > > submodule does not mean we fork necessarily. We could have > > all options: check for the module and use it if there, if not > > use one from system if not there install with pip .. > > But yea, I'm not sure what's best either. > > submodules create a dependency on an internet connection, too. Yes but this dependency is explicit. > So before you > add yet another submodule (which have a couple of other disadvantages), I > think you could also directly use the venv here. > > Thomas
On Tue, Jun 28, 2022 at 08:48:42AM +0100, Daniel P. Berrangé wrote: > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > > Following two are downloaded: > > > > > > > > > avocado-framework==88.1 > > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > > the net anyway. > > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > > > Currently I only need this: > > > > > > > https://github.com/python-tap/tappy > > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > > > It seems its only installed through pip: > > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > > the python src tree? Not sure ... > > > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > > will receive new releases through > > > > > https://pypi.org/project/tap.py/ > > > > > so forking is OK but someone has to keep this updated. > > > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > > bits, I think you should loop him into this discussion, too. > > > > > > > > Thomas > > > > > > submodule does not mean we fork necessarily. We could have > > > all options: check for the module and use it if there, if not > > > use one from system if not there install with pip .. > > > But yea, I'm not sure what's best either. > > > > submodules create a dependency on an internet connection, too. So before you > > add yet another submodule (which have a couple of other disadvantages), I > > think you could also directly use the venv here. > > Definitely not submodules. > > We need to get out of the mindset that submodules are needed for every new > dependancy we add. Submodules are only appropriate if the external project > is designed to be used as a copylib (eg the keycodemapdb tool), or if we > need to bundle in order to prevent a regression for previously deployed > QEMU installs where the dependancy is known not to exist on all our > supported platforms. > > This does not apply in this case, because the proposed use of tappy is > merely for a test case. Meson just needs to check if tappy exists and if > it does, then use it, otherwise skip the tests that need it. The user can > arrange to install tappy, as they do with the majority of other deps. I think that's reasonable, except I think we should make configure do the checks as that is when users see the output of what is available. By the way, it might make sense to have a script setting up developer dependencies for qemu for major platforms. > If John's venv stuff is relevant, then we don't even need the meson checks, > just delegate to the venv setup. IIUC venv does not install things itself. pip does. > Regardless, no submodules are needed or desirable. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jun 28, 2022 at 02:19:41PM +0530, Ani Sinha wrote: > > > On Tue, Jun 28, 2022 at 14:05 Ani Sinha <ani@anisinha.ca> wrote: > > On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote: > > > > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé < > berrange@redhat.com> wrote: > > >>> > > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and > pulling > > >>>>>>>>>>>> packages from the net during testing? > > >>>>>>>>>>> > > >>>>>>>>>>> We do that too. See requirements.txt in tests/ > > >>>>>>>>>>> Following two are downloaded: > > >>>>>>>>>>> avocado-framework==88.1 > > >>>>>>>>>>> pycdlib==1.11.0 > > >>>>>>>>>>> > > >>>>>>>>>>> Also see this line in Makefie.include: > > >>>>>>>>>>> > > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > >>>>>>>>>> > > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from > > >>>>>>>>>> the net anyway. > > >>>>>>>>>> Are the libraries in question not packaged on major distros? > > >>>>>>>>> > > >>>>>>>>> Currently I only need this: > > >>>>>>>>> https://github.com/python-tap/tappy > > >>>>>>>>> which is the basic TAP processing library for python. > > >>>>>>>>> > > >>>>>>>>> It seems its only installed through pip: > > >>>>>>>>> https://tappy.readthedocs.io/en/latest/ > > >>>>>>>>> > > >>>>>>>>> I do not think this is packaged by default. It's such a basic > library > > >>>>>>>>> for parsing test output that maybe we can keep this somewhere > within > > >>>>>>>>> the python src tree? Not sure ... > > >>>>>>>> > > >>>>>>>> It's pretty small for sure. Another submodule? > > >>>>>>> > > >>>>>>> Unlike BITS, this one is likely going to be maintained for a > while and > > >>>>>>> will receive new releases through > > >>>>>>> https://pypi.org/project/tap.py/ > > >>>>>>> so forking is OK but someone has to keep this updated. > > >>>>>>> > > >>>>>>> I am open to anything. Whatever feels right is fine to me. > > >>>>>> > > >>>>>> John Snow is currently working on the "Pythonification" of various > QEMU > > >>>>>> bits, I think you should loop him into this discussion, too. > > >>>>>> > > >>>>>> Thomas > > >>>>> > > >>>>> submodule does not mean we fork necessarily. We could have > > >>>>> all options: check for the module and use it if there, if not > > >>>>> use one from system if not there install with pip .. > > >>>>> But yea, I'm not sure what's best either. > > >>>> > > >>>> submodules create a dependency on an internet connection, too. So > before you > > >>>> add yet another submodule (which have a couple of other > disadvantages), I > > >>>> think you could also directly use the venv here. > > >>> > > >>> Definitely not submodules. > > >>> > > >>> We need to get out of the mindset that submodules are needed for > every new > > >>> dependancy we add. Submodules are only appropriate if the external > project > > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if > we > > >>> need to bundle in order to prevent a regression for previously > deployed > > >>> QEMU installs where the dependancy is known not to exist on all our > > >>> supported platforms. > > >>> > > >>> This does not apply in this case, because the proposed use of tappy > is > > >>> merely for a test case. Meson just needs to check if tappy exists and > if > > >>> it does, then use it, otherwise skip the tests that need it. The user > can > > >>> arrange to install tappy, as they do with the majority of other deps. > > >>> > > >>> If John's venv stuff is relevant, then we don't even need the meson > checks, > > >>> just delegate to the venv setup. > > >>> > > >>> Regardless, no submodules are needed or desirable. > > >> > > >> What about keeping biosbits stuff? Source or pre-built. > > > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > > IMHO, especially for grub as a GPL'd project we need to be extremely > > > clear about the exact corresponding source and build process for any > > > binary. > > > > > > For this kind of thing I would generally expect the distro to provide > > > packages that we consume. Looking at biosbits I see it is itself > > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > > even an ancient version of python as a submodule. > > > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > > too, all with dubious license compliance. I don't think this looks like > > > something we should have in qemu.git or qemu tarballs. It will also > > > make it challenging for the distro to take biosbits at all, unless > > > those 3rd party bundles can be eliminated in favour of using existing > > > builds their have packaged for grub, python, libffi, etc. > > > > So if this depends on some third party binary bits, I think this is > pretty > > similar to the tests in the avocado directory ... there we download third > > party binaries, too... Wouldn't it make sense to adapt your tests to that > > framework? > > I do not want to bring in the whole avocado framework because it would > unnecessarily make things complicated. I just need the qemu machine > python library and that is enough. For downloading third party stuff, > > we can simply wget things from somewhere. > > > https://pypi.org/project/wget/ > > That get_asset() call is an overkill for downloading two archives. > For biosbits if we are going this route then I feel a submodule is much better. It records which version exactly each qemu version wants. > > > > > > Thomas > > >
On Tue, Jun 28, 2022 at 02:05:19PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote: > > > > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > >>> > > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling > > >>>>>>>>>>>> packages from the net during testing? > > >>>>>>>>>>> > > >>>>>>>>>>> We do that too. See requirements.txt in tests/ > > >>>>>>>>>>> Following two are downloaded: > > >>>>>>>>>>> avocado-framework==88.1 > > >>>>>>>>>>> pycdlib==1.11.0 > > >>>>>>>>>>> > > >>>>>>>>>>> Also see this line in Makefie.include: > > >>>>>>>>>>> > > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > >>>>>>>>>> > > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from > > >>>>>>>>>> the net anyway. > > >>>>>>>>>> Are the libraries in question not packaged on major distros? > > >>>>>>>>> > > >>>>>>>>> Currently I only need this: > > >>>>>>>>> https://github.com/python-tap/tappy > > >>>>>>>>> which is the basic TAP processing library for python. > > >>>>>>>>> > > >>>>>>>>> It seems its only installed through pip: > > >>>>>>>>> https://tappy.readthedocs.io/en/latest/ > > >>>>>>>>> > > >>>>>>>>> I do not think this is packaged by default. It's such a basic library > > >>>>>>>>> for parsing test output that maybe we can keep this somewhere within > > >>>>>>>>> the python src tree? Not sure ... > > >>>>>>>> > > >>>>>>>> It's pretty small for sure. Another submodule? > > >>>>>>> > > >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and > > >>>>>>> will receive new releases through > > >>>>>>> https://pypi.org/project/tap.py/ > > >>>>>>> so forking is OK but someone has to keep this updated. > > >>>>>>> > > >>>>>>> I am open to anything. Whatever feels right is fine to me. > > >>>>>> > > >>>>>> John Snow is currently working on the "Pythonification" of various QEMU > > >>>>>> bits, I think you should loop him into this discussion, too. > > >>>>>> > > >>>>>> Thomas > > >>>>> > > >>>>> submodule does not mean we fork necessarily. We could have > > >>>>> all options: check for the module and use it if there, if not > > >>>>> use one from system if not there install with pip .. > > >>>>> But yea, I'm not sure what's best either. > > >>>> > > >>>> submodules create a dependency on an internet connection, too. So before you > > >>>> add yet another submodule (which have a couple of other disadvantages), I > > >>>> think you could also directly use the venv here. > > >>> > > >>> Definitely not submodules. > > >>> > > >>> We need to get out of the mindset that submodules are needed for every new > > >>> dependancy we add. Submodules are only appropriate if the external project > > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we > > >>> need to bundle in order to prevent a regression for previously deployed > > >>> QEMU installs where the dependancy is known not to exist on all our > > >>> supported platforms. > > >>> > > >>> This does not apply in this case, because the proposed use of tappy is > > >>> merely for a test case. Meson just needs to check if tappy exists and if > > >>> it does, then use it, otherwise skip the tests that need it. The user can > > >>> arrange to install tappy, as they do with the majority of other deps. > > >>> > > >>> If John's venv stuff is relevant, then we don't even need the meson checks, > > >>> just delegate to the venv setup. > > >>> > > >>> Regardless, no submodules are needed or desirable. > > >> > > >> What about keeping biosbits stuff? Source or pre-built. > > > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > > IMHO, especially for grub as a GPL'd project we need to be extremely > > > clear about the exact corresponding source and build process for any > > > binary. > > > > > > For this kind of thing I would generally expect the distro to provide > > > packages that we consume. Looking at biosbits I see it is itself > > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > > even an ancient version of python as a submodule. > > > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > > too, all with dubious license compliance. I don't think this looks like > > > something we should have in qemu.git or qemu tarballs. It will also > > > make it challenging for the distro to take biosbits at all, unless > > > those 3rd party bundles can be eliminated in favour of using existing > > > builds their have packaged for grub, python, libffi, etc. > > > > So if this depends on some third party binary bits, I think this is pretty > > similar to the tests in the avocado directory ... there we download third > > party binaries, too... Wouldn't it make sense to adapt your tests to that > > framework? > > I do not want to bring in the whole avocado framework because it would > unnecessarily make things complicated. I just need the qemu machine > python library and that is enough. For downloading third party stuff, > we can simply wget things from somewhere. Well avocado is already a part of QEMU's testing framework, so it is not bringing in a whole new framework. Arguably the introduction of python usage in qtest is what's introducing a new maint burden here. If it is actually booting a real guest image (from biosbits) and interacting with it, then it does feel like the scope of this testing is more appropriate to QEMU's avocado framework than qtest, especially given the desire to use python for it all. With regards, Daniel
On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > If it is actually booting a real guest image (from biosbits) and interacting > with it, then it does feel like the scope of this testing is more appropriate > to QEMU's avocado framework than qtest, especially given the desire to use > python for it all. > > With regards, > Daniel I feel avocado is directed towards booting full fledged guest OS. It makes it much easier to figure out guest issues but it also prone to false positives and is harder to debug as a result. Booting a minimal image like this shouldn't require that.
On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote: > So bundling a pre-built biosbits in QEMU appears to mean that we're in > turn going to unexpectedly bundle a bunch of other 3rd party projects > too, all with dubious license compliance. Well looks like classical mere aggregation to me ... license issues need to be figured out if we are to distribute things but I think this is basically what distros do anyway. And I doubt we want to support arbitrary versions of grub etc, they are very distro specific tools. I don't see why we can't have the resulting ISOs in some submodule - nothing requires us to distribute it in qemu tarballs.
On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote: > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > > > > > Following two are downloaded: > > > > > > > > > > > > avocado-framework==88.1 > > > > > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > > > > > the net anyway. > > > > > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > > > > > > > > > Currently I only need this: > > > > > > > > > > https://github.com/python-tap/tappy > > > > > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > > > > > > > > > It seems its only installed through pip: > > > > > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > > > > > the python src tree? Not sure ... > > > > > > > > > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > > > > > will receive new releases through > > > > > > > > https://pypi.org/project/tap.py/ > > > > > > > > so forking is OK but someone has to keep this updated. > > > > > > > > > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > > > > > bits, I think you should loop him into this discussion, too. > > > > > > > > > > > > > > Thomas > > > > > > > > > > > > submodule does not mean we fork necessarily. We could have > > > > > > all options: check for the module and use it if there, if not > > > > > > use one from system if not there install with pip .. > > > > > > But yea, I'm not sure what's best either. > > > > > > > > > > submodules create a dependency on an internet connection, too. So before you > > > > > add yet another submodule (which have a couple of other disadvantages), I > > > > > think you could also directly use the venv here. > > > > > > > > Definitely not submodules. > > > > > > > > We need to get out of the mindset that submodules are needed for every new > > > > dependancy we add. Submodules are only appropriate if the external project > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we > > > > need to bundle in order to prevent a regression for previously deployed > > > > QEMU installs where the dependancy is known not to exist on all our > > > > supported platforms. > > > > > > > > This does not apply in this case, because the proposed use of tappy is > > > > merely for a test case. Meson just needs to check if tappy exists and if > > > > it does, then use it, otherwise skip the tests that need it. The user can > > > > arrange to install tappy, as they do with the majority of other deps. > > > > > > > > If John's venv stuff is relevant, then we don't even need the meson checks, > > > > just delegate to the venv setup. > > > > > > > > Regardless, no submodules are needed or desirable. > > > > > > What about keeping biosbits stuff? Source or pre-built. > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > IMHO, especially for grub as a GPL'd project we need to be extremely > > clear about the exact corresponding source and build process for any > > binary. > > > > For this kind of thing I would generally expect the distro to provide > > packages that we consume. Looking at biosbits I see it is itself > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > even an ancient version of python as a submodule. > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > too, all with dubious license compliance. I don't think this looks like > > something we should have in qemu.git or qemu tarballs. It will also > > make it challenging for the distro to take biosbits at all, unless > > those 3rd party bundles can be eliminated in favour of using existing > > builds their have packaged for grub, python, libffi, etc. > > So if this depends on some third party binary bits, I think this is pretty > similar to the tests in the avocado directory ... there we download third > party binaries, too... Wouldn't it make sense to adapt your tests to that > framework? Now that you mention it, avocado does feel like a more appropriate fit. IIUC the biosbits project appears to be effectively providing a custom guest OS ISO image. IOW this testing is quite biased towards being integration testing which is the target of avocado, while qtest is much more to the unit testing end of the spectrum. This would avoid all the discussion and patches around introducing python to qtest With regards, Daniel
On Tue, Jun 28, 2022 at 06:12:48AM -0400, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote: > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > too, all with dubious license compliance. > > Well looks like classical mere aggregation to me ... license issues > need to be figured out if we are to distribute things but I think > this is basically what distros do anyway. > > And I doubt we want to support arbitrary versions of grub etc, > they are very distro specific tools. > I don't see why we can't have the resulting ISOs in some submodule - > nothing requires us to distribute it in qemu tarballs. If we don't distribute it in the tarballs then these tests won't be usable for testing release builds of QEMU which feels wrong to me. With regards, Daniel
On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > If it is actually booting a real guest image (from biosbits) and interacting > > with it, then it does feel like the scope of this testing is more appropriate > > to QEMU's avocado framework than qtest, especially given the desire to use > > python for it all. > > > > With regards, > > Daniel > > I feel avocado is directed towards booting full fledged guest OS. That's essentially what this is doing - its a custom guest OS rather than a common distro IIUC > It makes it much easier to figure out guest issues but it also > prone to false positives and is harder to debug as a result. > Booting a minimal image like this shouldn't require that. Well avocado is as reliable as the tests you write for it. The problems are largely around the images being used in avocado. If the biosbits testing system is reliable, then avocado will be too, and if they not reliable, then it will affect qtest too. With regards, Daniel
On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote: > Now that you mention it, avocado does feel like a more appropriate fit. > IIUC the biosbits project appears to be effectively providing a custom > guest OS ISO image. IOW this testing is quite biased towards being > integration testing which is the target of avocado, while qtest is much > more to the unit testing end of the spectrum. This would avoid all the > discussion and patches around introducing python to qtest > > With regards, > Daniel Yes it's an ISO image but not a full OS. So IMHO not really a integration test, more like a unit test for ACPI - current tests we have just compare ACPI to expected file which is okay but very limited.
On 28/06/2022 12.03, Michael S. Tsirkin wrote: [...] > For biosbits if we are going this route then I feel a submodule is much > better. It records which version exactly each qemu version wants. As far as I know, you can also specify the version when using pip, can't you? So that's not really an advantage here. On the contrary, submodules have a couple of disadvantages that I really dislike: - submodules do not get updated automatically when doing a "git checkout", we have to update them via a script instead. This causes e.g. trouble if you rsync your source tree to a machine that has no access to the internet and you forgot to update the submodule before the sync - the content of submodules is not added to the tarballs that get created on the git forges automatically. There were lots of requests from users in the past that tried to download a tarball from github and then wondered why they couldn't compile QEMU. - we include the submodule content in our release tarballs, so people get the impression that hte submodule content is part of the QEMU sources. This has two disadvantages: * We already got bug reports for the code in the submodule, where people did not understand that they should report that rather to the original project instead (i.e. you ship it - you own it) * People get the impression that QEMU is a huge monster application if they count the number of code lines, run their code scanner tools on the tarball contents, etc. Remember "nemu", for example, where one of the main complaints was that QEMU has too many lines of code? - If programs includes code via submodules, this gets a higher burder for distro maintainers, since they have to patch each and every package when there is a bug, instead of being able to fix it in one central place. So in my opinion we should avoid new submodules if there is an alternative. Thomas
On Tue, Jun 28, 2022 at 11:18:14AM +0100, Daniel P. Berrangé wrote: > On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > > If it is actually booting a real guest image (from biosbits) and interacting > > > with it, then it does feel like the scope of this testing is more appropriate > > > to QEMU's avocado framework than qtest, especially given the desire to use > > > python for it all. > > > > > > With regards, > > > Daniel > > > > I feel avocado is directed towards booting full fledged guest OS. > > That's essentially what this is doing - its a custom guest OS rather > than a common distro IIUC yes but then so is bios tables test - we generate the disk on the fly. > > It makes it much easier to figure out guest issues but it also > > prone to false positives and is harder to debug as a result. > > Booting a minimal image like this shouldn't require that. > > Well avocado is as reliable as the tests you write for it. The problems > are largely around the images being used in avocado. The idea of downloading large images as part of the test as opposed to part of setup is a large part of what makes it flaky - due to running into unpredictable latency and errors on both the internet and local IO side of things. > If the biosbits > testing system is reliable, then avocado will be too, and if they not > reliable, then it will affect qtest too. A larger system has more potential for bugs ;) I'm with Ani here. Using a system that is overkill will just make debugging more painful than it needs to be. > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > If it is actually booting a real guest image (from biosbits) and interacting > > with it, then it does feel like the scope of this testing is more appropriate > > to QEMU's avocado framework than qtest, especially given the desire to use > > python for it all. > > > > With regards, > > Daniel > > I feel avocado is directed towards booting full fledged guest OS. > It makes it much easier to figure out guest issues but it also > prone to false positives and is harder to debug as a result. > Booting a minimal image like this shouldn't require that. Yes 100% agree with Michael on this. Biobits is *not* booting any OS image. It runs off grub, that is, directly from bootloader stage. The interraction with the VM is minimal. > > -- > MST >
On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote: > > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > > > > > > Following two are downloaded: > > > > > > > > > > > > > avocado-framework==88.1 > > > > > > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > > > > > > the net anyway. > > > > > > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > > > > > > > > > > > Currently I only need this: > > > > > > > > > > > https://github.com/python-tap/tappy > > > > > > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > > > > > > > > > > > It seems its only installed through pip: > > > > > > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > > > > > > the python src tree? Not sure ... > > > > > > > > > > > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > > > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > > > > > > will receive new releases through > > > > > > > > > https://pypi.org/project/tap.py/ > > > > > > > > > so forking is OK but someone has to keep this updated. > > > > > > > > > > > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > > > > > > bits, I think you should loop him into this discussion, too. > > > > > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > submodule does not mean we fork necessarily. We could have > > > > > > > all options: check for the module and use it if there, if not > > > > > > > use one from system if not there install with pip .. > > > > > > > But yea, I'm not sure what's best either. > > > > > > > > > > > > submodules create a dependency on an internet connection, too. So before you > > > > > > add yet another submodule (which have a couple of other disadvantages), I > > > > > > think you could also directly use the venv here. > > > > > > > > > > Definitely not submodules. > > > > > > > > > > We need to get out of the mindset that submodules are needed for every new > > > > > dependancy we add. Submodules are only appropriate if the external project > > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we > > > > > need to bundle in order to prevent a regression for previously deployed > > > > > QEMU installs where the dependancy is known not to exist on all our > > > > > supported platforms. > > > > > > > > > > This does not apply in this case, because the proposed use of tappy is > > > > > merely for a test case. Meson just needs to check if tappy exists and if > > > > > it does, then use it, otherwise skip the tests that need it. The user can > > > > > arrange to install tappy, as they do with the majority of other deps. > > > > > > > > > > If John's venv stuff is relevant, then we don't even need the meson checks, > > > > > just delegate to the venv setup. > > > > > > > > > > Regardless, no submodules are needed or desirable. > > > > > > > > What about keeping biosbits stuff? Source or pre-built. > > > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > > IMHO, especially for grub as a GPL'd project we need to be extremely > > > clear about the exact corresponding source and build process for any > > > binary. > > > > > > For this kind of thing I would generally expect the distro to provide > > > packages that we consume. Looking at biosbits I see it is itself > > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > > even an ancient version of python as a submodule. > > > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > > too, all with dubious license compliance. I don't think this looks like > > > something we should have in qemu.git or qemu tarballs. It will also > > > make it challenging for the distro to take biosbits at all, unless > > > those 3rd party bundles can be eliminated in favour of using existing > > > builds their have packaged for grub, python, libffi, etc. > > > > So if this depends on some third party binary bits, I think this is pretty > > similar to the tests in the avocado directory ... there we download third > > party binaries, too... Wouldn't it make sense to adapt your tests to that > > framework? > > Now that you mention it, avocado does feel like a more appropriate fit. > IIUC the biosbits project appears to be effectively providing a custom > guest OS ISO image. IOW this testing is quite biased towards being > integration testing which is the target of avocado, while qtest is much > more to the unit testing end of the spectrum. This is more like unit testing than integration testing, now that you mention it. It tests only the bios, very narrowly and does not involve any OS at all. This would avoid all the > discussion and patches around introducing python to qtest > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On 28/06/2022 12.21, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote: >> Now that you mention it, avocado does feel like a more appropriate fit. >> IIUC the biosbits project appears to be effectively providing a custom >> guest OS ISO image. IOW this testing is quite biased towards being >> integration testing which is the target of avocado, while qtest is much >> more to the unit testing end of the spectrum. This would avoid all the >> discussion and patches around introducing python to qtest >> >> With regards, >> Daniel > > Yes it's an ISO image but not a full OS. The avocado framework is not meant for full OS testing only. We have a couple of tests there that just run a small firmware image. For example: - tests/avocado/machine_m68k_nextcube.py : Just runs the original firmware of the NexT Cube machine - tests/avocado/ppc_405.py : Just runs an U-Boot firmware image Some of the tests even do not download anything at all and just run a QEMU instance via python, e.g.: - tests/avocado/virtio_check_params.py - tests/avocado/x86_cpu_model_versions.py So I fail to see why biosbits should be so much different here. Of course we could argue whether the avocado framework is really such a great fit for the QEMU test suite (as I said in another mail, it rather feels like an oddball nowadays), but that's a different argument. Thomas
On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > [...] > > For biosbits if we are going this route then I feel a submodule is much > > better. It records which version exactly each qemu version wants. > > As far as I know, you can also specify the version when using pip, can't > you? So that's not really an advantage here. But of course if you do you do not get updates ;) You do however rely on a 3rd party to faithfully provide you correct code based on the version, and host it forever. > On the contrary, submodules have a couple of disadvantages that I really > dislike: > > - submodules do not get updated automatically when doing a "git checkout", > we have to update them via a script instead. This causes e.g. trouble if you > rsync your source tree to a machine that has no access to the internet and > you forgot to update the submodule before the sync how is pip better? > - the content of submodules is not added to the tarballs that get created on > the git forges automatically. There were lots of requests from users in the > past that tried to download a tarball from github and then wondered why they > couldn't compile QEMU. how is pip better here? > - we include the submodule content in our release tarballs, so people get > the impression that hte submodule content is part of the QEMU sources. This > has two disadvantages: > * We already got bug reports for the code in the submodule, > where people did not understand that they should report that > rather to the original project instead (i.e. you ship it - you > own it) > * People get the impression that QEMU is a huge monster > application if they count the number of code lines, run > their code scanner tools on the tarball contents, etc. > Remember "nemu", for example, where one of the main complaints > was that QEMU has too many lines of code? I think we can skip the checkout in the tarball if we like. If people want to run the test they can checkout then. > - If programs includes code via submodules, this gets a higher > burder for distro maintainers, since they have to patch each > and every package when there is a bug, instead of being able to > fix it in one central place. Come on, this is just a test. We *really* don't care if an ISO we use to test ACPI is using an exploitable version of grub. > So in my opinion we should avoid new submodules if there is an alternative. > > Thomas Interesting take generally, but I don't see how the logic applies in this case. Would appreciate hearing your answers to the above.
On Tue, Jun 28, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > > If it is actually booting a real guest image (from biosbits) and interacting > > > with it, then it does feel like the scope of this testing is more appropriate > > > to QEMU's avocado framework than qtest, especially given the desire to use > > > python for it all. > > > > > > With regards, > > > Daniel > > > > I feel avocado is directed towards booting full fledged guest OS. > > That's essentially what this is doing - its a custom guest OS rather Daniel, there is no OS :-) Like there is no kernel image, no persistent real file system (there is a fake one called pyfs), no packages, nothing. Its very minimalistic image narrowly targeted for one thing alone - to test the bios. > than a common distro IIUC > > > It makes it much easier to figure out guest issues but it also > > prone to false positives and is harder to debug as a result. > > Booting a minimal image like this shouldn't require that. > > Well avocado is as reliable as the tests you write for it. The problems > are largely around the images being used in avocado. If the biosbits > testing system is reliable, then avocado will be too, and if they not > reliable, then it will affect qtest too. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > > If it is actually booting a real guest image (from biosbits) and interacting > > > with it, then it does feel like the scope of this testing is more appropriate > > > to QEMU's avocado framework than qtest, especially given the desire to use > > > python for it all. > > > > > > With regards, > > > Daniel > > > > I feel avocado is directed towards booting full fledged guest OS. > > It makes it much easier to figure out guest issues but it also > > prone to false positives and is harder to debug as a result. > > Booting a minimal image like this shouldn't require that. > > Yes 100% agree with Michael on this. Biobits is *not* booting any OS > image. It runs off grub, that is, directly from bootloader stage. The > interraction with the VM is minimal. Just because it doesn't run a whole Linux kernel, doesn't make it not a guest OS image. It is merely unsual in that it can do everything it needs from grub stage, because it is just poking low level BIOS stuff and doesn't need a full OS like Linux on top. This is still functional integration testing IMHO and relevant to avocado in QEMU. With regards, Daniel
On Tue, 28 Jun 2022 at 11:38, Michael S. Tsirkin <mst@redhat.com> wrote: > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > - we include the submodule content in our release tarballs, so people get > > the impression that hte submodule content is part of the QEMU sources. This > > has two disadvantages: > > * We already got bug reports for the code in the submodule, > > where people did not understand that they should report that > > rather to the original project instead (i.e. you ship it - you > > own it) > > * People get the impression that QEMU is a huge monster > > application if they count the number of code lines, run > > their code scanner tools on the tarball contents, etc. > > Remember "nemu", for example, where one of the main complaints > > was that QEMU has too many lines of code? > > I think we can skip the checkout in the tarball if we like. > If people want to run the test they can checkout then. For tarballs and submodules, we want to provide the code in the cases where we're providing binary blobs, and for where it's required to build QEMU proper. Overall I think that the approach we use today for providing guest binaries (submodules with the code, pre-built blobs checked into git) is creaking at the seams and often awkward for downstream distros (who want to rebuild the binaries anyway). Plus submodules in general in git work really badly and awkwardly, and I'd rather we didn't add them unless we really must. We already have an approach for "tests that use binaries" -- the avocado test suites. Is that something we could use in this case ? thanks -- PMM
On Tue, Jun 28, 2022 at 4:00 PM Ani Sinha <ani@anisinha.ca> wrote: > > On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote: > > > On 28/06/2022 10.23, Daniel P. Berrangé wrote: > > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote: > > > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote: > > > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote: > > > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote: > > > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling > > > > > > > > > > > > > > > packages from the net during testing? > > > > > > > > > > > > > > > > > > > > > > > > > > > > We do that too. See requirements.txt in tests/ > > > > > > > > > > > > > > Following two are downloaded: > > > > > > > > > > > > > > avocado-framework==88.1 > > > > > > > > > > > > > > pycdlib==1.11.0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also see this line in Makefie.include: > > > > > > > > > > > > > > > > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) > > > > > > > > > > > > > > > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from > > > > > > > > > > > > > the net anyway. > > > > > > > > > > > > > Are the libraries in question not packaged on major distros? > > > > > > > > > > > > > > > > > > > > > > > > Currently I only need this: > > > > > > > > > > > > https://github.com/python-tap/tappy > > > > > > > > > > > > which is the basic TAP processing library for python. > > > > > > > > > > > > > > > > > > > > > > > > It seems its only installed through pip: > > > > > > > > > > > > https://tappy.readthedocs.io/en/latest/ > > > > > > > > > > > > > > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library > > > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within > > > > > > > > > > > > the python src tree? Not sure ... > > > > > > > > > > > > > > > > > > > > > > It's pretty small for sure. Another submodule? > > > > > > > > > > > > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and > > > > > > > > > > will receive new releases through > > > > > > > > > > https://pypi.org/project/tap.py/ > > > > > > > > > > so forking is OK but someone has to keep this updated. > > > > > > > > > > > > > > > > > > > > I am open to anything. Whatever feels right is fine to me. > > > > > > > > > > > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU > > > > > > > > > bits, I think you should loop him into this discussion, too. > > > > > > > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > submodule does not mean we fork necessarily. We could have > > > > > > > > all options: check for the module and use it if there, if not > > > > > > > > use one from system if not there install with pip .. > > > > > > > > But yea, I'm not sure what's best either. > > > > > > > > > > > > > > submodules create a dependency on an internet connection, too. So before you > > > > > > > add yet another submodule (which have a couple of other disadvantages), I > > > > > > > think you could also directly use the venv here. > > > > > > > > > > > > Definitely not submodules. > > > > > > > > > > > > We need to get out of the mindset that submodules are needed for every new > > > > > > dependancy we add. Submodules are only appropriate if the external project > > > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we > > > > > > need to bundle in order to prevent a regression for previously deployed > > > > > > QEMU installs where the dependancy is known not to exist on all our > > > > > > supported platforms. > > > > > > > > > > > > This does not apply in this case, because the proposed use of tappy is > > > > > > merely for a test case. Meson just needs to check if tappy exists and if > > > > > > it does, then use it, otherwise skip the tests that need it. The user can > > > > > > arrange to install tappy, as they do with the majority of other deps. > > > > > > > > > > > > If John's venv stuff is relevant, then we don't even need the meson checks, > > > > > > just delegate to the venv setup. > > > > > > > > > > > > Regardless, no submodules are needed or desirable. > > > > > > > > > > What about keeping biosbits stuff? Source or pre-built. > > > > > > > > Shipping them as pre-built binaries in QEMU is not a viable option > > > > IMHO, especially for grub as a GPL'd project we need to be extremely > > > > clear about the exact corresponding source and build process for any > > > > binary. > > > > > > > > For this kind of thing I would generally expect the distro to provide > > > > packages that we consume. Looking at biosbits I see it is itself > > > > bundling a bunch more 3rd party projects, libffi, grub2, and including > > > > even an ancient version of python as a submodule. > > > > > > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in > > > > turn going to unexpectedly bundle a bunch of other 3rd party projects > > > > too, all with dubious license compliance. I don't think this looks like > > > > something we should have in qemu.git or qemu tarballs. It will also > > > > make it challenging for the distro to take biosbits at all, unless > > > > those 3rd party bundles can be eliminated in favour of using existing > > > > builds their have packaged for grub, python, libffi, etc. > > > > > > So if this depends on some third party binary bits, I think this is pretty > > > similar to the tests in the avocado directory ... there we download third > > > party binaries, too... Wouldn't it make sense to adapt your tests to that > > > framework? > > > > Now that you mention it, avocado does feel like a more appropriate fit. > > IIUC the biosbits project appears to be effectively providing a custom > > guest OS ISO image. IOW this testing is quite biased towards being > > integration testing which is the target of avocado, while qtest is much > > more to the unit testing end of the spectrum. > > This is more like unit testing than integration testing, now that you > mention it. It tests only the bios, very narrowly and does not involve > any OS at all. Another thing to consider is that integration testing is further down the line? Not for once when submitting patches on acpi have I run them. However, every time I have run make check to make sure bios-tables-test passes and I am not breaking anything. It's much more useful to have this kind of thing part of make check before patch submitters can quickly check for failures either in bios-tables-test or in bits. Also its lot easier to add new acpi/smbios tests as a part of this when bios-tables-test and this one are closer together. > > This would avoid all the > > discussion and patches around introducing python to qtest > > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On 28/06/2022 12.30, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: >> On 28/06/2022 12.03, Michael S. Tsirkin wrote: >> [...] >>> For biosbits if we are going this route then I feel a submodule is much >>> better. It records which version exactly each qemu version wants. >> >> As far as I know, you can also specify the version when using pip, can't >> you? So that's not really an advantage here. > > But of course if you do you do not get updates ;) You do > however rely on a 3rd party to faithfully provide you > correct code based on the version, and host it forever. > >> On the contrary, submodules have a couple of disadvantages that I really >> dislike: >> >> - submodules do not get updated automatically when doing a "git checkout", >> we have to update them via a script instead. This causes e.g. trouble if you >> rsync your source tree to a machine that has no access to the internet and >> you forgot to update the submodule before the sync > > how is pip better? You don't end up with an inconsistent source tree in that case (which happens with submodules quite easily, at least for me it happened a couple of times already). Either the machine has an internet connection, so that pip can install the missing bits, or it does not and the test has to be skipped. But if I copy the wrong state of a submodule around, things get messed up quite easily in my experience. Ok, you could say that this is just my special setup with rsync, but already given the fact that "git checkout" creates an inconsistent state of your source tree until you run the script for updating the submodules the next time is an indication that submodules are rather a shaky thing (e.g. if you'd create a tarball for somebody else from your source tree right after doing a "git checkout"). >> - the content of submodules is not added to the tarballs that get created on >> the git forges automatically. There were lots of requests from users in the >> past that tried to download a tarball from github and then wondered why they >> couldn't compile QEMU. > > how is pip better here? You don't get incomplete/non-working tarballs in that case. >> - we include the submodule content in our release tarballs, so people get >> the impression that hte submodule content is part of the QEMU sources. This >> has two disadvantages: >> * We already got bug reports for the code in the submodule, >> where people did not understand that they should report that >> rather to the original project instead (i.e. you ship it - you >> own it) >> * People get the impression that QEMU is a huge monster >> application if they count the number of code lines, run >> their code scanner tools on the tarball contents, etc. >> Remember "nemu", for example, where one of the main complaints >> was that QEMU has too many lines of code? > > I think we can skip the checkout in the tarball if we like. > If people want to run the test they can checkout then. Release tarballs don't include the ".git" folder infrastructur, so everybody who downloads a tarball will simply never be able to run the test. > >> - If programs includes code via submodules, this gets a higher >> burder for distro maintainers, since they have to patch each >> and every package when there is a bug, instead of being able to >> fix it in one central place. > > Come on, this is just a test. We *really* don't care if an ISO > we use to test ACPI is using an exploitable version of grub. Wait, I thought we were only talking about tappy here? The ISO binaries should certainly *not* be bundled in the QEMU tarballs (they are too big already anyway, we should rather think of moving the firmware binaries out of the tarball instead). Thomas
On Tue, Jun 28, 2022 at 11:43:58AM +0100, Peter Maydell wrote: > On Tue, 28 Jun 2022 at 11:38, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > - we include the submodule content in our release tarballs, so people get > > > the impression that hte submodule content is part of the QEMU sources. This > > > has two disadvantages: > > > * We already got bug reports for the code in the submodule, > > > where people did not understand that they should report that > > > rather to the original project instead (i.e. you ship it - you > > > own it) > > > * People get the impression that QEMU is a huge monster > > > application if they count the number of code lines, run > > > their code scanner tools on the tarball contents, etc. > > > Remember "nemu", for example, where one of the main complaints > > > was that QEMU has too many lines of code? > > > > I think we can skip the checkout in the tarball if we like. > > If people want to run the test they can checkout then. > > For tarballs and submodules, we want to provide the code in the > cases where we're providing binary blobs, and for where it's > required to build QEMU proper. > > Overall I think that the approach we use today for providing > guest binaries (submodules with the code, pre-built blobs checked > into git) is creaking at the seams and often awkward for downstream > distros (who want to rebuild the binaries anyway). > > Plus submodules in general in git work really badly and awkwardly, > and I'd rather we didn't add them unless we really must. > > We already have an approach for "tests that use binaries" -- > the avocado test suites. Is that something we could use in this > case ? > > thanks > -- PMM From experience avocado is basically rolling its own reimplementation of git fetch. Result is weird failures when some corner case or another isn't handled perfectly. Kind of unavoidable since you can't have ISOs for a ton of guest OSes reasonably in a git repo. Well maybe with sparse checkouts you can. Haven't really looked at it. These specific tests are weird, kind of special in that they are developer oriented. I *would* like anyone making changes to ACPI to run them, and for CI to run them on patches, but I do not believe e.g. distros necessarily need them. git submodules are awkward basically because they are an automated wget. I don't think an explicit wget is much better ... but looks like I'm alone in this. Oh well. So it will be a weird dance of wget a tarball, unpack, generate ISO and run. God help you if you need to patch the test - it's wget all the way down.
On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote: > On 28/06/2022 12.30, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > [...] > > > > For biosbits if we are going this route then I feel a submodule is much > > > > better. It records which version exactly each qemu version wants. > > > > > > As far as I know, you can also specify the version when using pip, can't > > > you? So that's not really an advantage here. > > > > But of course if you do you do not get updates ;) You do > > however rely on a 3rd party to faithfully provide you > > correct code based on the version, and host it forever. > > > > > On the contrary, submodules have a couple of disadvantages that I really > > > dislike: > > > > > > - submodules do not get updated automatically when doing a "git checkout", > > > we have to update them via a script instead. This causes e.g. trouble if you > > > rsync your source tree to a machine that has no access to the internet and > > > you forgot to update the submodule before the sync > > > > how is pip better? > > You don't end up with an inconsistent source tree in that case (which > happens with submodules quite easily, at least for me it happened a couple > of times already). Either the machine has an internet connection, so that > pip can install the missing bits, or it does not and the test has to be > skipped. skipped tests are too easy to ignore ... > But if I copy the wrong state of a submodule around, things get > messed up quite easily in my experience. Ok, you could say that this is just > my special setup with rsync, but already given the fact that "git checkout" > creates an inconsistent state of your source tree until you run the script > for updating the submodules the next time is an indication that submodules > are rather a shaky thing (e.g. if you'd create a tarball for somebody else > from your source tree right after doing a "git checkout"). yea one has to remember to set submodule.recurse = true in .gitconfig I agree it's annoying, I guess they don't change it for compat reasons. > > > - the content of submodules is not added to the tarballs that get created on > > > the git forges automatically. There were lots of requests from users in the > > > past that tried to download a tarball from github and then wondered why they > > > couldn't compile QEMU. > > > > how is pip better here? > > You don't get incomplete/non-working tarballs in that case. So skip the test ;) > > > - we include the submodule content in our release tarballs, so people get > > > the impression that hte submodule content is part of the QEMU sources. This > > > has two disadvantages: > > > * We already got bug reports for the code in the submodule, > > > where people did not understand that they should report that > > > rather to the original project instead (i.e. you ship it - you > > > own it) > > > * People get the impression that QEMU is a huge monster > > > application if they count the number of code lines, run > > > their code scanner tools on the tarball contents, etc. > > > Remember "nemu", for example, where one of the main complaints > > > was that QEMU has too many lines of code? > > > > I think we can skip the checkout in the tarball if we like. > > If people want to run the test they can checkout then. > > Release tarballs don't include the ".git" folder infrastructur, so everybody > who downloads a tarball will simply never be able to run the test. I actually think I'm fine with that for this specific case. > > > > > - If programs includes code via submodules, this gets a higher > > > burder for distro maintainers, since they have to patch each > > > and every package when there is a bug, instead of being able to > > > fix it in one central place. > > > > Come on, this is just a test. We *really* don't care if an ISO > > we use to test ACPI is using an exploitable version of grub. > > Wait, I thought we were only talking about tappy here? The ISO binaries > should certainly *not* be bundled in the QEMU tarballs (they are too big > already anyway, we should rather think of moving the firmware binaries out > of the tarball instead). > > Thomas IIUC there are three things we are discussing - biosbits source - biosbits image - tappy
On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote: > On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > > > If it is actually booting a real guest image (from biosbits) and interacting > > > > with it, then it does feel like the scope of this testing is more appropriate > > > > to QEMU's avocado framework than qtest, especially given the desire to use > > > > python for it all. > > > > > > > > With regards, > > > > Daniel > > > > > > I feel avocado is directed towards booting full fledged guest OS. > > > It makes it much easier to figure out guest issues but it also > > > prone to false positives and is harder to debug as a result. > > > Booting a minimal image like this shouldn't require that. > > > > Yes 100% agree with Michael on this. Biobits is *not* booting any OS > > image. It runs off grub, that is, directly from bootloader stage. The > > interraction with the VM is minimal. > > Just because it doesn't run a whole Linux kernel, doesn't make it > not a guest OS image. It is merely unsual in that it can do everything > it needs from grub stage, because it is just poking low level BIOS > stuff and doesn't need a full OS like Linux on top. This is still > functional integration testing IMHO and relevant to avocado in QEMU. > > With regards, > Daniel I wonder why it needs grub BTW. Let's just load all of disk into ram and be done with it? > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote: > On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote: > > > > If it is actually booting a real guest image (from biosbits) and interacting > > > > with it, then it does feel like the scope of this testing is more appropriate > > > > to QEMU's avocado framework than qtest, especially given the desire to use > > > > python for it all. > > > > > > > > With regards, > > > > Daniel > > > > > > I feel avocado is directed towards booting full fledged guest OS. > > > It makes it much easier to figure out guest issues but it also > > > prone to false positives and is harder to debug as a result. > > > Booting a minimal image like this shouldn't require that. > > > > Yes 100% agree with Michael on this. Biobits is *not* booting any OS > > image. It runs off grub, that is, directly from bootloader stage. The > > interraction with the VM is minimal. > > Just because it doesn't run a whole Linux kernel, doesn't make it > not a guest OS image. It is merely unsual in that it can do everything > it needs from grub stage, because it is just poking low level BIOS > stuff and doesn't need a full OS like Linux on top. This is still > functional integration testing IMHO and relevant to avocado in QEMU. > > With regards, > Daniel I think the main difference is not even in how it works, it's in what it does. Which is check that ACPI tables are sane. Who cares about that? Well developers do when they change the tables. Users really don't because for users we have the expected tables in tree and we check against these.
On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote: > I think the main difference is not even in how it works, it's > in what it does. Which is check that ACPI tables are sane. > Who cares about that? Well developers do when they change the > tables. Users really don't because for users we have the expected > tables in tree and we check against these. It wants to build and run a big guest binary blob -- that to me is the main difference. Users don't much care about any of our tests, whether they're under 'make check' or 'make check-avocado' or the iotests framework. The reason to pick one framework or another is mostly I think whether the properties of the test are such that one framework works better. Avocado is (for better or worse) the one we have for dealing with "actually run a guest machine with a big lump of guest code in it". -- PMM
On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote: > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote: > > I think the main difference is not even in how it works, it's > > in what it does. Which is check that ACPI tables are sane. > > Who cares about that? Well developers do when they change the > > tables. Users really don't because for users we have the expected > > tables in tree and we check against these. > > It wants to build and run a big guest binary blob -- that to me is > the main difference. Users don't much care about any of our tests, perhaps but we do enforce patch submitters to make sure make check passes before submitting patches. make check-avocado is not run as part of make check, requires considerable disk space to download all guest images and hence generally not run by patch submitters. Making bits parts of avocado tests almost defeats the purpose of having this test at all. > whether they're under 'make check' or 'make check-avocado' or the > iotests framework. The reason to pick one framework or another > is mostly I think whether the properties of the test are such > that one framework works better. Avocado is (for better or worse) > the one we have for dealing with "actually run a guest machine > with a big lump of guest code in it". > > -- PMM
On 28/06/2022 13.14, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote: [...] >>> Come on, this is just a test. We *really* don't care if an ISO >>> we use to test ACPI is using an exploitable version of grub. >> >> Wait, I thought we were only talking about tappy here? The ISO binaries >> should certainly *not* be bundled in the QEMU tarballs (they are too big >> already anyway, we should rather think of moving the firmware binaries out >> of the tarball instead). >> >> Thomas > > IIUC there are three things we are discussing > - biosbits source > - biosbits image > - tappy Oh well, I missed that part of the discussion so far since the corresponding patches did not make it to the mailing list ¯\_(ツ)_/¯ Anyway, that's just another indication that it might not be the right fit for inclusion into the QEMU source tree. So either download it similar to (or included in) the avocado tests, or maybe another solution would be to have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ? Thomas
On 28/06/2022 14.36, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote: >> >> On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote: >>> I think the main difference is not even in how it works, it's >>> in what it does. Which is check that ACPI tables are sane. >>> Who cares about that? Well developers do when they change the >>> tables. Users really don't because for users we have the expected >>> tables in tree and we check against these. >> >> It wants to build and run a big guest binary blob -- that to me is >> the main difference. Users don't much care about any of our tests, > > perhaps but we do enforce patch submitters to make sure make check > passes before submitting patches. make check-avocado is not run as > part of make check, requires considerable disk space to download all > guest images and hence generally not run by patch submitters. Making > bits parts of avocado tests almost defeats the purpose of having this > test at all. You can define sane subsets of the avocado tests by using tags. For example, as I'm one of the s390x maintainers, I'm also only running the tests that are tagged with tags=arch:s390x there. You could simply introduce an "acpi" tag for your tests, too. Thomas
On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote: > > > I think the main difference is not even in how it works, it's > > > in what it does. Which is check that ACPI tables are sane. > > > Who cares about that? Well developers do when they change the > > > tables. Users really don't because for users we have the expected > > > tables in tree and we check against these. > > > > It wants to build and run a big guest binary blob -- that to me is > > the main difference. Users don't much care about any of our tests, > > perhaps but we do enforce patch submitters to make sure make check > passes before submitting patches. make check-avocado is not run as > part of make check, requires considerable disk space to download all > guest images and hence generally not run by patch submitters. Making > bits parts of avocado tests almost defeats the purpose of having this > test at all. This proposed biosbits test also involves a considerable download. The test is said to be irrelevant for anyone except those working on a fairly narrow set of QEMU firmware related bits. So by the same rationale we shouldn't impose that burden on everyone working on QEMU by having it in qtest. Making it entirely optional, only downloaded by avocado on demand, for the people who need to run the tests is best. With regards, Daniel
On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > I think the main difference is not even in how it works, it's > > > > in what it does. Which is check that ACPI tables are sane. > > > > Who cares about that? Well developers do when they change the > > > > tables. Users really don't because for users we have the expected > > > > tables in tree and we check against these. > > > > > > It wants to build and run a big guest binary blob -- that to me is > > > the main difference. Users don't much care about any of our tests, > > > > perhaps but we do enforce patch submitters to make sure make check > > passes before submitting patches. make check-avocado is not run as > > part of make check, requires considerable disk space to download all > > guest images and hence generally not run by patch submitters. Making > > bits parts of avocado tests almost defeats the purpose of having this > > test at all. > > This proposed biosbits test also involves a considerable download. I do not think 50 MB is "considerable" . Last time I tried to run avocado tests, my laptop ran out of disk space! > The test is said to be irrelevant for anyone except those working > on a fairly narrow set of QEMU firmware related bits. Well ok that is just a bad argument. You can say the same thing for most qtests. In fact, that is why most qtetes can run directly simply by passing QTEST_QEMU_BINARY in the environment. No need to go through make check. Same with the bits test. It can be run directly. So by the same > rationale we shouldn't impose that burden on everyone working on > QEMU by having it in qtest. So why burden everyone by having bios-tables-test when it only affects acpi/smbios developers? Making it entirely optional, only > downloaded by avocado on demand, for the people who need to run > the tests is best. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote: > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > This proposed biosbits test also involves a considerable download. > > I do not think 50 MB is "considerable" . Last time I tried to run > avocado tests, my laptop ran out of disk space! I think 50MB is pretty big. It might be smaller than some other avocado tests, but it's not exactly the "no binary involved" that most qtests are. > > The test is said to be irrelevant for anyone except those working > > on a fairly narrow set of QEMU firmware related bits. > > Well ok that is just a bad argument. You can say the same thing for > most qtests. In fact, that is why most qtetes can run directly simply > by passing QTEST_QEMU_BINARY in the environment. No need to go through > make check. Same with the bits test. It can be run directly. 'make check' is generally the small, fast, no-binary-blobs tests. Very few 'make check' tests even run code in the guest. > So by the same > > rationale we shouldn't impose that burden on everyone working on > > QEMU by having it in qtest. > > So why burden everyone by having bios-tables-test when it only affects > acpi/smbios developers? Because it's small and fast and doesn't have a binary blob to download. There are definitely some awkwardnesses with 'check-avocado', but we should work on fixing those, not use them as a reason to refuse to put tests into the avocado tests if that's where they fit best. -- PMM
On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org> wrote: > On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote: > > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> > wrote: > > > This proposed biosbits test also involves a considerable download. > > > > I do not think 50 MB is "considerable" . Last time I tried to run > > avocado tests, my laptop ran out of disk space! > > I think 50MB is pretty big. It might be smaller than some other > avocado tests, but it's not exactly the "no binary involved" > that most qtests are. Well bios-tables-test uses the binary blobs of the acpi tables. Only difference is that in this case, we could maintain them within the qemu tree. In this case the blob in slightly larger and comes from a third party. That is the difference. > > > > The test is said to be irrelevant for anyone except those working > > > on a fairly narrow set of QEMU firmware related bits. > > > > Well ok that is just a bad argument. You can say the same thing for > > most qtests. In fact, that is why most qtetes can run directly simply > > by passing QTEST_QEMU_BINARY in the environment. No need to go through > > make check. Same with the bits test. It can be run directly. > > 'make check' is generally the small, fast, no-binary-blobs tests. See above. > Very few 'make check' tests even run code in the guest. So bits test is similar here. It runs code in the guest vm. > > > So by the same > > > rationale we shouldn't impose that burden on everyone working on > > > QEMU by having it in qtest. > > > > So why burden everyone by having bios-tables-test when it only affects > > acpi/smbios developers? > > Because it's small and fast and doesn't have a binary blob to download. > > There are definitely some awkwardnesses with 'check-avocado', > but we should work on fixing those, not use them as a reason > to refuse to put tests into the avocado tests if that's where > they fit best. I think this test fits best in the qtrst not with the integration test framework. Very few path developers will ever run it and wrote new tests for it if we have it there. I would be terribly discouraged if that’s where this test landed. > > -- PMM >
On Tue, 28 Jun 2022 at 14:53, Ani Sinha <ani@anisinha.ca> wrote: > > > > On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org> wrote: >> >> On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote: >> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote: >> > > This proposed biosbits test also involves a considerable download. >> > >> > I do not think 50 MB is "considerable" . Last time I tried to run >> > avocado tests, my laptop ran out of disk space! >> >> I think 50MB is pretty big. It might be smaller than some other >> avocado tests, but it's not exactly the "no binary involved" >> that most qtests are. > > > Well bios-tables-test uses the binary blobs of the acpi tables. Only difference is that in this case, we could maintain them within the qemu tree. In this case the blob in slightly larger and comes from a third party. That is the difference. Yes. That is exactly the difference that means it should go in the avocado tests. -- PMM
On Tue, Jun 28, 2022 at 01:55:48PM +0100, Daniel P. Berrangé wrote: > On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > I think the main difference is not even in how it works, it's > > > > in what it does. Which is check that ACPI tables are sane. > > > > Who cares about that? Well developers do when they change the > > > > tables. Users really don't because for users we have the expected > > > > tables in tree and we check against these. > > > > > > It wants to build and run a big guest binary blob -- that to me is > > > the main difference. Users don't much care about any of our tests, > > > > perhaps but we do enforce patch submitters to make sure make check > > passes before submitting patches. make check-avocado is not run as > > part of make check, requires considerable disk space to download all > > guest images and hence generally not run by patch submitters. Making > > bits parts of avocado tests almost defeats the purpose of having this > > test at all. > > This proposed biosbits test also involves a considerable download. > The test is said to be irrelevant for anyone except those working > on a fairly narrow set of QEMU firmware related bits. So by the same > rationale we shouldn't impose that burden on everyone working on > QEMU by having it in qtest. Making it entirely optional, only > downloaded by avocado on demand, for the people who need to run > the tests is best. > > With regards, > Daniel Optional is fine. I would like to make life easy for developers working on ACPI though, the field is unforgiving enough as it is. How about we skip the test if the submodule is not checked out? > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jun 28, 2022 at 06:52:55PM +0530, Ani Sinha wrote: > So why burden everyone by having bios-tables-test when it only affects > acpi/smbios developers? No it doesn't. Any change in qemu can break ACPI - it's enough to register an MR overlapping the ACPI registers. So we have a basic test making sure ACPI is in expected state and have everyone run it.
On Tue, Jun 28, 2022 at 02:39:31PM +0200, Thomas Huth wrote: > On 28/06/2022 13.14, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote: > [...] > > > > Come on, this is just a test. We *really* don't care if an ISO > > > > we use to test ACPI is using an exploitable version of grub. > > > > > > Wait, I thought we were only talking about tappy here? The ISO binaries > > > should certainly *not* be bundled in the QEMU tarballs (they are too big > > > already anyway, we should rather think of moving the firmware binaries out > > > of the tarball instead). > > > > > > Thomas > > > > IIUC there are three things we are discussing > > - biosbits source > > - biosbits image > > - tappy > > Oh well, I missed that part of the discussion so far since the corresponding > patches did not make it to the mailing list ¯\_(ツ)_/¯ > > Anyway, that's just another indication that it might not be the right fit > for inclusion into the QEMU source tree. So either download it similar to > (or included in) the avocado tests, or maybe another solution would be to > have a separate "qemu-ci" or "qemu-testing" repository for stuff like this > ... ? > > Thomas I don't think anyone suggested sticking all those blobs in qemu.git: I proposed a submodule, others proposed the web (fetched with avocado or just wget). If it's a submodule we'll want to skip it in the tarball, and skip the test if not checked out. Whoever is trying to do development out of a tarball should just stop :).
On Tue, Jun 28, 2022 at 5:10 AM Michael S. Tsirkin <mst@redhat.com> wrote: > git submodules are awkward basically because they are an automated wget. > I don't think an explicit wget is much better ... but > looks like I'm alone in this. Oh well. > submodules are 90% of the hassles I have in upstreaming and updating the bsd-user stuff. They play terribly with branches, rebases, and any number of other things and are a pain to reset at times. They are easily the most brittle, error-prone and difficult to use feature of git. I hate them with a burning passion. You are not alone. Warner
On Tue, Jun 28, 2022 at 6:09 PM Thomas Huth <thuth@redhat.com> wrote: > > On 28/06/2022 13.14, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote: > [...] > >>> Come on, this is just a test. We *really* don't care if an ISO > >>> we use to test ACPI is using an exploitable version of grub. > >> > >> Wait, I thought we were only talking about tappy here? The ISO binaries > >> should certainly *not* be bundled in the QEMU tarballs (they are too big > >> already anyway, we should rather think of moving the firmware binaries out > >> of the tarball instead). > >> > >> Thomas > > > > IIUC there are three things we are discussing > > - biosbits source > > - biosbits image > > - tappy > > Oh well, I missed that part of the discussion so far since the corresponding > patches did not make it to the mailing list ¯\_(ツ)_/¯ > > Anyway, that's just another indication that it might not be the right fit > for inclusion into the QEMU source tree. So either download it similar to > (or included in) the avocado tests, or maybe another solution would be to > have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ? Yes, I think we should have a separate repo for all testing related stuff like blobs, guest images etc. Curently, some of them are in personal github accounts which is not idea IMHo. Bits stuff can reside in the same place.
On Tue, Jun 28, 2022 at 09:24:34PM +0530, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 6:09 PM Thomas Huth <thuth@redhat.com> wrote: > > > > On 28/06/2022 13.14, Michael S. Tsirkin wrote: > > > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote: > > [...] > > >>> Come on, this is just a test. We *really* don't care if an ISO > > >>> we use to test ACPI is using an exploitable version of grub. > > >> > > >> Wait, I thought we were only talking about tappy here? The ISO binaries > > >> should certainly *not* be bundled in the QEMU tarballs (they are too big > > >> already anyway, we should rather think of moving the firmware binaries out > > >> of the tarball instead). > > >> > > >> Thomas > > > > > > IIUC there are three things we are discussing > > > - biosbits source > > > - biosbits image > > > - tappy > > > > Oh well, I missed that part of the discussion so far since the corresponding > > patches did not make it to the mailing list ¯\_(ツ)_/¯ > > > > Anyway, that's just another indication that it might not be the right fit > > for inclusion into the QEMU source tree. So either download it similar to > > (or included in) the avocado tests, or maybe another solution would be to > > have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ? > > Yes, I think we should have a separate repo for all testing related > stuff like blobs, guest images etc. Curently, some of them are in > personal github accounts which is not idea IMHo. > > Bits stuff can reside in the same place. FYI, the reason much of this is intentionally NOT under the /qemu-project gitlab namespace is that we did not want to be responsible for distributing arbitrary binary blobs/images. That in turn makes the QEMU project responsible for license compliance, which is non-trivial todo correctly for much of this stuff. As such it is highly desirable to delegate both the hosting the binaries and source to the third party who builds it. I agree the use of personal github accounts is not nice, but it was the least worst solution identified. With regards, Daniel
On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote: > FYI, the reason much of this is intentionally NOT under the /qemu-project > gitlab namespace is that we did not want to be responsible for distributing > arbitrary binary blobs/images. That in turn makes the QEMU project responsible > for license compliance, which is non-trivial todo correctly for much of this > stuff. As such it is highly desirable to delegate both the hosting the > binaries and source to the third party who builds it. This might be understadable for random guest OS images which include tons of stuff and are thus hard to audit. But not to biosbits which has its own license (more or less bsd) + gpl for grub: https://github.com/biosbits/bits/blob/master/COPYING > I agree the use of personal github accounts is not nice, but it was the > least worst solution identified.
On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote: > > FYI, the reason much of this is intentionally NOT under the /qemu-project > > gitlab namespace is that we did not want to be responsible for distributing > > arbitrary binary blobs/images. That in turn makes the QEMU project responsible > > for license compliance, which is non-trivial todo correctly for much of this > > stuff. As such it is highly desirable to delegate both the hosting the > > binaries and source to the third party who builds it. > > This might be understadable for random guest OS images which include tons of stuff > and are thus hard to audit. But not to biosbits which has its own > license (more or less bsd) + gpl for grub: > https://github.com/biosbits/bits/blob/master/COPYING These are all the dependencies: https://github.com/biosbits/bits/tree/master/deps We can go through the licenses for each and make a determination. The audit would be lost easier because there is a bounded number of dependencies for bits. > > > I agree the use of personal github accounts is not nice, but it was the > > least worst solution identified. > > > -- > MST >
On 29/06/2022 08.28, Ani Sinha wrote: > On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote: >>> FYI, the reason much of this is intentionally NOT under the /qemu-project >>> gitlab namespace is that we did not want to be responsible for distributing >>> arbitrary binary blobs/images. That in turn makes the QEMU project responsible >>> for license compliance, which is non-trivial todo correctly for much of this >>> stuff. As such it is highly desirable to delegate both the hosting the >>> binaries and source to the third party who builds it. >> >> This might be understadable for random guest OS images which include tons of stuff >> and are thus hard to audit. But not to biosbits which has its own >> license (more or less bsd) + gpl for grub: >> https://github.com/biosbits/bits/blob/master/COPYING > > These are all the dependencies: > https://github.com/biosbits/bits/tree/master/deps > > We can go through the licenses for each and make a determination. The > audit would be lost easier because there is a bounded number of > dependencies for bits. That's quite a bit of dependencies already ... I don't think that we should put the burden of keeping up with the licenses of those projects to the QEMU project. So just make sure that the binaries are available somewhere and then include your test into the avocado framework or download via curl/wget as suggested in other mails. Thomas
On 28/06/2022 15.53, Ani Sinha wrote: > > > On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org > <mailto:peter.maydell@linaro.org>> wrote: > > On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca > <mailto:ani@anisinha.ca>> wrote: > > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé > <berrange@redhat.com <mailto:berrange@redhat.com>> wrote: > > > This proposed biosbits test also involves a considerable download. > > > > I do not think 50 MB is "considerable" . Last time I tried to run > > avocado tests, my laptop ran out of disk space! > > I think 50MB is pretty big. It might be smaller than some other > avocado tests, but it's not exactly the "no binary involved" > that most qtests are. > > > Well bios-tables-test uses the binary blobs of the acpi tables. Only > difference is that in this case, we could maintain them within the qemu > tree. In this case the blob in slightly larger and comes from a third party. > That is the difference. "slightly larger" ... it apparently contains already a complete grub and python 2.7 interpreter ... I'd consider that as *much* larger than the ~ 2k loc bios-tables-test ;-) > Very few 'make check' tests even run code in the guest. > > So bits test is similar here. It runs code in the guest vm. The qtests that run some TCG code only use some few lines of code. The tests that run third party firmware images are the avocado tests. > There are definitely some awkwardnesses with 'check-avocado', > but we should work on fixing those, not use them as a reason > to refuse to put tests into the avocado tests if that's where > they fit best. > > I think this test fits best in the qtrst not with the integration test > framework. I disagree. Third party binary blob tests are certainly nothing for the qtest directory. > Very few path developers will ever run it and wrote new tests for > it if we have it there. I would be terribly discouraged if that’s where this > test landed. I see your point - I'm also hardly ever running whole "make check-avocado" since it's too slow/fat/broken. But using this as a reason to stick your test into the qtest framework also does not work/scale: Being one of the s390x maintainers - What if I now also want my s390x related tests to be executed by everybody? The kernel + initrd images from tests/avocado/machine_s390_ccw_virtio.py are also not that big, ca. 50 MiB each. Should I now move that out of the avocade directory, too, and force everybody to consume it via a submodule? Then who's next? QEMU has 21 target architectures, if every maintainers adds a 50 MiB test to the tree, we're at more than a gigabyte of data already that you'd have to download before you were able to run "make check". This simply does not scale. So the avocado framework is currently still the best place that we have for such tests. You just have to get used to consume it via "tags" instead of running the whole "make check-avocado" suite. For example, being a s390x maintainer, I'm only running: make check-venv ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/ and that finishes within some few minutes and only downloads some few hundreds of megabytes. You could do something similar with acpi. You could even wrap it in a "make check-avocado-acpi" target if you don't like to execute avocado directly. I even wouldn't mind if you put your python stuff in a new directory like tests/pytests/ for example, as long as it downloads your binaries separately - as I wrote in another mail, the avocado framework rather looks like an oddball in our test framework nowadays since it uses a separate test runner and not the meson test harness, so having a new approach for python-based tests is maybe even a good idea. I just really want to avoid that this goes into tests/qtest (since it really does not belong there), and please don't add more external stuff via git submodules, that's really the wrong approach for this. Thomas
On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > I even wouldn't mind if you put your python stuff in a new directory like > tests/pytests/ for example, as long as it downloads your binaries separately > - as I wrote in another mail, the avocado framework rather looks like an > oddball in our test framework nowadays since it uses a separate test runner > and not the meson test harness, so having a new approach for python-based > tests is maybe even a good idea. I just really want to avoid that this goes > into tests/qtest (since it really does not belong there), and please don't > add more external stuff via git submodules, that's really the wrong approach > for this. I get it, people hate submodules with passion. I think trying another approach for testing that is neither avocado nor qtest is not too bad. As long as this is not user visible, we can allow ourselves space to experiment. OK so, how about this: - put it in a new directory: tests/roms? - create repo for a fork of biosbits under git.qemu.org - roll our own analog to git submodules: a script that clones the repo - new target make check-roms, if the clone exists locally - run the test, if not - skip it - as for using pre-generates ISOs as an optimization, I'm not sure how important that is, if yes - we can add another repo and another make target along the same lines
On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > > I even wouldn't mind if you put your python stuff in a new directory like > > tests/pytests/ for example, as long as it downloads your binaries separately > > - as I wrote in another mail, the avocado framework rather looks like an > > oddball in our test framework nowadays since it uses a separate test runner > > and not the meson test harness, so having a new approach for python-based > > tests is maybe even a good idea. I just really want to avoid that this goes > > into tests/qtest (since it really does not belong there), and please don't > > add more external stuff via git submodules, that's really the wrong approach > > for this. > > I get it, people hate submodules with passion. I think trying another > approach for testing that is neither avocado nor qtest is > not too bad. As long as this is not user visible, we can > allow ourselves space to experiment. > > OK so, how about this: > - put it in a new directory: tests/roms? > - create repo for a fork of biosbits under git.qemu.org > - roll our own analog to git submodules: a script > that clones the repo No need to clone the whole repo. We can simply download the binaries that the girlab CI job would generate from the bits sources in that repo. We need to clone if we are always building bits from source for every test. That is not necessary IMHO since much of the bits package would remain as is without modification. > - new target make check-roms, I think make pytest or some such is better and more generic if other such tests in other areas follow suit. if the clone exists locally - > run the test, if not - skip it if download of the bits binaries fail, skip it. > - as for using pre-generates ISOs as an optimization, > I'm not sure how important that is, if yes - > we can add another repo and another make target along the > same lines > > > > -- > MST >
On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote: > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > > > I even wouldn't mind if you put your python stuff in a new directory like > > > tests/pytests/ for example, as long as it downloads your binaries separately > > > - as I wrote in another mail, the avocado framework rather looks like an > > > oddball in our test framework nowadays since it uses a separate test runner > > > and not the meson test harness, so having a new approach for python-based > > > tests is maybe even a good idea. I just really want to avoid that this goes > > > into tests/qtest (since it really does not belong there), and please don't > > > add more external stuff via git submodules, that's really the wrong approach > > > for this. > > > > I get it, people hate submodules with passion. I think trying another > > approach for testing that is neither avocado nor qtest is > > not too bad. As long as this is not user visible, we can > > allow ourselves space to experiment. > > > > OK so, how about this: > > - put it in a new directory: tests/roms? > > - create repo for a fork of biosbits under git.qemu.org > > - roll our own analog to git submodules: a script > > that clones the repo > > No need to clone the whole repo. We can simply download the binaries > that the girlab CI job would generate from the bits sources in that > repo. > We need to clone if we are always building bits from source for every > test. That is not necessary IMHO since much of the bits package would > remain as is without modification. IMHO CI job idea isn't great since isn't versioned at all, is it? Also as long as test passes, sure. But if it fails one will need the sources to investigate. Let's start with building things from source. Add an option of prebuilt binaries as an optimization once things stabilize. > > - new target make check-roms, > > I think make pytest or some such is better and more generic if other > such tests in other areas follow suit. The name is not critical in my mind, but I think we need to decide what exactly differentiates it from other tests. > > if the clone exists locally - > > run the test, if not - skip it > > if download of the bits binaries fail, skip it. You seem to be recreating either git or avocado or both here. Personally I want something that works offline. > > - as for using pre-generates ISOs as an optimization, > > I'm not sure how important that is, if yes - > > we can add another repo and another make target along the > > same lines > > > > > > > > -- > > MST > >
On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote: > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > > > > I even wouldn't mind if you put your python stuff in a new directory like > > > > tests/pytests/ for example, as long as it downloads your binaries separately > > > > - as I wrote in another mail, the avocado framework rather looks like an > > > > oddball in our test framework nowadays since it uses a separate test runner > > > > and not the meson test harness, so having a new approach for python-based > > > > tests is maybe even a good idea. I just really want to avoid that this goes > > > > into tests/qtest (since it really does not belong there), and please don't > > > > add more external stuff via git submodules, that's really the wrong approach > > > > for this. > > > > > > I get it, people hate submodules with passion. I think trying another > > > approach for testing that is neither avocado nor qtest is > > > not too bad. As long as this is not user visible, we can > > > allow ourselves space to experiment. > > > > > > OK so, how about this: > > > - put it in a new directory: tests/roms? > > > - create repo for a fork of biosbits under git.qemu.org > > > - roll our own analog to git submodules: a script > > > that clones the repo > > > > No need to clone the whole repo. We can simply download the binaries > > that the girlab CI job would generate from the bits sources in that > > repo. > > We need to clone if we are always building bits from source for every > > test. That is not necessary IMHO since much of the bits package would > > remain as is without modification. > > IMHO CI job idea isn't great since isn't versioned at all, is it? bits is versioned yes, in a crude way. every time you make a commit in the top level repo, the version would increment by one. > Also as long as test passes, sure. But if it fails one will > need the sources to investigate. sources might also be needed to write the tests. > > Let's start with building things from source. hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to build depending on parallelity and build host. Add an option > of prebuilt binaries as an optimization once things > stabilize. > > > > > - new target make check-roms, > > > > I think make pytest or some such is better and more generic if other > > such tests in other areas follow suit. > > The name is not critical in my mind, but I think we need to decide > what exactly differentiates it from other tests. > > > > > > if the clone exists locally - > > > run the test, if not - skip it > > > > if download of the bits binaries fail, skip it. > > You seem to be recreating either git or avocado or both here. > > Personally I want something that works offline. > > > > > > - as for using pre-generates ISOs as an optimization, > > > I'm not sure how important that is, if yes - > > > we can add another repo and another make target along the > > > same lines > > > > > > > > > > > > -- > > > MST > > > >
On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote: > On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote: > > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > > > > > I even wouldn't mind if you put your python stuff in a new directory like > > > > > tests/pytests/ for example, as long as it downloads your binaries separately > > > > > - as I wrote in another mail, the avocado framework rather looks like an > > > > > oddball in our test framework nowadays since it uses a separate test runner > > > > > and not the meson test harness, so having a new approach for python-based > > > > > tests is maybe even a good idea. I just really want to avoid that this goes > > > > > into tests/qtest (since it really does not belong there), and please don't > > > > > add more external stuff via git submodules, that's really the wrong approach > > > > > for this. > > > > > > > > I get it, people hate submodules with passion. I think trying another > > > > approach for testing that is neither avocado nor qtest is > > > > not too bad. As long as this is not user visible, we can > > > > allow ourselves space to experiment. > > > > > > > > OK so, how about this: > > > > - put it in a new directory: tests/roms? > > > > - create repo for a fork of biosbits under git.qemu.org > > > > - roll our own analog to git submodules: a script > > > > that clones the repo > > > > > > No need to clone the whole repo. We can simply download the binaries > > > that the girlab CI job would generate from the bits sources in that > > > repo. > > > We need to clone if we are always building bits from source for every > > > test. That is not necessary IMHO since much of the bits package would > > > remain as is without modification. > > > > IMHO CI job idea isn't great since isn't versioned at all, is it? > > bits is versioned yes, in a crude way. every time you make a commit in > the top level repo, the version would increment by one. Is it easy to find out which source was this generated from? And is there a promise to keep these around indefinitely? > > Also as long as test passes, sure. But if it fails one will > > need the sources to investigate. > > sources might also be needed to write the tests. > > > > > Let's start with building things from source. > > hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to > build depending on parallelity and build host. Right. But whoever wants to use these just needs to do it once. > Add an option > > of prebuilt binaries as an optimization once things > > stabilize. > > > > > > > > - new target make check-roms, > > > > > > I think make pytest or some such is better and more generic if other > > > such tests in other areas follow suit. > > > > The name is not critical in my mind, but I think we need to decide > > what exactly differentiates it from other tests. > > > > > > > > > > if the clone exists locally - > > > > run the test, if not - skip it > > > > > > if download of the bits binaries fail, skip it. > > > > You seem to be recreating either git or avocado or both here. > > > > Personally I want something that works offline. > > > > > > > > > > - as for using pre-generates ISOs as an optimization, > > > > I'm not sure how important that is, if yes - > > > > we can add another repo and another make target along the > > > > same lines > > > > > > > > > > > > > > > > -- > > > > MST > > > > > >
On Fri, Jul 1, 2022 at 3:12 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote: > > On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote: > > > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > > > > > > I even wouldn't mind if you put your python stuff in a new directory like > > > > > > tests/pytests/ for example, as long as it downloads your binaries separately > > > > > > - as I wrote in another mail, the avocado framework rather looks like an > > > > > > oddball in our test framework nowadays since it uses a separate test runner > > > > > > and not the meson test harness, so having a new approach for python-based > > > > > > tests is maybe even a good idea. I just really want to avoid that this goes > > > > > > into tests/qtest (since it really does not belong there), and please don't > > > > > > add more external stuff via git submodules, that's really the wrong approach > > > > > > for this. > > > > > > > > > > I get it, people hate submodules with passion. I think trying another > > > > > approach for testing that is neither avocado nor qtest is > > > > > not too bad. As long as this is not user visible, we can > > > > > allow ourselves space to experiment. > > > > > > > > > > OK so, how about this: > > > > > - put it in a new directory: tests/roms? > > > > > - create repo for a fork of biosbits under git.qemu.org > > > > > - roll our own analog to git submodules: a script > > > > > that clones the repo > > > > > > > > No need to clone the whole repo. We can simply download the binaries > > > > that the girlab CI job would generate from the bits sources in that > > > > repo. > > > > We need to clone if we are always building bits from source for every > > > > test. That is not necessary IMHO since much of the bits package would > > > > remain as is without modification. > > > > > > IMHO CI job idea isn't great since isn't versioned at all, is it? > > > > bits is versioned yes, in a crude way. every time you make a commit in > > the top level repo, the version would increment by one. > > Is it easy to find out which source was this generated from? yes because this is how the version number is generated: expr 2000 + `git rev-list HEAD 2>/dev/null | wc -l` That means we simply count the number of commits in the top level directory. If we overwrote history by force pushing, then well yes all bets are off. > And is there a promise to keep these around indefinitely? We do not need to maintain multiple versions of bits or increment the version number every time we wrote a test. Typically new commits/changes will be made if we wanted to fix a bug or upgrade acpica or whatever. Test scripts are python based and can be pushed into bits while generating a new iso for testing purposes. > > > > Also as long as test passes, sure. But if it fails one will > > > need the sources to investigate. > > > > sources might also be needed to write the tests. > > > > > > > > Let's start with building things from source. > > > > hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to > > build depending on parallelity and build host. > > Right. But whoever wants to use these just needs to do it once. but I thought you were suggesting we built bits every time the test is run? > > > > Add an option > > > of prebuilt binaries as an optimization once things > > > stabilize. > > > > > > > > > > > - new target make check-roms, > > > > > > > > I think make pytest or some such is better and more generic if other > > > > such tests in other areas follow suit. > > > > > > The name is not critical in my mind, but I think we need to decide > > > what exactly differentiates it from other tests. > > > > > > > > > > > > > > if the clone exists locally - > > > > > run the test, if not - skip it > > > > > > > > if download of the bits binaries fail, skip it. > > > > > > You seem to be recreating either git or avocado or both here. > > > > > > Personally I want something that works offline. > > > > > > > > > > > > > > - as for using pre-generates ISOs as an optimization, > > > > > I'm not sure how important that is, if yes - > > > > > we can add another repo and another make target along the > > > > > same lines > > > > > > > > > > > > > > > > > > > > -- > > > > > MST > > > > > > > > >
On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> but I thought you were suggesting we built bits every time the test is run?
In my opinion 3 scenarios are worth supporting:
- people not touching ACPI, including users - simply don't run the tests,
comparing tables with expected output should be enough
- people making changes indirectly affecting ACPI -
use tests to validate that tables are still well formed,
using a pre built binary should be enough IMO
- people working on ACPI - use tests to analyse the tables,
building from source might be necessary for debugging,
sources change very rarely
- people developing the tests
building from source is required
So I would suggest basically two github repos, one with binaries one with
sources. We'll keep relevant hashes to use in the script.
All in all not that different from submodules but I guess
people have submodules and that is that.
And I personally would probably not tie it to CI whoever owns the
repository can worry about the builds, and I think keeping
things distributed is important.
So
- people not touching ACPI - make check should see directory not found
and skip the test
- people making changes indirectly affecting ACPI -
check out binaries and use
- people working on ACPI -
see that source directory is present, go there
and run make. should not rebuild each time right?
- people developing the tests
building from source is required
On Fri, Jul 1, 2022 at 5:37 AM Thomas Huth <thuth@redhat.com> wrote: > On 29/06/2022 08.28, Ani Sinha wrote: > > On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote: > >>> FYI, the reason much of this is intentionally NOT under the /qemu-project > >>> gitlab namespace is that we did not want to be responsible for distributing > >>> arbitrary binary blobs/images. That in turn makes the QEMU project responsible > >>> for license compliance, which is non-trivial todo correctly for much of this > >>> stuff. As such it is highly desirable to delegate both the hosting the > >>> binaries and source to the third party who builds it. > >> > >> This might be understadable for random guest OS images which include tons of stuff > >> and are thus hard to audit. But not to biosbits which has its own > >> license (more or less bsd) + gpl for grub: > >> https://github.com/biosbits/bits/blob/master/COPYING > > > > These are all the dependencies: > > https://github.com/biosbits/bits/tree/master/deps > > > > We can go through the licenses for each and make a determination. The > > audit would be lost easier because there is a bounded number of > > dependencies for bits. > > That's quite a bit of dependencies already ... I don't think that we should > put the burden of keeping up with the licenses of those projects to the QEMU > project. So just make sure that the binaries are available somewhere and > then include your test into the avocado framework or download via curl/wget > as suggested in other mails. "available somewhere" until they get moved or deleted... This happened a few times with archives/artifacts Avocado tests fetch. We ended up committing one or a set of URLs pointing to a file, its usual name and its hash. If you downloaded the file in time and don't flush the Avocado cache, you can keep testing. If it is too late (the file disappeared) and you have time to dig for such a file online, and are lucky because you found it and its hash matches, then you can run some tests. As a maintainer I find this feature useful, and find it worse to lose the ability to test something that used to work. My 2¢... Phil.
On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote: > > but I thought you were suggesting we built bits every time the test is run? > > In my opinion 3 scenarios are worth supporting: > - people not touching ACPI, including users - simply don't run the tests, > comparing tables with expected output should be enough > - people making changes indirectly affecting ACPI - > use tests to validate that tables are still well formed, > using a pre built binary should be enough IMO > - people working on ACPI - use tests to analyse the tables, > building from source might be necessary for debugging, > sources change very rarely > - people developing the tests > building from source is required > > So I would suggest basically two github repos, one with binaries one with > sources. We'll keep relevant hashes to use in the script. > All in all not that different from submodules but I guess > people have submodules and that is that. > > And I personally would probably not tie it to CI whoever owns the > repository can worry about the builds, and I think keeping > things distributed is important. > > So > - people not touching ACPI - make check should see directory not found > and skip the test > - people making changes indirectly affecting ACPI - > check out binaries and use > - people working on ACPI - > see that source directory is present, go there > and run make. should not rebuild each time right? > - people developing the tests > building from source is required Ok I have now committed a Dockerfile that has all the build time dependencies and builds bits and generates the tarballs/zip file that my test requires: https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile We just need to fork the repo and generate automated builds with this Dockerfile and put the binaries somewhere. This should also help developers requiring to rebuild bits when necessary. Oh and btw, I also made bits compliant with the latest gcc 11 compiler that comes with Ubuntu 22.04 LTS :-)
On Mon, Jul 4, 2022 at 7:02 PM Ani Sinha <ani@anisinha.ca> wrote: > > On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote: > > > but I thought you were suggesting we built bits every time the test is run? > > > > In my opinion 3 scenarios are worth supporting: > > - people not touching ACPI, including users - simply don't run the tests, > > comparing tables with expected output should be enough > > - people making changes indirectly affecting ACPI - > > use tests to validate that tables are still well formed, > > using a pre built binary should be enough IMO > > - people working on ACPI - use tests to analyse the tables, > > building from source might be necessary for debugging, > > sources change very rarely > > - people developing the tests > > building from source is required > > > > So I would suggest basically two github repos, one with binaries one with > > sources. We'll keep relevant hashes to use in the script. > > All in all not that different from submodules but I guess > > people have submodules and that is that. > > > > And I personally would probably not tie it to CI whoever owns the > > repository can worry about the builds, and I think keeping > > things distributed is important. > > > > So > > - people not touching ACPI - make check should see directory not found > > and skip the test > > - people making changes indirectly affecting ACPI - > > check out binaries and use > > - people working on ACPI - > > see that source directory is present, go there > > and run make. should not rebuild each time right? > > - people developing the tests > > building from source is required > > Ok I have now committed a Dockerfile that has all the build time > dependencies and builds bits and generates the tarballs/zip file that > my test requires: > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > We just need to fork the repo and generate automated builds with this > Dockerfile and put the binaries somewhere. This should also help > developers requiring to rebuild bits when necessary. > Oh and btw, I also made bits compliant with the latest gcc 11 compiler > that comes with Ubuntu 22.04 LTS :-) Pushed a build script here: https://github.com/ani-sinha/bits/commit/90b99ef05d55ead4b33b2fb19ad07dfb9682ec92 and the bios bits binaries are in this branch: https://github.com/ani-sinha/bits/tree/bits-builds
On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote: > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote: > > > > I even wouldn't mind if you put your python stuff in a new directory like > > > > tests/pytests/ for example, as long as it downloads your binaries separately > > > > - as I wrote in another mail, the avocado framework rather looks like an > > > > oddball in our test framework nowadays since it uses a separate test runner > > > > and not the meson test harness, so having a new approach for python-based > > > > tests is maybe even a good idea. I just really want to avoid that this goes > > > > into tests/qtest (since it really does not belong there), and please don't > > > > add more external stuff via git submodules, that's really the wrong approach > > > > for this. > > > > > > I get it, people hate submodules with passion. I think trying another > > > approach for testing that is neither avocado nor qtest is > > > not too bad. As long as this is not user visible, we can > > > allow ourselves space to experiment. > > > > > > OK so, how about this: > > > - put it in a new directory: tests/roms? > > > - create repo for a fork of biosbits under git.qemu.org > > > - roll our own analog to git submodules: a script > > > that clones the repo > > > > No need to clone the whole repo. We can simply download the binaries > > that the girlab CI job would generate from the bits sources in that > > repo. > > We need to clone if we are always building bits from source for every > > test. That is not necessary IMHO since much of the bits package would > > remain as is without modification. > > IMHO CI job idea isn't great since isn't versioned at all, is it? > Also as long as test passes, sure. But if it fails one will > need the sources to investigate. > > Let's start with building things from source. Add an option > of prebuilt binaries as an optimization once things > stabilize. > > > > > - new target make check-roms, > > > > I think make pytest or some such is better and more generic if other > > such tests in other areas follow suit. > > The name is not critical in my mind, but I think we need to decide > what exactly differentiates it from other tests. > > > > > > if the clone exists locally - > > > run the test, if not - skip it > > > > if download of the bits binaries fail, skip it. > > You seem to be recreating either git or avocado or both here. > > Personally I want something that works offline. Well we need to make a call as to how to make things work offline, if that's what we wanted. I have made changes in v2 to download the binaries from my github repo. It works great. If download fails, it skips the test. If those files are available before, it will not attempt to download them. https://pastebin.com/nrcdN8WS and https://pastebin.com/m7Dx3PZk The Dockerfile and build script I have in the repo will generate these artifacts. So if you wanted to run the test offline, either build those from source and place them where the test expects to find them or build them from source using Dockerfile/build script and place them there before running the test. > > > > > > - as for using pre-generates ISOs as an optimization, > > > I'm not sure how important that is, if yes - > > > we can add another repo and another make target along the > > > same lines > > > > > > > > > > > > -- > > > MST > > > >
On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > [...] > > For biosbits if we are going this route then I feel a submodule is much > > better. It records which version exactly each qemu version wants. > > As far as I know, you can also specify the version when using pip, can't > you? So that's not really an advantage here. > > On the contrary, submodules have a couple of disadvantages that I really > dislike: > > - submodules do not get updated automatically when doing a "git checkout", > we have to update them via a script instead. This causes e.g. trouble if you > rsync your source tree to a machine that has no access to the internet and > you forgot to update the submodule before the sync > > - the content of submodules is not added to the tarballs that get created on > the git forges automatically. There were lots of requests from users in the > past that tried to download a tarball from github and then wondered why they > couldn't compile QEMU. > > - we include the submodule content in our release tarballs, so people get > the impression that hte submodule content is part of the QEMU sources. This > has two disadvantages: > * We already got bug reports for the code in the submodule, > where people did not understand that they should report that > rather to the original project instead (i.e. you ship it - you > own it) > * People get the impression that QEMU is a huge monster > application if they count the number of code lines, run > their code scanner tools on the tarball contents, etc. > Remember "nemu", for example, where one of the main complaints > was that QEMU has too many lines of code? > > - If programs includes code via submodules, this gets a higher > burder for distro maintainers, since they have to patch each > and every package when there is a bug, instead of being able to > fix it in one central place. > > So in my opinion we should avoid new submodules if there is an alternative. > > Thomas So looking at the latest proposals downloading files from CI, checksumming them etc etc. No auto checkout, not added automatically either, right? This seems to be the only difference: - we include the submodule content in our release tarballs How about we just fix that? Thomas would that address your concern at least wrt tests?
On 28/09/2022 11.26, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: >> On 28/06/2022 12.03, Michael S. Tsirkin wrote: >> [...] >>> For biosbits if we are going this route then I feel a submodule is much >>> better. It records which version exactly each qemu version wants. >> >> As far as I know, you can also specify the version when using pip, can't >> you? So that's not really an advantage here. >> >> On the contrary, submodules have a couple of disadvantages that I really >> dislike: >> >> - submodules do not get updated automatically when doing a "git checkout", >> we have to update them via a script instead. This causes e.g. trouble if you >> rsync your source tree to a machine that has no access to the internet and >> you forgot to update the submodule before the sync >> >> - the content of submodules is not added to the tarballs that get created on >> the git forges automatically. There were lots of requests from users in the >> past that tried to download a tarball from github and then wondered why they >> couldn't compile QEMU. >> >> - we include the submodule content in our release tarballs, so people get >> the impression that hte submodule content is part of the QEMU sources. This >> has two disadvantages: >> * We already got bug reports for the code in the submodule, >> where people did not understand that they should report that >> rather to the original project instead (i.e. you ship it - you >> own it) >> * People get the impression that QEMU is a huge monster >> application if they count the number of code lines, run >> their code scanner tools on the tarball contents, etc. >> Remember "nemu", for example, where one of the main complaints >> was that QEMU has too many lines of code? >> >> - If programs includes code via submodules, this gets a higher >> burder for distro maintainers, since they have to patch each >> and every package when there is a bug, instead of being able to >> fix it in one central place. >> >> So in my opinion we should avoid new submodules if there is an alternative. >> >> Thomas > > So looking at the latest proposals downloading files from CI, > checksumming them etc etc. No auto checkout, not added automatically > either, right? > > This seems to be the only difference: > - we include the submodule content in our release tarballs > > How about we just fix that? Thomas would that address your > concern at least wrt tests? If I'm not forced to checkout that submodule, and if we don't add it to the release tarball, I guess there's not much left I can complain about ;-) Thomas
On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote: > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > [...] > > > For biosbits if we are going this route then I feel a submodule is much > > > better. It records which version exactly each qemu version wants. > > > > As far as I know, you can also specify the version when using pip, can't > > you? So that's not really an advantage here. > > > > On the contrary, submodules have a couple of disadvantages that I really > > dislike: > > > > - submodules do not get updated automatically when doing a "git checkout", > > we have to update them via a script instead. This causes e.g. trouble if you > > rsync your source tree to a machine that has no access to the internet and > > you forgot to update the submodule before the sync > > > > - the content of submodules is not added to the tarballs that get created on > > the git forges automatically. There were lots of requests from users in the > > past that tried to download a tarball from github and then wondered why they > > couldn't compile QEMU. > > > > - we include the submodule content in our release tarballs, so people get > > the impression that hte submodule content is part of the QEMU sources. This > > has two disadvantages: > > * We already got bug reports for the code in the submodule, > > where people did not understand that they should report that > > rather to the original project instead (i.e. you ship it - you > > own it) > > * People get the impression that QEMU is a huge monster > > application if they count the number of code lines, run > > their code scanner tools on the tarball contents, etc. > > Remember "nemu", for example, where one of the main complaints > > was that QEMU has too many lines of code? > > > > - If programs includes code via submodules, this gets a higher > > burder for distro maintainers, since they have to patch each > > and every package when there is a bug, instead of being able to > > fix it in one central place. > > > > So in my opinion we should avoid new submodules if there is an alternative. > > > > Thomas > > So looking at the latest proposals downloading files from CI, > checksumming them etc etc. No auto checkout, not added automatically > either, right? > > This seems to be the only difference: > - we include the submodule content in our release tarballs That's just one of the issues with submodules. Working with them in general is not a pleasant experiance. Thomas pointed out some of the issues, such as 'git checkout' ignoring submodules, requiring extra steps to sync them. The flipside on tarballs is that the auto-generated tarballs from gitlab do not include submodules, so the tests will be missing the files they wanted. There's also the perenial problem that developers frequently send patches that mistakenly include submodule changes, which is related to the way that 'git checkout' doesn't sync submodule state when switching branches. I'd really like to see us doing more to eliminate as much use of submodules as is possible over time. With regards, Daniel
On Wed, Sep 28, 2022 at 11:33:52AM +0200, Thomas Huth wrote: > On 28/09/2022 11.26, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > [...] > > > > For biosbits if we are going this route then I feel a submodule is much > > > > better. It records which version exactly each qemu version wants. > > > > > > As far as I know, you can also specify the version when using pip, can't > > > you? So that's not really an advantage here. > > > > > > On the contrary, submodules have a couple of disadvantages that I really > > > dislike: > > > > > > - submodules do not get updated automatically when doing a "git checkout", > > > we have to update them via a script instead. This causes e.g. trouble if you > > > rsync your source tree to a machine that has no access to the internet and > > > you forgot to update the submodule before the sync > > > > > > - the content of submodules is not added to the tarballs that get created on > > > the git forges automatically. There were lots of requests from users in the > > > past that tried to download a tarball from github and then wondered why they > > > couldn't compile QEMU. > > > > > > - we include the submodule content in our release tarballs, so people get > > > the impression that hte submodule content is part of the QEMU sources. This > > > has two disadvantages: > > > * We already got bug reports for the code in the submodule, > > > where people did not understand that they should report that > > > rather to the original project instead (i.e. you ship it - you > > > own it) > > > * People get the impression that QEMU is a huge monster > > > application if they count the number of code lines, run > > > their code scanner tools on the tarball contents, etc. > > > Remember "nemu", for example, where one of the main complaints > > > was that QEMU has too many lines of code? > > > > > > - If programs includes code via submodules, this gets a higher > > > burder for distro maintainers, since they have to patch each > > > and every package when there is a bug, instead of being able to > > > fix it in one central place. > > > > > > So in my opinion we should avoid new submodules if there is an alternative. > > > > > > Thomas > > > > So looking at the latest proposals downloading files from CI, > > checksumming them etc etc. No auto checkout, not added automatically > > either, right? > > > > This seems to be the only difference: > > - we include the submodule content in our release tarballs > > > > How about we just fix that? Thomas would that address your > > concern at least wrt tests? > > If I'm not forced to checkout that submodule, I think the make check script can do that? > and if we don't add it to the > release tarball, I guess there's not much left I can complain about ;-) > > Thomas
On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote: > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > [...] > > > > For biosbits if we are going this route then I feel a submodule is much > > > > better. It records which version exactly each qemu version wants. > > > > > > As far as I know, you can also specify the version when using pip, can't > > > you? So that's not really an advantage here. > > > > > > On the contrary, submodules have a couple of disadvantages that I really > > > dislike: > > > > > > - submodules do not get updated automatically when doing a "git checkout", > > > we have to update them via a script instead. This causes e.g. trouble if you > > > rsync your source tree to a machine that has no access to the internet and > > > you forgot to update the submodule before the sync > > > > > > - the content of submodules is not added to the tarballs that get created on > > > the git forges automatically. There were lots of requests from users in the > > > past that tried to download a tarball from github and then wondered why they > > > couldn't compile QEMU. > > > > > > - we include the submodule content in our release tarballs, so people get > > > the impression that hte submodule content is part of the QEMU sources. This > > > has two disadvantages: > > > * We already got bug reports for the code in the submodule, > > > where people did not understand that they should report that > > > rather to the original project instead (i.e. you ship it - you > > > own it) > > > * People get the impression that QEMU is a huge monster > > > application if they count the number of code lines, run > > > their code scanner tools on the tarball contents, etc. > > > Remember "nemu", for example, where one of the main complaints > > > was that QEMU has too many lines of code? > > > > > > - If programs includes code via submodules, this gets a higher > > > burder for distro maintainers, since they have to patch each > > > and every package when there is a bug, instead of being able to > > > fix it in one central place. > > > > > > So in my opinion we should avoid new submodules if there is an alternative. > > > > > > Thomas > > > > So looking at the latest proposals downloading files from CI, > > checksumming them etc etc. No auto checkout, not added automatically > > either, right? > > > > This seems to be the only difference: > > - we include the submodule content in our release tarballs > > That's just one of the issues with submodules. Working with them in general > is not a pleasant experiance. This is what I asked about at the maintainers summit. I'd like to map the issues and see if there's anything we can do to solve them. In particular we will likely keep using submodules where we historically did so it's time well spent. I agree generally but the big question is what to replace these with. Below I assume the replacement is a script such as avocado or pytest with its own hashing, calling wget internally etc etc. > Thomas pointed out some of the issues, such > as 'git checkout' ignoring submodules, requiring extra steps to sync them. Not different from a home grown SCM as part of test script, right? > The flipside on tarballs is that the auto-generated tarballs from gitlab > do not include submodules, so the tests will be missing the files they > wanted. Not different from a home grown SCM as part of test script, right? > There's also the perenial problem that developers frequently send > patches that mistakenly include submodule changes, OK, so the thing to do would be to look for ways to exclude submodule changes from git commits. > which is related to the > way that 'git checkout' doesn't sync submodule state when switching branches. Not different from a home grown SCM as part of test script, right? > I'd really like to see us doing more to eliminate as much use of submodules > as is possible over time. Or try to fix the problems, right? > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 28/09/2022 11.47, Michael S. Tsirkin wrote: > On Wed, Sep 28, 2022 at 11:33:52AM +0200, Thomas Huth wrote: >> On 28/09/2022 11.26, Michael S. Tsirkin wrote: >>> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: >>>> On 28/06/2022 12.03, Michael S. Tsirkin wrote: >>>> [...] >>>>> For biosbits if we are going this route then I feel a submodule is much >>>>> better. It records which version exactly each qemu version wants. >>>> >>>> As far as I know, you can also specify the version when using pip, can't >>>> you? So that's not really an advantage here. >>>> >>>> On the contrary, submodules have a couple of disadvantages that I really >>>> dislike: >>>> >>>> - submodules do not get updated automatically when doing a "git checkout", >>>> we have to update them via a script instead. This causes e.g. trouble if you >>>> rsync your source tree to a machine that has no access to the internet and >>>> you forgot to update the submodule before the sync >>>> >>>> - the content of submodules is not added to the tarballs that get created on >>>> the git forges automatically. There were lots of requests from users in the >>>> past that tried to download a tarball from github and then wondered why they >>>> couldn't compile QEMU. >>>> >>>> - we include the submodule content in our release tarballs, so people get >>>> the impression that hte submodule content is part of the QEMU sources. This >>>> has two disadvantages: >>>> * We already got bug reports for the code in the submodule, >>>> where people did not understand that they should report that >>>> rather to the original project instead (i.e. you ship it - you >>>> own it) >>>> * People get the impression that QEMU is a huge monster >>>> application if they count the number of code lines, run >>>> their code scanner tools on the tarball contents, etc. >>>> Remember "nemu", for example, where one of the main complaints >>>> was that QEMU has too many lines of code? >>>> >>>> - If programs includes code via submodules, this gets a higher >>>> burder for distro maintainers, since they have to patch each >>>> and every package when there is a bug, instead of being able to >>>> fix it in one central place. >>>> >>>> So in my opinion we should avoid new submodules if there is an alternative. >>>> >>>> Thomas >>> >>> So looking at the latest proposals downloading files from CI, >>> checksumming them etc etc. No auto checkout, not added automatically >>> either, right? >>> >>> This seems to be the only difference: >>> - we include the submodule content in our release tarballs >>> >>> How about we just fix that? Thomas would that address your >>> concern at least wrt tests? >> >> If I'm not forced to checkout that submodule, > > I think the make check script can do that? Yes, currently there are only some submodules that are always checked out: dtc, meson, tests/fp/berkeley-softfloat-3, tests/fp/berkeley-testfloat-3 and ui/keycodemapdb. And for example the ones in the roms/ directory are not checked out by default. Thomas
On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote: > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote: > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > > [...] > > > > > For biosbits if we are going this route then I feel a submodule is much > > > > > better. It records which version exactly each qemu version wants. > > > > > > > > As far as I know, you can also specify the version when using pip, can't > > > > you? So that's not really an advantage here. > > > > > > > > On the contrary, submodules have a couple of disadvantages that I really > > > > dislike: > > > > > > > > - submodules do not get updated automatically when doing a "git checkout", > > > > we have to update them via a script instead. This causes e.g. trouble if you > > > > rsync your source tree to a machine that has no access to the internet and > > > > you forgot to update the submodule before the sync > > > > > > > > - the content of submodules is not added to the tarballs that get created on > > > > the git forges automatically. There were lots of requests from users in the > > > > past that tried to download a tarball from github and then wondered why they > > > > couldn't compile QEMU. > > > > > > > > - we include the submodule content in our release tarballs, so people get > > > > the impression that hte submodule content is part of the QEMU sources. This > > > > has two disadvantages: > > > > * We already got bug reports for the code in the submodule, > > > > where people did not understand that they should report that > > > > rather to the original project instead (i.e. you ship it - you > > > > own it) > > > > * People get the impression that QEMU is a huge monster > > > > application if they count the number of code lines, run > > > > their code scanner tools on the tarball contents, etc. > > > > Remember "nemu", for example, where one of the main complaints > > > > was that QEMU has too many lines of code? > > > > > > > > - If programs includes code via submodules, this gets a higher > > > > burder for distro maintainers, since they have to patch each > > > > and every package when there is a bug, instead of being able to > > > > fix it in one central place. > > > > > > > > So in my opinion we should avoid new submodules if there is an alternative. > > > > > > > > Thomas > > > > > > So looking at the latest proposals downloading files from CI, > > > checksumming them etc etc. No auto checkout, not added automatically > > > either, right? > > > > > > This seems to be the only difference: > > > - we include the submodule content in our release tarballs > > > > That's just one of the issues with submodules. Working with them in general > > is not a pleasant experiance. > > This is what I asked about at the maintainers summit. > I'd like to map the issues and see if there's anything > we can do to solve them. In particular we will likely > keep using submodules where we historically did > so it's time well spent. > > I agree generally but the big question is what to replace these with. > Below I assume the replacement is a script such as avocado or pytest > with its own hashing, calling wget internally etc etc. > > > > Thomas pointed out some of the issues, such > > as 'git checkout' ignoring submodules, requiring extra steps to sync them. > > > Not different from a home grown SCM as part of test script, right? We're not building a home grown SCM as part of a test script, so this answer is irrelevant. > > There's also the perenial problem that developers frequently send > > patches that mistakenly include submodule changes, > > OK, so the thing to do would be to look for ways to exclude submodule changes > from git commits. If someone wants to make git suck less with submodules great, but needs someone to actually do the work. > > I'd really like to see us doing more to eliminate as much use of submodules > > as is possible over time. > > Or try to fix the problems, right? Again needs someone to actually make it happen. Meanwhile QEMU already has an integrated test harness in the form of Avocado that does everything needed. If Avocado had just been used for this biosbits test in the first place, the test would likely have already been merged to QEMU, instead of us having this never ending debate on how to re-invent an alternative to what already avocado does. With regards, Daniel
On Wed, Sep 28, 2022 at 10:57:47AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote: > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote: > > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > > > [...] > > > > > > For biosbits if we are going this route then I feel a submodule is much > > > > > > better. It records which version exactly each qemu version wants. > > > > > > > > > > As far as I know, you can also specify the version when using pip, can't > > > > > you? So that's not really an advantage here. > > > > > > > > > > On the contrary, submodules have a couple of disadvantages that I really > > > > > dislike: > > > > > > > > > > - submodules do not get updated automatically when doing a "git checkout", > > > > > we have to update them via a script instead. This causes e.g. trouble if you > > > > > rsync your source tree to a machine that has no access to the internet and > > > > > you forgot to update the submodule before the sync > > > > > > > > > > - the content of submodules is not added to the tarballs that get created on > > > > > the git forges automatically. There were lots of requests from users in the > > > > > past that tried to download a tarball from github and then wondered why they > > > > > couldn't compile QEMU. > > > > > > > > > > - we include the submodule content in our release tarballs, so people get > > > > > the impression that hte submodule content is part of the QEMU sources. This > > > > > has two disadvantages: > > > > > * We already got bug reports for the code in the submodule, > > > > > where people did not understand that they should report that > > > > > rather to the original project instead (i.e. you ship it - you > > > > > own it) > > > > > * People get the impression that QEMU is a huge monster > > > > > application if they count the number of code lines, run > > > > > their code scanner tools on the tarball contents, etc. > > > > > Remember "nemu", for example, where one of the main complaints > > > > > was that QEMU has too many lines of code? > > > > > > > > > > - If programs includes code via submodules, this gets a higher > > > > > burder for distro maintainers, since they have to patch each > > > > > and every package when there is a bug, instead of being able to > > > > > fix it in one central place. > > > > > > > > > > So in my opinion we should avoid new submodules if there is an alternative. > > > > > > > > > > Thomas > > > > > > > > So looking at the latest proposals downloading files from CI, > > > > checksumming them etc etc. No auto checkout, not added automatically > > > > either, right? > > > > > > > > This seems to be the only difference: > > > > - we include the submodule content in our release tarballs > > > > > > That's just one of the issues with submodules. Working with them in general > > > is not a pleasant experiance. > > > > This is what I asked about at the maintainers summit. > > I'd like to map the issues and see if there's anything > > we can do to solve them. In particular we will likely > > keep using submodules where we historically did > > so it's time well spent. > > > > I agree generally but the big question is what to replace these with. > > Below I assume the replacement is a script such as avocado or pytest > > with its own hashing, calling wget internally etc etc. > > > > > > > Thomas pointed out some of the issues, such > > > as 'git checkout' ignoring submodules, requiring extra steps to sync them. > > > > > > Not different from a home grown SCM as part of test script, right? > > We're not building a home grown SCM as part of a test script, so > this answer is irrelevant. If you are managing contents of files using code then you have a content managing system :) > > > There's also the perenial problem that developers frequently send > > > patches that mistakenly include submodule changes, > > > > OK, so the thing to do would be to look for ways to exclude submodule changes > > from git commits. > > If someone wants to make git suck less with submodules great, but needs > someone to actually do the work. I have some experience with git, might be able to if I can figure out what the issues are. Or maybe there's already a flag to do what we want git is pretty configurable. > > > I'd really like to see us doing more to eliminate as much use of submodules > > > as is possible over time. > > > > Or try to fix the problems, right? > > Again needs someone to actually make it happen. > > Meanwhile QEMU already has an integrated test harness in the form > of Avocado that does everything needed. If Avocado had just been > used for this biosbits test in the first place, the test would > likely have already been merged to QEMU, instead of us having this > never ending debate on how to re-invent an alternative to what > already avocado does. > > With regards, > Daniel I just think there's a ton of difference between linux guest images which we want to keep running indefinitely and a unit test firmware which we want to version with qemu. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > There's also the perenial problem that developers frequently send > patches that mistakenly include submodule changes, which is related to the > way that 'git checkout' doesn't sync submodule state when switching branches. Do you happen to know how exactly that happens?
On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote: > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > There's also the perenial problem that developers frequently send > > patches that mistakenly include submodule changes, which is related to the > > way that 'git checkout' doesn't sync submodule state when switching branches. > > Do you happen to know how exactly that happens? For any given branch the submodule is synced to a given git commit hash. If the submodule checkout is not synced to the same commit hash it will show as dirty, and if you git add this pending change, it'll record that new submodule commit hash. Seeing dirty state is common when you switch between branches, either side of a git master change that updated a submodule. With regards, Daniel
On Wed, Sep 28, 2022 at 6:44 AM Michael S. Tsirkin <mst@redhat.com> wrote: > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > There's also the perenial problem that developers frequently send > > patches that mistakenly include submodule changes, which is related to > the > > way that 'git checkout' doesn't sync submodule state when switching > branches. > > Do you happen to know how exactly that happens? > Here's how it happens to me. % git checkout master % git checkout -b bsd-user-2022q4 # bring changes in, doesn't matter # Build a few times # time passes, master upstream evolves, submodules move % git checkout master % git pull # master fast forwards % git rebase -i master bsd-user-2022q4 at this point I have to do unnatural git submodule recursive things to make it stop complaining about how all the submodules are now 'modified' because they have moved on the master branch. It is real, lasting, lingering pain and a large part of the reason that I refuse to allow FreeBSD to use submodules[*]. It's just too much extra hassle for no gain at all (at least for me, I see only the pain and no benefits to me or anybody else). I've sent at least three review requests that have this pollution in it since it's hard to avoid if you press forward after the rebase not to have something swept up into an innocuous change. Warner [*] They are also generally a poor match for FreeBSD since we rarely import things verbatim, unchanged from upstreams that have well-oiled CI to give us assurance that things are good...
On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote: > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > > There's also the perenial problem that developers frequently send > > > patches that mistakenly include submodule changes, which is related to the > > > way that 'git checkout' doesn't sync submodule state when switching branches. > > > > Do you happen to know how exactly that happens? > > For any given branch the submodule is synced to a given git commit hash. > If the submodule checkout is not synced to the same commit hash it will > show as dirty, and if you git add this pending change, it'll record that > new submodule commit hash. Seeing dirty state is common when you switch > between branches, either side of a git master change that updated a > submodule. > > With regards, > Daniel I see. It is interesting. So apparently what you want is ignore submodule changes, right? If yes this is close to what we want: git submodule update --checkout testsub git update-index --skip-worktree testsub A script checking out the submodule can just run this command. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Sep 28, 2022 at 7:09 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote: > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote: > > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > > > [...] > > > > > > For biosbits if we are going this route then I feel a submodule > is much > > > > > > better. It records which version exactly each qemu version > wants. > > > > > > > > > > As far as I know, you can also specify the version when using pip, > can't > > > > > you? So that's not really an advantage here. > > > > > > > > > > On the contrary, submodules have a couple of disadvantages that I > really > > > > > dislike: > > > > > > > > > > - submodules do not get updated automatically when doing a "git > checkout", > > > > > we have to update them via a script instead. This causes e.g. > trouble if you > > > > > rsync your source tree to a machine that has no access to the > internet and > > > > > you forgot to update the submodule before the sync > > > > > > > > > > - the content of submodules is not added to the tarballs that get > created on > > > > > the git forges automatically. There were lots of requests from > users in the > > > > > past that tried to download a tarball from github and then > wondered why they > > > > > couldn't compile QEMU. > > > > > > > > > > - we include the submodule content in our release tarballs, so > people get > > > > > the impression that hte submodule content is part of the QEMU > sources. This > > > > > has two disadvantages: > > > > > * We already got bug reports for the code in the submodule, > > > > > where people did not understand that they should report that > > > > > rather to the original project instead (i.e. you ship it - you > > > > > own it) > > > > > * People get the impression that QEMU is a huge monster > > > > > application if they count the number of code lines, run > > > > > their code scanner tools on the tarball contents, etc. > > > > > Remember "nemu", for example, where one of the main complaints > > > > > was that QEMU has too many lines of code? > > > > > > > > > > - If programs includes code via submodules, this gets a higher > > > > > burder for distro maintainers, since they have to patch each > > > > > and every package when there is a bug, instead of being able to > > > > > fix it in one central place. > > > > > > > > > > So in my opinion we should avoid new submodules if there is an > alternative. > > > > > > > > > > Thomas > > > > > > > > So looking at the latest proposals downloading files from CI, > > > > checksumming them etc etc. No auto checkout, not added automatically > > > > either, right? > > > > > > > > This seems to be the only difference: > > > > - we include the submodule content in our release tarballs > > > > > > That's just one of the issues with submodules. Working with them in > general > > > is not a pleasant experiance. > > > > This is what I asked about at the maintainers summit. > > I'd like to map the issues and see if there's anything > > we can do to solve them. In particular we will likely > > keep using submodules where we historically did > > so it's time well spent. > > > > I agree generally but the big question is what to replace these with. > > Below I assume the replacement is a script such as avocado or pytest > > with its own hashing, calling wget internally etc etc. > > > > > > > Thomas pointed out some of the issues, such > > > as 'git checkout' ignoring submodules, requiring extra steps to sync > them. > > > > > > Not different from a home grown SCM as part of test script, right? > > We're not building a home grown SCM as part of a test script, so > this answer is irrelevant. > > > > There's also the perenial problem that developers frequently send > > > patches that mistakenly include submodule changes, > > > > OK, so the thing to do would be to look for ways to exclude submodule > changes > > from git commits. > > If someone wants to make git suck less with submodules great, but needs > someone to actually do the work. > A big part of the problem is knowing which of the following commands I have to do to undo the uncommitted changes, the committed changes, the staged changes, etc: git submodule update --init --recursive git submodule update --init git submodule foreach --recursive git reset --hard git submodule foreach --recursive git clean -xfd git submodule foreach --recursive git clean -xfg (all of these are in my history, I honestly don't know the difference between the last two). And each 'oops' takes time away from upstreaming bsd-user I don't really have that much of. I've wasted hours on this over the past year between all the different ways it can screw up. To be fair, it is a relatively small fraction of the time, but as you can tell from the animation in my email it inspires much passion. Warner > > > I'd really like to see us doing more to eliminate as much use of > submodules > > > as is possible over time.p > > > > Or try to fix the problems, right? > > Again needs someone to actually make it happen. > > Meanwhile QEMU already has an integrated test harness in the form > of Avocado that does everything needed. If Avocado had just been > used for this biosbits test in the first place, the test would > likely have already been merged to QEMU, instead of us having this > never ending debate on how to re-invent an alternative to what > already avocado does. > > With regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > > >
On Wed, Sep 28, 2022 at 07:15:53AM -0600, Warner Losh wrote: > > > On Wed, Sep 28, 2022 at 7:09 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote: > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote: > > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote: > > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote: > > > > > [...] > > > > > > For biosbits if we are going this route then I feel a submodule > is much > > > > > > better. It records which version exactly each qemu version > wants. > > > > > > > > > > As far as I know, you can also specify the version when using pip, > can't > > > > > you? So that's not really an advantage here. > > > > > > > > > > On the contrary, submodules have a couple of disadvantages that I > really > > > > > dislike: > > > > > > > > > > - submodules do not get updated automatically when doing a "git > checkout", > > > > > we have to update them via a script instead. This causes e.g. > trouble if you > > > > > rsync your source tree to a machine that has no access to the > internet and > > > > > you forgot to update the submodule before the sync > > > > > > > > > > - the content of submodules is not added to the tarballs that get > created on > > > > > the git forges automatically. There were lots of requests from > users in the > > > > > past that tried to download a tarball from github and then wondered > why they > > > > > couldn't compile QEMU. > > > > > > > > > > - we include the submodule content in our release tarballs, so > people get > > > > > the impression that hte submodule content is part of the QEMU > sources. This > > > > > has two disadvantages: > > > > > * We already got bug reports for the code in the submodule, > > > > > where people did not understand that they should report that > > > > > rather to the original project instead (i.e. you ship it - you > > > > > own it) > > > > > * People get the impression that QEMU is a huge monster > > > > > application if they count the number of code lines, run > > > > > their code scanner tools on the tarball contents, etc. > > > > > Remember "nemu", for example, where one of the main complaints > > > > > was that QEMU has too many lines of code? > > > > > > > > > > - If programs includes code via submodules, this gets a higher > > > > > burder for distro maintainers, since they have to patch each > > > > > and every package when there is a bug, instead of being able to > > > > > fix it in one central place. > > > > > > > > > > So in my opinion we should avoid new submodules if there is an > alternative. > > > > > > > > > > Thomas > > > > > > > > So looking at the latest proposals downloading files from CI, > > > > checksumming them etc etc. No auto checkout, not added automatically > > > > either, right? > > > > > > > > This seems to be the only difference: > > > > - we include the submodule content in our release tarballs > > > > > > That's just one of the issues with submodules. Working with them in > general > > > is not a pleasant experiance. > > > > This is what I asked about at the maintainers summit. > > I'd like to map the issues and see if there's anything > > we can do to solve them. In particular we will likely > > keep using submodules where we historically did > > so it's time well spent. > > > > I agree generally but the big question is what to replace these with. > > Below I assume the replacement is a script such as avocado or pytest > > with its own hashing, calling wget internally etc etc. > > > > > > > Thomas pointed out some of the issues, such > > > as 'git checkout' ignoring submodules, requiring extra steps to sync > them. > > > > > > Not different from a home grown SCM as part of test script, right? > > We're not building a home grown SCM as part of a test script, so > this answer is irrelevant. > > > > There's also the perenial problem that developers frequently send > > > patches that mistakenly include submodule changes, > > > > OK, so the thing to do would be to look for ways to exclude submodule > changes > > from git commits. > > If someone wants to make git suck less with submodules great, but needs > someone to actually do the work. > > > A big part of the problem is knowing which of the following commands I have to > do to undo the uncommitted changes, the committed changes, the staged changes, > etc: > > git submodule update --init --recursive > git submodule update --init > git submodule foreach --recursive git reset --hard > git submodule foreach --recursive git clean -xfd > git submodule foreach --recursive git clean -xfg > > (all of these are in my history, I honestly don't know the difference between > the last two). > And each 'oops' takes time away from upstreaming bsd-user I don't really have > that > much of. I've wasted hours on this over the past year between all the different > ways > it can screw up. > > To be fair, it is a relatively small fraction of the time, but as you can tell > from the > animation in my email it inspires much passion. > > Warner > OK this is understandable. I think for what we do in qemu (never change upstream really) git submodule update --checkout testsub git update-index --skip-worktree testsub is more or less what most people want. This assumes you have a script that basically does git submodule deinit XYZ git submodule update --init XYZ git update-index --skip-worktree XYZ each time, and do not need to work on the submodule proper. > > > I'd really like to see us doing more to eliminate as much use of > submodules > > > as is possible over time.p > > > > Or try to fix the problems, right? > > Again needs someone to actually make it happen. > > Meanwhile QEMU already has an integrated test harness in the form > of Avocado that does everything needed. If Avocado had just been > used for this biosbits test in the first place, the test would > likely have already been merged to QEMU, instead of us having this > never ending debate on how to re-invent an alternative to what > already avocado does. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange > :| > |: https://libvirt.org -o- https://fstop138.berrange.com > :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange > :| > > >
On Wed, 28 Sept 2022 at 15:29, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote: > > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote: > > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > > > There's also the perenial problem that developers frequently send > > > > patches that mistakenly include submodule changes, which is related to the > > > > way that 'git checkout' doesn't sync submodule state when switching branches. > > > > > > Do you happen to know how exactly that happens? > > > > For any given branch the submodule is synced to a given git commit hash. > > If the submodule checkout is not synced to the same commit hash it will > > show as dirty, and if you git add this pending change, it'll record that > > new submodule commit hash. Seeing dirty state is common when you switch > > between branches, either side of a git master change that updated a > > submodule. > I see. It is interesting. > > So apparently what you want is ignore submodule > changes, right? If yes this is close to what we want: > > git submodule update --checkout testsub > git update-index --skip-worktree testsub > > A script checking out the submodule can just run this > command. The problem happens not when you check out the submodule, but when you do basic not-submodule-related git operations like "git checkout my-working-branch". The fix would be if git itself automatically kept submodule state in sync with the main repo's branch. But it doesn't, so the UI is a massive beartrap. -- PMM
On Wed, Sep 28, 2022 at 04:07:40PM +0100, Peter Maydell wrote: > On Wed, 28 Sept 2022 at 15:29, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote: > > > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote: > > > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote: > > > > > There's also the perenial problem that developers frequently send > > > > > patches that mistakenly include submodule changes, which is related to the > > > > > way that 'git checkout' doesn't sync submodule state when switching branches. > > > > > > > > Do you happen to know how exactly that happens? > > > > > > For any given branch the submodule is synced to a given git commit hash. > > > If the submodule checkout is not synced to the same commit hash it will > > > show as dirty, and if you git add this pending change, it'll record that > > > new submodule commit hash. Seeing dirty state is common when you switch > > > between branches, either side of a git master change that updated a > > > submodule. > > I see. It is interesting. > > > > So apparently what you want is ignore submodule > > changes, right? If yes this is close to what we want: > > > > git submodule update --checkout testsub > > git update-index --skip-worktree testsub > > > > A script checking out the submodule can just run this > > command. > > The problem happens not when you check out the submodule, but when > you do basic not-submodule-related git operations like > "git checkout my-working-branch". The fix would be if git itself > automatically kept submodule state in sync with the main repo's > branch. But it doesn't, so the UI is a massive beartrap. > > -- PMM If this is what you want, you can make it do that too: $ cat .gitmodules [submodule "testsub"] path = testsub recurse = true url = /tmp/testsub/ git checkout (and a bunch of other commands) will now recurse into this submodule.
Hello, On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote: > > git submodules are awkward basically because they are an automated wget. > I don't think an explicit wget is much better ... but > looks like I'm alone in this. Oh well. > So it will be a weird dance of wget a tarball, unpack, generate > ISO and run. God help you if you need to patch the test - it's > wget all the way down. That's the problem - the submodules are not automated. They are half-automated, and the rules for when the automation works and for when the automation falls apart are not intellibible for the general Joe Developer. You might spend a few days studying how they behave exactly, and then you will know. But unless you will use them every day you will forget again, because the rules do not lend themselves to some abstraction easily understood by humans. Thanks Michal
On Wed, Sep 28, 2022 at 10:48:03PM +0200, Michal Suchánek wrote: > Hello, > > On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote: > > > > > git submodules are awkward basically because they are an automated wget. > > I don't think an explicit wget is much better ... but > > looks like I'm alone in this. Oh well. > > So it will be a weird dance of wget a tarball, unpack, generate > > ISO and run. God help you if you need to patch the test - it's > > wget all the way down. > > That's the problem - the submodules are not automated. They are > half-automated, and the rules for when the automation works and for when > the automation falls apart are not intellibible for the general Joe > Developer. > > You might spend a few days studying how they behave exactly, and then you > will know. But unless you will use them every day you will forget again, > because the rules do not lend themselves to some abstraction easily > understood by humans. > > Thanks > > Michal But why would learning a different tool be easier?
On Wed, Sep 28, 2022 at 05:07:48PM -0400, Michael S. Tsirkin wrote: > On Wed, Sep 28, 2022 at 10:48:03PM +0200, Michal Suchánek wrote: > > Hello, > > > > On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote: > > > > > > > > git submodules are awkward basically because they are an automated wget. > > > I don't think an explicit wget is much better ... but > > > looks like I'm alone in this. Oh well. > > > So it will be a weird dance of wget a tarball, unpack, generate > > > ISO and run. God help you if you need to patch the test - it's > > > wget all the way down. > > > > That's the problem - the submodules are not automated. They are > > half-automated, and the rules for when the automation works and for when > > the automation falls apart are not intellibible for the general Joe > > Developer. > > > > You might spend a few days studying how they behave exactly, and then you > > will know. But unless you will use them every day you will forget again, > > because the rules do not lend themselves to some abstraction easily > > understood by humans. > > > > Thanks > > > > Michal > > But why would learning a different tool be easier? a) it's working lends itself to explaining in human-intelligible concepts b) it's used elsewhere as well Thanks Michal
diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README new file mode 100644 index 0000000000..97b15f1665 --- /dev/null +++ b/tests/qtest/acpi-bits/README @@ -0,0 +1,168 @@ +============================================================================= +ACPI/SMBIOS QTESTS USING BIOSBITS +============================================================================= + +Biosbits is a software written by Josh Triplett that can be downloaded by +visiting https://biosbits.org/. The github codebase can be found here: +https://github.com/biosbits/bits/tree/master. It is a software that exercizes +the bios components such as acpi and smbios tables directly through acpica +bios interpreter (a freely available C based library written by Intel, +downloadable from https://acpica.org/ and is included with biosbits) without an +operating system getting involved in between. +There are several advantages to directly testing the bios in a real physical +machine or VM as opposed to indirectly discovering bios issues through the +operating system. For one thing, the OSes tend to hide bios problems from the +end user. The other is that we have more control of what we wanted to test +and how by directly using acpica interpreter on top of the bios on a running +system. More details on the inspiration for developing biosbits and its real +life uses can be found in (a) and (b). +This directory contains QEMU qtests written in python that exercizes the QEMU +bios components using biosbits and reports test failures. + +These tests use python virtual environment. In debian/ubuntu system, the tests +would require python3.8-venv and python3-pip packages to be installed. + +A brief description of the contents of this directory follows: + +├── acpi-bits-test.py +├── acpi-bits-test-venv.sh +├── bits-config +│ ├── bits-cfg.txt +│ └── meson.build +├── bits-tests +│ ├── meson.build +│ ├── smbios.py +│ ├── smilatency.py +│ ├── testacpi.py +│ └── testcpuid.py +├── meson.build +├── prebuilt +│ ├── bits-2095-grub.tar.gz +│ ├── bits-2095.zip +│ └── meson.build +├── README +└── requirements.txt + +acpi-bits: + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual + environment necessary for the actual python test script to run. Amongst + other things, it makes sure that QEMU python library is available within + that venv so that QEMU machines can be forked. QEMU python library can be + found within python/ directory in QEMU source. + After setting up the virtual environment, it runs the python test script + from within that environment. + If you want to enable verbose mode only for bits test and run make check, + one trick is to add V=1 before the call to execute the python script in + this file. + - acpi-bits-test.py: This is the main python test script that generates a + biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports + test failures. This is the script one would be interested in if they wanted + to add or change some component of the log parsing, add a new commandline to + how QEMU is spawned etc. Test writers typically would not need to modify + this script unless they wanted to enhance or change the log parsing for + their tests. + - requirements.txt: This text file contains the dependent python libraries + needed for the tests to run. If a new dependent library is needed, it would + be added here as a new entry and then acpi-bits-test-venv.sh would install + it when setting up the python virtual environment using pip. + - README: This text file. + +acpi-bits/bits-config: + This location contains biosbits config files that determine how the software + runs the tests. + - bits-config.txt: this is the biosbits config file that determines what tests + or actions are performed by bits. The description of the config options are + provided in the file itself. + +acpi-bits/prebuilt: + This location contains prebuilt biosbits binaries that are used to generate + the bits iso file for testing. + - bits-2095.zip: The contents from this zip archive are the main contents of + the iso file that are used for testing. This binary zip archive also + contains the full source of the bits software including the full text of + the license agreement listed here: + https://github.com/biosbits/bits/blob/master/COPYING + The source tarball can be found in this location in the zip file: + boot/src/bits-2095.tar.gz + The additional changes beyond those that are present in the official + biosbits github repository can be found here: + https://github.com/ani-sinha/bits/tree/bits-qemu-logging + + Basically these changes can be categorized into the following: + (a) changes to make sure biosbits builds with the latest gcc compiler + (gcc 9.4). + (b) upgrade of acpica to the latest version (march 2022). + (c) send bits logs to the debug IO port at addresss 0x403 so that isa + debugcon can be used to collect the logs. + (d) send a gub halt command to shutdown the VM once all the tests have been + executed. + + This zip file is automatically generated by the bits build system. It can + be found in the bits build directory and it is suffixed by the bits build + number. + Normally, there is no need to make any alteration to this zip archive + unless one wanted to change the bits software itself (for example to add + a new config option or change actions of existing options or change the + debug IO port etc). When such changes are made and a new biosbits software + is needed to be generated, please refresh this zip archive as well as the + grub tarball at the same time. Tests will break if changes are made to bits + that are incompatible with existing behavior. So care must be taken to make + sure that the change is compatible with existing bits software as much as + possible. When a new config option is introduced for example, bits must + be upadated here first before introducing the new config option through + a later commit. + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub + scripts and modules to the prebuilt directory. These prebuilt grub + artifacts are required in order to generate a bootable bits iso file that + can run tests. + In order to generate this tar archive, please put the following two + directories that can be found in the bits build directory in a single + tar archive file named as bits-<n>-grub.tar.gz where n is the version of + bits that generated the archive: + + grub-inst-x86_64-efi + grub-inst + + This version should be the same as the version number of bits that generated + bits-<n>.zip file. In other words, the two files must be in sync and should + come from the same build of biosbits of the same version. Things may still + work if they come from different versions but mixing version numbers are + not recommended. + There is normally no need to regenerate this archive unless some fixes or + changes have gone into the grub that are part of biosbits. + +acpi-bits/bits-tests: + This directory contains biosbits python based tests that are run from within + the biosbits environment in the spawned VM. New additions of test cases can + be made in the appropriate test file. For example, new acpi tests can go + into testacpi.py and one would call testsuite.add_test() to register the new + test so that it gets executed as a part of the ACPI tests. + It might be occassionally necessary to disable some subtests or add a new + test that belongs to a test suite not already present in this directory. To + do this, please extract the bits source from the zip file mentioned above. + Copy the test suite/script that needs modification (addition of new tests + or disabling them) from boot/python location of the extracted directory + into this directory. + + step (a): copy unmodified test script to this directory. + step (b): update meson.build and add this file to the list. + Commit (a) and (b) together in the same commit. + + step (c): perform modifications to the test. + Commit (c) in a separate commit. + + The test framework will then use your modified test script to run the test. + No further changes would be needed. Please check the logs to make sure that + appropriate changes have taken effect. + +meson.build files makes sure that the bits qtests are appropriately integrated +into the QEMU qtest framework and are run with "make check-qtest". +Currently, the bits test is configured to run only for x86_64 architectures. Bits +has been built only for this architecture. + + +Author: Ani Sinha <ani@anisinha.ca> + +References: +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf +(b) https://www.youtube.com/watch?v=36QIepyUuhg
The README file is added describing the directory structure and the purpose of every file it contains. It also describes how to add new tests, make changes to existing tests or bits config files or regenerate the bits software. Signed-off-by: Ani Sinha <ani@anisinha.ca> --- tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 tests/qtest/acpi-bits/README