| Message ID | 20191206140012.15517-1-wainersm@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | tests/acceptance: Add boot vmlinux test | expand |
On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote: > This series add a new acceptance test: boot an uncompressed > Linux kernel built with CONFIG_PVH, so checking the PVH > capability introduced in QEMU 4.0 works. > > The test implementation depends on [1] which is likely released > on next Avocado. So that will need a version 2 of this > series to bump Avocado version. > Right, the Avocado bits have been merged, so unless a major reversal comes (not expected at all), it will be on Avocado 74.0. > Also I want to use this as an example of a scenario that test > assets could be better managed. As you see on patch 01 the > kernel is built at test time on localhost. While Avocado provides > an API to easily fetch and build it, the whole process takes > reasonable time - besides the fact that localhost must have > all build dependencies installed. How could it be done better? > This is clearly not a "kernel build" test, so we should avoid building the kernel as part of the "PVH boot" test. The problem you expose here is a very real, and each possible solution has its own problems, unfortunately. The best solution IMO would be to find a "well known" distribution of such a kernel. Maybe something maintained by the Xen project or one of its commercial products? The second best solution is to have a helper script (using the Avocado utils API is fine) that will build a kernel and create/populate the test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux. The test can cancel itself if it doesn't find a kernel there. The third best solution IMO is for this test to require a parameter telling where the CONFIG_PVH enabled vmlinux kernel image is. Also notice that, with a custom built kernel image (a different one for each user), the result of the test as a general indicator to the QEMU codebase diminishes quite a bit. > Nonetheless I argue in favor of merging this as is, and > gradually implement corrections to improve the tests assets > management. Also if eventually the test is proven to unacceptable > slow down the Travis CI then I can add a guard to skip it. > I'm pretty sure that building the kernel will cause a major slow down of the Travis CI results for the "acceptance" block (and thus for the overall results). But, it may also time it out (there's a 50 minutes deadline). I hope this brings some ideas, and thanks for coming up with interesting problems! :) - Cleber. > [1] https://github.com/avocado-framework/avocado/pull/3389 > > Wainer dos Santos Moschetta (2): > tests/acceptance: Add PVH boot test > .travis.yml: Add kernel build deps for acceptance tests > > .travis.yml | 2 ++ > tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 50 insertions(+) > create mode 100644 tests/acceptance/pvh.py > > -- > 2.21.0 >
Cleber Rosa <crosa@redhat.com> writes: > On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote: >> This series add a new acceptance test: boot an uncompressed >> Linux kernel built with CONFIG_PVH, so checking the PVH >> capability introduced in QEMU 4.0 works. >> >> The test implementation depends on [1] which is likely released >> on next Avocado. So that will need a version 2 of this >> series to bump Avocado version. >> > > Right, the Avocado bits have been merged, so unless a major reversal > comes (not expected at all), it will be on Avocado 74.0. > >> Also I want to use this as an example of a scenario that test >> assets could be better managed. As you see on patch 01 the >> kernel is built at test time on localhost. While Avocado provides >> an API to easily fetch and build it, the whole process takes >> reasonable time - besides the fact that localhost must have >> all build dependencies installed. How could it be done better? >> > > This is clearly not a "kernel build" test, so we should avoid building > the kernel as part of the "PVH boot" test. The problem you expose > here is a very real, and each possible solution has its own problems, > unfortunately. > > The best solution IMO would be to find a "well known" distribution of > such a kernel. Maybe something maintained by the Xen project or one > of its commercial products? > > The second best solution is to have a helper script (using the Avocado > utils API is fine) that will build a kernel and create/populate the > test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux. > The test can cancel itself if it doesn't find a kernel there. > > The third best solution IMO is for this test to require a parameter > telling where the CONFIG_PVH enabled vmlinux kernel image is. > > Also notice that, with a custom built kernel image (a different one > for each user), the result of the test as a general indicator to the > QEMU codebase diminishes quite a bit. I can see a use case for making it easier for developers to build test cases otherwise everyone has their own particular variant of the kernel and buildroot/busybox which they have in their own farm. However as Cleber has noted they make poor standardised tests given the variation in kernel builds you can get. That said I think there are better targets. kvm-unit-tests can be built against a range of architectures and are fashioned as individual unit tests for specific functionality. It would be useful to make it easy for a developer to regenerate the test assets to re-run a test someone else has found to fail. >> Nonetheless I argue in favor of merging this as is, and >> gradually implement corrections to improve the tests assets >> management. Also if eventually the test is proven to unacceptable >> slow down the Travis CI then I can add a guard to skip it. >> > > I'm pretty sure that building the kernel will cause a major slow down > of the Travis CI results for the "acceptance" block (and thus for the > overall results). But, it may also time it out (there's a 50 minutes > deadline). > > I hope this brings some ideas, and thanks for coming up with > interesting problems! :) > > - Cleber. > >> [1] https://github.com/avocado-framework/avocado/pull/3389 >> >> Wainer dos Santos Moschetta (2): >> tests/acceptance: Add PVH boot test >> .travis.yml: Add kernel build deps for acceptance tests >> >> .travis.yml | 2 ++ >> tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 50 insertions(+) >> create mode 100644 tests/acceptance/pvh.py >> >> -- >> 2.21.0 >>