| Message ID | 20180201022046.9425-1-famz@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 01/02/2018 03:20, Fam Zheng wrote: > To make our efforts on QEMU testing easier to consume by contributors, > let's add a document. For example, Patchew reports build errors on > patches that should be relatively easy to reproduce with a few steps, and > it is much nicer if there is such a documentation that it can refer to. > > This focuses on how to run existing tests and how to write new test > cases, without going into the frameworks themselves. > > The VM based testing section is moved from tests/vm/README which now > is a single line pointing to the new doc. > > Signed-off-by: Fam Zheng <famz@redhat.com> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Queued, thanks. Paolo > --- > > v3: Fix "development", "reclamation" and "tests". [Eric] > Fix "libqtest". [Andrew] > Add Stefan's reviewed-by. > > v2: Fix spelling errors and improve wordings. [Stefan, Eric, Thomas] > Example test name "test-foo -> foo-test". The mass renaming will be > done in another series. [Thomas, Eric] > Document how to debug unit tests and qtests. [Eric] > Document group permission setup for docker, and the privilege risks. > [Alex] > Update tests/vm/README. > --- > docs/devel/testing.rst | 486 +++++++++++++++++++++++++++++++++++++++++++++++++ > tests/vm/README | 90 +-------- > 2 files changed, 487 insertions(+), 89 deletions(-) > create mode 100644 docs/devel/testing.rst > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > new file mode 100644 > index 0000000000..0ca1a2d4b5 > --- /dev/null > +++ b/docs/devel/testing.rst > @@ -0,0 +1,486 @@ > +=============== > +Testing in QEMU > +=============== > + > +This document describes the testing infrastructure in QEMU. > + > +Testing with "make check" > +========================= > + > +The "make check" testing family includes most of the C based tests in QEMU. For > +a quick help, run ``make check-help`` from the source tree. > + > +The usual way to run these tests is: > + > +.. code:: > + > + make check > + > +which includes QAPI schema tests, unit tests, and QTests. Different sub-types > +of "make check" tests will be explained below. > + > +Before running tests, it is best to build QEMU programs first. Some tests > +expect the executables to exist and will fail with obscure messages if they > +cannot find them. > + > +Unit tests > +---------- > + > +Unit tests, which can be invoked with ``make check-unit``, are simple C tests > +that typically link to individual QEMU object files and exercise them by > +calling exported functions. > + > +If you are writing new code in QEMU, consider adding a unit test, especially > +for utility modules that are relatively stateless or have few dependencies. To > +add a new unit test: > + > +1. Create a new source file. For example, ``tests/foo-test.c``. > + > +2. Write the test. Normally you would include the header file which exports > + the module API, then verify the interface behaves as expected from your > + test. The test code should be organized with the glib testing framework. > + Copying and modifying an existing test is usually a good idea. > + > +3. Add the test to ``tests/Makefile.include``. First, name the unit test > + program and add it to ``$(check-unit-y)``; then add a rule to build the > + executable. Optionally, you can add a magical variable to support ``gcov``. > + For example: > + > +.. code:: > + > + check-unit-y += tests/foo-test$(EXESUF) > + tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y) > + ... > + gcov-files-foo-test-y = util/foo.c > + > +Since unit tests don't require environment variables, the simplest way to debug > +a unit test failure is often directly invoking it or even running it under > +``gdb``. However there can still be differences in behavior between ``make`` > +invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment > +variable (which affects memory reclamation and catches invalid pointers better) > +and gtester options. If necessary, you can run > + > +.. code:: > + make check-unit V=1 > + > +and copy the actual command line which executes the unit test, then run > +it from the command line. > + > +QTest > +----- > + > +QTest is a device emulation testing framework. It can be very useful to test > +device models; it could also control certain aspects of QEMU (such as virtual > +clock stepping), with a special purpose "qtest" protocol. Refer to the > +documentation in ``qtest.c`` for more details of the protocol. > + > +QTest cases can be executed with > + > +.. code:: > + > + make check-qtest > + > +The QTest library is implemented by ``tests/libqtest.c`` and the API is defined > +in ``tests/libqtest.h``. > + > +Consider adding a new QTest case when you are introducing a new virtual > +hardware, or extending one if you are adding functionalities to an existing > +virtual device. > + > +On top of libqtest, a higher level library, ``libqos``, was created to > +encapsulate common tasks of device drivers, such as memory management and > +communicating with system buses or devices. Many virtual device tests use > +libqos instead of directly calling into libqtest. > + > +Steps to add a new QTest case are: > + > +1. Create a new source file for the test. (More than one file can be added as > + necessary.) For example, ``tests/test-foo-device.c``. > + > +2. Write the test code with the glib and libqtest/libqos API. See also existing > + tests and the library headers for reference. > + > +3. Register the new test in ``tests/Makefile.include``. Add the test executable > + name to an appropriate ``check-qtest-*-y`` variable. For example: > + > + ``check-qtest-generic-y = tests/test-foo-device$(EXESUF)`` > + > +4. Add object dependencies of the executable in the Makefile, including the > + test source file(s) and other interesting objects. For example: > + > + ``tests/test-foo-device$(EXESUF): tests/test-foo-device.o $(libqos-obj-y)`` > + > +Debugging a QTest failure is slightly harder than the unit test because the > +tests look up QEMU program names in the environment variables, such as > +``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy > +to attach gdb to the QEMU process spawned from the test. But manual invoking > +and using gdb on the test is still simple to do: find out the actual command > +from the output of > + > +.. code:: > + make check-qtest V=1 > + > +which you can run manually. > + > +QAPI schema tests > +----------------- > + > +The QAPI schema tests validate the QAPI parser used by QMP, by feeding > +predefined input to the parser and comparing the result with the reference > +output. > + > +The input/output data is managed under the ``tests/qapi-schema`` directory. > +Each test case includes four files that have a common base name: > + > + * ``${casename}.json`` - the file contains the JSON input for feeding the > + parser > + * ``${casename}.out`` - the file contains the expected stdout from the parser > + * ``${casename}.err`` - the file contains the expected stderr from the parser > + * ``${casename}.exit`` - the expected error code > + > +Consider adding a new QAPI schema test when you are making a change on the QAPI > +parser (either fixing a bug or extending/modifying the syntax). To do this: > + > +1. Add four files for the new case as explained above. For example: > + > + ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. > + > +2. Add the new test in ``tests/Makefile.include``. For example: > + > + ``qapi-schema += foo.json`` > + > +check-block > +----------- > + > +``make check-block`` is a legacy command to invoke block layer iotests and is > +rarely used. See "QEMU iotests" section below for more information. > + > +GCC gcov support > +---------------- > + > +``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the > +tested code. To use it, configure QEMU with ``--enable-gcov`` option and build. > +Then run ``make check`` as usual. There will be additional ``gcov`` output as > +the testing goes on, showing the test coverage percentage numbers per analyzed > +source file. More detailed reports can be obtained by running ``gcov`` command > +on the output files under ``$build_dir/tests/``, please read the ``gcov`` > +documentation for more information. > + > +QEMU iotests > +============ > + > +QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing > +framework widely used to test block layer related features. It is higher level > +than "make check" tests and 99% of the code is written in bash or Python > +scripts. The testing success criteria is golden output comparison, and the > +test files are named with numbers. > + > +To run iotests, make sure QEMU is built successfully, then switch to the > +``tests/qemu-iotests`` directory under the build directory, and run ``./check`` > +with desired arguments from there. > + > +By default, "raw" format and "file" protocol is used; all tests will be > +executed, except the unsupported ones. You can override the format and protocol > +with arguments: > + > +.. code:: > + > + # test with qcow2 format > + ./check -qcow2 > + # or test a different protocol > + ./check -nbd > + > +It's also possible to list test numbers explicitly: > + > +.. code:: > + > + # run selected cases with qcow2 format > + ./check -qcow2 001 030 153 > + > +Cache mode can be selected with the "-c" option, which may help reveal bugs > +that are specific to certain cache mode. > + > +More options are supported by the ``./check`` script, run ``./check -h`` for > +help. > + > +Writing a new test case > +----------------------- > + > +Consider writing a tests case when you are making any changes to the block > +layer. An iotest case is usually the choice for that. There are already many > +test cases, so it is possible that extending one of them may achieve the goal > +and save the boilerplate to create one. (Unfortunately, there isn't a 100% > +reliable way to find a related one out of hundreds of tests. One approach is > +using ``git grep``.) > + > +Usually an iotest case consists of two files. One is an executable that > +produces output to stdout and stderr, the other is the expected reference > +output. They are given the same number in file names. E.g. Test script ``055`` > +and reference output ``055.out``. > + > +In rare cases, when outputs differ between cache mode ``none`` and others, a > +``.out.nocache`` file is added. In other cases, when outputs differ between > +image formats, more than one ``.out`` files are created ending with the > +respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. > + > +There isn't a hard rule about how to write a test script, but a new test is > +usually a (copy and) modification of an existing case. There are a few > +commonly used ways to create a test: > + > +* A Bash script. It will make use of several environmental variables related > + to the testing procedure, and could source a group of ``common.*`` libraries > + for some common helper routines. > + > +* A Python unittest script. Import ``iotests`` and create a subclass of > + ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of > + this approach is that the output is too scarce, and the script is considered > + harder to debug. > + > +* A simple Python script without using unittest module. This could also import > + ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit > + from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest > + execution. This is a combination of 1 and 2. > + > +Pick the language per your preference since both Bash and Python have > +comparable library support for invoking and interacting with QEMU programs. If > +you opt for Python, it is strongly recommended to write Python 3 compatible > +code. > + > +Docker based tests > +================== > + > +Introduction > +------------ > + > +The Docker testing framework in QEMU utilizes public Docker images to build and > +test QEMU in predefined and widely accessible Linux environments. This makes > +it possible to expand the test coverage across distros, toolchain flavors and > +library versions. > + > +Prerequisites > +------------- > + > +Install "docker" with the system package manager and start the Docker service > +on your development machine, then make sure you have the privilege to run > +Docker commands. Typically it means setting up passwordless ``sudo docker`` > +command or login as root. For example: > + > +.. code:: > + > + $ sudo yum install docker > + $ # or `apt-get install docker` for Ubuntu, etc. > + $ sudo systemctl start docker > + $ sudo docker ps > + > +The last command should print an empty table, to verify the system is ready. > + > +An alternative method to set up permissions is by adding the current user to > +"docker" group and making the docker daemon socket file (by default > +``/var/run/docker.sock``) accessible to the group: > + > +.. code:: > + > + $ sudo groupadd docker > + $ sudo usermod $USER -G docker > + $ sudo chown :docker /var/run/docker.sock > + > +Note that any one of above configurations makes it possible for the user to > +exploit the whole host with Docker bind mounting or other privileged > +operations. So only do it on development machines. > + > +Quickstart > +---------- > + > +From source tree, type ``make docker`` to see the help. Testing can be started > +without configuring or building QEMU (``configure`` and ``make`` are done in > +the container, with parameters defined by the make target): > + > +.. code:: > + > + make docker-test-build@min-glib > + > +This will create a container instance using the ``min-glib`` image (the image > +is downloaded and initialized automatically), in which the ``test-build`` job > +is executed. > + > +Images > +------ > + > +Along with many other images, the ``min-glib`` image is defined in a Dockerfile > +in ``tests/docker/dockefiles/``, called ``min-glib.docker``. ``make docker`` > +command will list all the available images. > + > +To add a new image, simply create a new ``.docker`` file under the > +``tests/docker/dockerfiles/`` directory. > + > +A ``.pre`` script can be added beside the ``.docker`` file, which will be > +executed before building the image under the build context directory. This is > +mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, > +for example, to make qemu-user powered cross build containers work. > + > +Tests > +----- > + > +Different tests are added to cover various configurations to build and test > +QEMU. Docker tests are the executables under ``tests/docker`` named > +``test-*``. They are typically shell scripts and are built on top of a shell > +library, ``tests/docker/common.rc``, which provides helpers to find the QEMU > +source and build it. > + > +The full list of tests is printed in the ``make docker`` help. > + > +Tools > +----- > + > +There are executables that are created to run in a specific Docker environment. > +This makes it easy to write scripts that have heavy or special dependencies, > +but are still very easy to use. > + > +Currently the only tool is ``travis``, which mimics the Travis-CI tests in a > +container. It runs in the ``travis`` image: > + > +.. code:: > + > + make docker-travis@travis > + > +Debugging a Docker test failure > +------------------------------- > + > +When CI tasks, maintainers or yourself report a Docker test failure, follow the > +below steps to debug it: > + > +1. Locally reproduce the failure with the reported command line. E.g. run > + ``make docker-test-mingw@fedora J=8``. > +2. Add "V=1" to the command line, try again, to see the verbose output. > +3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt > + in the container right before testing starts. You could either manually > + build QEMU and run tests from there, or press Ctrl-D to let the Docker > + testing continue. > +4. If you press Ctrl-D, the same building and testing procedure will begin, and > + will hopefully run into the error again. After that, you will be dropped to > + the prompt for debug. > + > +Options > +------- > + > +Various options can be used to affect how Docker tests are done. The full > +list is in the ``make docker`` help text. The frequently used ones are: > + > +* ``V=1``: the same as in top level ``make``. It will be propagated to the > + container and enable verbose output. > +* ``J=$N``: the number of parallel tasks in make commands in the container, > + similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in > + top level ``make`` will not be propagated into the container.) > +* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test > + failure" section. > + > +VM testing > +========== > + > +This test suite contains scripts that bootstrap various guest images that have > +necessary packages to build QEMU. The basic usage is documented in ``Makefile`` > +help which is displayed with ``make vm-test``. > + > +Quickstart > +---------- > + > +Run ``make vm-test`` to list available make targets. Invoke a specific make > +command to run build test in an image. For example, ``make vm-build-freebsd`` > +will build the source tree in the FreeBSD image. The command can be executed > +from either the source tree or the build dir; if the former, ``./configure`` is > +not needed. The command will then generate the test image in ``./tests/vm/`` > +under the working directory. > + > +Note: images created by the scripts accept a well-known RSA key pair for SSH > +access, so they SHOULD NOT be exposed to external interfaces if you are > +concerned about attackers taking control of the guest and potentially > +exploiting a QEMU security bug to compromise the host. > + > +QEMU binary > +----------- > + > +By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there > +isn't one, or if it is older than 2.10, the test won't work. In this case, > +provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. > + > +Make jobs > +--------- > + > +The ``-j$X`` option in the make command line is not propagated into the VM, > +specify ``J=$X`` to control the make jobs in the guest. > + > +Debugging > +--------- > + > +Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive > +debugging and verbose output. If this is not enough, see the next section. > + > +Manual invocation > +----------------- > + > +Each guest script is an executable script with the same command line options. > +For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: > + > +.. code:: > + > + $ cd $QEMU_SRC/tests/vm > + > + # To bootstrap the image > + $ ./netbsd --build-image --image /var/tmp/netbsd.img > + <...> > + > + # To run an arbitrary command in guest (the output will not be echoed unless > + # --debug is added) > + $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a > + > + # To build QEMU in guest > + $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC > + > + # To get to an interactive shell > + $ ./netbsd --interactive --image /var/tmp/netbsd.img sh > + > +Adding new guests > +----------------- > + > +Please look at existing guest scripts for how to add new guests. > + > +Most importantly, create a subclass of BaseVM and implement ``build_image()`` > +method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from > +the script's ``main()``. > + > +* Usually in ``build_image()``, a template image is downloaded from a > + predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and > + the checksum, so consider using it. > + > +* Once the image is downloaded, users, SSH server and QEMU build deps should > + be set up: > + > + - Root password set to ``BaseVM.ROOT_PASS`` > + - User ``BaseVM.GUEST_USER`` is created, and password set to > + ``BaseVM.GUEST_PASS`` > + - SSH service is enabled and started on boot, > + ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` > + file of both root and the normal user > + - DHCP client service is enabled and started on boot, so that it can > + automatically configure the virtio-net-pci NIC and communicate with QEMU > + user net (10.0.2.2) > + - Necessary packages are installed to untar the source tarball and build > + QEMU > + > +* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that > + untars a raw virtio-blk block device, which is the tarball data blob of the > + QEMU source tree, then configure/build it. Running "make check" is also > + recommended. > + > +Image fuzzer testing > +==================== > + > +An image fuzzer was added to exercise format drivers. Currently only qcow2 is > +supported. To start the fuzzer, run > + > +.. code:: > + > + tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 > + > +Alternatively, some command different from "qemu-img info" can be tested, by > +changing the ``-c`` option. > diff --git a/tests/vm/README b/tests/vm/README > index ae53dce6ee..f9c04cc0e7 100644 > --- a/tests/vm/README > +++ b/tests/vm/README > @@ -1,89 +1 @@ > -=== VM test suite to run build in guests === > - > -== Intro == > - > -This test suite contains scripts that bootstrap various guest images that have > -necessary packages to build QEMU. The basic usage is documented in Makefile > -help which is displayed with "make vm-test". > - > -== Quick start == > - > -Run "make vm-test" to list available make targets. Invoke a specific make > -command to run build test in an image. For example, "make vm-build-freebsd" > -will build the source tree in the FreeBSD image. The command can be executed > -from either the source tree or the build dir; if the former, ./configure is not > -needed. The command will then generate the test image in ./tests/vm/ under the > -working directory. > - > -Note: images created by the scripts accept a well-known RSA key pair for SSH > -access, so they SHOULD NOT be exposed to external interfaces if you are > -concerned about attackers taking control of the guest and potentially > -exploiting a QEMU security bug to compromise the host. > - > -== QEMU binary == > - > -By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there > -isn't one, or if it is older than 2.10, the test won't work. In this case, > -provide the QEMU binary in env var: QEMU=/path/to/qemu-2.10+. > - > -== Make jobs == > - > -The "-j$X" option in the make command line is not propagated into the VM, > -specify "J=$X" to control the make jobs in the guest. > - > -== Debugging == > - > -Add "DEBUG=1" and/or "V=1" to the make command to allow interactive debugging > -and verbose output. If this is not enough, see the next section. > - > -== Manual invocation == > - > -Each guest script is an executable script with the same command line options. > -For example to work with the netbsd guest, use $QEMU_SRC/tests/vm/netbsd: > - > - $ cd $QEMU_SRC/tests/vm > - > - # To bootstrap the image > - $ ./netbsd --build-image --image /var/tmp/netbsd.img > - <...> > - > - # To run an arbitrary command in guest (the output will not be echoed unless > - # --debug is added) > - $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a > - > - # To build QEMU in guest > - $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC > - > - # To get to an interactive shell > - $ ./netbsd --interactive --image /var/tmp/netbsd.img sh > - > -== Adding new guests == > - > -Please look at existing guest scripts for how to add new guests. > - > -Most importantly, create a subclass of BaseVM and implement build_image() > -method and define BUILD_SCRIPT, then finally call basevm.main() from the > -script's main(). > - > - - Usually in build_image(), a template image is downloaded from a predefined > - URL. BaseVM._download_with_cache() takes care of the cache and the > - checksum, so consider using it. > - > - - Once the image is downloaded, users, SSH server and QEMU build deps should > - be set up: > - > - * Root password set to BaseVM.ROOT_PASS > - * User BaseVM.GUEST_USER is created, and password set to BaseVM.GUEST_PASS > - * SSH service is enabled and started on boot, > - $QEMU_SRC/tests/keys/id_rsa.pub is added to ssh's "authorized_keys" file > - of both root and the normal user > - * DHCP client service is enabled and started on boot, so that it can > - automatically configure the virtio-net-pci NIC and communicate with QEMU > - user net (10.0.2.2) > - * Necessary packages are installed to untar the source tarball and build > - QEMU > - > - - Write a proper BUILD_SCRIPT template, which should be a shell script that > - untars a raw virtio-blk block device, which is the tarball data blob of the > - QEMU source tree, then configure/build it. Running "make check" is also > - recommended. > +See docs/devel/testing.rst for help. >
On Wed, 02/07 15:31, Paolo Bonzini wrote: > On 01/02/2018 03:20, Fam Zheng wrote: > > To make our efforts on QEMU testing easier to consume by contributors, > > let's add a document. For example, Patchew reports build errors on > > patches that should be relatively easy to reproduce with a few steps, and > > it is much nicer if there is such a documentation that it can refer to. > > > > This focuses on how to run existing tests and how to write new test > > cases, without going into the frameworks themselves. > > > > The VM based testing section is moved from tests/vm/README which now > > is a single line pointing to the new doc. > > > > Signed-off-by: Fam Zheng <famz@redhat.com> > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > > Queued, thanks. Thank you, I've included this in the pull request I just sent. :) Fam
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst new file mode 100644 index 0000000000..0ca1a2d4b5 --- /dev/null +++ b/docs/devel/testing.rst @@ -0,0 +1,486 @@ +=============== +Testing in QEMU +=============== + +This document describes the testing infrastructure in QEMU. + +Testing with "make check" +========================= + +The "make check" testing family includes most of the C based tests in QEMU. For +a quick help, run ``make check-help`` from the source tree. + +The usual way to run these tests is: + +.. code:: + + make check + +which includes QAPI schema tests, unit tests, and QTests. Different sub-types +of "make check" tests will be explained below. + +Before running tests, it is best to build QEMU programs first. Some tests +expect the executables to exist and will fail with obscure messages if they +cannot find them. + +Unit tests +---------- + +Unit tests, which can be invoked with ``make check-unit``, are simple C tests +that typically link to individual QEMU object files and exercise them by +calling exported functions. + +If you are writing new code in QEMU, consider adding a unit test, especially +for utility modules that are relatively stateless or have few dependencies. To +add a new unit test: + +1. Create a new source file. For example, ``tests/foo-test.c``. + +2. Write the test. Normally you would include the header file which exports + the module API, then verify the interface behaves as expected from your + test. The test code should be organized with the glib testing framework. + Copying and modifying an existing test is usually a good idea. + +3. Add the test to ``tests/Makefile.include``. First, name the unit test + program and add it to ``$(check-unit-y)``; then add a rule to build the + executable. Optionally, you can add a magical variable to support ``gcov``. + For example: + +.. code:: + + check-unit-y += tests/foo-test$(EXESUF) + tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y) + ... + gcov-files-foo-test-y = util/foo.c + +Since unit tests don't require environment variables, the simplest way to debug +a unit test failure is often directly invoking it or even running it under +``gdb``. However there can still be differences in behavior between ``make`` +invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment +variable (which affects memory reclamation and catches invalid pointers better) +and gtester options. If necessary, you can run + +.. code:: + make check-unit V=1 + +and copy the actual command line which executes the unit test, then run +it from the command line. + +QTest +----- + +QTest is a device emulation testing framework. It can be very useful to test +device models; it could also control certain aspects of QEMU (such as virtual +clock stepping), with a special purpose "qtest" protocol. Refer to the +documentation in ``qtest.c`` for more details of the protocol. + +QTest cases can be executed with + +.. code:: + + make check-qtest + +The QTest library is implemented by ``tests/libqtest.c`` and the API is defined +in ``tests/libqtest.h``. + +Consider adding a new QTest case when you are introducing a new virtual +hardware, or extending one if you are adding functionalities to an existing +virtual device. + +On top of libqtest, a higher level library, ``libqos``, was created to +encapsulate common tasks of device drivers, such as memory management and +communicating with system buses or devices. Many virtual device tests use +libqos instead of directly calling into libqtest. + +Steps to add a new QTest case are: + +1. Create a new source file for the test. (More than one file can be added as + necessary.) For example, ``tests/test-foo-device.c``. + +2. Write the test code with the glib and libqtest/libqos API. See also existing + tests and the library headers for reference. + +3. Register the new test in ``tests/Makefile.include``. Add the test executable + name to an appropriate ``check-qtest-*-y`` variable. For example: + + ``check-qtest-generic-y = tests/test-foo-device$(EXESUF)`` + +4. Add object dependencies of the executable in the Makefile, including the + test source file(s) and other interesting objects. For example: + + ``tests/test-foo-device$(EXESUF): tests/test-foo-device.o $(libqos-obj-y)`` + +Debugging a QTest failure is slightly harder than the unit test because the +tests look up QEMU program names in the environment variables, such as +``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy +to attach gdb to the QEMU process spawned from the test. But manual invoking +and using gdb on the test is still simple to do: find out the actual command +from the output of + +.. code:: + make check-qtest V=1 + +which you can run manually. + +QAPI schema tests +----------------- + +The QAPI schema tests validate the QAPI parser used by QMP, by feeding +predefined input to the parser and comparing the result with the reference +output. + +The input/output data is managed under the ``tests/qapi-schema`` directory. +Each test case includes four files that have a common base name: + + * ``${casename}.json`` - the file contains the JSON input for feeding the + parser + * ``${casename}.out`` - the file contains the expected stdout from the parser + * ``${casename}.err`` - the file contains the expected stderr from the parser + * ``${casename}.exit`` - the expected error code + +Consider adding a new QAPI schema test when you are making a change on the QAPI +parser (either fixing a bug or extending/modifying the syntax). To do this: + +1. Add four files for the new case as explained above. For example: + + ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. + +2. Add the new test in ``tests/Makefile.include``. For example: + + ``qapi-schema += foo.json`` + +check-block +----------- + +``make check-block`` is a legacy command to invoke block layer iotests and is +rarely used. See "QEMU iotests" section below for more information. + +GCC gcov support +---------------- + +``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the +tested code. To use it, configure QEMU with ``--enable-gcov`` option and build. +Then run ``make check`` as usual. There will be additional ``gcov`` output as +the testing goes on, showing the test coverage percentage numbers per analyzed +source file. More detailed reports can be obtained by running ``gcov`` command +on the output files under ``$build_dir/tests/``, please read the ``gcov`` +documentation for more information. + +QEMU iotests +============ + +QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing +framework widely used to test block layer related features. It is higher level +than "make check" tests and 99% of the code is written in bash or Python +scripts. The testing success criteria is golden output comparison, and the +test files are named with numbers. + +To run iotests, make sure QEMU is built successfully, then switch to the +``tests/qemu-iotests`` directory under the build directory, and run ``./check`` +with desired arguments from there. + +By default, "raw" format and "file" protocol is used; all tests will be +executed, except the unsupported ones. You can override the format and protocol +with arguments: + +.. code:: + + # test with qcow2 format + ./check -qcow2 + # or test a different protocol + ./check -nbd + +It's also possible to list test numbers explicitly: + +.. code:: + + # run selected cases with qcow2 format + ./check -qcow2 001 030 153 + +Cache mode can be selected with the "-c" option, which may help reveal bugs +that are specific to certain cache mode. + +More options are supported by the ``./check`` script, run ``./check -h`` for +help. + +Writing a new test case +----------------------- + +Consider writing a tests case when you are making any changes to the block +layer. An iotest case is usually the choice for that. There are already many +test cases, so it is possible that extending one of them may achieve the goal +and save the boilerplate to create one. (Unfortunately, there isn't a 100% +reliable way to find a related one out of hundreds of tests. One approach is +using ``git grep``.) + +Usually an iotest case consists of two files. One is an executable that +produces output to stdout and stderr, the other is the expected reference +output. They are given the same number in file names. E.g. Test script ``055`` +and reference output ``055.out``. + +In rare cases, when outputs differ between cache mode ``none`` and others, a +``.out.nocache`` file is added. In other cases, when outputs differ between +image formats, more than one ``.out`` files are created ending with the +respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. + +There isn't a hard rule about how to write a test script, but a new test is +usually a (copy and) modification of an existing case. There are a few +commonly used ways to create a test: + +* A Bash script. It will make use of several environmental variables related + to the testing procedure, and could source a group of ``common.*`` libraries + for some common helper routines. + +* A Python unittest script. Import ``iotests`` and create a subclass of + ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of + this approach is that the output is too scarce, and the script is considered + harder to debug. + +* A simple Python script without using unittest module. This could also import + ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit + from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest + execution. This is a combination of 1 and 2. + +Pick the language per your preference since both Bash and Python have +comparable library support for invoking and interacting with QEMU programs. If +you opt for Python, it is strongly recommended to write Python 3 compatible +code. + +Docker based tests +================== + +Introduction +------------ + +The Docker testing framework in QEMU utilizes public Docker images to build and +test QEMU in predefined and widely accessible Linux environments. This makes +it possible to expand the test coverage across distros, toolchain flavors and +library versions. + +Prerequisites +------------- + +Install "docker" with the system package manager and start the Docker service +on your development machine, then make sure you have the privilege to run +Docker commands. Typically it means setting up passwordless ``sudo docker`` +command or login as root. For example: + +.. code:: + + $ sudo yum install docker + $ # or `apt-get install docker` for Ubuntu, etc. + $ sudo systemctl start docker + $ sudo docker ps + +The last command should print an empty table, to verify the system is ready. + +An alternative method to set up permissions is by adding the current user to +"docker" group and making the docker daemon socket file (by default +``/var/run/docker.sock``) accessible to the group: + +.. code:: + + $ sudo groupadd docker + $ sudo usermod $USER -G docker + $ sudo chown :docker /var/run/docker.sock + +Note that any one of above configurations makes it possible for the user to +exploit the whole host with Docker bind mounting or other privileged +operations. So only do it on development machines. + +Quickstart +---------- + +From source tree, type ``make docker`` to see the help. Testing can be started +without configuring or building QEMU (``configure`` and ``make`` are done in +the container, with parameters defined by the make target): + +.. code:: + + make docker-test-build@min-glib + +This will create a container instance using the ``min-glib`` image (the image +is downloaded and initialized automatically), in which the ``test-build`` job +is executed. + +Images +------ + +Along with many other images, the ``min-glib`` image is defined in a Dockerfile +in ``tests/docker/dockefiles/``, called ``min-glib.docker``. ``make docker`` +command will list all the available images. + +To add a new image, simply create a new ``.docker`` file under the +``tests/docker/dockerfiles/`` directory. + +A ``.pre`` script can be added beside the ``.docker`` file, which will be +executed before building the image under the build context directory. This is +mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, +for example, to make qemu-user powered cross build containers work. + +Tests +----- + +Different tests are added to cover various configurations to build and test +QEMU. Docker tests are the executables under ``tests/docker`` named +``test-*``. They are typically shell scripts and are built on top of a shell +library, ``tests/docker/common.rc``, which provides helpers to find the QEMU +source and build it. + +The full list of tests is printed in the ``make docker`` help. + +Tools +----- + +There are executables that are created to run in a specific Docker environment. +This makes it easy to write scripts that have heavy or special dependencies, +but are still very easy to use. + +Currently the only tool is ``travis``, which mimics the Travis-CI tests in a +container. It runs in the ``travis`` image: + +.. code:: + + make docker-travis@travis + +Debugging a Docker test failure +------------------------------- + +When CI tasks, maintainers or yourself report a Docker test failure, follow the +below steps to debug it: + +1. Locally reproduce the failure with the reported command line. E.g. run + ``make docker-test-mingw@fedora J=8``. +2. Add "V=1" to the command line, try again, to see the verbose output. +3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt + in the container right before testing starts. You could either manually + build QEMU and run tests from there, or press Ctrl-D to let the Docker + testing continue. +4. If you press Ctrl-D, the same building and testing procedure will begin, and + will hopefully run into the error again. After that, you will be dropped to + the prompt for debug. + +Options +------- + +Various options can be used to affect how Docker tests are done. The full +list is in the ``make docker`` help text. The frequently used ones are: + +* ``V=1``: the same as in top level ``make``. It will be propagated to the + container and enable verbose output. +* ``J=$N``: the number of parallel tasks in make commands in the container, + similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in + top level ``make`` will not be propagated into the container.) +* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test + failure" section. + +VM testing +========== + +This test suite contains scripts that bootstrap various guest images that have +necessary packages to build QEMU. The basic usage is documented in ``Makefile`` +help which is displayed with ``make vm-test``. + +Quickstart +---------- + +Run ``make vm-test`` to list available make targets. Invoke a specific make +command to run build test in an image. For example, ``make vm-build-freebsd`` +will build the source tree in the FreeBSD image. The command can be executed +from either the source tree or the build dir; if the former, ``./configure`` is +not needed. The command will then generate the test image in ``./tests/vm/`` +under the working directory. + +Note: images created by the scripts accept a well-known RSA key pair for SSH +access, so they SHOULD NOT be exposed to external interfaces if you are +concerned about attackers taking control of the guest and potentially +exploiting a QEMU security bug to compromise the host. + +QEMU binary +----------- + +By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there +isn't one, or if it is older than 2.10, the test won't work. In this case, +provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. + +Make jobs +--------- + +The ``-j$X`` option in the make command line is not propagated into the VM, +specify ``J=$X`` to control the make jobs in the guest. + +Debugging +--------- + +Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive +debugging and verbose output. If this is not enough, see the next section. + +Manual invocation +----------------- + +Each guest script is an executable script with the same command line options. +For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: + +.. code:: + + $ cd $QEMU_SRC/tests/vm + + # To bootstrap the image + $ ./netbsd --build-image --image /var/tmp/netbsd.img + <...> + + # To run an arbitrary command in guest (the output will not be echoed unless + # --debug is added) + $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a + + # To build QEMU in guest + $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC + + # To get to an interactive shell + $ ./netbsd --interactive --image /var/tmp/netbsd.img sh + +Adding new guests +----------------- + +Please look at existing guest scripts for how to add new guests. + +Most importantly, create a subclass of BaseVM and implement ``build_image()`` +method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from +the script's ``main()``. + +* Usually in ``build_image()``, a template image is downloaded from a + predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and + the checksum, so consider using it. + +* Once the image is downloaded, users, SSH server and QEMU build deps should + be set up: + + - Root password set to ``BaseVM.ROOT_PASS`` + - User ``BaseVM.GUEST_USER`` is created, and password set to + ``BaseVM.GUEST_PASS`` + - SSH service is enabled and started on boot, + ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` + file of both root and the normal user + - DHCP client service is enabled and started on boot, so that it can + automatically configure the virtio-net-pci NIC and communicate with QEMU + user net (10.0.2.2) + - Necessary packages are installed to untar the source tarball and build + QEMU + +* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that + untars a raw virtio-blk block device, which is the tarball data blob of the + QEMU source tree, then configure/build it. Running "make check" is also + recommended. + +Image fuzzer testing +==================== + +An image fuzzer was added to exercise format drivers. Currently only qcow2 is +supported. To start the fuzzer, run + +.. code:: + + tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 + +Alternatively, some command different from "qemu-img info" can be tested, by +changing the ``-c`` option. diff --git a/tests/vm/README b/tests/vm/README index ae53dce6ee..f9c04cc0e7 100644 --- a/tests/vm/README +++ b/tests/vm/README @@ -1,89 +1 @@ -=== VM test suite to run build in guests === - -== Intro == - -This test suite contains scripts that bootstrap various guest images that have -necessary packages to build QEMU. The basic usage is documented in Makefile -help which is displayed with "make vm-test". - -== Quick start == - -Run "make vm-test" to list available make targets. Invoke a specific make -command to run build test in an image. For example, "make vm-build-freebsd" -will build the source tree in the FreeBSD image. The command can be executed -from either the source tree or the build dir; if the former, ./configure is not -needed. The command will then generate the test image in ./tests/vm/ under the -working directory. - -Note: images created by the scripts accept a well-known RSA key pair for SSH -access, so they SHOULD NOT be exposed to external interfaces if you are -concerned about attackers taking control of the guest and potentially -exploiting a QEMU security bug to compromise the host. - -== QEMU binary == - -By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there -isn't one, or if it is older than 2.10, the test won't work. In this case, -provide the QEMU binary in env var: QEMU=/path/to/qemu-2.10+. - -== Make jobs == - -The "-j$X" option in the make command line is not propagated into the VM, -specify "J=$X" to control the make jobs in the guest. - -== Debugging == - -Add "DEBUG=1" and/or "V=1" to the make command to allow interactive debugging -and verbose output. If this is not enough, see the next section. - -== Manual invocation == - -Each guest script is an executable script with the same command line options. -For example to work with the netbsd guest, use $QEMU_SRC/tests/vm/netbsd: - - $ cd $QEMU_SRC/tests/vm - - # To bootstrap the image - $ ./netbsd --build-image --image /var/tmp/netbsd.img - <...> - - # To run an arbitrary command in guest (the output will not be echoed unless - # --debug is added) - $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a - - # To build QEMU in guest - $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC - - # To get to an interactive shell - $ ./netbsd --interactive --image /var/tmp/netbsd.img sh - -== Adding new guests == - -Please look at existing guest scripts for how to add new guests. - -Most importantly, create a subclass of BaseVM and implement build_image() -method and define BUILD_SCRIPT, then finally call basevm.main() from the -script's main(). - - - Usually in build_image(), a template image is downloaded from a predefined - URL. BaseVM._download_with_cache() takes care of the cache and the - checksum, so consider using it. - - - Once the image is downloaded, users, SSH server and QEMU build deps should - be set up: - - * Root password set to BaseVM.ROOT_PASS - * User BaseVM.GUEST_USER is created, and password set to BaseVM.GUEST_PASS - * SSH service is enabled and started on boot, - $QEMU_SRC/tests/keys/id_rsa.pub is added to ssh's "authorized_keys" file - of both root and the normal user - * DHCP client service is enabled and started on boot, so that it can - automatically configure the virtio-net-pci NIC and communicate with QEMU - user net (10.0.2.2) - * Necessary packages are installed to untar the source tarball and build - QEMU - - - Write a proper BUILD_SCRIPT template, which should be a shell script that - untars a raw virtio-blk block device, which is the tarball data blob of the - QEMU source tree, then configure/build it. Running "make check" is also - recommended. +See docs/devel/testing.rst for help.