[0/9] Avocado tests: filter out tests using TCG booting full blown distros

Message

Cleber Rosa Feb. 25, 2022, 9:01 p.m. UTC

It was previously reported[1] and discussed that tests booting full
blown distros and relying on TCG would take too much time to run,
especially in the environments given by GitLab CI's shared runners.

This is an implementation of a proposal to exclude those tests from
being run by default on `make check-avocado` invocations.  To make it
extra clear, all tests are still available, but those that are tagged
with "accel:tcg" and "boots:distro", are filtered out by default on
`make check-avocado`.

This is the situation of the Avocado GitLab CI jobs with and without
the changes in this PS:

                         +------------------+--- ----------------+
                         |        Now[2]    |       Before[3]    |
+------------------------+------------------+--------------------+
|          Job           |  Length |  Tests |  Length  |  Tests  |
|         Name           | (mm:ss) |   Run  |  (mm:ss) |   Run   |
+------------------------+------------------+-------------------+
|avocado-system-alpine   |  06:33      16   |   20:30       18   |
|avocado-system-debian   |  12:06      24   |   13:05       24   |
|avocado-system-centos   |  09:58      41   |   24:15       44   |
|avocado-system-fedora   |  08:50      35   |   08:59       35   |
|avocado-system-opensuse |  08:09      38   |   27:21       42   |
|avocado-system-ubuntu   |  06:52      16   |   18:52       18   |
|avocado-cfi-x86_64      |  05:43      27   |   15:07       29   |
+------------------------+------------------+--------------------+
|TOTALS                  |  58:11     197   | 2:08:09      210   |
+------------------------+------------------+--------------------+

Assuming the jobs run in parallel, the overall wait time for all the
Avocado jobs to complete is now ~12 minutes.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg07271.html
[2] https://gitlab.com/cleber.gnu/qemu/-/pipelines/479720240
[3] https://gitlab.com/qemu-project/qemu/-/pipelines/478580581

Cleber Rosa (9):
  Avocado GitLab CI jobs: don't reset TARGETS and simplify commands
  Avocado tests: use logging namespace that is preserved in test logs
  Avocado migration test: adapt to "utils.network" API namespace change
  Avocado: bump to version 95.0
  tests/avocado/linux_ssh_mips_malta.py: add missing accel (tcg) tag
  tests/avocado/virtiofs_submounts.py: shared_dir may not exist
  Avocado tests: improve documentation on tag filtering
  Avocado tests: classify tests based on what it's booted
  Avocado tests: don't run tests with TCG that boot full blown distros

 .gitlab-ci.d/buildtest-template.yml       |  3 ++
 .gitlab-ci.d/buildtest.yml                |  9 ----
 docs/devel/testing.rst                    | 22 +++++++++
 tests/Makefile.include                    |  6 ++-
 tests/avocado/avocado_qemu/__init__.py    | 10 ++---
 tests/avocado/boot_linux.py               |  4 ++
 tests/avocado/boot_linux_console.py       | 54 +++++++++++++++++++++++
 tests/avocado/boot_xen.py                 |  3 ++
 tests/avocado/hotplug_cpu.py              |  1 +
 tests/avocado/intel_iommu.py              |  1 +
 tests/avocado/linux_initrd.py             |  5 ++-
 tests/avocado/linux_ssh_mips_malta.py     |  5 +++
 tests/avocado/machine_arm_canona1100.py   |  1 +
 tests/avocado/machine_arm_integratorcp.py |  7 ++-
 tests/avocado/machine_arm_n8x0.py         |  2 +
 tests/avocado/machine_avr6.py             |  1 +
 tests/avocado/machine_m68k_nextcube.py    |  1 +
 tests/avocado/machine_microblaze.py       |  1 +
 tests/avocado/machine_mips_fuloong2e.py   |  1 +
 tests/avocado/machine_mips_loongson3v.py  |  1 +
 tests/avocado/machine_mips_malta.py       |  6 ++-
 tests/avocado/machine_rx_gdbsim.py        |  2 +
 tests/avocado/machine_s390_ccw_virtio.py  |  4 ++
 tests/avocado/machine_sparc64_sun4u.py    |  1 +
 tests/avocado/machine_sparc_leon3.py      |  1 +
 tests/avocado/migration.py                |  4 +-
 tests/avocado/multiprocess.py             |  4 ++
 tests/avocado/ppc_405.py                  |  2 +
 tests/avocado/ppc_bamboo.py               |  2 +
 tests/avocado/ppc_mpc8544ds.py            |  1 +
 tests/avocado/ppc_prep_40p.py             |  1 +
 tests/avocado/ppc_pseries.py              |  1 +
 tests/avocado/ppc_virtex_ml507.py         |  1 +
 tests/avocado/replay_kernel.py            | 33 ++++++++++++--
 tests/avocado/replay_linux.py             |  6 +--
 tests/avocado/reverse_debugging.py        |  6 +--
 tests/avocado/smmu.py                     |  1 +
 tests/avocado/tcg_plugins.py              |  3 ++
 tests/avocado/tesseract_utils.py          |  6 +--
 tests/avocado/virtio-gpu.py               |  2 +
 tests/avocado/virtio_check_params.py      |  3 +-
 tests/avocado/virtiofs_submounts.py       |  8 ++--
 tests/requirements.txt                    |  2 +-
 43 files changed, 197 insertions(+), 41 deletions(-)