[v2,0/9] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

Message

Cleber Rosa Aug. 6, 2024, 5:31 p.m. UTC

This is a *long* overdue update of the Avocado version used in QEMU.
It comes a time where the role of the runner and the libraries are
being discussed and questioned.

These exact commits have been staging on my side for over 30 days now,
and I was exceeding what I should in terms of testing before posting.
I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Some examples of runs in the CI can be seen below:

* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
   RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 432.63 s

* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
   RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 148.99 s

Changes from v1 (references are from v1):

This version first applies the Avocado version bump (now PATCH 1),
adding to that patch the serial execution behavior.  Avocado 103.0 LTS
can run all existing tests in QEMU without any changes, so it's safe
to pick PATCH 1 *only* from this series if needed.  A GitLab CI job that
does only that can be seen here:

 https://gitlab.com/cleber.gnu/qemu/-/pipelines/1402633650

Details on changes:

 * Moved "Bump avocado to 103.0" to first patch

 * Patches already applied:
    - [PATCH 1/13] tests/avocado: mips: fallback to HTTP given certificate expiration
    - [PATCH 2/13] tests/avocado: mips: add hint for fetchasset plugin
    - [PATCH 8/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

  * Dropped patches:
    - [PATCH 3/13] tests/avocado/intel_iommu.py: increase timeout
    - [PATCH 6/13] tests/avocado: use more distinct names for assets
      - Replaced with "tests/avocado: simplify parameters on fetch_asset with name only"
    - [PATCH 7/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

  * [PATCH 5/13] tests/avocado: machine aarch64: standardize location and RO access:
    - Fixed rebase mistake

  * [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
    - Use "snapshot=on" instead of new copy of file

  * New patches:
    - tests/avocado: apply proper skipUnless decorator
    - tests/avocado: simplify parameters on fetch_asset with name only

Cleber Rosa (9):
  Bump avocado to 103.0
  tests/avocado: apply proper skipUnless decorator
  tests/avocado: add cdrom permission related tests
  tests/avocado: machine aarch64: standardize location and RO access
  tests/avocado: simplify parameters on fetch_asset with name only
  tests/avocado/boot_xen.py: fetch kernel during test setUp()
  tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
  tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
  Avocado tests: allow for parallel execution of tests

 docs/devel/testing.rst                   | 12 +++++++
 pythondeps.toml                          |  2 +-
 tests/Makefile.include                   |  6 +++-
 tests/avocado/boot_linux_console.py      |  1 -
 tests/avocado/boot_xen.py                | 13 ++++----
 tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
 tests/avocado/intel_iommu.py             |  1 -
 tests/avocado/kvm_xen_guest.py           |  2 +-
 tests/avocado/linux_initrd.py            |  1 -
 tests/avocado/machine_aarch64_sbsaref.py |  4 +--
 tests/avocado/machine_aarch64_virt.py    | 14 ++++----
 tests/avocado/machine_aspeed.py          |  2 --
 tests/avocado/machine_mips_malta.py      |  2 --
 tests/avocado/machine_rx_gdbsim.py       |  2 --
 tests/avocado/netdev-ethtool.py          |  2 +-
 tests/avocado/reverse_debugging.py       |  4 ---
 tests/avocado/smmu.py                    |  1 -
 tests/avocado/tuxrun_baselines.py        | 16 ++++-----
 18 files changed, 82 insertions(+), 44 deletions(-)
 create mode 100644 tests/avocado/cdrom.py

Comments

Marcin Juszkiewicz Aug. 6, 2024, 6:20 p.m. UTC | #1

On 6.08.2024 19:31, Cleber Rosa wrote:
> This is along  overdue update of the Avocado version used in QEMU.
> It comes a time where the role of the runner and the libraries are
> being discussed and questioned.
> 
> These exact commits have been staging on my side for over 30 days now,
> and I was exceeding what I should in terms of testing before posting.
> I apologize for the miscalculation.
> 
> Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Tested-by: Marcin Juszkiewicz <[e-mail]>

SBSA Reference Platform tests can be done in 1/3rd of time is a nice update.

Serial run:
real    6m20,324s
user    12m18,446s
sys     0m36,686s

Parallel (4) run:
real    2m22,658s
user    11m50,514s
sys     0m26,088s