| Message ID | 20210924185506.2542588-2-crosa@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Acceptance Tests: use Avocado 91.0 features and other improvements | expand |
On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa <crosa@redhat.com> wrote: > > Avocado 91.0, brings, among other changes, a switch to a new runner > implementation, known as "nrunner". While my personal testing shows > that it's suitable for running the QEMU tests, there are some > considerations that should be addressed before the change. > > For instance, the "nrunner" implementation will run tests in parallel > by default. With more tests running, and resources shared, some tests > may fail with timeouts on some environments. > > So, for now, let's bump the Avocado version to allow for the other > features to be used. And the "nrunner" implementation to be used by > those that choose to do so. > > More information can be found at the release notes at: > https://avocado-framework.readthedocs.io/en/latest/releases/91_0.html > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > docs/devel/testing.rst | 12 ++++++++++++ > tests/Makefile.include | 2 ++ > tests/requirements.txt | 2 +- > 3 files changed, 15 insertions(+), 1 deletion(-) > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index 4a0abbf23d..2818c58ff8 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -740,6 +740,18 @@ may be invoked by running: > > tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ > > +Running tests in parallel > +------------------------- > + > +The current ``make check-acceptance`` target continues to use the > +Avocado runner implementation simply known as "runner". But, it's now > +possible to opt-in and choose the "nrunner" implementation, which, > +among other things, allows for parallel execution of tests: > + > +.. code:: > + > + make AVOCADO_RUNNER=nrunner check-acceptance The examples in the documentation are all setting the environment variable at the end of the `make` command. For consistency, I would move the definition of the AVOCADO_RUNNER to the end of the command. > + > Manual Installation > ------------------- > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 6e16c05f10..ac289a2e41 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -92,6 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > # Any number of command separated loggers are accepted. For more > # information please refer to "avocado --help". > AVOCADO_SHOW=app > +AVOCADO_RUNNER=runner > AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS))) > > $(TESTS_VENV_DIR): $(TESTS_VENV_REQ) > @@ -128,6 +129,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images > $(call quiet-command, \ > $(TESTS_VENV_DIR)/bin/python -m avocado \ > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > + --test-runner=$(AVOCADO_RUNNER) \ > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > $(AVOCADO_TAGS) \ > $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ > diff --git a/tests/requirements.txt b/tests/requirements.txt > index a21b59b443..40af24c664 100644 > --- a/tests/requirements.txt > +++ b/tests/requirements.txt > @@ -1,5 +1,5 @@ > # Add Python module requirements, one per line, to be installed > # in the tests/venv Python virtual environment. For more info, > # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > -avocado-framework==88.1 > +avocado-framework==91.0 > pycdlib==1.11.0 > -- > 2.31.1 > Apart from a tiny documentation change, the code is welcome as it allows using the new Avocado runner. Reviewed-by: Willian Rampazzo <willianr@redhat.com>
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 4a0abbf23d..2818c58ff8 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -740,6 +740,18 @@ may be invoked by running: tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ +Running tests in parallel +------------------------- + +The current ``make check-acceptance`` target continues to use the +Avocado runner implementation simply known as "runner". But, it's now +possible to opt-in and choose the "nrunner" implementation, which, +among other things, allows for parallel execution of tests: + +.. code:: + + make AVOCADO_RUNNER=nrunner check-acceptance + Manual Installation ------------------- diff --git a/tests/Makefile.include b/tests/Makefile.include index 6e16c05f10..ac289a2e41 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -92,6 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help". AVOCADO_SHOW=app +AVOCADO_RUNNER=runner AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS))) $(TESTS_VENV_DIR): $(TESTS_VENV_REQ) @@ -128,6 +129,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images $(call quiet-command, \ $(TESTS_VENV_DIR)/bin/python -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ + --test-runner=$(AVOCADO_RUNNER) \ --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ $(AVOCADO_TAGS) \ $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ diff --git a/tests/requirements.txt b/tests/requirements.txt index a21b59b443..40af24c664 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,5 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==88.1 +avocado-framework==91.0 pycdlib==1.11.0
Avocado 91.0, brings, among other changes, a switch to a new runner implementation, known as "nrunner". While my personal testing shows that it's suitable for running the QEMU tests, there are some considerations that should be addressed before the change. For instance, the "nrunner" implementation will run tests in parallel by default. With more tests running, and resources shared, some tests may fail with timeouts on some environments. So, for now, let's bump the Avocado version to allow for the other features to be used. And the "nrunner" implementation to be used by those that choose to do so. More information can be found at the release notes at: https://avocado-framework.readthedocs.io/en/latest/releases/91_0.html Signed-off-by: Cleber Rosa <crosa@redhat.com> --- docs/devel/testing.rst | 12 ++++++++++++ tests/Makefile.include | 2 ++ tests/requirements.txt | 2 +- 3 files changed, 15 insertions(+), 1 deletion(-)