| Message ID | 20191113112649.14322-6-thuth@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Improvements for the Travis CI | expand |
Thomas Huth <thuth@redhat.com> writes: > While working on the travis.yml file, I've run into cases where > all tests are reported as "SKIP" by the run_test.sh script (e.g. > when QEMU could not be started). This should not result in a > successful test run, so mark it as failed if not at least one > test passed. But doesn't this mean you could have everything fail except one pass and still report success? > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > .travis.yml | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/.travis.yml b/.travis.yml > index 9ceb04d..aacf7d2 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -115,3 +115,4 @@ script: > - make -j3 > - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt > - if grep -q FAIL results.txt ; then exit 1 ; fi > + - if ! grep -q PASS results.txt ; then exit 1 ; fi -- Alex Bennée
On 13/11/2019 16.06, Alex Bennée wrote: > > Thomas Huth <thuth@redhat.com> writes: > >> While working on the travis.yml file, I've run into cases where >> all tests are reported as "SKIP" by the run_test.sh script (e.g. >> when QEMU could not be started). This should not result in a >> successful test run, so mark it as failed if not at least one >> test passed. > > But doesn't this mean you could have everything fail except one pass and > still report success? The FAILs are already handled one line earlier... >> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> .travis.yml | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/.travis.yml b/.travis.yml >> index 9ceb04d..aacf7d2 100644 >> --- a/.travis.yml >> +++ b/.travis.yml >> @@ -115,3 +115,4 @@ script: >> - make -j3 >> - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt >> - if grep -q FAIL results.txt ; then exit 1 ; fi ... here -----------^ >> + - if ! grep -q PASS results.txt ; then exit 1 ; fi Maybe it would also be nicer to provide proper exit values in the run_tests.sh script, but the logic there gives me a bad headache... grep'ing for FAIL and PASS in the yml script is way more easy. Thomas
Thomas Huth <thuth@redhat.com> writes: > On 13/11/2019 16.06, Alex Bennée wrote: >> >> Thomas Huth <thuth@redhat.com> writes: >> >>> While working on the travis.yml file, I've run into cases where >>> all tests are reported as "SKIP" by the run_test.sh script (e.g. >>> when QEMU could not be started). This should not result in a >>> successful test run, so mark it as failed if not at least one >>> test passed. >> >> But doesn't this mean you could have everything fail except one pass and >> still report success? > > The FAILs are already handled one line earlier... Oops I should have noticed that: Reviewed-by: Alex Bennée <alex.bennee@linaro.org> > >>> >>> Signed-off-by: Thomas Huth <thuth@redhat.com> >>> --- >>> .travis.yml | 1 + >>> 1 file changed, 1 insertion(+) >>> >>> diff --git a/.travis.yml b/.travis.yml >>> index 9ceb04d..aacf7d2 100644 >>> --- a/.travis.yml >>> +++ b/.travis.yml >>> @@ -115,3 +115,4 @@ script: >>> - make -j3 >>> - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt >>> - if grep -q FAIL results.txt ; then exit 1 ; fi > > ... here -----------^ > >>> + - if ! grep -q PASS results.txt ; then exit 1 ; fi > > Maybe it would also be nicer to provide proper exit values in the > run_tests.sh script, but the logic there gives me a bad headache... > grep'ing for FAIL and PASS in the yml script is way more easy. > > Thomas -- Alex Bennée
diff --git a/.travis.yml b/.travis.yml index 9ceb04d..aacf7d2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -115,3 +115,4 @@ script: - make -j3 - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt - if grep -q FAIL results.txt ; then exit 1 ; fi + - if ! grep -q PASS results.txt ; then exit 1 ; fi
While working on the travis.yml file, I've run into cases where all tests are reported as "SKIP" by the run_test.sh script (e.g. when QEMU could not be started). This should not result in a successful test run, so mark it as failed if not at least one test passed. Signed-off-by: Thomas Huth <thuth@redhat.com> --- .travis.yml | 1 + 1 file changed, 1 insertion(+)