| Message ID | 20170616175704.7832-17-anthony.perard@citrix.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Anthony PERARD writes ("[OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"):
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Again, does this not mean we're going to suffer a maintenance burden
as tempest grows new inapplicable tests ?
Other possibilities that come to my mind: ideally the tempest tests
would have metadata so that particular classes of tests could be
skipped. Alternatively, if they aren't disruptive or very slow, if
run, we could run them anyway as substeps.
Which makes me think: maybe the tempest tests want to be substeps
anyway. Is that possible ? (Does tempest speak subunit or
something ?)
Ian.
On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote: > Anthony PERARD writes ("[OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"): > > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> > > Again, does this not mean we're going to suffer a maintenance burden > as tempest grows new inapplicable tests ? That exactly what is happening with the OpenStack CI loop, from time to time, there are new tests, that can fail, and diffenrent ways to fix this (fix the bug, fix a configuration, or just skip the tests). Recently, we've actually push the list of tests to skip into nova.git, but I think it is only available in master. > Other possibilities that come to my mind: ideally the tempest tests > would have metadata so that particular classes of tests could be > skipped. Alternatively, if they aren't disruptive or very slow, if > run, we could run them anyway as substeps. > > Which makes me think: maybe the tempest tests want to be substeps > anyway. Is that possible ? (Does tempest speak subunit or > something ?) I think it is subunit, yes. And I've got the command that is been run by tempest (as it's printed on stdout). running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list then several: running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpzSNsrB
Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"):
> On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote:
> > Again, does this not mean we're going to suffer a maintenance burden
> > as tempest grows new inapplicable tests ?
>
> That exactly what is happening with the OpenStack CI loop, from time to
> time, there are new tests, that can fail, and diffenrent ways to fix
> this (fix the bug, fix a configuration, or just skip the tests).
>
> Recently, we've actually push the list of tests to skip into nova.git,
> but I think it is only available in master.
Aha. Well, if the stable branch is stable then the set of tests to
skip there is probably stable too ? And on master it's built-in ? So
this actually won't be a problem - in the sense that this will be
approximately the last necessary update to the list of tests to skip ?
> > Which makes me think: maybe the tempest tests want to be substeps
> > anyway. Is that possible ? (Does tempest speak subunit or
> > something ?)
>
> I think it is subunit, yes. And I've got the command that is been run by
> tempest (as it's printed on stdout).
>
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
>
> then several:
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpzSNsrB
It would be really good to have those individual subunit results as
substeps. There is no subunit parser in osstest yet but we could have
one.
What version of subunit does it print out ? The subunit v1 protocol
is lovely and simple but there is a Second System :-/.
I guess that tempest doesn't stop on the first failed test ? So
perhaps we could just tolerate the failed-but-not-skipped tests ?
Ian.
On Fri, Jun 23, 2017 at 04:38:05PM +0100, Ian Jackson wrote: > Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"): > > On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote: > > > Again, does this not mean we're going to suffer a maintenance burden > > > as tempest grows new inapplicable tests ? > > > > That exactly what is happening with the OpenStack CI loop, from time to > > time, there are new tests, that can fail, and diffenrent ways to fix > > this (fix the bug, fix a configuration, or just skip the tests). > > > > Recently, we've actually push the list of tests to skip into nova.git, > > but I think it is only available in master. > > Aha. Well, if the stable branch is stable then the set of tests to > skip there is probably stable too ? And on master it's built-in ? So > this actually won't be a problem - in the sense that this will be > approximately the last necessary update to the list of tests to skip ? The built-in list is to be consume by the CI, it is not used automatically. It a bash script to be sourced. I have not yet try to have osstest use it. That will be for later. > > > Which makes me think: maybe the tempest tests want to be substeps > > > anyway. Is that possible ? (Does tempest speak subunit or > > > something ?) > > > > I think it is subunit, yes. And I've got the command that is been run by > > tempest (as it's printed on stdout). > > > > running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ > > OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ > > OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ > > OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ > > ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list > > > > then several: > > running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ > > OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ > > OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ > > OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ > > ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpzSNsrB > > It would be really good to have those individual subunit results as > substeps. There is no subunit parser in osstest yet but we could have > one. > > What version of subunit does it print out ? The subunit v1 protocol > is lovely and simple but there is a Second System :-/. Right now, tempest print on stdout something much simpler, lines liked this: {1} tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario [71.009969s] ... ok Followed by summary of all test, and the output of failed tests. But it can be told to print subunit v2, which can be pipped to a different program to transform into want we want, including simple cvs, subunit v1, and other. I thing we could try to parse subunit v1, grab the start and finish time of a test, the result, and the output of the tests. And have all that as subunit. During my manual tests, `tempest run --subunit ... | subunit-2to1` only print the stuff about a tests once it is finish, so I guess we'll have to rewrite substep_start() (or another function) to take time as parameter. > I guess that tempest doesn't stop on the first failed test ? So > perhaps we could just tolerate the failed-but-not-skipped tests ? It does not, it run everything, no matter how many test fails.
Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"):
> On Fri, Jun 23, 2017 at 04:38:05PM +0100, Ian Jackson wrote:
> > Aha. Well, if the stable branch is stable then the set of tests to
> > skip there is probably stable too ? And on master it's built-in ? So
> > this actually won't be a problem - in the sense that this will be
> > approximately the last necessary update to the list of tests to skip ?
>
> The built-in list is to be consume by the CI, it is not used
> automatically. It a bash script to be sourced. I have not yet try to
> have osstest use it. That will be for later.
Hah.
> Right now, tempest print on stdout something much simpler, lines liked
> this:
> {1} tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario [71.009969s] ... ok
> Followed by summary of all test, and the output of failed tests.
>
> But it can be told to print subunit v2, which can be pipped to a different
> program to transform into want we want, including simple cvs, subunit v1, and
> other.
>
> I thing we could try to parse subunit v1, grab the start and finish time
> of a test, the result, and the output of the tests. And have all that as
> subunit. During my manual tests, `tempest run --subunit ... | subunit-2to1`
> only print the stuff about a tests once it is finish, so I guess we'll
> have to rewrite substep_start() (or another function) to take time as
> parameter.
That sounds quite good really.
I guess you don't want to write a parser for subunit v2 but if there
is one in Perl already in some package in Debian we could use it. Or
subunit-2to1 is fine.
Ian.
On Fri, Jun 30, 2017 at 05:32:56PM +0100, Ian Jackson wrote: > Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"): > > But it can be told to print subunit v2, which can be pipped to a different > > program to transform into want we want, including simple cvs, subunit v1, and > > other. > > > > I thing we could try to parse subunit v1, grab the start and finish time > > of a test, the result, and the output of the tests. And have all that as > > subunit. During my manual tests, `tempest run --subunit ... | subunit-2to1` > > only print the stuff about a tests once it is finish, so I guess we'll > > have to rewrite substep_start() (or another function) to take time as > > parameter. > > That sounds quite good really. > > I guess you don't want to write a parser for subunit v2 but if there > is one in Perl already in some package in Debian we could use it. Or > subunit-2to1 is fine. There is one, libsubunit-perl, but its only for subunit v1 :-(. So I guess I'll use both, the package and subunit-2to1.
On Fri, Jun 30, 2017 at 06:58:43PM +0100, Anthony PERARD wrote: > On Fri, Jun 30, 2017 at 05:32:56PM +0100, Ian Jackson wrote: > > Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests"): > > > But it can be told to print subunit v2, which can be pipped to a different > > > program to transform into want we want, including simple cvs, subunit v1, and > > > other. > > > > > > I thing we could try to parse subunit v1, grab the start and finish time > > > of a test, the result, and the output of the tests. And have all that as > > > subunit. During my manual tests, `tempest run --subunit ... | subunit-2to1` > > > only print the stuff about a tests once it is finish, so I guess we'll > > > have to rewrite substep_start() (or another function) to take time as > > > parameter. > > > > That sounds quite good really. > > > > I guess you don't want to write a parser for subunit v2 but if there > > is one in Perl already in some package in Debian we could use it. Or > > subunit-2to1 is fine. > > There is one, libsubunit-perl, but its only for subunit v1 :-(. So I > guess I'll use both, the package and subunit-2to1. :( the package does not work with the output of `Tempest | subunit-2to1` there are a few things that are not parsed properly. That the parser: https://github.com/testing-cabal/subunit/blob/master/perl/lib/Subunit.pm I'm kind of tempted to write one that's will be easier to integrate with Osstest, especially because subunit v1 is simple enough.
diff --git a/ts-openstack-tempest b/ts-openstack-tempest index b95043a..ae3662f 100755 --- a/ts-openstack-tempest +++ b/ts-openstack-tempest @@ -31,23 +31,20 @@ sub tempest() { my $scenario = 'tempest.scenario'; my $volume_boot_pattern = "$scenario.test_volume_boot_pattern.TestVolumeBootPattern"; - my $shelve_instance = "$scenario.test_shelve_instance.TestShelveInstance"; - - # Ignore tests which try to boot a guest with /dev/vda as boot device name. - push @ignored_tests, - "^\Q$volume_boot_pattern.test_volume_boot_pattern\E"; - push @ignored_tests, - "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E"; - push @ignored_tests, - "^\Q$shelve_instance.test_shelve_volume_backed_instance\E"; # Those tests access a volume through iSCSI. This does not work when both # the server and client of iSCSI are on the same Xen host (both in dom0), # Linux 4.0 is the first Linux to have a fix. push @ignored_tests, - "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E"; + "^\Q${volume_boot_pattern}.test_volume_boot_pattern\E"; + push @ignored_tests, + "^\Q${volume_boot_pattern}.test_create_ebs_image_and_check_boot\E"; + + # See nova.git:devstack/tempest-dsvm-tempest-xen-rc + push @ignored_tests, + "^\Qtempest.api.compute.admin.test_volume_swap.TestVolumeSwap.test_volume_swap\E"; push @ignored_tests, - "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E"; + "^\Qtempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_paused_server\E"; # This regex below select the tests to run and exclude the ones marked as # slow as well as the explicit tests listed above. It is based on the one
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> --- ts-openstack-tempest | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-)