| Message ID | 1442490164-16728-1-git-send-email-thomas.wood@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: > Add a script to take a piglit results file and create a list of tests > that ran in under 60 seconds. This list can be used by the --test-list > option of piglit. And passed? -Chris
On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote: > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: >> Add a script to take a piglit results file and create a list of tests >> that ran in under 60 seconds. This list can be used by the --test-list >> option of piglit. > > And passed? Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on. > -Chris > > -- > Chris Wilson, Intel Open Source Technology Centre > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote: > On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote: > > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: > >> Add a script to take a piglit results file and create a list of tests > >> that ran in under 60 seconds. This list can be used by the --test-list > >> option of piglit. > > > > And passed? > > Any status except "incomplete", which normally means a reboot was > required. It might also be worth noting that tests skipped on one > platform may not skip on another, so the output only really applies to > the platform the results were initially produced on. My point is that the timing can only be expected to be consistent for a pass. A failure may be quick, but success may take a few hours. -Chris
This script measures the duration of the subtests. But in case of gem_concurrent_all, for instance, there are 3780 subtests for which, although many shorter than 60s, the total duration will not qualify this for a quick test (~2h). Thank you, Gabriel
On 17 September 2015 at 13:43, Chris Wilson <chris@chris-wilson.co.uk> wrote: > On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote: >> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote: >> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: >> >> Add a script to take a piglit results file and create a list of tests >> >> that ran in under 60 seconds. This list can be used by the --test-list >> >> option of piglit. >> > >> > And passed? >> >> Any status except "incomplete", which normally means a reboot was >> required. It might also be worth noting that tests skipped on one >> platform may not skip on another, so the output only really applies to >> the platform the results were initially produced on. > > My point is that the timing can only be expected to be consistent for a > pass. A failure may be quick, but success may take a few hours. I think including failed tests depends on whether the "quick" run should be able to detect fixes from previous runs. Perhaps an option to exclude anything other than successful tests might be useful. There is also the timeout mechanism in piglit/igt that will prevent any test from running for more than 10 minutes. > -Chris > > -- > Chris Wilson, Intel Open Source Technology Centre
On 17 September 2015 at 15:01, Feceoru, Gabriel <gabriel.feceoru@intel.com> wrote: > This script measures the duration of the subtests. But in case of gem_concurrent_all, > for instance, there are 3780 subtests for which, although many shorter than 60s, the total > duration will not qualify this for a quick test (~2h). For this specific case, there has been some discussion about hiding some of the combinatorial subtests from the default testing list. kms_frontbuffer_tracking already has an option to show hidden subtests and pm_rpm has an option to reduce the amount of work in the stress tests. A common approach to hiding or reducing the number of stress tests for quick testing would be useful. > > Thank you, > Gabriel > ________________________________________ > From: Intel-gfx [intel-gfx-bounces@lists.freedesktop.org] on behalf of Chris Wilson [chris@chris-wilson.co.uk] > Sent: Thursday, September 17, 2015 3:43 PM > To: Wood, Thomas > Cc: Intel Graphics Development > Subject: Re: [Intel-gfx] [PATCH i-g-t v2] scripts: add quick-testlist.py > > On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote: >> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote: >> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: >> >> Add a script to take a piglit results file and create a list of tests >> >> that ran in under 60 seconds. This list can be used by the --test-list >> >> option of piglit. >> > >> > And passed? >> >> Any status except "incomplete", which normally means a reboot was >> required. It might also be worth noting that tests skipped on one >> platform may not skip on another, so the output only really applies to >> the platform the results were initially produced on. > > My point is that the timing can only be expected to be consistent for a > pass. A failure may be quick, but success may take a few hours. > -Chris > > -- > Chris Wilson, Intel Open Source Technology Centre > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
On Thu, Sep 17, 2015 at 04:47:06PM +0100, Thomas Wood wrote: > On 17 September 2015 at 13:43, Chris Wilson <chris@chris-wilson.co.uk> wrote: > > On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote: > >> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote: > >> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: > >> >> Add a script to take a piglit results file and create a list of tests > >> >> that ran in under 60 seconds. This list can be used by the --test-list > >> >> option of piglit. > >> > > >> > And passed? > >> > >> Any status except "incomplete", which normally means a reboot was > >> required. It might also be worth noting that tests skipped on one > >> platform may not skip on another, so the output only really applies to > >> the platform the results were initially produced on. > > > > My point is that the timing can only be expected to be consistent for a > > pass. A failure may be quick, but success may take a few hours. > > I think including failed tests depends on whether the "quick" run > should be able to detect fixes from previous runs. Perhaps an option > to exclude anything other than successful tests might be useful. There > is also the timeout mechanism in piglit/igt that will prevent any test > from running for more than 10 minutes. Yeah, for the fast run we also need to reduce the piglit timeout to 60s (and kill anything that timed out ofc too). Otherwise there's indeed a gap where a testcase that takes too long could sneak in. -Daniel
On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote: > Add a script to take a piglit results file and create a list of tests > that ran in under 60 seconds. This list can be used by the --test-list > option of piglit. > > v2: exclude incomplete tests > > Signed-off-by: Thomas Wood <thomas.wood@intel.com> I think this might be generally useful for piglit uses, maybe in contrib/? Anyway adding piglit m-l. -Daniel > --- > scripts/quick-testlist.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 46 insertions(+) > create mode 100755 scripts/quick-testlist.py > > diff --git a/scripts/quick-testlist.py b/scripts/quick-testlist.py > new file mode 100755 > index 0000000..0dd4c69 > --- /dev/null > +++ b/scripts/quick-testlist.py > @@ -0,0 +1,46 @@ > +#!/usr/bin/env python > +# > +# Copyright 2015 Intel Corporation > +# > +# Permission is hereby granted, free of charge, to any person obtaining a > +# copy of this software and associated documentation files (the "Software"), > +# to deal in the Software without restriction, including without limitation > +# the rights to use, copy, modify, merge, publish, distribute, sublicense, > +# and/or sell copies of the Software, and to permit persons to whom the > +# Software is furnished to do so, subject to the following conditions: > +# > +# The above copyright notice and this permission notice (including the next > +# paragraph) shall be included in all copies or substantial portions of the > +# Software. > +# > +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR > +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, > +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL > +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER > +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING > +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS > +# IN THE SOFTWARE. > + > +from __future__ import print_function > +import json > +import sys > + > +def filter_results(filename): > + with open(filename) as data: > + json_data = json.load(data) > + > + for test_name in json_data["tests"]: > + if json_data["tests"][test_name]["result"] == "incomplete": > + continue > + if json_data["tests"][test_name]["time"] < 60: > + print(test_name) > + > + > +if len(sys.argv) < 2: > + print("Usage: quick-testlist.py RESULTS") > + print("Read piglit results from RESULTS and print the tests that executed" > + " in under 60 seconds, excluding any incomplete tests. The list can" > + " be used by the --test-list option of piglit.") > + sys.exit(1) > + > +filter_results(sys.argv[1]) > -- > 1.9.1 > > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
diff --git a/scripts/quick-testlist.py b/scripts/quick-testlist.py new file mode 100755 index 0000000..0dd4c69 --- /dev/null +++ b/scripts/quick-testlist.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python +# +# Copyright 2015 Intel Corporation +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice (including the next +# paragraph) shall be included in all copies or substantial portions of the +# Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +# IN THE SOFTWARE. + +from __future__ import print_function +import json +import sys + +def filter_results(filename): + with open(filename) as data: + json_data = json.load(data) + + for test_name in json_data["tests"]: + if json_data["tests"][test_name]["result"] == "incomplete": + continue + if json_data["tests"][test_name]["time"] < 60: + print(test_name) + + +if len(sys.argv) < 2: + print("Usage: quick-testlist.py RESULTS") + print("Read piglit results from RESULTS and print the tests that executed" + " in under 60 seconds, excluding any incomplete tests. The list can" + " be used by the --test-list option of piglit.") + sys.exit(1) + +filter_results(sys.argv[1])
Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit. v2: exclude incomplete tests Signed-off-by: Thomas Wood <thomas.wood@intel.com> --- scripts/quick-testlist.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100755 scripts/quick-testlist.py