| Message ID | 1453979137-12522-1-git-send-email-derek.j.morton@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 28/01/16 11:05, Derek Morton wrote: > Added support for specifying arbitary lists of subtests to run, or > to exclude from being run if prefixed by ^ or !. > > subtest1,subtest2 Will run subtest1 and subtest2 > ^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except > subtest1 and subtest2. > > Any subtest string not starting ! or ^ and not containing a comma is > treated as a normal wildcard expression. > > This is required mainly on android to exclude subtests that test > features that do not exist in the android driver while still being able > to run other subtests in the binary when a wildcard expression is > insufficient. > > v2: Use comma as list separator (Ville Syrjala) > support both ^ and ! as not operators (Dave Gordon) > > Signed-off-by: Derek Morton <derek.j.morton@intel.com> > --- > lib/igt_core.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- > 1 file changed, 46 insertions(+), 2 deletions(-) > > diff --git a/lib/igt_core.c b/lib/igt_core.c > index 6b69bb7..5d243af 100644 > --- a/lib/igt_core.c > +++ b/lib/igt_core.c > @@ -207,7 +207,15 @@ > * To do that obtain the lists of subtests with "--list-subtests", which can be > * run as non-root and doesn't require the i915 driver to be loaded (or any > * intel gpu to be present). Then individual subtests can be run with > - * "--run-subtest". Usage help for tests with subtests can be obtained with the > + * "--run-subtest". --run-subtest accepts wildcard characters. A list of > + * subtests to run may be specified by using ',' as a separator. > + * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests except... > + * > + * - --run-subtest basic* will run all subtests starting basic. > + * - --run-subtest subtest1,subtest2 will run only subtest1 and subtest2 > + * - --run-subtest ^subtest1,subtest2 will run all those except subtest1 and subtest2 > + * > + * Usage help for tests with subtests can be obtained with the > * "--help" command line option. > */ This is the syntax option 1 from my previous mail (but not allowing multiple --run-subtest options?), which I think is a bit less powerful and a bit less desirable because of the perhaps unexpected precedence of OR above NOT (^sub1,sub2 => NOT(sub1 OR sub2). Do you think the advantages of option 2 (full generality, can express any boolean, expected precedence of operators) would make it worth the extra effort? Whichever we pick now, it will be set in stone as it will be very inconvenient to edit every script using the feature once it's been in use for a while! Another idea: have you considered bash(1) extglob syntax? See the manpage for bash, section "Pattern Matching". It's quite powerful, familiar to least to script writers, and easy to try out in bash: $ : enable extglob mode $ shopt -s extglob $ : create an array of test names $ declare -a tests=( starting phase1 phase2 wibble phase3 wobble done ) $ : see what matches a pattern: $ function match() { for w in "${tests[@]}" ; do [[ "$w" == $1 ]] && echo "$w" ; done ; } $ match '*' starting phase1 phase2 wibble phase3 wobble done $ match 'phase*' phase1 phase2 phase3 $ match 'w?bble' wibble wobble $ match '[!pw]*' starting done $ match '@(starting|done)' starting done $ match '!(starting|done)' phase1 phase2 wibble phase3 wobble $ match '!(phase*)' starting wibble wobble done $ match '!(phase*|w?bble)' starting done $ : etc, etc, ad lib .Dave.
Hi, I have spoken to Dave. I do not want to implement multiple --run-subtest as it is not something I require and I do not think it is worth adding unless someone has a specific need for it. However I understand that whatever is done should not make that harder in the future to add multiple --run-subtest and so the --run-subtest should always be additive. The issue seems to be how to specify "add all subtests except ..." in a 'standard' way. The fnmatch() used currently only supports *?[] so there is no way of specifying 'not'. What would be the feeling if I was to replace fnmatch() with support for --run-subtest supporting proper regex expressions by including regex.h and using regexec()? As far as I can see the impact of that would be a change of behaviour of * and ? in the expression which might not be desirable if the wildcard expressions are in use. As an alternative I could add a new parameter --run-subtest-regex which allows the list of subtests to be run to be specified with a regex. Would that be acceptable? //Derek > > >-----Original Message----- >From: Gordon, David S >Sent: Thursday, January 28, 2016 12:58 PM >To: Morton, Derek J; intel-gfx@lists.freedesktop.org >Cc: daniel.vetter@ffwll.ch >Subject: Re: [Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality. > >On 28/01/16 11:05, Derek Morton wrote: >> Added support for specifying arbitary lists of subtests to run, or to >> exclude from being run if prefixed by ^ or !. >> >> subtest1,subtest2 Will run subtest1 and subtest2 >> ^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except >> subtest1 and subtest2. >> >> Any subtest string not starting ! or ^ and not containing a comma is >> treated as a normal wildcard expression. >> >> This is required mainly on android to exclude subtests that test >> features that do not exist in the android driver while still being >> able to run other subtests in the binary when a wildcard expression is >> insufficient. >> >> v2: Use comma as list separator (Ville Syrjala) support both ^ and ! >> as not operators (Dave Gordon) >> >> Signed-off-by: Derek Morton <derek.j.morton@intel.com> >> --- >> lib/igt_core.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- >> 1 file changed, 46 insertions(+), 2 deletions(-) >> >> diff --git a/lib/igt_core.c b/lib/igt_core.c index 6b69bb7..5d243af >> 100644 >> --- a/lib/igt_core.c >> +++ b/lib/igt_core.c >> @@ -207,7 +207,15 @@ >> * To do that obtain the lists of subtests with "--list-subtests", which can be >> * run as non-root and doesn't require the i915 driver to be loaded (or any >> * intel gpu to be present). Then individual subtests can be run >> with >> - * "--run-subtest". Usage help for tests with subtests can be >> obtained with the >> + * "--run-subtest". --run-subtest accepts wildcard characters. A list >> + of >> + * subtests to run may be specified by using ',' as a separator. >> + * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests except... >> + * >> + * - --run-subtest basic* will run all subtests starting basic. >> + * - --run-subtest subtest1,subtest2 will run only subtest1 and >> + subtest2 >> + * - --run-subtest ^subtest1,subtest2 will run all those except >> + subtest1 and subtest2 >> + * >> + * Usage help for tests with subtests can be obtained with the >> * "--help" command line option. >> */ > >This is the syntax option 1 from my previous mail (but not allowing multiple --run-subtest options?), which I think is a bit less powerful and a bit less desirable because of the perhaps unexpected precedence of OR above NOT (^sub1,sub2 => NOT(sub1 OR sub2). > >Do you think the advantages of option 2 (full generality, can express any boolean, expected precedence of operators) would make it worth the extra effort? > >Whichever we pick now, it will be set in stone as it will be very inconvenient to edit every script using the feature once it's been in use for a while! > >Another idea: have you considered bash(1) extglob syntax? See the manpage for bash, section "Pattern Matching". It's quite powerful, familiar to least to script writers, and easy to try out in bash: > >$ : enable extglob mode >$ shopt -s extglob >$ : create an array of test names >$ declare -a tests=( starting phase1 phase2 wibble phase3 wobble done ) $ : see what matches a pattern: >$ function match() { for w in "${tests[@]}" ; do [[ "$w" == $1 ]] && echo "$w" ; done ; } $ match '*' >starting >phase1 >phase2 >wibble >phase3 >wobble >done >$ match 'phase*' >phase1 >phase2 >phase3 >$ match 'w?bble' >wibble >wobble >$ match '[!pw]*' >starting >done >$ match '@(starting|done)' >starting >done >$ match '!(starting|done)' >phase1 >phase2 >wibble >phase3 >wobble >$ match '!(phase*)' >starting >wibble >wobble >done >$ match '!(phase*|w?bble)' >starting >done >$ : etc, etc, ad lib > >.Dave. > >
On 28/01/16 12:57, Dave Gordon wrote: > On 28/01/16 11:05, Derek Morton wrote: >> Added support for specifying arbitary lists of subtests to run, or >> to exclude from being run if prefixed by ^ or !. >> >> subtest1,subtest2 Will run subtest1 and subtest2 >> ^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except >> subtest1 and subtest2. >> >> Any subtest string not starting ! or ^ and not containing a comma is >> treated as a normal wildcard expression. >> >> This is required mainly on android to exclude subtests that test >> features that do not exist in the android driver while still being able >> to run other subtests in the binary when a wildcard expression is >> insufficient. >> >> v2: Use comma as list separator (Ville Syrjala) >> support both ^ and ! as not operators (Dave Gordon) >> >> Signed-off-by: Derek Morton <derek.j.morton@intel.com> >> --- >> lib/igt_core.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- >> 1 file changed, 46 insertions(+), 2 deletions(-) >> >> diff --git a/lib/igt_core.c b/lib/igt_core.c >> index 6b69bb7..5d243af 100644 >> --- a/lib/igt_core.c >> +++ b/lib/igt_core.c >> @@ -207,7 +207,15 @@ >> * To do that obtain the lists of subtests with "--list-subtests", >> which can be >> * run as non-root and doesn't require the i915 driver to be loaded >> (or any >> * intel gpu to be present). Then individual subtests can be run with >> - * "--run-subtest". Usage help for tests with subtests can be >> obtained with the >> + * "--run-subtest". --run-subtest accepts wildcard characters. A list of >> + * subtests to run may be specified by using ',' as a separator. >> + * A prefix of ^ or ! may be added to invert the logic, e.g. run all >> tests except... >> + * >> + * - --run-subtest basic* will run all subtests starting basic. >> + * - --run-subtest subtest1,subtest2 will run only subtest1 and subtest2 >> + * - --run-subtest ^subtest1,subtest2 will run all those except >> subtest1 and subtest2 >> + * >> + * Usage help for tests with subtests can be obtained with the >> * "--help" command line option. >> */ > > This is the syntax option 1 from my previous mail (but not allowing > multiple --run-subtest options?), which I think is a bit less powerful > and a bit less desirable because of the perhaps unexpected precedence of > OR above NOT (^sub1,sub2 => NOT(sub1 OR sub2). > > Do you think the advantages of option 2 (full generality, can express > any boolean, expected precedence of operators) would make it worth the > extra effort? > > Whichever we pick now, it will be set in stone as it will be very > inconvenient to edit every script using the feature once it's been in > use for a while! > > Another idea: have you considered bash(1) extglob syntax? See the > manpage for bash, section "Pattern Matching". It's quite powerful, > familiar to least to script writers, and easy to try out in bash: GNU libc has a globbing function that lets you override readdir() and friends so you can use it on a string or list of strings. http://www.gnu.org/software/libc/manual/html_mono/libc.html#Globbing Perl's Text::Glob implements glob(3) style matching that can be used to match against text, rather than fetching names from a filesystem. Or have a look at wildmat(3) as standardised by RFC3977, or better still uwildmat(3). https://tools.ietf.org/html/rfc3977#section-4 http://linux.die.net/man/3/uwildmat That seems to give about the right tradeoff of complexity vs flexibility! .Dave.
diff --git a/lib/igt_core.c b/lib/igt_core.c index 6b69bb7..5d243af 100644 --- a/lib/igt_core.c +++ b/lib/igt_core.c @@ -207,7 +207,15 @@ * To do that obtain the lists of subtests with "--list-subtests", which can be * run as non-root and doesn't require the i915 driver to be loaded (or any * intel gpu to be present). Then individual subtests can be run with - * "--run-subtest". Usage help for tests with subtests can be obtained with the + * "--run-subtest". --run-subtest accepts wildcard characters. A list of + * subtests to run may be specified by using ',' as a separator. + * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests except... + * + * - --run-subtest basic* will run all subtests starting basic. + * - --run-subtest subtest1,subtest2 will run only subtest1 and subtest2 + * - --run-subtest ^subtest1,subtest2 will run all those except subtest1 and subtest2 + * + * Usage help for tests with subtests can be obtained with the * "--help" command line option. */ @@ -786,6 +794,41 @@ void igt_simple_init_parse_opts(int *argc, char **argv, extra_opt_handler, handler_data); } +static bool check_testlist(const char *subtest_name) +{ + char *p; + + /* Run subtests not in list + * Look for subtest_name in list of form ^subtest1,subtest2,subtest3 + * or !test1,subtest2,subtest3 + * return true if not found. + */ + if ((run_single_subtest[0] == '^') || (run_single_subtest[0] == '!')) { + char prefix = run_single_subtest[0]; + p = strstr(run_single_subtest, subtest_name); + if (!((p) && + ((*(p-1) == prefix) || (*(p-1) == ',')) && + ((*(p+strlen(subtest_name)) == ',') || (*(p+strlen(subtest_name)) == '\0')))) + return true; + } + /* Run subtests in list + * Look for subtest_name in list of form subtest1,subtest2,subtest3 + * return true if found. + */ + else if (strstr(run_single_subtest, ",")) { + p = strstr(run_single_subtest, subtest_name); + if ((p) && + ((p == run_single_subtest) || (*(p-1) == ',')) && + ((*(p+strlen(subtest_name)) == ',') || (*(p+strlen(subtest_name)) == '\0'))) + return true; + } + /* Run subtests that match shell wildcard */ + else if (fnmatch(run_single_subtest, subtest_name, 0) == 0) + return true; + + return false; +} + /* * Note: Testcases which use these helpers MUST NOT output anything to stdout * outside of places protected by igt_run_subtest checks - the piglit @@ -814,7 +857,8 @@ bool __igt_run_subtest(const char *subtest_name) } if (run_single_subtest) { - if (fnmatch(run_single_subtest, subtest_name, 0) != 0) + + if (check_testlist(subtest_name) == false) return false; else run_single_subtest_found = true;
Added support for specifying arbitary lists of subtests to run, or to exclude from being run if prefixed by ^ or !. subtest1,subtest2 Will run subtest1 and subtest2 ^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except subtest1 and subtest2. Any subtest string not starting ! or ^ and not containing a comma is treated as a normal wildcard expression. This is required mainly on android to exclude subtests that test features that do not exist in the android driver while still being able to run other subtests in the binary when a wildcard expression is insufficient. v2: Use comma as list separator (Ville Syrjala) support both ^ and ! as not operators (Dave Gordon) Signed-off-by: Derek Morton <derek.j.morton@intel.com> --- lib/igt_core.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)