| Message ID | 20201025214842.5924-1-linux@rasmusvillemoes.dk (mailing list archive) |
|---|---|
| Headers | show |
| Series | deterministic random testing | expand |
On Sun, Oct 25, 2020 at 10:48:38PM +0100, Rasmus Villemoes wrote: > This is a bit of a mixed bag. > > The background is that I have some sort() and list_sort() rework > planned, but as part of that series I want to extend their their test > suites somewhat to make sure I don't goof up - and I want to use lots > of random list lengths with random contents to increase the chance of > somebody eventually hitting "hey, sort() is broken when the length is > 3 less than a power of 2 and only the last two elements are out of > order". But when such a case is hit, it's vitally important that the > developer can reproduce the exact same test case, which means using a > deterministic sequence of random numbers. > > Since Petr noticed [1] the non-determinism in test_printf in > connection with Arpitha's work on rewriting it to kunit, this prompted > me to use test_printf as a first place to apply that principle, and > get the infrastructure in place that will avoid repeating the "module > parameter/seed the rnd_state/report the seed used" boilerplate in each > module. > > Shuah, assuming the kselftest_module.h changes are ok, I think it's > most natural if you carry these patches, though I'd be happy with any > other route as well. Completely in favour of this. Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> One note though. AFAIU the global variables are always being used in the modules that include the corresponding header. Otherwise we might have an extra warning(s). I believe you have compiled with W=1 to exclude other cases. > [1] https://lore.kernel.org/lkml/20200821113710.GA26290@alley/ > > > Rasmus Villemoes (4): > prandom.h: add *_state variant of prandom_u32_max > kselftest_module.h: unconditionally expand the KSTM_MODULE_GLOBALS() > macro > kselftest_module.h: add struct rnd_state and seed parameter > lib/test_printf.c: use deterministic sequence of random numbers > > Documentation/dev-tools/kselftest.rst | 2 -- > include/linux/prandom.h | 29 ++++++++++++++++ > lib/test_bitmap.c | 3 -- > lib/test_printf.c | 13 ++++--- > lib/test_strscpy.c | 2 -- > tools/testing/selftests/kselftest_module.h | 40 ++++++++++++++++++---- > 6 files changed, 72 insertions(+), 17 deletions(-) > > -- > 2.23.0 >
On 26/10/2020 11.59, Andy Shevchenko wrote: > On Sun, Oct 25, 2020 at 10:48:38PM +0100, Rasmus Villemoes wrote: >> This is a bit of a mixed bag. >> >> The background is that I have some sort() and list_sort() rework >> planned, but as part of that series I want to extend their their test >> suites somewhat to make sure I don't goof up - and I want to use lots >> of random list lengths with random contents to increase the chance of >> somebody eventually hitting "hey, sort() is broken when the length is >> 3 less than a power of 2 and only the last two elements are out of >> order". But when such a case is hit, it's vitally important that the >> developer can reproduce the exact same test case, which means using a >> deterministic sequence of random numbers. >> >> Since Petr noticed [1] the non-determinism in test_printf in >> connection with Arpitha's work on rewriting it to kunit, this prompted >> me to use test_printf as a first place to apply that principle, and >> get the infrastructure in place that will avoid repeating the "module >> parameter/seed the rnd_state/report the seed used" boilerplate in each >> module. >> >> Shuah, assuming the kselftest_module.h changes are ok, I think it's >> most natural if you carry these patches, though I'd be happy with any >> other route as well. > > Completely in favour of this. > > Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Thanks. > One note though. AFAIU the global variables are always being used in the > modules that include the corresponding header. Otherwise we might have an extra > warning(s). I believe you have compiled with W=1 to exclude other cases. Yes, I unconditionally define the two new variables. gcc doesn't warn about them being unused, since they are referenced from inside a if (0) {} block. And when those references are the only ones, gcc is smart enough to elide the static variables completely, so they don't even take up space in .data (or .init.data) - you can verify by running nm on test_printf.o and test_bitmap.o - the former has 'seed' and 'rnd_state' symbols, the latter does not. I did it that way to reduce the need for explicit preprocessor conditionals inside C functions. Rasmus
This is a bit of a mixed bag. The background is that I have some sort() and list_sort() rework planned, but as part of that series I want to extend their their test suites somewhat to make sure I don't goof up - and I want to use lots of random list lengths with random contents to increase the chance of somebody eventually hitting "hey, sort() is broken when the length is 3 less than a power of 2 and only the last two elements are out of order". But when such a case is hit, it's vitally important that the developer can reproduce the exact same test case, which means using a deterministic sequence of random numbers. Since Petr noticed [1] the non-determinism in test_printf in connection with Arpitha's work on rewriting it to kunit, this prompted me to use test_printf as a first place to apply that principle, and get the infrastructure in place that will avoid repeating the "module parameter/seed the rnd_state/report the seed used" boilerplate in each module. Shuah, assuming the kselftest_module.h changes are ok, I think it's most natural if you carry these patches, though I'd be happy with any other route as well. [1] https://lore.kernel.org/lkml/20200821113710.GA26290@alley/ Rasmus Villemoes (4): prandom.h: add *_state variant of prandom_u32_max kselftest_module.h: unconditionally expand the KSTM_MODULE_GLOBALS() macro kselftest_module.h: add struct rnd_state and seed parameter lib/test_printf.c: use deterministic sequence of random numbers Documentation/dev-tools/kselftest.rst | 2 -- include/linux/prandom.h | 29 ++++++++++++++++ lib/test_bitmap.c | 3 -- lib/test_printf.c | 13 ++++--- lib/test_strscpy.c | 2 -- tools/testing/selftests/kselftest_module.h | 40 ++++++++++++++++++---- 6 files changed, 72 insertions(+), 17 deletions(-)