| Message ID | 20201025214842.5924-4-linux@rasmusvillemoes.dk (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | deterministic random testing | expand |
On Sun 2020-10-25 22:48:41, Rasmus Villemoes wrote: > Some test suites make use of random numbers to increase the test > coverage when the test suite gets run on different machines and > increase the chance of some corner case bug being discovered - and I'm > planning on extending some existing ones in that direction as > well. However, should a bug be found this way, it's important that the > exact same series of tests can be repeated to verify the bug is > fixed. That means the random numbers must be obtained > deterministically from a generator private to the test module. > > To avoid adding boilerplate to various test modules, put some logic > into kselftest_module.h: If the module declares that it will use > random numbers, add a "seed" module parameter. If not explicitly given > when the module is loaded (or via kernel command line), obtain a > random one. In either case, print the seed used, and repeat that > information if there was at least one test failing. > > Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk> > --- > tools/testing/selftests/kselftest_module.h | 35 ++++++++++++++++++++-- > 1 file changed, 32 insertions(+), 3 deletions(-) > > diff --git a/tools/testing/selftests/kselftest_module.h b/tools/testing/selftests/kselftest_module.h > index c81c0b0c054befaf665b..43f3ca58fcd550b8ac83 100644 > --- a/tools/testing/selftests/kselftest_module.h > +++ b/tools/testing/selftests/kselftest_module.h > @@ -3,14 +3,31 @@ > #define __KSELFTEST_MODULE_H > > #include <linux/module.h> > +#include <linux/prandom.h> > +#include <linux/random.h> > > /* > * Test framework for writing test modules to be loaded by kselftest. > * See Documentation/dev-tools/kselftest.rst for an example test module. > */ > > +/* > + * If the test module makes use of random numbers, define KSTM_RANDOM > + * to 1 before including this header. Then a module parameter "seed" > + * will be defined. If not given, a random one will be obtained. In > + * either case, the used seed is reported, so the exact same series of > + * tests can be repeated by loading the module with that seed > + * given. > + */ > + > +#ifndef KSTM_RANDOM > +#define KSTM_RANDOM 0 > +#endif > + > static unsigned int total_tests __initdata; > static unsigned int failed_tests __initdata; > +static struct rnd_state rnd_state __initdata; > +static u64 seed __initdata; > > #define KSTM_CHECK_ZERO(x) do { \ > total_tests++; \ > @@ -22,11 +39,13 @@ static unsigned int failed_tests __initdata; > > static inline int kstm_report(unsigned int total_tests, unsigned int failed_tests) > { > - if (failed_tests == 0) > + if (failed_tests == 0) { > pr_info("all %u tests passed\n", total_tests); > - else > + } else { > pr_warn("failed %u out of %u tests\n", failed_tests, total_tests); > - > + if (KSTM_RANDOM) > + pr_info("random seed used was 0x%016llx\n", seed); I have a bit mixed feelings about this. It is genial and dirty hack at the same time ;-) Well, it is basically the same approach as with IS_ENABLED(CONFIG_bla_bla). Reviewed-by: Petr Mladek <pmladek@suse.com> Best Regards, Petr
diff --git a/tools/testing/selftests/kselftest_module.h b/tools/testing/selftests/kselftest_module.h index c81c0b0c054befaf665b..43f3ca58fcd550b8ac83 100644 --- a/tools/testing/selftests/kselftest_module.h +++ b/tools/testing/selftests/kselftest_module.h @@ -3,14 +3,31 @@ #define __KSELFTEST_MODULE_H #include <linux/module.h> +#include <linux/prandom.h> +#include <linux/random.h> /* * Test framework for writing test modules to be loaded by kselftest. * See Documentation/dev-tools/kselftest.rst for an example test module. */ +/* + * If the test module makes use of random numbers, define KSTM_RANDOM + * to 1 before including this header. Then a module parameter "seed" + * will be defined. If not given, a random one will be obtained. In + * either case, the used seed is reported, so the exact same series of + * tests can be repeated by loading the module with that seed + * given. + */ + +#ifndef KSTM_RANDOM +#define KSTM_RANDOM 0 +#endif + static unsigned int total_tests __initdata; static unsigned int failed_tests __initdata; +static struct rnd_state rnd_state __initdata; +static u64 seed __initdata; #define KSTM_CHECK_ZERO(x) do { \ total_tests++; \ @@ -22,11 +39,13 @@ static unsigned int failed_tests __initdata; static inline int kstm_report(unsigned int total_tests, unsigned int failed_tests) { - if (failed_tests == 0) + if (failed_tests == 0) { pr_info("all %u tests passed\n", total_tests); - else + } else { pr_warn("failed %u out of %u tests\n", failed_tests, total_tests); - + if (KSTM_RANDOM) + pr_info("random seed used was 0x%016llx\n", seed); + } return failed_tests ? -EINVAL : 0; } @@ -34,6 +53,12 @@ static inline int kstm_report(unsigned int total_tests, unsigned int failed_test static int __init __module##_init(void) \ { \ pr_info("loaded.\n"); \ + if (KSTM_RANDOM) { \ + if (!seed) \ + seed = get_random_u64(); \ + prandom_seed_state(&rnd_state, seed); \ + pr_info("random seed = 0x%016llx\n", seed); \ + } \ selftest(); \ return kstm_report(total_tests, failed_tests); \ } \ @@ -44,4 +69,8 @@ static void __exit __module##_exit(void) \ module_init(__module##_init); \ module_exit(__module##_exit) +#if KSTM_RANDOM +module_param(seed, ullong, 0444); +#endif + #endif /* __KSELFTEST_MODULE_H */
Some test suites make use of random numbers to increase the test coverage when the test suite gets run on different machines and increase the chance of some corner case bug being discovered - and I'm planning on extending some existing ones in that direction as well. However, should a bug be found this way, it's important that the exact same series of tests can be repeated to verify the bug is fixed. That means the random numbers must be obtained deterministically from a generator private to the test module. To avoid adding boilerplate to various test modules, put some logic into kselftest_module.h: If the module declares that it will use random numbers, add a "seed" module parameter. If not explicitly given when the module is loaded (or via kernel command line), obtain a random one. In either case, print the seed used, and repeat that information if there was at least one test failing. Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk> --- tools/testing/selftests/kselftest_module.h | 35 ++++++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-)