| Message ID | 20230725212522.1622716-1-rmoar@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | kunit: Add test attributes API | expand |
On 25/07/23 18:25, 'Rae Moar' via KUnit Development wrote: > Hello everyone, > > This patch series adds a test attributes framework to KUnit. > > There has been interest in filtering out "slow" KUnit tests. Most notably, > a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude a > particularly slow memcpy test > (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). > > This attributes framework can be used to save and access test associated > data, including whether a test is slow. These attributes are reportable > (via KTAP and command line output) and are also filterable. > > This framework is designed to allow for the addition of other attributes in > the future. These attributes could include whether the test can be run > concurrently, test file path, etc. > > To try out the framework I suggest running: > "./tools/testing/kunit/kunit.py run --filter speed!=slow" > > This patch series was originally sent out as an RFC. Here is a link to the > RFC v2: > https://lore.kernel.org/all/20230707210947.1208717-1-rmoar@google.com/ > > Thanks! > Rae > Great addition to the KUnit framework! On all the patch set Reviewed-by: Arthur Grillo <arthurgrillo@riseup.net> Best Regards, ~Arthur Grillo > Rae Moar (9): > kunit: Add test attributes API structure > kunit: Add speed attribute > kunit: Add module attribute > kunit: Add ability to filter attributes > kunit: tool: Add command line interface to filter and report > attributes > kunit: memcpy: Mark tests as slow using test attributes > kunit: time: Mark test as slow using test attributes > kunit: add tests for filtering attributes > kunit: Add documentation of KUnit test attributes > > Documentation/dev-tools/kunit/run_wrapper.rst | 12 + > .../dev-tools/kunit/running_tips.rst | 166 +++++++ > include/kunit/attributes.h | 50 +++ > include/kunit/test.h | 70 ++- > kernel/time/time_test.c | 2 +- > lib/Kconfig.debug | 3 + > lib/kunit/Makefile | 3 +- > lib/kunit/attributes.c | 418 ++++++++++++++++++ > lib/kunit/executor.c | 114 ++++- > lib/kunit/executor_test.c | 128 +++++- > lib/kunit/kunit-example-test.c | 9 + > lib/kunit/test.c | 27 +- > lib/memcpy_kunit.c | 8 +- > tools/testing/kunit/kunit.py | 70 ++- > tools/testing/kunit/kunit_kernel.py | 8 +- > tools/testing/kunit/kunit_parser.py | 11 +- > tools/testing/kunit/kunit_tool_test.py | 39 +- > 17 files changed, 1062 insertions(+), 76 deletions(-) > create mode 100644 include/kunit/attributes.h > create mode 100644 lib/kunit/attributes.c > > > base-commit: 64bd4641310c41a1ecf07c13c67bc0ed61045dfd
On 25/07/23 18:25, 'Rae Moar' via KUnit Development wrote: > Hello everyone, > > This patch series adds a test attributes framework to KUnit. > > There has been interest in filtering out "slow" KUnit tests. Most notably, > a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude a > particularly slow memcpy test > (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). > > This attributes framework can be used to save and access test associated > data, including whether a test is slow. These attributes are reportable > (via KTAP and command line output) and are also filterable. > > This framework is designed to allow for the addition of other attributes in > the future. These attributes could include whether the test can be run > concurrently, test file path, etc. > > To try out the framework I suggest running: > "./tools/testing/kunit/kunit.py run --filter speed!=slow" > > This patch series was originally sent out as an RFC. Here is a link to the > RFC v2: > https://lore.kernel.org/all/20230707210947.1208717-1-rmoar@google.com/ > > Thanks! > Rae > Great addition to the KUnit framework! On all the patch set Reviewed-by: Arthur Grillo <arthurgrillo@riseup.net> Best Regards, ~Arthur Grillo > Rae Moar (9): > kunit: Add test attributes API structure > kunit: Add speed attribute > kunit: Add module attribute > kunit: Add ability to filter attributes > kunit: tool: Add command line interface to filter and report > attributes > kunit: memcpy: Mark tests as slow using test attributes > kunit: time: Mark test as slow using test attributes > kunit: add tests for filtering attributes > kunit: Add documentation of KUnit test attributes > > Documentation/dev-tools/kunit/run_wrapper.rst | 12 + > .../dev-tools/kunit/running_tips.rst | 166 +++++++ > include/kunit/attributes.h | 50 +++ > include/kunit/test.h | 70 ++- > kernel/time/time_test.c | 2 +- > lib/Kconfig.debug | 3 + > lib/kunit/Makefile | 3 +- > lib/kunit/attributes.c | 418 ++++++++++++++++++ > lib/kunit/executor.c | 114 ++++- > lib/kunit/executor_test.c | 128 +++++- > lib/kunit/kunit-example-test.c | 9 + > lib/kunit/test.c | 27 +- > lib/memcpy_kunit.c | 8 +- > tools/testing/kunit/kunit.py | 70 ++- > tools/testing/kunit/kunit_kernel.py | 8 +- > tools/testing/kunit/kunit_parser.py | 11 +- > tools/testing/kunit/kunit_tool_test.py | 39 +- > 17 files changed, 1062 insertions(+), 76 deletions(-) > create mode 100644 include/kunit/attributes.h > create mode 100644 lib/kunit/attributes.c > > > base-commit: 64bd4641310c41a1ecf07c13c67bc0ed61045dfd
On Tue, Jul 25, 2023 at 09:25:11PM +0000, Rae Moar wrote: > Hello everyone, > > This patch series adds a test attributes framework to KUnit. > With this series in linux-next, almost 50% of my boot tests crash, and many others show warnings tracebacks. Example crash from x86_64 boot attempt attached. Guenter --- [ 6.165419] ok 6 lib_sort [ 6.166044] KTAP version 1 [ 6.166139] # Subtest: kunit_executor_test [ 6.166247] # module: kunit [ 6.166261] 1..8 [ 6.168252] ok 1 parse_filter_test [ 6.169579] BUG: unable to handle page fault for address: ffffd757e80004c8 [ 6.169872] #PF: supervisor read access in kernel mode [ 6.169990] #PF: error_code(0x0000) - not-present page [ 6.170142] PGD ffd4067 P4D ffd4067 PUD 0 [ 6.170346] Oops: 0000 [#1] PREEMPT SMP PTI [ 6.170521] CPU: 0 PID: 182 Comm: kunit_try_catch Tainted: G N 6.5.0-rc3-next-20230728 #1 [ 6.170735] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014 [ 6.171024] RIP: 0010:kfree+0x61/0x140 [ 6.171313] Code: 80 48 01 e8 0f 82 ec 00 00 00 48 c7 c2 00 00 00 80 48 2b 15 29 bf aa 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06 48 03 05 07 bf aa 01 <48> 8b 50 08 48 89 c7 f6 c2 01 0f 85 b3 00 00 00 66 90 48 8b 07 f6 [ 6.171715] RSP: 0000:ffffb48f80267dc8 EFLAGS: 00000286 [ 6.171847] RAX: ffffd757e80004c0 RBX: ffffa34583aef5d0 RCX: 0000000000000000 [ 6.171998] RDX: 00005cba00000000 RSI: ffffffff95cb1ad5 RDI: ffffb48f80013cd0 [ 6.172146] RBP: ffffb48f80013cd0 R08: 0000000000000001 R09: 0000000000000001 [ 6.172293] R10: 0000000000000001 R11: 0000000000000001 R12: ffffb48f80267e74 [ 6.172445] R13: 0000000000000000 R14: ffffa34583bf4000 R15: 0000000000000002 [ 6.172620] FS: 0000000000000000(0000) GS:ffffa3458f800000(0000) knlGS:0000000000000000 [ 6.172793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6.172917] CR2: ffffd757e80004c8 CR3: 0000000009048000 CR4: 00000000007506f0 [ 6.173089] PKRU: 55555554 [ 6.173177] Call Trace: [ 6.173313] <TASK> [ 6.173494] ? __die+0x1f/0x70 [ 6.173600] ? page_fault_oops+0x14a/0x460 [ 6.173693] ? search_exception_tables+0x37/0x50 [ 6.173793] ? fixup_exception+0x21/0x310 [ 6.173887] ? exc_page_fault+0xee/0x1c0 [ 6.173977] ? asm_exc_page_fault+0x26/0x30 [ 6.174076] ? kunit_filter_suites+0x3a5/0x460 [ 6.174192] ? kfree+0x61/0x140 [ 6.174271] ? kfree+0x106/0x140 [ 6.174353] kunit_filter_suites+0x3a5/0x460 [ 6.174473] ? __pfx_kunit_generic_run_threadfn_adapter+0x10/0x10 [ 6.174604] filter_suites_test+0xea/0x2c0 [ 6.174702] ? __pfx_kunit_generic_run_threadfn_adapter+0x10/0x10 [ 6.174828] kunit_generic_run_threadfn_adapter+0x15/0x20 [ 6.174941] kthread+0xef/0x120 [ 6.175012] ? __pfx_kthread+0x10/0x10 [ 6.175098] ret_from_fork+0x2f/0x50 [ 6.175177] ? __pfx_kthread+0x10/0x10 [ 6.175260] ret_from_fork_asm+0x1b/0x30 [ 6.175387] </TASK> [ 6.175466] Modules linked in: [ 6.175640] CR2: ffffd757e80004c8 [ 6.175887] ---[ end trace 0000000000000000 ]--- [ 6.176029] RIP: 0010:kfree+0x61/0x140 [ 6.176119] Code: 80 48 01 e8 0f 82 ec 00 00 00 48 c7 c2 00 00 00 80 48 2b 15 29 bf aa 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06 48 03 05 07 bf aa 01 <48> 8b 50 08 48 89 c7 f6 c2 01 0f 85 b3 00 00 00 66 90 48 8b 07 f6 [ 6.176478] RSP: 0000:ffffb48f80267dc8 EFLAGS: 00000286 [ 6.176589] RAX: ffffd757e80004c0 RBX: ffffa34583aef5d0 RCX: 0000000000000000 [ 6.176729] RDX: 00005cba00000000 RSI: ffffffff95cb1ad5 RDI: ffffb48f80013cd0 [ 6.176869] RBP: ffffb48f80013cd0 R08: 0000000000000001 R09: 0000000000000001 [ 6.177008] R10: 0000000000000001 R11: 0000000000000001 R12: ffffb48f80267e74 [ 6.177150] R13: 0000000000000000 R14: ffffa34583bf4000 R15: 0000000000000002 [ 6.177290] FS: 0000000000000000(0000) GS:ffffa3458f800000(0000) knlGS:0000000000000000 [ 6.177448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6.177564] CR2: ffffd757e80004c8 CR3: 0000000009048000 CR4: 00000000007506f0 [ 6.177704] PKRU: 55555554 [ 6.177807] note: kunit_try_catch[182] exited with irqs disabled
On Fri, 28 Jul 2023 at 18:38, Guenter Roeck <linux@roeck-us.net> wrote: > > On Tue, Jul 25, 2023 at 09:25:11PM +0000, Rae Moar wrote: > > Hello everyone, > > > > This patch series adds a test attributes framework to KUnit. > > > > With this series in linux-next, almost 50% of my boot tests crash, > and many others show warnings tracebacks. > > Example crash from x86_64 boot attempt attached. > > Guenter > > --- Hmm... It's not reproducing here (even with KASAN), which is strange. Does the patch here help? https://lore.kernel.org/linux-kselftest/20230729010003.4058582-1-ruanjinjie@huawei.com/ -- David > [ 6.165419] ok 6 lib_sort > [ 6.166044] KTAP version 1 > [ 6.166139] # Subtest: kunit_executor_test > [ 6.166247] # module: kunit > [ 6.166261] 1..8 > [ 6.168252] ok 1 parse_filter_test > [ 6.169579] BUG: unable to handle page fault for address: ffffd757e80004c8 > [ 6.169872] #PF: supervisor read access in kernel mode > [ 6.169990] #PF: error_code(0x0000) - not-present page > [ 6.170142] PGD ffd4067 P4D ffd4067 PUD 0 > [ 6.170346] Oops: 0000 [#1] PREEMPT SMP PTI > [ 6.170521] CPU: 0 PID: 182 Comm: kunit_try_catch Tainted: G N 6.5.0-rc3-next-20230728 #1 > [ 6.170735] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014 > [ 6.171024] RIP: 0010:kfree+0x61/0x140 > [ 6.171313] Code: 80 48 01 e8 0f 82 ec 00 00 00 48 c7 c2 00 00 00 80 48 2b 15 29 bf aa 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06 48 03 05 07 bf aa 01 <48> 8b 50 08 48 89 c7 f6 c2 01 0f 85 b3 00 00 00 66 90 48 8b 07 f6 > [ 6.171715] RSP: 0000:ffffb48f80267dc8 EFLAGS: 00000286 > [ 6.171847] RAX: ffffd757e80004c0 RBX: ffffa34583aef5d0 RCX: 0000000000000000 > [ 6.171998] RDX: 00005cba00000000 RSI: ffffffff95cb1ad5 RDI: ffffb48f80013cd0 > [ 6.172146] RBP: ffffb48f80013cd0 R08: 0000000000000001 R09: 0000000000000001 > [ 6.172293] R10: 0000000000000001 R11: 0000000000000001 R12: ffffb48f80267e74 > [ 6.172445] R13: 0000000000000000 R14: ffffa34583bf4000 R15: 0000000000000002 > [ 6.172620] FS: 0000000000000000(0000) GS:ffffa3458f800000(0000) knlGS:0000000000000000 > [ 6.172793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 6.172917] CR2: ffffd757e80004c8 CR3: 0000000009048000 CR4: 00000000007506f0 > [ 6.173089] PKRU: 55555554 > [ 6.173177] Call Trace: > [ 6.173313] <TASK> > [ 6.173494] ? __die+0x1f/0x70 > [ 6.173600] ? page_fault_oops+0x14a/0x460 > [ 6.173693] ? search_exception_tables+0x37/0x50 > [ 6.173793] ? fixup_exception+0x21/0x310 > [ 6.173887] ? exc_page_fault+0xee/0x1c0 > [ 6.173977] ? asm_exc_page_fault+0x26/0x30 > [ 6.174076] ? kunit_filter_suites+0x3a5/0x460 > [ 6.174192] ? kfree+0x61/0x140 > [ 6.174271] ? kfree+0x106/0x140 > [ 6.174353] kunit_filter_suites+0x3a5/0x460 > [ 6.174473] ? __pfx_kunit_generic_run_threadfn_adapter+0x10/0x10 > [ 6.174604] filter_suites_test+0xea/0x2c0 > [ 6.174702] ? __pfx_kunit_generic_run_threadfn_adapter+0x10/0x10 > [ 6.174828] kunit_generic_run_threadfn_adapter+0x15/0x20 > [ 6.174941] kthread+0xef/0x120 > [ 6.175012] ? __pfx_kthread+0x10/0x10 > [ 6.175098] ret_from_fork+0x2f/0x50 > [ 6.175177] ? __pfx_kthread+0x10/0x10 > [ 6.175260] ret_from_fork_asm+0x1b/0x30 > [ 6.175387] </TASK> > [ 6.175466] Modules linked in: > [ 6.175640] CR2: ffffd757e80004c8 > [ 6.175887] ---[ end trace 0000000000000000 ]--- > [ 6.176029] RIP: 0010:kfree+0x61/0x140 > [ 6.176119] Code: 80 48 01 e8 0f 82 ec 00 00 00 48 c7 c2 00 00 00 80 48 2b 15 29 bf aa 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06 48 03 05 07 bf aa 01 <48> 8b 50 08 48 89 c7 f6 c2 01 0f 85 b3 00 00 00 66 90 48 8b 07 f6 > [ 6.176478] RSP: 0000:ffffb48f80267dc8 EFLAGS: 00000286 > [ 6.176589] RAX: ffffd757e80004c0 RBX: ffffa34583aef5d0 RCX: 0000000000000000 > [ 6.176729] RDX: 00005cba00000000 RSI: ffffffff95cb1ad5 RDI: ffffb48f80013cd0 > [ 6.176869] RBP: ffffb48f80013cd0 R08: 0000000000000001 R09: 0000000000000001 > [ 6.177008] R10: 0000000000000001 R11: 0000000000000001 R12: ffffb48f80267e74 > [ 6.177150] R13: 0000000000000000 R14: ffffa34583bf4000 R15: 0000000000000002 > [ 6.177290] FS: 0000000000000000(0000) GS:ffffa3458f800000(0000) knlGS:0000000000000000 > [ 6.177448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 6.177564] CR2: ffffd757e80004c8 CR3: 0000000009048000 CR4: 00000000007506f0 > [ 6.177704] PKRU: 55555554 > [ 6.177807] note: kunit_try_catch[182] exited with irqs disabled > > -- > You received this message because you are subscribed to the Google Groups "KUnit Development" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/59dc28b1-298c-4e7b-b35f-0b94071f4cb5%40roeck-us.net.
On 7/29/23 00:53, David Gow wrote: > On Fri, 28 Jul 2023 at 18:38, Guenter Roeck <linux@roeck-us.net> wrote: >> >> On Tue, Jul 25, 2023 at 09:25:11PM +0000, Rae Moar wrote: >>> Hello everyone, >>> >>> This patch series adds a test attributes framework to KUnit. >>> >> >> With this series in linux-next, almost 50% of my boot tests crash, >> and many others show warnings tracebacks. >> >> Example crash from x86_64 boot attempt attached. >> >> Guenter >> >> --- > > Hmm... It's not reproducing here (even with KASAN), which is strange. > > Does the patch here help? > https://lore.kernel.org/linux-kselftest/20230729010003.4058582-1-ruanjinjie@huawei.com/ > Yes, it does. Guenter
Hello everyone, This patch series adds a test attributes framework to KUnit. There has been interest in filtering out "slow" KUnit tests. Most notably, a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude a particularly slow memcpy test (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). This attributes framework can be used to save and access test associated data, including whether a test is slow. These attributes are reportable (via KTAP and command line output) and are also filterable. This framework is designed to allow for the addition of other attributes in the future. These attributes could include whether the test can be run concurrently, test file path, etc. To try out the framework I suggest running: "./tools/testing/kunit/kunit.py run --filter speed!=slow" This patch series was originally sent out as an RFC. Here is a link to the RFC v2: https://lore.kernel.org/all/20230707210947.1208717-1-rmoar@google.com/ Thanks! Rae Rae Moar (9): kunit: Add test attributes API structure kunit: Add speed attribute kunit: Add module attribute kunit: Add ability to filter attributes kunit: tool: Add command line interface to filter and report attributes kunit: memcpy: Mark tests as slow using test attributes kunit: time: Mark test as slow using test attributes kunit: add tests for filtering attributes kunit: Add documentation of KUnit test attributes Documentation/dev-tools/kunit/run_wrapper.rst | 12 + .../dev-tools/kunit/running_tips.rst | 166 +++++++ include/kunit/attributes.h | 50 +++ include/kunit/test.h | 70 ++- kernel/time/time_test.c | 2 +- lib/Kconfig.debug | 3 + lib/kunit/Makefile | 3 +- lib/kunit/attributes.c | 418 ++++++++++++++++++ lib/kunit/executor.c | 114 ++++- lib/kunit/executor_test.c | 128 +++++- lib/kunit/kunit-example-test.c | 9 + lib/kunit/test.c | 27 +- lib/memcpy_kunit.c | 8 +- tools/testing/kunit/kunit.py | 70 ++- tools/testing/kunit/kunit_kernel.py | 8 +- tools/testing/kunit/kunit_parser.py | 11 +- tools/testing/kunit/kunit_tool_test.py | 39 +- 17 files changed, 1062 insertions(+), 76 deletions(-) create mode 100644 include/kunit/attributes.h create mode 100644 lib/kunit/attributes.c base-commit: 64bd4641310c41a1ecf07c13c67bc0ed61045dfd