| Message ID | 20230302013822.1808711-3-sboyd@kernel.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | clk: Add kunit tests for fixed rate and parent data | expand |
On Thu, 2 Mar 2023 at 09:38, Stephen Boyd <sboyd@kernel.org> wrote: > > To fully exercise common clk framework code in KUnit we need to > associate 'struct device' pointers with 'struct device_node' pointers so > that things like clk_get() can parse DT nodes for 'clocks' and so that > clk providers can use DT to provide clks; the most common mode of > operation for clk providers. > > Adding support to KUnit so that it loads a DTB is fairly simple after > commit b31297f04e86 ("um: Add devicetree support"). We can simply pass a > pre-compiled deviectree blob (DTB) on the kunit.py commandline and UML > will load it. The problem is that tests won't know that the commandline > has been modified, nor that a DTB has been loaded. Take a different > approach so that tests can skip if a DTB hasn't been loaded. > > Reuse the Makefile logic from the OF unittests to build a DTB into the > kernel. This DTB will be for the mythical machine "linux,kunit", i.e. > the devicetree for the KUnit "board". In practice, it is a dtsi file > that will gather includes for kunit tests that rely in part on a > devicetree being loaded. The devicetree should only be loaded if > CONFIG_OF_KUNIT=y. Make that a choice config parallel to the existing > CONFIG_OF_UNITTEST so that only one devicetree can be loaded in the > system at a time. Similarly, the kernel commandline option to load a > DTB is ignored if CONFIG_OF_KUNIT is enabled so that only one DTB is > loaded at a time. This feels a little bit like it's just papering over the real problem, which is that there's no way tests can skip themselves if no DTB is loaded. That being said, I do think that there's probably some sense in supporting the compiled-in DTB as well (it's definitely simpler than patching kunit.py to always pass the extra command-line option in, for example). But maybe it'd be nice to have the command-line option override the built-in one if present. > > Add a simple unit test to confirm that the DTB loading worked. Future > tests will add to the kunit.dtsi file to include their specific test > nodes. > > Cc: Richard Weinberger <richard@nod.at> > Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com> > Cc: Johannes Berg <johannes@sipsolutions.net> > Cc: Vincent Whitchurch <vincent.whitchurch@axis.com> > Cc: Rob Herring <robh+dt@kernel.org> > Cc: Frank Rowand <frowand.list@gmail.com> > Signed-off-by: Stephen Boyd <sboyd@kernel.org> > --- > arch/um/kernel/dtb.c | 29 +++++++++++++++-- > drivers/of/Kconfig | 26 ++++++++++++++++ > drivers/of/Makefile | 1 + > drivers/of/kunit/.kunitconfig | 4 +++ > drivers/of/kunit/Makefile | 4 +++ > drivers/of/kunit/kunit.dtsi | 8 +++++ > drivers/of/kunit/kunit.dtso | 4 +++ > drivers/of/kunit/uml_dtb_test.c | 55 +++++++++++++++++++++++++++++++++ > 8 files changed, 128 insertions(+), 3 deletions(-) > create mode 100644 drivers/of/kunit/.kunitconfig > create mode 100644 drivers/of/kunit/Makefile > create mode 100644 drivers/of/kunit/kunit.dtsi > create mode 100644 drivers/of/kunit/kunit.dtso > create mode 100644 drivers/of/kunit/uml_dtb_test.c > > diff --git a/arch/um/kernel/dtb.c b/arch/um/kernel/dtb.c > index 484141b06938..ee63951b12df 100644 > --- a/arch/um/kernel/dtb.c > +++ b/arch/um/kernel/dtb.c > @@ -15,9 +15,32 @@ void uml_dtb_init(void) > long long size; > void *area; > > - area = uml_load_file(dtb, &size); > - if (!area) > - return; > + if (IS_ENABLED(CONFIG_OF_KUNIT)) { > + /* > + * __dtbo_kunit_begin[] and __dtbo_kunit_end[] are magically > + * created by cmd_dt_S_dtbo in scripts/Makefile.lib from the > + * drivers/of/kunit/kunit.dtsi file. > + */ > + extern uint8_t __dtbo_kunit_begin[]; > + extern uint8_t __dtbo_kunit_end[]; > + > + size = __dtbo_kunit_end - __dtbo_kunit_begin; > + if (!size) { > + pr_warn("%s: kunit testcases is empty\n", __func__); > + return; > + } > + > + /* creating copy */ > + area = memblock_alloc(size, 8); > + if (!area) > + return; > + > + memcpy(area, __dtbo_kunit_begin, size); > + } else { I think this should probably override the KUnit dtb if present (so, try to load the dtb, and fallback to the builtin one). If not, I think we should at least print a warning if a DTB is specified on the command-line, but we're not using it. > + area = uml_load_file(dtb, &size); > + if (!area) > + return; > + } > > if (!early_init_dt_scan(area)) { > pr_err("invalid DTB %s\n", dtb); > diff --git a/drivers/of/Kconfig b/drivers/of/Kconfig > index 80b5fd44ab1c..1f968b6a3dde 100644 > --- a/drivers/of/Kconfig > +++ b/drivers/of/Kconfig > @@ -12,6 +12,20 @@ menuconfig OF > > if OF > > +choice > + prompt "Devicetree Runtime Tests" > + default OF_UNITTEST > + > +config OF_KUNIT > + bool "Devicetree KUnit support" if KUNIT > + depends on UML > + select IRQ_DOMAIN > + select OF_EARLY_FLATTREE > + help > + This option builds in KUnit test cases that rely on device tree infrastructure. > + A fake Device Tree Blob (DTB) is loaded on the UML kernel running KUnit so that > + KUnit tests can test device tree dependent code. > + > config OF_UNITTEST > bool "Device Tree runtime unit tests" > depends on !SPARC > @@ -25,6 +39,18 @@ config OF_UNITTEST > > If unsure, say N here, but this option is safe to enable. > > +endchoice > + > +config OF_DTB_KUNIT_TEST > + tristate "Devicetree KUnit DTB Test" if !KUNIT_ALL_TESTS > + depends on KUNIT > + default KUNIT_ALL_TESTS > + help > + This option builds unit tests for the "linux,kunit" DTB built into > + the UML kernel image. > + > + If unsure, say N here, but this option is safe to enable. > + > config OF_ALL_DTBS > bool "Build all Device Tree Blobs" > depends on COMPILE_TEST > diff --git a/drivers/of/Makefile b/drivers/of/Makefile > index e0360a44306e..16eef3fdf60a 100644 > --- a/drivers/of/Makefile > +++ b/drivers/of/Makefile > @@ -19,4 +19,5 @@ obj-y += kexec.o > endif > endif > > +obj-y += kunit/ > obj-$(CONFIG_OF_UNITTEST) += unittest-data/ > diff --git a/drivers/of/kunit/.kunitconfig b/drivers/of/kunit/.kunitconfig > new file mode 100644 > index 000000000000..1def0ad30d29 > --- /dev/null > +++ b/drivers/of/kunit/.kunitconfig > @@ -0,0 +1,4 @@ > +CONFIG_KUNIT=y > +CONFIG_OF=y > +CONFIG_OF_KUNIT=y > +CONFIG_OF_DTB_KUNIT_TEST=y > diff --git a/drivers/of/kunit/Makefile b/drivers/of/kunit/Makefile > new file mode 100644 > index 000000000000..ffe0447e1ac7 > --- /dev/null > +++ b/drivers/of/kunit/Makefile > @@ -0,0 +1,4 @@ > +# SPDX-License-Identifier: GPL-2.0 > +obj-$(CONFIG_OF_KUNIT) += kunit.dtbo.o > + > +obj-$(CONFIG_OF_DTB_KUNIT_TEST) += uml_dtb_test.o > diff --git a/drivers/of/kunit/kunit.dtsi b/drivers/of/kunit/kunit.dtsi > new file mode 100644 > index 000000000000..82f6c3e2b8d5 > --- /dev/null > +++ b/drivers/of/kunit/kunit.dtsi > @@ -0,0 +1,8 @@ > +// SPDX-License-Identifier: GPL-2.0 > + > +/ { > + model = "KUnit UML"; > + compatible = "linux,kunit"; > +}; > + > +/* Include testcase dtsi files below */ > diff --git a/drivers/of/kunit/kunit.dtso b/drivers/of/kunit/kunit.dtso > new file mode 100644 > index 000000000000..50187e8d1422 > --- /dev/null > +++ b/drivers/of/kunit/kunit.dtso > @@ -0,0 +1,4 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/dts-v1/; > + > +#include "kunit.dtsi" > diff --git a/drivers/of/kunit/uml_dtb_test.c b/drivers/of/kunit/uml_dtb_test.c > new file mode 100644 > index 000000000000..8966c9ebf51f > --- /dev/null > +++ b/drivers/of/kunit/uml_dtb_test.c > @@ -0,0 +1,55 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * KUnit tests for DTB loading on UML > + */ > +#include <linux/kconfig.h> > +#include <linux/of.h> > +#include <linux/of_fdt.h> > + > +#include <kunit/test.h> > + > +/* > + * Test that of_machine_is_compatible() returns positive int when loaded DTB > + * matches. > + */ > +static void uml_dtb_of_machine_compatible_test(struct kunit *test) > +{ > + KUNIT_EXPECT_GT(test, of_machine_is_compatible("linux,kunit"), 0); > +} > + > +/* > + * Test that of_flat_dt_get_machine_name() returns the expected 'model' from the > + * loaded DTB. > + */ > +static void uml_dtb_of_flat_dt_get_machine_name_test(struct kunit *test) > +{ > + KUNIT_EXPECT_STREQ(test, of_flat_dt_get_machine_name(), "KUnit UML"); > +} > + > +static struct kunit_case uml_dtb_test_cases[] = { > + KUNIT_CASE(uml_dtb_of_machine_compatible_test), > + KUNIT_CASE(uml_dtb_of_flat_dt_get_machine_name_test), > + {} > +}; > + > +static int uml_dtb_test_init(struct kunit *test) > +{ > + if (!IS_ENABLED(CONFIG_OF_KUNIT)) > + kunit_skip(test, "requires CONFIG_OF_KUNIT"); > + > + return 0; > +} > + > +/* > + * Test suite to confirm DTB is loaded on UML. > + */ > +static struct kunit_suite uml_dtb_suite = { > + .name = "uml_dtb", > + .init = uml_dtb_test_init, > + .test_cases = uml_dtb_test_cases, > +}; > + > +kunit_test_suites( > + ¨_dtb_suite, > +); > +MODULE_LICENSE("GPL"); > -- > https://git.kernel.org/pub/scm/linux/kernel/git/clk/linux.git/ > https://git.kernel.org/pub/scm/linux/kernel/git/sboyd/spmi.git > > -- > You received this message because you are subscribed to the Google Groups "KUnit Development" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/20230302013822.1808711-3-sboyd%40kernel.org.
On Wed, Mar 01, 2023 at 05:38:15PM -0800, Stephen Boyd wrote: > To fully exercise common clk framework code in KUnit we need to > associate 'struct device' pointers with 'struct device_node' pointers so > that things like clk_get() can parse DT nodes for 'clocks' and so that > clk providers can use DT to provide clks; the most common mode of > operation for clk providers. > > Adding support to KUnit so that it loads a DTB is fairly simple after > commit b31297f04e86 ("um: Add devicetree support"). We can simply pass a > pre-compiled deviectree blob (DTB) on the kunit.py commandline and UML > will load it. The problem is that tests won't know that the commandline > has been modified, nor that a DTB has been loaded. Take a different > approach so that tests can skip if a DTB hasn't been loaded. > > Reuse the Makefile logic from the OF unittests to build a DTB into the > kernel. This DTB will be for the mythical machine "linux,kunit", i.e. > the devicetree for the KUnit "board". In practice, it is a dtsi file > that will gather includes for kunit tests that rely in part on a > devicetree being loaded. The devicetree should only be loaded if > CONFIG_OF_KUNIT=y. Make that a choice config parallel to the existing > CONFIG_OF_UNITTEST so that only one devicetree can be loaded in the > system at a time. Similarly, the kernel commandline option to load a > DTB is ignored if CONFIG_OF_KUNIT is enabled so that only one DTB is > loaded at a time. > > Add a simple unit test to confirm that the DTB loading worked. Future > tests will add to the kunit.dtsi file to include their specific test > nodes. > > Cc: Richard Weinberger <richard@nod.at> > Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com> > Cc: Johannes Berg <johannes@sipsolutions.net> > Cc: Vincent Whitchurch <vincent.whitchurch@axis.com> > Cc: Rob Herring <robh+dt@kernel.org> > Cc: Frank Rowand <frowand.list@gmail.com> > Signed-off-by: Stephen Boyd <sboyd@kernel.org> > --- > arch/um/kernel/dtb.c | 29 +++++++++++++++-- > drivers/of/Kconfig | 26 ++++++++++++++++ > drivers/of/Makefile | 1 + > drivers/of/kunit/.kunitconfig | 4 +++ > drivers/of/kunit/Makefile | 4 +++ > drivers/of/kunit/kunit.dtsi | 8 +++++ > drivers/of/kunit/kunit.dtso | 4 +++ > drivers/of/kunit/uml_dtb_test.c | 55 +++++++++++++++++++++++++++++++++ > 8 files changed, 128 insertions(+), 3 deletions(-) > create mode 100644 drivers/of/kunit/.kunitconfig > create mode 100644 drivers/of/kunit/Makefile > create mode 100644 drivers/of/kunit/kunit.dtsi > create mode 100644 drivers/of/kunit/kunit.dtso > create mode 100644 drivers/of/kunit/uml_dtb_test.c > > diff --git a/arch/um/kernel/dtb.c b/arch/um/kernel/dtb.c > index 484141b06938..ee63951b12df 100644 > --- a/arch/um/kernel/dtb.c > +++ b/arch/um/kernel/dtb.c > @@ -15,9 +15,32 @@ void uml_dtb_init(void) > long long size; > void *area; > > - area = uml_load_file(dtb, &size); > - if (!area) > - return; > + if (IS_ENABLED(CONFIG_OF_KUNIT)) { > + /* > + * __dtbo_kunit_begin[] and __dtbo_kunit_end[] are magically > + * created by cmd_dt_S_dtbo in scripts/Makefile.lib from the > + * drivers/of/kunit/kunit.dtsi file. > + */ > + extern uint8_t __dtbo_kunit_begin[]; > + extern uint8_t __dtbo_kunit_end[]; > + > + size = __dtbo_kunit_end - __dtbo_kunit_begin; > + if (!size) { > + pr_warn("%s: kunit testcases is empty\n", __func__); > + return; > + } > + > + /* creating copy */ > + area = memblock_alloc(size, 8); > + if (!area) > + return; > + > + memcpy(area, __dtbo_kunit_begin, size); > + } else { > + area = uml_load_file(dtb, &size); > + if (!area) > + return; > + } > > if (!early_init_dt_scan(area)) { > pr_err("invalid DTB %s\n", dtb); > diff --git a/drivers/of/Kconfig b/drivers/of/Kconfig > index 80b5fd44ab1c..1f968b6a3dde 100644 > --- a/drivers/of/Kconfig > +++ b/drivers/of/Kconfig > @@ -12,6 +12,20 @@ menuconfig OF > > if OF > > +choice No. This needs to be reworked such that a kernel rebuild is not needed to run different tests. I suspect that the overlay approach will do that for you. > + prompt "Devicetree Runtime Tests" > + default OF_UNITTEST > + > +config OF_KUNIT > + bool "Devicetree KUnit support" if KUNIT > + depends on UML This is not a great dependency either... Rob
On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: > > Quoting David Gow (2023-03-02 23:15:04) > > On Thu, 2 Mar 2023 at 09:38, Stephen Boyd <sboyd@kernel.org> wrote: > > > > > > To fully exercise common clk framework code in KUnit we need to > > > associate 'struct device' pointers with 'struct device_node' pointers so > > > that things like clk_get() can parse DT nodes for 'clocks' and so that > > > clk providers can use DT to provide clks; the most common mode of > > > operation for clk providers. > > > > > > Adding support to KUnit so that it loads a DTB is fairly simple after > > > commit b31297f04e86 ("um: Add devicetree support"). We can simply pass a > > > pre-compiled deviectree blob (DTB) on the kunit.py commandline and UML > > > will load it. The problem is that tests won't know that the commandline > > > has been modified, nor that a DTB has been loaded. Take a different > > > approach so that tests can skip if a DTB hasn't been loaded. > > > > > > Reuse the Makefile logic from the OF unittests to build a DTB into the > > > kernel. This DTB will be for the mythical machine "linux,kunit", i.e. > > > the devicetree for the KUnit "board". In practice, it is a dtsi file > > > that will gather includes for kunit tests that rely in part on a > > > devicetree being loaded. The devicetree should only be loaded if > > > CONFIG_OF_KUNIT=y. Make that a choice config parallel to the existing > > > CONFIG_OF_UNITTEST so that only one devicetree can be loaded in the > > > system at a time. Similarly, the kernel commandline option to load a > > > DTB is ignored if CONFIG_OF_KUNIT is enabled so that only one DTB is > > > loaded at a time. > > > > This feels a little bit like it's just papering over the real problem, > > which is that there's no way tests can skip themselves if no DTB is > > loaded. > > Hmm. I think you're suggesting that the unit test data be loaded > whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for > CONFIG_OF and skip if it isn't enabled? > More of the opposite: that we should have some way of supporting tests which might want to use a DTB other than the built-in one. Mostly for non-UML situations where an actual devicetree is needed to even boot far enough to get test output (so we wouldn't be able to override it with a compiled-in test one). I think moving to overlays probably will render this idea obsolete: but the thought was to give test code a way to check for the required devicetree nodes at runtime, and skip the test if they weren't found. That way, the failure mode for trying to boot this on something which required another device tree for, e.g., serial, would be "these tests are skipped because the wrong device tree is loaded", not "I get no output because serial isn't working". Again, though, it's only really needed for non-UML, and just loading overlays as needed should be much more sensible anyway. > > > > That being said, I do think that there's probably some sense in > > supporting the compiled-in DTB as well (it's definitely simpler than > > patching kunit.py to always pass the extra command-line option in, for > > example). > > But maybe it'd be nice to have the command-line option override the > > built-in one if present. > > Got it. I need to test loading another DTB on the commandline still, but > I think this won't be a problem. We'll load the unittest-data DTB even > with KUnit on UML, so assuming that works on UML right now it should be > unchanged by this series once I resend. Again, moving to overlays should render this mostly obsolete, no? Or am I misunderstanding how the overlay stuff will work? One possible future advantage of being able to test with custom DTs at boot time would be for fuzzing (provide random DT properties, see what happens in the test). We've got some vague plans to support a way of passing custom data to tests to support this kind of case (though, if we're using overlays, maybe the test could just patch those if we wanted to do that). Cheers, -- David
On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: > > Quoting David Gow (2023-03-10 00:09:48) > > On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: > > > > > > > > > Hmm. I think you're suggesting that the unit test data be loaded > > > whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for > > > CONFIG_OF and skip if it isn't enabled? > > > > > > > More of the opposite: that we should have some way of supporting tests > > which might want to use a DTB other than the built-in one. Mostly for > > non-UML situations where an actual devicetree is needed to even boot > > far enough to get test output (so we wouldn't be able to override it > > with a compiled-in test one). > > Ok, got it. > > > > > I think moving to overlays probably will render this idea obsolete: > > but the thought was to give test code a way to check for the required > > devicetree nodes at runtime, and skip the test if they weren't found. > > That way, the failure mode for trying to boot this on something which > > required another device tree for, e.g., serial, would be "these tests > > are skipped because the wrong device tree is loaded", not "I get no > > output because serial isn't working". > > > > Again, though, it's only really needed for non-UML, and just loading > > overlays as needed should be much more sensible anyway. > > I still have one niggle here. Loading overlays requires > CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when > CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled > in each test, but I'm thinking it may be better to simply call > kunit_skip() from the overlay loading function if the config is > disabled. This way tests can simply call the overlay loading function > and we'll halt the test immediately if the config isn't enabled. > That sounds sensible, though there is a potential pitfall. If kunit_skip() is called directly from overlay code, might introduce a dependency on kunit.ko from the DT overlay, which we might not want. The solution there is either to have a kunit wrapper function (so the call is already in kunit.ko), or to have a hook to skip the current test (which probably makes sense to do anyway, but I think the wrapper is the better option). > > > > > > > > > > That being said, I do think that there's probably some sense in > > > > supporting the compiled-in DTB as well (it's definitely simpler than > > > > patching kunit.py to always pass the extra command-line option in, for > > > > example). > > > > But maybe it'd be nice to have the command-line option override the > > > > built-in one if present. > > > > > > Got it. I need to test loading another DTB on the commandline still, but > > > I think this won't be a problem. We'll load the unittest-data DTB even > > > with KUnit on UML, so assuming that works on UML right now it should be > > > unchanged by this series once I resend. > > > > Again, moving to overlays should render this mostly obsolete, no? Or > > am I misunderstanding how the overlay stuff will work? > > Right, overlays make it largely a moot issue. The way the OF unit tests > work today is by grafting a DTB onto the live tree. I'm reusing that > logic to graft a container node target for kunit tests to add their > overlays too. It will be clearer once I post v2. > > > > > One possible future advantage of being able to test with custom DTs at > > boot time would be for fuzzing (provide random DT properties, see what > > happens in the test). We've got some vague plans to support a way of > > passing custom data to tests to support this kind of case (though, if > > we're using overlays, maybe the test could just patch those if we > > wanted to do that). > > Ah ok. I can see someone making a fuzzer that modifies devicetree > properties randomly, e.g. using different strings for clock-names. > > This reminds me of another issue I ran into. I wanted to test adding the > same platform device to the platform bus twice to confirm that the > second device can't be added. That prints a warning, which makes > kunit.py think that the test has failed because it printed a warning. Is > there some way to avoid that? I want something like > > KUNIT_EXPECT_WARNING(test, <call some function>) > > so I can test error cases. Hmm... I'd've thought that shouldn't be a problem: kunit.py should ignore most messages during a test, unless it can't find a valid result line. What does the raw KTAP output look like? (You can get it from kunit.py by passing the --raw_output option). That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something we've wanted for a while. I think that the KASAN folks have been working on something similar using console tracepoints: https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ Cheers, -- David
On 3/11/23 00:42, David Gow wrote: > On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: >> >> Quoting David Gow (2023-03-10 00:09:48) >>> On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: >>>> >>>> >>>> Hmm. I think you're suggesting that the unit test data be loaded >>>> whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for >>>> CONFIG_OF and skip if it isn't enabled? >>>> >>> >>> More of the opposite: that we should have some way of supporting tests >>> which might want to use a DTB other than the built-in one. Mostly for >>> non-UML situations where an actual devicetree is needed to even boot >>> far enough to get test output (so we wouldn't be able to override it >>> with a compiled-in test one). >> >> Ok, got it. >> >>> >>> I think moving to overlays probably will render this idea obsolete: >>> but the thought was to give test code a way to check for the required >>> devicetree nodes at runtime, and skip the test if they weren't found. >>> That way, the failure mode for trying to boot this on something which >>> required another device tree for, e.g., serial, would be "these tests >>> are skipped because the wrong device tree is loaded", not "I get no >>> output because serial isn't working". >>> >>> Again, though, it's only really needed for non-UML, and just loading >>> overlays as needed should be much more sensible anyway. >> >> I still have one niggle here. Loading overlays requires >> CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when >> CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled >> in each test, but I'm thinking it may be better to simply call >> kunit_skip() from the overlay loading function if the config is >> disabled. This way tests can simply call the overlay loading function >> and we'll halt the test immediately if the config isn't enabled. >> > > That sounds sensible, though there is a potential pitfall. If > kunit_skip() is called directly from overlay code, might introduce a > dependency on kunit.ko from the DT overlay, which we might not want. > The solution there is either to have a kunit wrapper function (so the > call is already in kunit.ko), or to have a hook to skip the current > test (which probably makes sense to do anyway, but I think the wrapper > is the better option). > > >>> >>>>> >>>>> That being said, I do think that there's probably some sense in >>>>> supporting the compiled-in DTB as well (it's definitely simpler than >>>>> patching kunit.py to always pass the extra command-line option in, for >>>>> example). >>>>> But maybe it'd be nice to have the command-line option override the >>>>> built-in one if present. >>>> >>>> Got it. I need to test loading another DTB on the commandline still, but >>>> I think this won't be a problem. We'll load the unittest-data DTB even >>>> with KUnit on UML, so assuming that works on UML right now it should be >>>> unchanged by this series once I resend. >>> >>> Again, moving to overlays should render this mostly obsolete, no? Or >>> am I misunderstanding how the overlay stuff will work? >> >> Right, overlays make it largely a moot issue. The way the OF unit tests >> work today is by grafting a DTB onto the live tree. I'm reusing that >> logic to graft a container node target for kunit tests to add their >> overlays too. It will be clearer once I post v2. >> >>> >>> One possible future advantage of being able to test with custom DTs at >>> boot time would be for fuzzing (provide random DT properties, see what >>> happens in the test). We've got some vague plans to support a way of >>> passing custom data to tests to support this kind of case (though, if >>> we're using overlays, maybe the test could just patch those if we >>> wanted to do that). >> >> Ah ok. I can see someone making a fuzzer that modifies devicetree >> properties randomly, e.g. using different strings for clock-names. >> >> This reminds me of another issue I ran into. I wanted to test adding the >> same platform device to the platform bus twice to confirm that the >> second device can't be added. That prints a warning, which makes >> kunit.py think that the test has failed because it printed a warning. Is >> there some way to avoid that? I want something like >> >> KUNIT_EXPECT_WARNING(test, <call some function>) >> >> so I can test error cases. DT unittests already have a similar concept. A test can report that a kernel warning (or any other specific text) either (1) must occur for the test to pass or (2) must _not_ occur for the test to pass. The check for the kernel warning is done by the test output parsing program scripts/dtc/of_unittest_expect. The reporting by a test of an expected error in drivers/of/unittest.c is done by EXPECT_BEGIN() and EXPECT_END(). These have been in unittest for a long time. The reporting by a test of a not expected to occur error is done by EXPECT_NOT_BEGIN() and EXPECT_NOT_END(). These are added to unittest in linux 6.3-rc1. I discussed this concept in one of the early TAP / KTAP discussion threads and expect to start a discussion thread on this specific topic in the KTAP Specification V2 context. I expect the discussion to result in a different implementation than what DT unittests are using (bike shedding likely to ensue) but whatever is agreed to should be easy for DT to switch to. > > Hmm... I'd've thought that shouldn't be a problem: kunit.py should > ignore most messages during a test, unless it can't find a valid > result line. What does the raw KTAP output look like? (You can get it > from kunit.py by passing the --raw_output option). > > That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something > we've wanted for a while. I think that the KASAN folks have been > working on something similar using console tracepoints: > https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ > > Cheers, > -- David
On 3/13/23 11:02, Frank Rowand wrote: > On 3/11/23 00:42, David Gow wrote: >> On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: >>> >>> Quoting David Gow (2023-03-10 00:09:48) >>>> On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: >>>>> >>>>> >>>>> Hmm. I think you're suggesting that the unit test data be loaded >>>>> whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for >>>>> CONFIG_OF and skip if it isn't enabled? >>>>> >>>> >>>> More of the opposite: that we should have some way of supporting tests >>>> which might want to use a DTB other than the built-in one. Mostly for >>>> non-UML situations where an actual devicetree is needed to even boot >>>> far enough to get test output (so we wouldn't be able to override it >>>> with a compiled-in test one). >>> >>> Ok, got it. >>> >>>> >>>> I think moving to overlays probably will render this idea obsolete: >>>> but the thought was to give test code a way to check for the required >>>> devicetree nodes at runtime, and skip the test if they weren't found. >>>> That way, the failure mode for trying to boot this on something which >>>> required another device tree for, e.g., serial, would be "these tests >>>> are skipped because the wrong device tree is loaded", not "I get no >>>> output because serial isn't working". >>>> >>>> Again, though, it's only really needed for non-UML, and just loading >>>> overlays as needed should be much more sensible anyway. >>> >>> I still have one niggle here. Loading overlays requires >>> CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when >>> CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled >>> in each test, but I'm thinking it may be better to simply call >>> kunit_skip() from the overlay loading function if the config is >>> disabled. This way tests can simply call the overlay loading function >>> and we'll halt the test immediately if the config isn't enabled. >>> >> >> That sounds sensible, though there is a potential pitfall. If >> kunit_skip() is called directly from overlay code, might introduce a >> dependency on kunit.ko from the DT overlay, which we might not want. >> The solution there is either to have a kunit wrapper function (so the >> call is already in kunit.ko), or to have a hook to skip the current >> test (which probably makes sense to do anyway, but I think the wrapper >> is the better option). >> >> >>>> >>>>>> >>>>>> That being said, I do think that there's probably some sense in >>>>>> supporting the compiled-in DTB as well (it's definitely simpler than >>>>>> patching kunit.py to always pass the extra command-line option in, for >>>>>> example). >>>>>> But maybe it'd be nice to have the command-line option override the >>>>>> built-in one if present. >>>>> >>>>> Got it. I need to test loading another DTB on the commandline still, but >>>>> I think this won't be a problem. We'll load the unittest-data DTB even >>>>> with KUnit on UML, so assuming that works on UML right now it should be >>>>> unchanged by this series once I resend. >>>> >>>> Again, moving to overlays should render this mostly obsolete, no? Or >>>> am I misunderstanding how the overlay stuff will work? >>> >>> Right, overlays make it largely a moot issue. The way the OF unit tests >>> work today is by grafting a DTB onto the live tree. I'm reusing that >>> logic to graft a container node target for kunit tests to add their >>> overlays too. It will be clearer once I post v2. >>> >>>> >>>> One possible future advantage of being able to test with custom DTs at >>>> boot time would be for fuzzing (provide random DT properties, see what >>>> happens in the test). We've got some vague plans to support a way of >>>> passing custom data to tests to support this kind of case (though, if >>>> we're using overlays, maybe the test could just patch those if we >>>> wanted to do that). >>> >>> Ah ok. I can see someone making a fuzzer that modifies devicetree >>> properties randomly, e.g. using different strings for clock-names. >>> >>> This reminds me of another issue I ran into. I wanted to test adding the >>> same platform device to the platform bus twice to confirm that the >>> second device can't be added. That prints a warning, which makes >>> kunit.py think that the test has failed because it printed a warning. Is >>> there some way to avoid that? I want something like >>> >>> KUNIT_EXPECT_WARNING(test, <call some function>) >>> >>> so I can test error cases. > > DT unittests already have a similar concept. A test can report that a > kernel warning (or any other specific text) either (1) must occur for the > test to pass or (2) must _not_ occur for the test to pass. The check > for the kernel warning is done by the test output parsing program > scripts/dtc/of_unittest_expect. > > The reporting by a test of an expected error in drivers/of/unittest.c > is done by EXPECT_BEGIN() and EXPECT_END(). These have been in > unittest for a long time. > > The reporting by a test of a not expected to occur error is done > by EXPECT_NOT_BEGIN() and EXPECT_NOT_END(). These are added to > unittest in linux 6.3-rc1. > > I discussed this concept in one of the early TAP / KTAP discussion The link to the early KTAP discussion on this concept is: https://lore.kernel.org/all/d38bf9f9-8a39-87a6-8ce7-d37e4a641675@gmail.com/T/#u > threads and expect to start a discussion thread on this specific > topic in the KTAP Specification V2 context. I expect the discussion > to result in a different implementation than what DT unittests are > using (bike shedding likely to ensue) but whatever is agreed to > should be easy for DT to switch to. The link to the KTAP Specification Version 2 process and progress is: https://elinux.org/Test_Results_Format_Notes#KTAP_version_2 -Frank > >> >> Hmm... I'd've thought that shouldn't be a problem: kunit.py should >> ignore most messages during a test, unless it can't find a valid >> result line. What does the raw KTAP output look like? (You can get it >> from kunit.py by passing the --raw_output option). >> >> That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something >> we've wanted for a while. I think that the KASAN folks have been >> working on something similar using console tracepoints: >> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ >> >> Cheers, >> -- David >
On Tue, 14 Mar 2023 at 12:28, Frank Rowand <frowand.list@gmail.com> wrote: > > On 3/13/23 11:02, Frank Rowand wrote: > > On 3/11/23 00:42, David Gow wrote: > >> On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: > >>> > >>> Quoting David Gow (2023-03-10 00:09:48) > >>>> On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: > >>>>> > >>>>> > >>>>> Hmm. I think you're suggesting that the unit test data be loaded > >>>>> whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for > >>>>> CONFIG_OF and skip if it isn't enabled? > >>>>> > >>>> > >>>> More of the opposite: that we should have some way of supporting tests > >>>> which might want to use a DTB other than the built-in one. Mostly for > >>>> non-UML situations where an actual devicetree is needed to even boot > >>>> far enough to get test output (so we wouldn't be able to override it > >>>> with a compiled-in test one). > >>> > >>> Ok, got it. > >>> > >>>> > >>>> I think moving to overlays probably will render this idea obsolete: > >>>> but the thought was to give test code a way to check for the required > >>>> devicetree nodes at runtime, and skip the test if they weren't found. > >>>> That way, the failure mode for trying to boot this on something which > >>>> required another device tree for, e.g., serial, would be "these tests > >>>> are skipped because the wrong device tree is loaded", not "I get no > >>>> output because serial isn't working". > >>>> > >>>> Again, though, it's only really needed for non-UML, and just loading > >>>> overlays as needed should be much more sensible anyway. > >>> > >>> I still have one niggle here. Loading overlays requires > >>> CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when > >>> CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled > >>> in each test, but I'm thinking it may be better to simply call > >>> kunit_skip() from the overlay loading function if the config is > >>> disabled. This way tests can simply call the overlay loading function > >>> and we'll halt the test immediately if the config isn't enabled. > >>> > >> > >> That sounds sensible, though there is a potential pitfall. If > >> kunit_skip() is called directly from overlay code, might introduce a > >> dependency on kunit.ko from the DT overlay, which we might not want. > >> The solution there is either to have a kunit wrapper function (so the > >> call is already in kunit.ko), or to have a hook to skip the current > >> test (which probably makes sense to do anyway, but I think the wrapper > >> is the better option). > >> > >> > >>>> > >>>>>> > >>>>>> That being said, I do think that there's probably some sense in > >>>>>> supporting the compiled-in DTB as well (it's definitely simpler than > >>>>>> patching kunit.py to always pass the extra command-line option in, for > >>>>>> example). > >>>>>> But maybe it'd be nice to have the command-line option override the > >>>>>> built-in one if present. > >>>>> > >>>>> Got it. I need to test loading another DTB on the commandline still, but > >>>>> I think this won't be a problem. We'll load the unittest-data DTB even > >>>>> with KUnit on UML, so assuming that works on UML right now it should be > >>>>> unchanged by this series once I resend. > >>>> > >>>> Again, moving to overlays should render this mostly obsolete, no? Or > >>>> am I misunderstanding how the overlay stuff will work? > >>> > >>> Right, overlays make it largely a moot issue. The way the OF unit tests > >>> work today is by grafting a DTB onto the live tree. I'm reusing that > >>> logic to graft a container node target for kunit tests to add their > >>> overlays too. It will be clearer once I post v2. > >>> > >>>> > >>>> One possible future advantage of being able to test with custom DTs at > >>>> boot time would be for fuzzing (provide random DT properties, see what > >>>> happens in the test). We've got some vague plans to support a way of > >>>> passing custom data to tests to support this kind of case (though, if > >>>> we're using overlays, maybe the test could just patch those if we > >>>> wanted to do that). > >>> > >>> Ah ok. I can see someone making a fuzzer that modifies devicetree > >>> properties randomly, e.g. using different strings for clock-names. > >>> > >>> This reminds me of another issue I ran into. I wanted to test adding the > >>> same platform device to the platform bus twice to confirm that the > >>> second device can't be added. That prints a warning, which makes > >>> kunit.py think that the test has failed because it printed a warning. Is > >>> there some way to avoid that? I want something like > >>> > >>> KUNIT_EXPECT_WARNING(test, <call some function>) > >>> > >>> so I can test error cases. > > > > DT unittests already have a similar concept. A test can report that a > > kernel warning (or any other specific text) either (1) must occur for the > > test to pass or (2) must _not_ occur for the test to pass. The check > > for the kernel warning is done by the test output parsing program > > scripts/dtc/of_unittest_expect. > > > > The reporting by a test of an expected error in drivers/of/unittest.c > > is done by EXPECT_BEGIN() and EXPECT_END(). These have been in > > unittest for a long time. > > > > The reporting by a test of a not expected to occur error is done > > by EXPECT_NOT_BEGIN() and EXPECT_NOT_END(). These are added to > > unittest in linux 6.3-rc1. > > > > I discussed this concept in one of the early TAP / KTAP discussion > > The link to the early KTAP discussion on this concept is: > > https://lore.kernel.org/all/d38bf9f9-8a39-87a6-8ce7-d37e4a641675@gmail.com/T/#u > > Thanks -- I'd totally forgotten about that! I still personally would prefer a way of checking this from within the kernel, as if we're just printing out "EXPECT: " lines, then it's not possible to know if a test passes just from the raw results (and things like statistics can't be updated without a separate tool like kunit.py parsing the KTAP. Indeed, my personal preference is that this log-based way of doing expectations is probably best kept as a last resort. i.e., 1. Try to add a hook to the code which prints the message, which can then fail the test (or set a flag for the test to check later). This probably needs some better KUnit-side helpers to be truly ergonomic, but at least avoids too strict a dependency on the exact formatting of the log messages. 2. If that doesn't work, use console tracepoints or similar to implement an EXPECT_BEGIN() / EXPECT_END() or similar API entirely within the kernel. 3. Only if we can't come up with a working way of doing the former options, resort to adding "EXPECT:" lines and having a parser pick up on this. One of the downsides of doing "EXPECT" lines in KTAP is that it'll suddenly be much more dependent on the exact layout of the tests, as we'd need to be able to override a test result if an expectation fails (at least, to maintain the KUnit structure). And overriding a result which is already in the output seems really, really ugly. There's a patch to the KASAN tests to move from doing option 1 to option 2 above (in order to better support RCU, which didn't work with the hook): https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ > > threads and expect to start a discussion thread on this specific > > topic in the KTAP Specification V2 context. I expect the discussion > > to result in a different implementation than what DT unittests are > > using (bike shedding likely to ensue) but whatever is agreed to > > should be easy for DT to switch to. > > The link to the KTAP Specification Version 2 process and progress is: > > https://elinux.org/Test_Results_Format_Notes#KTAP_version_2 > Thanks! We've got a few more KTAP ideas to air, so will hopefully send those out soon! Cheers, -- David > > > >> > >> Hmm... I'd've thought that shouldn't be a problem: kunit.py should > >> ignore most messages during a test, unless it can't find a valid > >> result line. What does the raw KTAP output look like? (You can get it > >> from kunit.py by passing the --raw_output option). > >> > >> That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something > >> we've wanted for a while. I think that the KASAN folks have been > >> working on something similar using console tracepoints: > >> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ > >> > >> Cheers, > >> -- David > > >
On 3/15/23 02:04, David Gow wrote: > On Tue, 14 Mar 2023 at 12:28, Frank Rowand <frowand.list@gmail.com> wrote: >> >> On 3/13/23 11:02, Frank Rowand wrote: >>> On 3/11/23 00:42, David Gow wrote: >>>> On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: >>>>> >>>>> Quoting David Gow (2023-03-10 00:09:48) >>>>>> On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: >>>>>>> >>>>>>> >>>>>>> Hmm. I think you're suggesting that the unit test data be loaded >>>>>>> whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for >>>>>>> CONFIG_OF and skip if it isn't enabled? >>>>>>> >>>>>> >>>>>> More of the opposite: that we should have some way of supporting tests >>>>>> which might want to use a DTB other than the built-in one. Mostly for >>>>>> non-UML situations where an actual devicetree is needed to even boot >>>>>> far enough to get test output (so we wouldn't be able to override it >>>>>> with a compiled-in test one). >>>>> >>>>> Ok, got it. >>>>> >>>>>> >>>>>> I think moving to overlays probably will render this idea obsolete: >>>>>> but the thought was to give test code a way to check for the required >>>>>> devicetree nodes at runtime, and skip the test if they weren't found. >>>>>> That way, the failure mode for trying to boot this on something which >>>>>> required another device tree for, e.g., serial, would be "these tests >>>>>> are skipped because the wrong device tree is loaded", not "I get no >>>>>> output because serial isn't working". >>>>>> >>>>>> Again, though, it's only really needed for non-UML, and just loading >>>>>> overlays as needed should be much more sensible anyway. >>>>> >>>>> I still have one niggle here. Loading overlays requires >>>>> CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when >>>>> CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled >>>>> in each test, but I'm thinking it may be better to simply call >>>>> kunit_skip() from the overlay loading function if the config is >>>>> disabled. This way tests can simply call the overlay loading function >>>>> and we'll halt the test immediately if the config isn't enabled. >>>>> >>>> >>>> That sounds sensible, though there is a potential pitfall. If >>>> kunit_skip() is called directly from overlay code, might introduce a >>>> dependency on kunit.ko from the DT overlay, which we might not want. >>>> The solution there is either to have a kunit wrapper function (so the >>>> call is already in kunit.ko), or to have a hook to skip the current >>>> test (which probably makes sense to do anyway, but I think the wrapper >>>> is the better option). >>>> >>>> >>>>>> >>>>>>>> >>>>>>>> That being said, I do think that there's probably some sense in >>>>>>>> supporting the compiled-in DTB as well (it's definitely simpler than >>>>>>>> patching kunit.py to always pass the extra command-line option in, for >>>>>>>> example). >>>>>>>> But maybe it'd be nice to have the command-line option override the >>>>>>>> built-in one if present. >>>>>>> >>>>>>> Got it. I need to test loading another DTB on the commandline still, but >>>>>>> I think this won't be a problem. We'll load the unittest-data DTB even >>>>>>> with KUnit on UML, so assuming that works on UML right now it should be >>>>>>> unchanged by this series once I resend. >>>>>> >>>>>> Again, moving to overlays should render this mostly obsolete, no? Or >>>>>> am I misunderstanding how the overlay stuff will work? >>>>> >>>>> Right, overlays make it largely a moot issue. The way the OF unit tests >>>>> work today is by grafting a DTB onto the live tree. I'm reusing that >>>>> logic to graft a container node target for kunit tests to add their >>>>> overlays too. It will be clearer once I post v2. >>>>> >>>>>> >>>>>> One possible future advantage of being able to test with custom DTs at >>>>>> boot time would be for fuzzing (provide random DT properties, see what >>>>>> happens in the test). We've got some vague plans to support a way of >>>>>> passing custom data to tests to support this kind of case (though, if >>>>>> we're using overlays, maybe the test could just patch those if we >>>>>> wanted to do that). >>>>> >>>>> Ah ok. I can see someone making a fuzzer that modifies devicetree >>>>> properties randomly, e.g. using different strings for clock-names. >>>>> >>>>> This reminds me of another issue I ran into. I wanted to test adding the >>>>> same platform device to the platform bus twice to confirm that the >>>>> second device can't be added. That prints a warning, which makes >>>>> kunit.py think that the test has failed because it printed a warning. Is >>>>> there some way to avoid that? I want something like >>>>> >>>>> KUNIT_EXPECT_WARNING(test, <call some function>) >>>>> >>>>> so I can test error cases. >>> >>> DT unittests already have a similar concept. A test can report that a >>> kernel warning (or any other specific text) either (1) must occur for the >>> test to pass or (2) must _not_ occur for the test to pass. The check >>> for the kernel warning is done by the test output parsing program >>> scripts/dtc/of_unittest_expect. >>> >>> The reporting by a test of an expected error in drivers/of/unittest.c >>> is done by EXPECT_BEGIN() and EXPECT_END(). These have been in >>> unittest for a long time. >>> >>> The reporting by a test of a not expected to occur error is done >>> by EXPECT_NOT_BEGIN() and EXPECT_NOT_END(). These are added to >>> unittest in linux 6.3-rc1. >>> >>> I discussed this concept in one of the early TAP / KTAP discussion >> >> The link to the early KTAP discussion on this concept is: >> >> https://lore.kernel.org/all/d38bf9f9-8a39-87a6-8ce7-d37e4a641675@gmail.com/T/#u >> >> > > Thanks -- I'd totally forgotten about that! > > I still personally would prefer a way of checking this from within the > kernel, as if we're just printing out "EXPECT: " lines, then it's not > possible to know if a test passes just from the raw results (and > things like statistics can't be updated without a separate tool like > kunit.py parsing the KTAP. Yes, I totally agree with that. If there is a reasonable way to implement. But in the DT unittest world, I have not found a reasonable way. Adding hooks is suggested below, but for DT unittest _I_ (opinion) do not find that reasonable. I voice no vote for kunit - that decision is up to the kunit crowd. > > Indeed, my personal preference is that this log-based way of doing > expectations is probably best kept as a last resort. i.e., > 1. Try to add a hook to the code which prints the message, which can > then fail the test (or set a flag for the test to check later). This > probably needs some better KUnit-side helpers to be truly ergonomic, > but at least avoids too strict a dependency on the exact formatting of > the log messages. I'm not a fan of hooks. I see them as a maintenance burden, dependent upon the source version of the object being tested, yet another thing that can go wrong, and adds complexity to creating a test environment and running the test. Again, this just a personal opinion, and I'm not voting for or against this for kunit. > 2. If that doesn't work, use console tracepoints or similar to > implement an EXPECT_BEGIN() / EXPECT_END() or similar API entirely > within the kernel. Isn't this just another hook? So same opinion. > 3. Only if we can't come up with a working way of doing the former > options, resort to adding "EXPECT:" lines and having a parser pick up > on this. Again, don't let my opinion affect the voting between 1, 2, 3, or other for kunit. > > One of the downsides of doing "EXPECT" lines in KTAP is that it'll > suddenly be much more dependent on the exact layout of the tests, as > we'd need to be able to override a test result if an expectation fails > (at least, to maintain the KUnit structure). And overriding a result > which is already in the output seems really, really ugly. I don't understand "dependent on the exact layout of the tests". If you are saying that the test result parser has to figure out which test result to override, that has not been an issue in the cases that I use EXPECTs in DT unittest. The EXPECT begin and EXPECT end have always immediately surrounded a single test, so when the parser processes the EXPECT end, only the most recent test result could be overridden. This has worked because the kernel warning and error messages have been from kernel action that happens synchronously with the test. If the test prods the kernel in a way that results in the kernel performing an asynchronous activity (eg in another thread), then it becomes more complex to structure the EXPECT end -- I would imagine that the test would have to block on the asynchronous activity just before reporting the normal KTAP status result for the test (and the EXPECT end would normally be just after reporting the KTAP status result for the test). I agree with overriding being ugly. For the DT unittest results parser, the EXPECT summary results are reported separately from the individual test summary results. The parser also flags the EXPECT failure in line with the normal individual test result lines. I see both parsing results as valid, and as a policy choice for each test parser. > > There's a patch to the KASAN tests to move from doing option 1 to > option 2 above (in order to better support RCU, which didn't work with > the hook): > https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ > > >>> threads and expect to start a discussion thread on this specific >>> topic in the KTAP Specification V2 context. I expect the discussion >>> to result in a different implementation than what DT unittests are >>> using (bike shedding likely to ensue) but whatever is agreed to >>> should be easy for DT to switch to. >> >> The link to the KTAP Specification Version 2 process and progress is: >> >> https://elinux.org/Test_Results_Format_Notes#KTAP_version_2 >> > > Thanks! We've got a few more KTAP ideas to air, so will hopefully send > those out soon! Glad to hear, I'm hoping that process starts progressing a bit. -Frank > > Cheers, > -- David > >>> >>>> >>>> Hmm... I'd've thought that shouldn't be a problem: kunit.py should >>>> ignore most messages during a test, unless it can't find a valid >>>> result line. What does the raw KTAP output look like? (You can get it >>>> from kunit.py by passing the --raw_output option). >>>> >>>> That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something >>>> we've wanted for a while. I think that the KASAN folks have been >>>> working on something similar using console tracepoints: >>>> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ >>>> >>>> Cheers, >>>> -- David >>> >>
On 3/15/23 16:35, Frank Rowand wrote: > On 3/15/23 02:04, David Gow wrote: >> On Tue, 14 Mar 2023 at 12:28, Frank Rowand <frowand.list@gmail.com> wrote: >>> >>> On 3/13/23 11:02, Frank Rowand wrote: >>>> On 3/11/23 00:42, David Gow wrote: >>>>> On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: >>>>>> >>>>>> Quoting David Gow (2023-03-10 00:09:48) >>>>>>> On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hmm. I think you're suggesting that the unit test data be loaded >>>>>>>> whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for >>>>>>>> CONFIG_OF and skip if it isn't enabled? >>>>>>>> >>>>>>> >>>>>>> More of the opposite: that we should have some way of supporting tests >>>>>>> which might want to use a DTB other than the built-in one. Mostly for >>>>>>> non-UML situations where an actual devicetree is needed to even boot >>>>>>> far enough to get test output (so we wouldn't be able to override it >>>>>>> with a compiled-in test one). >>>>>> >>>>>> Ok, got it. >>>>>> >>>>>>> >>>>>>> I think moving to overlays probably will render this idea obsolete: >>>>>>> but the thought was to give test code a way to check for the required >>>>>>> devicetree nodes at runtime, and skip the test if they weren't found. >>>>>>> That way, the failure mode for trying to boot this on something which >>>>>>> required another device tree for, e.g., serial, would be "these tests >>>>>>> are skipped because the wrong device tree is loaded", not "I get no >>>>>>> output because serial isn't working". >>>>>>> >>>>>>> Again, though, it's only really needed for non-UML, and just loading >>>>>>> overlays as needed should be much more sensible anyway. >>>>>> >>>>>> I still have one niggle here. Loading overlays requires >>>>>> CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when >>>>>> CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled >>>>>> in each test, but I'm thinking it may be better to simply call >>>>>> kunit_skip() from the overlay loading function if the config is >>>>>> disabled. This way tests can simply call the overlay loading function >>>>>> and we'll halt the test immediately if the config isn't enabled. >>>>>> >>>>> >>>>> That sounds sensible, though there is a potential pitfall. If >>>>> kunit_skip() is called directly from overlay code, might introduce a >>>>> dependency on kunit.ko from the DT overlay, which we might not want. >>>>> The solution there is either to have a kunit wrapper function (so the >>>>> call is already in kunit.ko), or to have a hook to skip the current >>>>> test (which probably makes sense to do anyway, but I think the wrapper >>>>> is the better option). >>>>> >>>>> >>>>>>> >>>>>>>>> >>>>>>>>> That being said, I do think that there's probably some sense in >>>>>>>>> supporting the compiled-in DTB as well (it's definitely simpler than >>>>>>>>> patching kunit.py to always pass the extra command-line option in, for >>>>>>>>> example). >>>>>>>>> But maybe it'd be nice to have the command-line option override the >>>>>>>>> built-in one if present. >>>>>>>> >>>>>>>> Got it. I need to test loading another DTB on the commandline still, but >>>>>>>> I think this won't be a problem. We'll load the unittest-data DTB even >>>>>>>> with KUnit on UML, so assuming that works on UML right now it should be >>>>>>>> unchanged by this series once I resend. >>>>>>> >>>>>>> Again, moving to overlays should render this mostly obsolete, no? Or >>>>>>> am I misunderstanding how the overlay stuff will work? >>>>>> >>>>>> Right, overlays make it largely a moot issue. The way the OF unit tests >>>>>> work today is by grafting a DTB onto the live tree. I'm reusing that >>>>>> logic to graft a container node target for kunit tests to add their >>>>>> overlays too. It will be clearer once I post v2. >>>>>> >>>>>>> >>>>>>> One possible future advantage of being able to test with custom DTs at >>>>>>> boot time would be for fuzzing (provide random DT properties, see what >>>>>>> happens in the test). We've got some vague plans to support a way of >>>>>>> passing custom data to tests to support this kind of case (though, if >>>>>>> we're using overlays, maybe the test could just patch those if we >>>>>>> wanted to do that). >>>>>> >>>>>> Ah ok. I can see someone making a fuzzer that modifies devicetree >>>>>> properties randomly, e.g. using different strings for clock-names. >>>>>> >>>>>> This reminds me of another issue I ran into. I wanted to test adding the >>>>>> same platform device to the platform bus twice to confirm that the >>>>>> second device can't be added. That prints a warning, which makes >>>>>> kunit.py think that the test has failed because it printed a warning. Is >>>>>> there some way to avoid that? I want something like >>>>>> >>>>>> KUNIT_EXPECT_WARNING(test, <call some function>) >>>>>> >>>>>> so I can test error cases. >>>> >>>> DT unittests already have a similar concept. A test can report that a >>>> kernel warning (or any other specific text) either (1) must occur for the >>>> test to pass or (2) must _not_ occur for the test to pass. The check >>>> for the kernel warning is done by the test output parsing program >>>> scripts/dtc/of_unittest_expect. >>>> >>>> The reporting by a test of an expected error in drivers/of/unittest.c >>>> is done by EXPECT_BEGIN() and EXPECT_END(). These have been in >>>> unittest for a long time. >>>> >>>> The reporting by a test of a not expected to occur error is done >>>> by EXPECT_NOT_BEGIN() and EXPECT_NOT_END(). These are added to >>>> unittest in linux 6.3-rc1. >>>> >>>> I discussed this concept in one of the early TAP / KTAP discussion >>> >>> The link to the early KTAP discussion on this concept is: >>> >>> https://lore.kernel.org/all/d38bf9f9-8a39-87a6-8ce7-d37e4a641675@gmail.com/T/#u >>> >>> >> >> Thanks -- I'd totally forgotten about that! >> > >> I still personally would prefer a way of checking this from within the >> kernel, as if we're just printing out "EXPECT: " lines, then it's not >> possible to know if a test passes just from the raw results (and >> things like statistics can't be updated without a separate tool like >> kunit.py parsing the KTAP. > > Yes, I totally agree with that. If there is a reasonable way to > implement. But in the DT unittest world, I have not found a > reasonable way. Adding hooks is suggested below, but for DT > unittest _I_ (opinion) do not find that reasonable. I voice no > vote for kunit - that decision is up to the kunit crowd. > >> >> Indeed, my personal preference is that this log-based way of doing >> expectations is probably best kept as a last resort. i.e., >> 1. Try to add a hook to the code which prints the message, which can >> then fail the test (or set a flag for the test to check later). This >> probably needs some better KUnit-side helpers to be truly ergonomic, >> but at least avoids too strict a dependency on the exact formatting of >> the log messages. > > I'm not a fan of hooks. I see them as a maintenance burden, dependent > upon the source version of the object being tested, yet another > thing that can go wrong, and adds complexity to creating a test > environment and running the test. Again, this just a personal > opinion, and I'm not voting for or against this for kunit. > >> 2. If that doesn't work, use console tracepoints or similar to >> implement an EXPECT_BEGIN() / EXPECT_END() or similar API entirely >> within the kernel. > > Isn't this just another hook? So same opinion. > >> 3. Only if we can't come up with a working way of doing the former >> options, resort to adding "EXPECT:" lines and having a parser pick up >> on this. Adding one more thought here so I don't forget it before the topic picks up again in the KTAP version 2 world... The test parser could generate an artificial subtest test case status line or normal test case status line to report the result of the EXPECT. This also is ugly because it is creating a new requirement on the parser vs the expectations in the KTAP plan line (the plan line could include the EXPECT in the number of tests count, but then the raw KTAP test output would be missing the artificial EXPECT test result). No need to hash out details here, just a thought... -Frank > > Again, don't let my opinion affect the voting between 1, 2, 3, or other > for kunit. > >> >> One of the downsides of doing "EXPECT" lines in KTAP is that it'll >> suddenly be much more dependent on the exact layout of the tests, as >> we'd need to be able to override a test result if an expectation fails >> (at least, to maintain the KUnit structure). And overriding a result >> which is already in the output seems really, really ugly. > > I don't understand "dependent on the exact layout of the tests". > If you are saying that the test result parser has to figure out > which test result to override, that has not been an issue in > the cases that I use EXPECTs in DT unittest. The EXPECT begin and > EXPECT end have always immediately surrounded a single test, so > when the parser processes the EXPECT end, only the most recent > test result could be overridden. This has worked because the > kernel warning and error messages have been from kernel action > that happens synchronously with the test. If the test prods the > kernel in a way that results in the kernel performing an > asynchronous activity (eg in another thread), then it becomes > more complex to structure the EXPECT end -- I would imagine that the > test would have to block on the asynchronous activity just before > reporting the normal KTAP status result for the test (and the > EXPECT end would normally be just after reporting the KTAP > status result for the test). > > I agree with overriding being ugly. For the DT unittest results > parser, the EXPECT summary results are reported separately from > the individual test summary results. The parser also flags the > EXPECT failure in line with the normal individual test result > lines. > > I see both parsing results as valid, and as a policy choice for > each test parser. > >> >> There's a patch to the KASAN tests to move from doing option 1 to >> option 2 above (in order to better support RCU, which didn't work with >> the hook): >> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ >> >> >>>> threads and expect to start a discussion thread on this specific >>>> topic in the KTAP Specification V2 context. I expect the discussion >>>> to result in a different implementation than what DT unittests are >>>> using (bike shedding likely to ensue) but whatever is agreed to >>>> should be easy for DT to switch to. >>> >>> The link to the KTAP Specification Version 2 process and progress is: >>> >>> https://elinux.org/Test_Results_Format_Notes#KTAP_version_2 >>> >> >> Thanks! We've got a few more KTAP ideas to air, so will hopefully send >> those out soon! > > Glad to hear, I'm hoping that process starts progressing a bit. > > -Frank > >> >> Cheers, >> -- David >> >>>> >>>>> >>>>> Hmm... I'd've thought that shouldn't be a problem: kunit.py should >>>>> ignore most messages during a test, unless it can't find a valid >>>>> result line. What does the raw KTAP output look like? (You can get it >>>>> from kunit.py by passing the --raw_output option). >>>>> >>>>> That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something >>>>> we've wanted for a while. I think that the KASAN folks have been >>>>> working on something similar using console tracepoints: >>>>> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ >>>>> >>>>> Cheers, >>>>> -- David >>>> >>> >
On Thu, 16 Mar 2023 at 08:45, Frank Rowand <frowand.list@gmail.com> wrote: > > On 3/15/23 16:35, Frank Rowand wrote: > > On 3/15/23 02:04, David Gow wrote: > >> On Tue, 14 Mar 2023 at 12:28, Frank Rowand <frowand.list@gmail.com> wrote: > >>> > >>> On 3/13/23 11:02, Frank Rowand wrote: > >>>> On 3/11/23 00:42, David Gow wrote: > >>>>> On Sat, 11 Mar 2023 at 07:34, Stephen Boyd <sboyd@kernel.org> wrote: > >>>>>> > >>>>>> Quoting David Gow (2023-03-10 00:09:48) > >>>>>>> On Fri, 10 Mar 2023 at 07:19, Stephen Boyd <sboyd@kernel.org> wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> Hmm. I think you're suggesting that the unit test data be loaded > >>>>>>>> whenever CONFIG_OF=y and CONFIG_KUNIT=y. Then tests can check for > >>>>>>>> CONFIG_OF and skip if it isn't enabled? > >>>>>>>> > >>>>>>> > >>>>>>> More of the opposite: that we should have some way of supporting tests > >>>>>>> which might want to use a DTB other than the built-in one. Mostly for > >>>>>>> non-UML situations where an actual devicetree is needed to even boot > >>>>>>> far enough to get test output (so we wouldn't be able to override it > >>>>>>> with a compiled-in test one). > >>>>>> > >>>>>> Ok, got it. > >>>>>> > >>>>>>> > >>>>>>> I think moving to overlays probably will render this idea obsolete: > >>>>>>> but the thought was to give test code a way to check for the required > >>>>>>> devicetree nodes at runtime, and skip the test if they weren't found. > >>>>>>> That way, the failure mode for trying to boot this on something which > >>>>>>> required another device tree for, e.g., serial, would be "these tests > >>>>>>> are skipped because the wrong device tree is loaded", not "I get no > >>>>>>> output because serial isn't working". > >>>>>>> > >>>>>>> Again, though, it's only really needed for non-UML, and just loading > >>>>>>> overlays as needed should be much more sensible anyway. > >>>>>> > >>>>>> I still have one niggle here. Loading overlays requires > >>>>>> CONFIG_OF_OVERLAY, and the overlay loading API returns -ENOTSUPP when > >>>>>> CONFIG_OF_OVERLAY=n. For now I'm checking for the config being enabled > >>>>>> in each test, but I'm thinking it may be better to simply call > >>>>>> kunit_skip() from the overlay loading function if the config is > >>>>>> disabled. This way tests can simply call the overlay loading function > >>>>>> and we'll halt the test immediately if the config isn't enabled. > >>>>>> > >>>>> > >>>>> That sounds sensible, though there is a potential pitfall. If > >>>>> kunit_skip() is called directly from overlay code, might introduce a > >>>>> dependency on kunit.ko from the DT overlay, which we might not want. > >>>>> The solution there is either to have a kunit wrapper function (so the > >>>>> call is already in kunit.ko), or to have a hook to skip the current > >>>>> test (which probably makes sense to do anyway, but I think the wrapper > >>>>> is the better option). > >>>>> > >>>>> > >>>>>>> > >>>>>>>>> > >>>>>>>>> That being said, I do think that there's probably some sense in > >>>>>>>>> supporting the compiled-in DTB as well (it's definitely simpler than > >>>>>>>>> patching kunit.py to always pass the extra command-line option in, for > >>>>>>>>> example). > >>>>>>>>> But maybe it'd be nice to have the command-line option override the > >>>>>>>>> built-in one if present. > >>>>>>>> > >>>>>>>> Got it. I need to test loading another DTB on the commandline still, but > >>>>>>>> I think this won't be a problem. We'll load the unittest-data DTB even > >>>>>>>> with KUnit on UML, so assuming that works on UML right now it should be > >>>>>>>> unchanged by this series once I resend. > >>>>>>> > >>>>>>> Again, moving to overlays should render this mostly obsolete, no? Or > >>>>>>> am I misunderstanding how the overlay stuff will work? > >>>>>> > >>>>>> Right, overlays make it largely a moot issue. The way the OF unit tests > >>>>>> work today is by grafting a DTB onto the live tree. I'm reusing that > >>>>>> logic to graft a container node target for kunit tests to add their > >>>>>> overlays too. It will be clearer once I post v2. > >>>>>> > >>>>>>> > >>>>>>> One possible future advantage of being able to test with custom DTs at > >>>>>>> boot time would be for fuzzing (provide random DT properties, see what > >>>>>>> happens in the test). We've got some vague plans to support a way of > >>>>>>> passing custom data to tests to support this kind of case (though, if > >>>>>>> we're using overlays, maybe the test could just patch those if we > >>>>>>> wanted to do that). > >>>>>> > >>>>>> Ah ok. I can see someone making a fuzzer that modifies devicetree > >>>>>> properties randomly, e.g. using different strings for clock-names. > >>>>>> > >>>>>> This reminds me of another issue I ran into. I wanted to test adding the > >>>>>> same platform device to the platform bus twice to confirm that the > >>>>>> second device can't be added. That prints a warning, which makes > >>>>>> kunit.py think that the test has failed because it printed a warning. Is > >>>>>> there some way to avoid that? I want something like > >>>>>> > >>>>>> KUNIT_EXPECT_WARNING(test, <call some function>) > >>>>>> > >>>>>> so I can test error cases. > >>>> > >>>> DT unittests already have a similar concept. A test can report that a > >>>> kernel warning (or any other specific text) either (1) must occur for the > >>>> test to pass or (2) must _not_ occur for the test to pass. The check > >>>> for the kernel warning is done by the test output parsing program > >>>> scripts/dtc/of_unittest_expect. > >>>> > >>>> The reporting by a test of an expected error in drivers/of/unittest.c > >>>> is done by EXPECT_BEGIN() and EXPECT_END(). These have been in > >>>> unittest for a long time. > >>>> > >>>> The reporting by a test of a not expected to occur error is done > >>>> by EXPECT_NOT_BEGIN() and EXPECT_NOT_END(). These are added to > >>>> unittest in linux 6.3-rc1. > >>>> > >>>> I discussed this concept in one of the early TAP / KTAP discussion > >>> > >>> The link to the early KTAP discussion on this concept is: > >>> > >>> https://lore.kernel.org/all/d38bf9f9-8a39-87a6-8ce7-d37e4a641675@gmail.com/T/#u > >>> > >>> > >> > >> Thanks -- I'd totally forgotten about that! > >> > > > >> I still personally would prefer a way of checking this from within the > >> kernel, as if we're just printing out "EXPECT: " lines, then it's not > >> possible to know if a test passes just from the raw results (and > >> things like statistics can't be updated without a separate tool like > >> kunit.py parsing the KTAP. > > > > Yes, I totally agree with that. If there is a reasonable way to > > implement. But in the DT unittest world, I have not found a > > reasonable way. Adding hooks is suggested below, but for DT > > unittest _I_ (opinion) do not find that reasonable. I voice no > > vote for kunit - that decision is up to the kunit crowd. > > > >> > >> Indeed, my personal preference is that this log-based way of doing > >> expectations is probably best kept as a last resort. i.e., > >> 1. Try to add a hook to the code which prints the message, which can > >> then fail the test (or set a flag for the test to check later). This > >> probably needs some better KUnit-side helpers to be truly ergonomic, > >> but at least avoids too strict a dependency on the exact formatting of > >> the log messages. > > > > I'm not a fan of hooks. I see them as a maintenance burden, dependent > > upon the source version of the object being tested, yet another > > thing that can go wrong, and adds complexity to creating a test > > environment and running the test. Again, this just a personal > > opinion, and I'm not voting for or against this for kunit. > > I definitely agree that they've got their problems, and aren't the right solution for every test. That being said, I think a few of those problems also apply to expecting individual error lines, which are basically doing the same thing, just less explicitly (though with the advantage of serving another real-world purpose). > >> 2. If that doesn't work, use console tracepoints or similar to > >> implement an EXPECT_BEGIN() / EXPECT_END() or similar API entirely > >> within the kernel. > > > > Isn't this just another hook? So same opinion. > > I see what you mean. I guess the distinction I was trying to draw was between a specific hook, implemented as such explicitly on both sides, and a generic mechanism for console-message-based expectations. So the difference is that, once the EXPECT_BEGIN()/EXPECT_END() macros have been implemented this way, the uses of them should look the same. > >> 3. Only if we can't come up with a working way of doing the former > >> options, resort to adding "EXPECT:" lines and having a parser pick up > >> on this. > > Adding one more thought here so I don't forget it before the topic picks > up again in the KTAP version 2 world... > > The test parser could generate an artificial subtest test case status line > or normal test case status line to report the result of the EXPECT. This > also is ugly because it is creating a new requirement on the parser vs the > expectations in the KTAP plan line (the plan line could include the EXPECT > in the number of tests count, but then the raw KTAP test output would be > missing the artificial EXPECT test result). No need to hash out details > here, just a thought... > Yeah, I think if we have this done at the parser level, our options are either that or an override. For KUnit, I think the override makes much more sense, as logically, any expectation would be considered part of an existing test case. Maybe we could treat an override as a "subtest" of the current test, but injecting a test case anywhere else would go pretty seriously against the KUnit model. > -Frank > > > > > Again, don't let my opinion affect the voting between 1, 2, 3, or other > > for kunit. > > > >> > >> One of the downsides of doing "EXPECT" lines in KTAP is that it'll > >> suddenly be much more dependent on the exact layout of the tests, as > >> we'd need to be able to override a test result if an expectation fails > >> (at least, to maintain the KUnit structure). And overriding a result > >> which is already in the output seems really, really ugly. > > > > I don't understand "dependent on the exact layout of the tests". > > If you are saying that the test result parser has to figure out > > which test result to override, that has not been an issue in > > the cases that I use EXPECTs in DT unittest. The EXPECT begin and > > EXPECT end have always immediately surrounded a single test, so > > when the parser processes the EXPECT end, only the most recent > > test result could be overridden. This has worked because the > > kernel warning and error messages have been from kernel action > > that happens synchronously with the test. If the test prods the > > kernel in a way that results in the kernel performing an > > asynchronous activity (eg in another thread), then it becomes > > more complex to structure the EXPECT end -- I would imagine that the > > test would have to block on the asynchronous activity just before > > reporting the normal KTAP status result for the test (and the > > EXPECT end would normally be just after reporting the KTAP > > status result for the test). Okay: I agree 100% with you there. I think the difference I was thinking of is more whether expectations are considered part of an existing test (which may need overriding), or exist as their own separate test result (as you suggest in the follow-up email above). > > > > I agree with overriding being ugly. For the DT unittest results > > parser, the EXPECT summary results are reported separately from > > the individual test summary results. The parser also flags the > > EXPECT failure in line with the normal individual test result > > lines. > > > > I see both parsing results as valid, and as a policy choice for > > each test parser. > > I agree that each parser should have some leeway here, but do think we want to make sure the results have some sensible, standardised interpretation, so that we can use parsers interchangeably without getting totally inconsistent results. That's the big advantage of standardisation, after all. My philosophical objection to overriding is that it's really confusing to have an "ok" line in the results, indicating that a test has passed, when the test has in fact failed (because the expectation doesn't match). This gets worse when subtests are considered, and we have these misleading results bubbled up to reported overall "suite" results. I guess one solution is to have an extra layer of parsing, which takes raw kernel output, verifies the expectations, and then outputs "processed KTAP", with all of the final results resolved. But that's ugly in a way, too. > >> > >> There's a patch to the KASAN tests to move from doing option 1 to > >> option 2 above (in order to better support RCU, which didn't work with > >> the hook): > >> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ > >> > >> > >>>> threads and expect to start a discussion thread on this specific > >>>> topic in the KTAP Specification V2 context. I expect the discussion > >>>> to result in a different implementation than what DT unittests are > >>>> using (bike shedding likely to ensue) but whatever is agreed to > >>>> should be easy for DT to switch to. > >>> > >>> The link to the KTAP Specification Version 2 process and progress is: > >>> > >>> https://elinux.org/Test_Results_Format_Notes#KTAP_version_2 > >>> > >> > >> Thanks! We've got a few more KTAP ideas to air, so will hopefully send > >> those out soon! > > > > Glad to hear, I'm hoping that process starts progressing a bit. > > Yeah. I'll shift further discussion of this to a KTAP proposal: I don't want to derail this thread too much further. We'll keep looking at fully in-kernel ways of achieving similar things in the meantime. Cheers, -- David > > -Frank > > > >> > >> Cheers, > >> -- David > >> > >>>> > >>>>> > >>>>> Hmm... I'd've thought that shouldn't be a problem: kunit.py should > >>>>> ignore most messages during a test, unless it can't find a valid > >>>>> result line. What does the raw KTAP output look like? (You can get it > >>>>> from kunit.py by passing the --raw_output option). > >>>>> > >>>>> That being said, a KUNIT_EXPECT_LOG_MESSAGE() or similar is something > >>>>> we've wanted for a while. I think that the KASAN folks have been > >>>>> working on something similar using console tracepoints: > >>>>> https://lore.kernel.org/all/ebf96ea600050f00ed567e80505ae8f242633640.1666113393.git.andreyknvl@google.com/ > >>>>> > >>>>> Cheers, > >>>>> -- David > >>>> > >>> > > >
diff --git a/arch/um/kernel/dtb.c b/arch/um/kernel/dtb.c index 484141b06938..ee63951b12df 100644 --- a/arch/um/kernel/dtb.c +++ b/arch/um/kernel/dtb.c @@ -15,9 +15,32 @@ void uml_dtb_init(void) long long size; void *area; - area = uml_load_file(dtb, &size); - if (!area) - return; + if (IS_ENABLED(CONFIG_OF_KUNIT)) { + /* + * __dtbo_kunit_begin[] and __dtbo_kunit_end[] are magically + * created by cmd_dt_S_dtbo in scripts/Makefile.lib from the + * drivers/of/kunit/kunit.dtsi file. + */ + extern uint8_t __dtbo_kunit_begin[]; + extern uint8_t __dtbo_kunit_end[]; + + size = __dtbo_kunit_end - __dtbo_kunit_begin; + if (!size) { + pr_warn("%s: kunit testcases is empty\n", __func__); + return; + } + + /* creating copy */ + area = memblock_alloc(size, 8); + if (!area) + return; + + memcpy(area, __dtbo_kunit_begin, size); + } else { + area = uml_load_file(dtb, &size); + if (!area) + return; + } if (!early_init_dt_scan(area)) { pr_err("invalid DTB %s\n", dtb); diff --git a/drivers/of/Kconfig b/drivers/of/Kconfig index 80b5fd44ab1c..1f968b6a3dde 100644 --- a/drivers/of/Kconfig +++ b/drivers/of/Kconfig @@ -12,6 +12,20 @@ menuconfig OF if OF +choice + prompt "Devicetree Runtime Tests" + default OF_UNITTEST + +config OF_KUNIT + bool "Devicetree KUnit support" if KUNIT + depends on UML + select IRQ_DOMAIN + select OF_EARLY_FLATTREE + help + This option builds in KUnit test cases that rely on device tree infrastructure. + A fake Device Tree Blob (DTB) is loaded on the UML kernel running KUnit so that + KUnit tests can test device tree dependent code. + config OF_UNITTEST bool "Device Tree runtime unit tests" depends on !SPARC @@ -25,6 +39,18 @@ config OF_UNITTEST If unsure, say N here, but this option is safe to enable. +endchoice + +config OF_DTB_KUNIT_TEST + tristate "Devicetree KUnit DTB Test" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + This option builds unit tests for the "linux,kunit" DTB built into + the UML kernel image. + + If unsure, say N here, but this option is safe to enable. + config OF_ALL_DTBS bool "Build all Device Tree Blobs" depends on COMPILE_TEST diff --git a/drivers/of/Makefile b/drivers/of/Makefile index e0360a44306e..16eef3fdf60a 100644 --- a/drivers/of/Makefile +++ b/drivers/of/Makefile @@ -19,4 +19,5 @@ obj-y += kexec.o endif endif +obj-y += kunit/ obj-$(CONFIG_OF_UNITTEST) += unittest-data/ diff --git a/drivers/of/kunit/.kunitconfig b/drivers/of/kunit/.kunitconfig new file mode 100644 index 000000000000..1def0ad30d29 --- /dev/null +++ b/drivers/of/kunit/.kunitconfig @@ -0,0 +1,4 @@ +CONFIG_KUNIT=y +CONFIG_OF=y +CONFIG_OF_KUNIT=y +CONFIG_OF_DTB_KUNIT_TEST=y diff --git a/drivers/of/kunit/Makefile b/drivers/of/kunit/Makefile new file mode 100644 index 000000000000..ffe0447e1ac7 --- /dev/null +++ b/drivers/of/kunit/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_OF_KUNIT) += kunit.dtbo.o + +obj-$(CONFIG_OF_DTB_KUNIT_TEST) += uml_dtb_test.o diff --git a/drivers/of/kunit/kunit.dtsi b/drivers/of/kunit/kunit.dtsi new file mode 100644 index 000000000000..82f6c3e2b8d5 --- /dev/null +++ b/drivers/of/kunit/kunit.dtsi @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 + +/ { + model = "KUnit UML"; + compatible = "linux,kunit"; +}; + +/* Include testcase dtsi files below */ diff --git a/drivers/of/kunit/kunit.dtso b/drivers/of/kunit/kunit.dtso new file mode 100644 index 000000000000..50187e8d1422 --- /dev/null +++ b/drivers/of/kunit/kunit.dtso @@ -0,0 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 +/dts-v1/; + +#include "kunit.dtsi" diff --git a/drivers/of/kunit/uml_dtb_test.c b/drivers/of/kunit/uml_dtb_test.c new file mode 100644 index 000000000000..8966c9ebf51f --- /dev/null +++ b/drivers/of/kunit/uml_dtb_test.c @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KUnit tests for DTB loading on UML + */ +#include <linux/kconfig.h> +#include <linux/of.h> +#include <linux/of_fdt.h> + +#include <kunit/test.h> + +/* + * Test that of_machine_is_compatible() returns positive int when loaded DTB + * matches. + */ +static void uml_dtb_of_machine_compatible_test(struct kunit *test) +{ + KUNIT_EXPECT_GT(test, of_machine_is_compatible("linux,kunit"), 0); +} + +/* + * Test that of_flat_dt_get_machine_name() returns the expected 'model' from the + * loaded DTB. + */ +static void uml_dtb_of_flat_dt_get_machine_name_test(struct kunit *test) +{ + KUNIT_EXPECT_STREQ(test, of_flat_dt_get_machine_name(), "KUnit UML"); +} + +static struct kunit_case uml_dtb_test_cases[] = { + KUNIT_CASE(uml_dtb_of_machine_compatible_test), + KUNIT_CASE(uml_dtb_of_flat_dt_get_machine_name_test), + {} +}; + +static int uml_dtb_test_init(struct kunit *test) +{ + if (!IS_ENABLED(CONFIG_OF_KUNIT)) + kunit_skip(test, "requires CONFIG_OF_KUNIT"); + + return 0; +} + +/* + * Test suite to confirm DTB is loaded on UML. + */ +static struct kunit_suite uml_dtb_suite = { + .name = "uml_dtb", + .init = uml_dtb_test_init, + .test_cases = uml_dtb_test_cases, +}; + +kunit_test_suites( + ¨_dtb_suite, +); +MODULE_LICENSE("GPL");
To fully exercise common clk framework code in KUnit we need to associate 'struct device' pointers with 'struct device_node' pointers so that things like clk_get() can parse DT nodes for 'clocks' and so that clk providers can use DT to provide clks; the most common mode of operation for clk providers. Adding support to KUnit so that it loads a DTB is fairly simple after commit b31297f04e86 ("um: Add devicetree support"). We can simply pass a pre-compiled deviectree blob (DTB) on the kunit.py commandline and UML will load it. The problem is that tests won't know that the commandline has been modified, nor that a DTB has been loaded. Take a different approach so that tests can skip if a DTB hasn't been loaded. Reuse the Makefile logic from the OF unittests to build a DTB into the kernel. This DTB will be for the mythical machine "linux,kunit", i.e. the devicetree for the KUnit "board". In practice, it is a dtsi file that will gather includes for kunit tests that rely in part on a devicetree being loaded. The devicetree should only be loaded if CONFIG_OF_KUNIT=y. Make that a choice config parallel to the existing CONFIG_OF_UNITTEST so that only one devicetree can be loaded in the system at a time. Similarly, the kernel commandline option to load a DTB is ignored if CONFIG_OF_KUNIT is enabled so that only one DTB is loaded at a time. Add a simple unit test to confirm that the DTB loading worked. Future tests will add to the kunit.dtsi file to include their specific test nodes. Cc: Richard Weinberger <richard@nod.at> Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com> Cc: Johannes Berg <johannes@sipsolutions.net> Cc: Vincent Whitchurch <vincent.whitchurch@axis.com> Cc: Rob Herring <robh+dt@kernel.org> Cc: Frank Rowand <frowand.list@gmail.com> Signed-off-by: Stephen Boyd <sboyd@kernel.org> --- arch/um/kernel/dtb.c | 29 +++++++++++++++-- drivers/of/Kconfig | 26 ++++++++++++++++ drivers/of/Makefile | 1 + drivers/of/kunit/.kunitconfig | 4 +++ drivers/of/kunit/Makefile | 4 +++ drivers/of/kunit/kunit.dtsi | 8 +++++ drivers/of/kunit/kunit.dtso | 4 +++ drivers/of/kunit/uml_dtb_test.c | 55 +++++++++++++++++++++++++++++++++ 8 files changed, 128 insertions(+), 3 deletions(-) create mode 100644 drivers/of/kunit/.kunitconfig create mode 100644 drivers/of/kunit/Makefile create mode 100644 drivers/of/kunit/kunit.dtsi create mode 100644 drivers/of/kunit/kunit.dtso create mode 100644 drivers/of/kunit/uml_dtb_test.c