[v7,kunit-next,0/4] kunit: add debugfs representation to show results

Message

Alan Maguire March 13, 2020, 2:44 p.m. UTC

When kunit tests are run on native (i.e. non-UML) environments, the results
of test execution are often intermixed with dmesg output.  This patch
series attempts to solve this by providing a debugfs representation
of the results of the last test run, available as

/sys/kernel/debug/kunit/<testsuite>/results

Changes since v6:

- fixed regexp parsing in kunit_parser.py to ensure test results are read
  successfully with 4-space indentation (Brendan, patch 3)

Changes since v5:

- replaced undefined behaviour use of snprintf(buf, ..., buf) in
  kunit_log() with a function to append string to existing log
  (Frank, patch 1)
- added clarification on log size limitations to documentation
  (Frank, patch 4)

Changes since v4:

- added suite-level log expectations to kunit log test (Brendan, patch 2)
- added log expectations (of it being NULL) for case where
  CONFIG_KUNIT_DEBUGFS=n to kunit log test (patch 2)
- added patch 3 which replaces subtest tab indentation with 4 space
  indentation as per TAP 14 spec (Frank, patch 3)

Changes since v3:

- added CONFIG_KUNIT_DEBUGFS to support conditional compilation of debugfs
  representation, including string logging (Frank, patch 1)
- removed unneeded NULL check for test_case in
  kunit_suite_for_each_test_case() (Frank, patch 1)
- added kunit log test to verify logging multiple strings works
  (Frank, patch 2)
- rephrased description of results file (Frank, patch 3)

Changes since v2:

- updated kunit_status2str() to kunit_status_to_string() and made it
  static inline in include/kunit/test.h (Brendan)
- added log string to struct kunit_suite and kunit_case, with log
  pointer in struct kunit pointing at the case log.  This allows us
  to collect kunit_[err|info|warning]() messages at the same time
  as we printk() them.  This solves for the most part the sharing
  of log messages between test execution and debugfs since we
  just print the suite log (which contains the test suite preamble)
  and the individual test logs.  The only exception is the suite-level
  status, which we cannot store in the suite log as it would mean
  we'd print the suite and its status prior to the suite's results.
  (Brendan, patch 1)
- dropped debugfs-based kunit run patch for now so as not to cause
  problems with tests currently under development (Brendan)
- fixed doc issues with code block (Brendan, patch 3)

Changes since v1:
 - trimmed unneeded include files in lib/kunit/debugfs.c (Greg)
 - renamed global debugfs functions to be prefixed with kunit_ (Greg)
 - removed error checking for debugfs operations (Greg)

Alan Maguire (4):
  kunit: add debugfs /sys/kernel/debug/kunit/<suite>/results display
  kunit: add log test
  kunit: subtests should be indented 4 spaces according to TAP
  kunit: update documentation to describe debugfs representation

 Documentation/dev-tools/kunit/usage.rst |  14 +++
 include/kunit/test.h                    |  59 +++++++++++--
 lib/kunit/Kconfig                       |   8 ++
 lib/kunit/Makefile                      |   4 +
 lib/kunit/assert.c                      |  79 ++++++++---------
 lib/kunit/debugfs.c                     | 116 +++++++++++++++++++++++++
 lib/kunit/debugfs.h                     |  30 +++++++
 lib/kunit/kunit-test.c                  |  45 +++++++++-
 lib/kunit/test.c                        | 147 +++++++++++++++++++++++++-------
 tools/testing/kunit/kunit_parser.py     |  10 +--
 10 files changed, 426 insertions(+), 86 deletions(-)
 create mode 100644 lib/kunit/debugfs.c
 create mode 100644 lib/kunit/debugfs.h

Comments

Brendan Higgins March 14, 2020, 12:03 a.m. UTC | #1

On Fri, Mar 13, 2020 at 7:44 AM Alan Maguire <[e-mail]> wrote:
>
> When kunit tests are run on native (i.e. non-UML) environments, the results
> of test execution are often intermixed with dmesg output.  This patch
> series attempts to solve this by providing a debugfs representation
> of the results of the last test run, available as
>
> /sys/kernel/debug/kunit/<testsuite>/results
>
> Changes since v6:
>
> - fixed regexp parsing in kunit_parser.py to ensure test results are read
>   successfully with 4-space indentation (Brendan, patch 3)
>
> Changes since v5:
>
> - replaced undefined behaviour use of snprintf(buf, ..., buf) in
>   kunit_log() with a function to append string to existing log
>   (Frank, patch 1)
> - added clarification on log size limitations to documentation
>   (Frank, patch 4)
>
> Changes since v4:
>
> - added suite-level log expectations to kunit log test (Brendan, patch 2)
> - added log expectations (of it being NULL) for case where
>   CONFIG_KUNIT_DEBUGFS=n to kunit log test (patch 2)
> - added patch 3 which replaces subtest tab indentation with 4 space
>   indentation as per TAP 14 spec (Frank, patch 3)
>
> Changes since v3:
>
> - added CONFIG_KUNIT_DEBUGFS to support conditional compilation of debugfs
>   representation, including string logging (Frank, patch 1)
> - removed unneeded NULL check for test_case in
>   kunit_suite_for_each_test_case() (Frank, patch 1)
> - added kunit log test to verify logging multiple strings works
>   (Frank, patch 2)
> - rephrased description of results file (Frank, patch 3)
>
> Changes since v2:
>
> - updated kunit_status2str() to kunit_status_to_string() and made it
>   static inline in include/kunit/test.h (Brendan)
> - added log string to struct kunit_suite and kunit_case, with log
>   pointer in struct kunit pointing at the case log.  This allows us
>   to collect kunit_[err|info|warning]() messages at the same time
>   as we printk() them.  This solves for the most part the sharing
>   of log messages between test execution and debugfs since we
>   just print the suite log (which contains the test suite preamble)
>   and the individual test logs.  The only exception is the suite-level
>   status, which we cannot store in the suite log as it would mean
>   we'd print the suite and its status prior to the suite's results.
>   (Brendan, patch 1)
> - dropped debugfs-based kunit run patch for now so as not to cause
>   problems with tests currently under development (Brendan)
> - fixed doc issues with code block (Brendan, patch 3)
>
> Changes since v1:
>  - trimmed unneeded include files in lib/kunit/debugfs.c (Greg)
>  - renamed global debugfs functions to be prefixed with kunit_ (Greg)
>  - removed error checking for debugfs operations (Greg)

Everything looks good to me. I tried running kunit_tool and everything
looked good there well.