[RFC,v2,00/10] MAINTAINERS: Introduce V: entry for tests

Message

Nikolai Kondrashov Dec. 5, 2023, 6:02 p.m. UTC

Alright, here's a second version, attempting to address as many concerns as
possible. It's likely I've missed something, though.

Changes from v1:

* Make scripts/get_maintainer.pl survive querying missing files, giving a
  warning instead. This is necessary to enable scripts/checkpatch.pl to query
  MAINTAINERS about files being deleted.
* Start with the minimal change just documenting the V: entry, which accepts
  test commands directly, and tweaking the tools to deal with that.
* However, require the commands accept the -h/--help option so that users have
  an easier time getting *some* help. The run_checks.py missing that is the
  reason why the patch proposing it for kunit subsystem is marked "DONOTMERGE"
  in this version. We can drop that requirement, or soften the language, if
  there's opposition.
* Have a *separate* patch documenting 'Tested-with:' as the next (early)
  change. Mention that you can add a '#' followed by a results URL, on the
  end. Adjust the V: docs/checks to exclude '#'.
* Have a *separate* patch making scripts/checkpatch.pl propose the execution
  of the test suite defined in MAINTAINERS whenever the corresponding
  subsystem is changed.
* However, use 'CHECK', instead of 'WARNING', to allow submitters specify the
  exact (and potentially slightly different) command they used, and not have
  checkpatch.pl complain too loudly that they didn't run the (exact
  MAINTAINERS-specified) command. This unfortunately means that unless you use
  --strict, you won't see the message. We'll try to address that in a new
  change at the end.
* Have a *separate* patch introducing the test catalog and accepting
  references to that everywhere, with a special syntax to distinguish them
  from verbatim/direct commands. The syntax is prepending the test name with a
  '*' (just like C pointer dereference). Make checkpatch.pl handle that.
* Drop the recommendation to have the "Docs" and "Sources" fields in test
  descriptions, as the description text should focus on giving a good
  introduction and not prompt the user to go somewhere else immediately. They
  both can be referenced in the text where and how is appropriate.
* Generally keep the previous changes adding V: entries and test suite docs,
  and try to accommodate all the requests, but refine the "Summary" fields to
  fit the checkpatch.pl messages better.
* Have a separate patch cataloguing the complete kunit suite.
* Finally, add a patch introducing the "proposal strength" keywords
  (SUGGESTED/RECOMMENDED/REQUIRED) to the syntax of V: entries, which directly
  affect which level of checkpatch.pl message missing 'Tested-with:' tags
  would generate: CHECK/WARNING/ERROR respectively. This allows subsystems to
  disable checkpatch.pl WARNINGS/ERRORS, and keep their test proposals
  inobtrusive, if they so wish. E.g. if they expect people to change their
  commands often. At the same time allow stricter workflows for subsystems
  with more uniform testing. Or e.g. for subsystems which expect the tests to
  explain their parameters in their output, and the submitters to upload and
  link their results in their 'Tested-with:' tags.

That seems to be all, but I'm sure I forgot something :D

Anyway, send me more corrections and I'll try to address them, but it's likely
going to happen next year only.

Nick
---
Nikolai Kondrashov (9):
      get_maintainer: Survive querying missing files
      MAINTAINERS: Introduce V: entry for tests
      MAINTAINERS: Propose kunit core tests for framework changes
      docs: submitting-patches: Introduce Tested-with:
      checkpatch: Propose tests to execute
      MAINTAINERS: Support referencing test docs in V:
      MAINTAINERS: Propose kvm-xfstests smoke for ext4
      docs: tests: Document kunit in general
      MAINTAINERS: Add proposal strength to V: entries

Mark Brown (1):
      MAINTAINERS: Propose kunit tests for regmap

 Documentation/process/index.rst              |   1 +
 Documentation/process/submitting-patches.rst |  46 +++++++
 Documentation/process/tests.rst              |  96 +++++++++++++++
 MAINTAINERS                                  |  17 +++
 scripts/checkpatch.pl                        | 174 ++++++++++++++++++++++++++-
 scripts/get_maintainer.pl                    |  23 +++-
 scripts/parse-maintainers.pl                 |   3 +-
 7 files changed, 355 insertions(+), 5 deletions(-)
---

Comments

Nikolai Kondrashov Jan. 8, 2024, 10:42 a.m. UTC | #1

On 12/5/23 20:02, Nikolai Kondrashov wrote:
> Alright, here's a second version, attempting to address as many concerns as
> possible. It's likely I've missed something, though.

Happy New Year, everyone! Hope your holidays went smoothly and peacefully.

Is anyone still interested in this effort? Any more feedback you have in mind?
Something to change?

Thanks!
Nick