[v2,00/39] Implement kernel-doc in Python

Message

Mauro Carvalho Chehab Feb. 24, 2025, 9:08 a.m. UTC

Hi Jon,

This changeset contains the kernel-doc.py script to replace the verable
kernel-doc originally written in Perl. It replaces the first version and the
second series I sent on the top of it.

I tried to stay as close as possible of the original Perl implementation
on the first patch introducing kernel-doc.py, as it helps to double check
if each function was  properly translated to Python.  This have been 
helpful debugging troubles that happened during the conversion.

I worked hard to make it bug-compatible with the original one. Still, its
output has a couple of differences from the original one:

- The tab expansion works better with the Python script. With that, some
  outputs that contain tabs at kernel-doc markups are now different;

- The new script  works better stripping blank lines. So, there are a couple
  of empty new lines that are now stripped with this version;

- There is a buggy logic at kernel-doc to strip empty description and
  return sections. I was not able to replicate the exact behavior. So, I ended
  adding an extra logic to strip empty sections with a different algorithm.

Yet, on my tests, the results are compatible with the venerable script
output for all .. kernel-doc tags found in Documentation/. I double-checked
this by adding support to output the kernel-doc commands when V=1, and
then I ran a diff between kernel-doc.pl and kernel-doc.py for the same
command lines.

On version 2, kerneldoc.py Sphinx extension now uses the Python classes
if KERNELDOC  environment var ends with kernel-doc.py. It  keeps a cache
of previously-parsed files to avoid performance penalties when the same
file is added on multiple places.

This series contains:

- 4 patches fixing some kernel-doc issues. One of them is for media, but
   I prefer to have this merged via your tree, as it suppresses a warning
   that happens after the changes;

- 2 cleanup patches for Perl kernel-doc;

- 2 patches renaming kernel-doc to kernel-doc.pl and adding a symlink.
  I opted to have the symlink in separate to make easier to review, but
  feel free to merge them on a single patch if you want.

The remaining patches add the new script, converts it into a library and
addresses lots of minor compatibility issues. The last patch changes
Sphinx extension to directly use the Python classes.

The only patch that doesn't belong to this series is a patch dropping
kernel-doc.py. I opted to keep it for now, as it can help to better
test the new tools.

With such changes, if one wants to build docs with the old script,
all it is needed is to use KERNELDOC parameter, e.g.:

	$ make KERNELDOC=scripts/kernel-doc.pl htmldocs

Mauro Carvalho Chehab (39):
  include/asm-generic/io.h: fix kerneldoc markup
  drivers: media: intel-ipu3.h: fix identation on a kernel-doc markup
  drivers: firewire: firewire-cdev.h: fix identation on a kernel-doc
    markup
  docs: driver-api/infiniband.rst: fix Kerneldoc markup
  scripts/kernel-doc: don't add not needed new lines
  scripts/kernel-doc: drop dead code for Wcontents_before_sections
  scripts/kernel-doc: rename it to scripts/kernel-doc.pl
  scripts/kernel-doc: add a symlink to the Perl version of kernel-doc
  scripts/kernel-doc.py: add a Python parser
  scripts/kernel-doc.py: output warnings the same way as kerneldoc
  scripts/kernel-doc.py: better handle empty sections
  scripts/kernel-doc.py: properly handle struct_group macros
  scripts/kernel-doc.py: move regex methods to a separate file
  scripts/kernel-doc.py: move KernelDoc class to a separate file
  scripts/kernel-doc.py: move KernelFiles class to a separate file
  scripts/kernel-doc.py: move output classes to a separate file
  scripts/kernel-doc.py: convert message output to an interactor
  scripts/kernel-doc.py: move file lists to the parser function
  scripts/kernel-doc.py: implement support for -no-doc-sections
  scripts/kernel-doc.py: fix line number output
  scripts/kernel-doc.py: fix handling of doc output check
  scripts/kernel-doc.py: properly handle out_section for ReST
  scripts/kernel-doc.py: postpone warnings to the output plugin
  docs: add a .pylintrc file with sys path for docs scripts
  docs: sphinx: kerneldoc: verbose kernel-doc command if V=1
  docs: sphinx: kerneldoc: ignore "\" characters from options
  docs: sphinx: kerneldoc: use kernel-doc.py script
  scripts/kernel-doc.py: Set an output format for --none
  scripts/kernel-doc.py: adjust some coding style issues
  scripts/lib/kdoc/kdoc_parser.py: fix Python compat with < v3.13
  scripts/kernel-doc.py: move modulename to man class
  scripts/kernel-doc.py: properly handle KBUILD_BUILD_TIMESTAMP
  scripts/lib/kdoc/kdoc_parser.py: remove a python 3.9 dependency
  scripts/kernel-doc.py: Properly handle Werror and exit codes
  scripts/kernel-doc.py: some coding style cleanups
  scripts/kernel-doc: switch to use kernel-doc.py
  scripts/lib/kdoc/kdoc_files.py: allow filtering output per fname
  scripts/kernel_doc.py: better handle exported symbols
  docs: sphinx: kerneldoc: Use python class if available

 .pylintrc                                     |    2 +
 Documentation/Makefile                        |    2 +-
 Documentation/conf.py                         |    2 +-
 Documentation/driver-api/infiniband.rst       |   16 +-
 Documentation/sphinx/kerneldoc.py             |  183 +-
 .../media/ipu3/include/uapi/intel-ipu3.h      |    3 +-
 include/asm-generic/io.h                      |    6 +-
 include/uapi/linux/firewire-cdev.h            |    3 +-
 scripts/kernel-doc                            | 2447 +----------------
 scripts/kernel-doc.pl                         | 2439 ++++++++++++++++
 scripts/kernel-doc.py                         |  240 ++
 scripts/lib/kdoc/kdoc_files.py                |  281 ++
 scripts/lib/kdoc/kdoc_output.py               |  792 ++++++
 scripts/lib/kdoc/kdoc_parser.py               | 1714 ++++++++++++
 scripts/lib/kdoc/kdoc_re.py                   |  273 ++
 15 files changed, 5930 insertions(+), 2473 deletions(-)
 create mode 100644 .pylintrc
 mode change 100755 => 120000 scripts/kernel-doc
 create mode 100755 scripts/kernel-doc.pl
 create mode 100755 scripts/kernel-doc.py
 create mode 100755 scripts/lib/kdoc/kdoc_files.py
 create mode 100755 scripts/lib/kdoc/kdoc_output.py
 create mode 100755 scripts/lib/kdoc/kdoc_parser.py
 create mode 100755 scripts/lib/kdoc/kdoc_re.py