[00/27] Implement kernel-doc in Python

Message

Mauro Carvalho Chehab Feb. 19, 2025, 8:32 a.m. UTC

Hi Jon,

This changeset contains the kernel-doc.py script to replace the verable
kernel-doc originally written in Perl.

As the previous versions, I tried to stay as close as possible of the original
Perl implementation, as it helps to double check if each function was 
properly translated to Python.  This have been helpful debugging troubles
that happened during the conversion.

I worked hard to make it bug-compatible with the original one. Still, its
output has a couple of differences from the original one:

- The tab expansion works better with the Python script. With that, some
  outputs that contain tabs at kernel-doc markups are now different;

- The new script  works better stripping blank lines. So, there are a couple
  of empty new lines that are now stripped with this version;

- There is a buggy logic at kernel-doc to strip empty description and
  return sections. I was not able to replicate the exact behavior. So, I ended
  adding an extra logic to strip empty sections with a different algorithm.

Yet, on my tests, the results are compatible with the venerable script
output for all .. kernel-doc tags found in Documentation/. I double-checked
this by adding support to output the kernel-doc commands when V=1, and
then I ran a diff between kernel-doc.pl and kernel-doc.py for the same
command lines.

This version uses a minimal integration scenario: it just replaces the
exec file from the Perl to th Python version.

This series contains:

- 4 patches fixing some kernel-doc issues. One of them is for media, but
   I prefer to have this merged via your tree, as it suppresses a warning
   that happens after the changes;

- 2 cleanup patches for Perl kernel-doc;

- 2 patches renaming kernel-doc to kernel-doc.pl and adding a symlink.
  I opted to have the symlink in separate to make easier to review, but
  feel free to merge them on a single patch if you want;

- 15 patches with the new script. The first one is the new tool on a single
   file. The other ones split it into a library. Then, there are several bug
   fixes to make its output compatible with the original script;

- 1 patch adding a .pylintrc file to teach pylint about scripts/lib/* dirs;

- 2 patches adding some extra functionality to Sphinx kerneldoc extension;

- 1 patch switching Sphinx to use the new tool.

What is missing:

- a patch droping kernel-doc.pl;
- a patch renaming kernel-doc.py to kernel-doc (or changing the symlink).

I opted to not do those final changes here, as this way we can better
test the tools.

With such changes, if one wants to build docs with the old script,
all it is needed is to use KERNELDOC parameter, e.g.:

	$ make KERNELDOC=scripts/kernel-doc.pl htmldocs

Will make Sphinx use the original version.

Mauro Carvalho Chehab (27):
  include/asm-generic/io.h: fix kerneldoc markup
  drivers: media: intel-ipu3.h: fix identation on a kernel-doc markup
  drivers: firewire: firewire-cdev.h: fix identation on a kernel-doc
    markup
  docs: driver-api/infiniband.rst: fix Kerneldoc markup
  scripts/kernel-doc: don't add not needed new lines
  scripts/kernel-doc: drop dead code for Wcontents_before_sections
  scripts/kernel-doc: rename it to scripts/kernel-doc.pl
  scripts/kernel-doc: add a symlink to the Perl version of kernel-doc
  scripts/kernel-doc.py: add a Python parser
  scripts/kernel-doc.py: output warnings the same way as kerneldoc
  scripts/kernel-doc.py: better handle empty sections
  scripts/kernel-doc.py: properly handle struct_group macros
  scripts/kernel-doc.py: move regex methods to a separate file
  scripts/kernel-doc.py: move KernelDoc class to a separate file
  scripts/kernel-doc.py: move KernelFiles class to a separate file
  scripts/kernel-doc.py: move output classes to a separate file
  scripts/kernel-doc.py: convert message output to an interactor
  scripts/kernel-doc.py: move file lists to the parser function
  scripts/kernel-doc.py: implement support for -no-doc-sections
  scripts/kernel-doc.py: fix line number output
  scripts/kernel-doc.py: fix handling of doc output check
  scripts/kernel-doc.py: properly handle out_section for ReST
  scripts/kernel-doc.py: postpone warnings to the output plugin
  docs: add a .pylintrc file with sys path for docs scripts
  docs: sphinx: kerneldoc: verbose kernel-doc command if V=1
  docs: sphinx: kerneldoc: ignore "\" characters from options
  docs: sphinx: kerneldoc: use kernel-doc.py script

 .pylintrc                                     |    2 +
 Documentation/Makefile                        |    2 +-
 Documentation/conf.py                         |    2 +-
 Documentation/driver-api/infiniband.rst       |   16 +-
 Documentation/sphinx/kerneldoc.py             |   46 +
 .../media/ipu3/include/uapi/intel-ipu3.h      |    3 +-
 include/asm-generic/io.h                      |    6 +-
 include/uapi/linux/firewire-cdev.h            |    3 +-
 scripts/kernel-doc                            | 2447 +----------------
 scripts/kernel-doc.pl                         | 2439 ++++++++++++++++
 scripts/kernel-doc.py                         |  224 ++
 scripts/lib/kdoc/kdoc_files.py                |  274 ++
 scripts/lib/kdoc/kdoc_output.py               |  753 +++++
 scripts/lib/kdoc/kdoc_parser.py               | 1702 ++++++++++++
 scripts/lib/kdoc/kdoc_re.py                   |  272 ++
 15 files changed, 5730 insertions(+), 2461 deletions(-)
 create mode 100644 .pylintrc
 mode change 100755 => 120000 scripts/kernel-doc
 create mode 100755 scripts/kernel-doc.pl
 create mode 100755 scripts/kernel-doc.py
 create mode 100755 scripts/lib/kdoc/kdoc_files.py
 create mode 100755 scripts/lib/kdoc/kdoc_output.py
 create mode 100755 scripts/lib/kdoc/kdoc_parser.py
 create mode 100755 scripts/lib/kdoc/kdoc_re.py