| Message ID | cover.1744106241.git.mchehab+huawei@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Implement kernel-doc in Python | expand |
Jon, Em Tue, 8 Apr 2025 18:09:03 +0800 Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu: > Mauro Carvalho Chehab (32): > scripts/kernel-doc: rename it to scripts/kernel-doc.pl > scripts/kernel-doc: add a symlink to the Perl version of kernel-doc Forgot to mention at the description. After review, it makes sense to merge those two patches into one. Having them in separate is good for review, but merging them makes sense from bisectability PoV. Regards, Mauro
On Tue, 08 Apr 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote: > Hi Jon, > > This changeset contains the kernel-doc.py script to replace the verable > kernel-doc originally written in Perl. It replaces the first version and the > second series I sent on the top of it. Yay! Thanks for doing this. I believe this will make contributing to kernel-doc more accessible in the long run. > I tried to stay as close as possible of the original Perl implementation > on the first patch introducing kernel-doc.py, as it helps to double check > if each function was properly translated to Python. This have been > helpful debugging troubles that happened during the conversion. > > I worked hard to make it bug-compatible with the original one. Still, its > output has a couple of differences from the original one: > > - The tab expansion works better with the Python script. With that, some > outputs that contain tabs at kernel-doc markups are now different; > > - The new script works better stripping blank lines. So, there are a couple > of empty new lines that are now stripped with this version; > > - There is a buggy logic at kernel-doc to strip empty description and > return sections. I was not able to replicate the exact behavior. So, I ended > adding an extra logic to strip empty sections with a different algorithm. > > Yet, on my tests, the results are compatible with the venerable script > output for all .. kernel-doc tags found in Documentation/. I double-checked > this by adding support to output the kernel-doc commands when V=1, and > then I ran a diff between kernel-doc.pl and kernel-doc.py for the same > command lines. > > The only patch that doesn't belong to this series is a patch dropping > kernel-doc.pl. I opted to keep it for now, as it can help to better > test the new tools. > > With such changes, if one wants to build docs with the old script, > all it is needed is to use KERNELDOC parameter, e.g.: > > $ make KERNELDOC=scripts/kernel-doc.pl htmldocs I guess that's good for double checking that the python version reproduces the output of the old version, warts and all. And it could be used standalone for comparing the output for .[ch] files directly instead of going through Sphinx. But once we're reasonably sure the new one works fine, I think the natural follow-up will be to import the kernel-doc python module from the kernel-doc Sphinx extension instead of running it with subprocess.Popen(). It'll bypass an absolutely insane amount of forks, python interpreter launches and module imports. It'll also open the door for passing the results in python native structures instead of text, also making it possible to cache parse results instead of parsing the source files for every kernel-doc directive in rst. Another idea regarding code organization, again for future. Maybe we should have a scripts/python/ directory structure, so we can point python path there, and be able to import stuff from there? And reasonably share code between modules. And have linters handle it recursively, etc, etc. Anyway, I applaud the work, and I regret that I don't have time to review it in detail. Regardless, I think the matching output is the most important part. BR, Jani. > --- > > v3: > - rebased on the top of v6.15-rc1; > - Removed patches that weren't touching kernel-doc and its Sphinx extension; > - The "Re" class was renamed to "KernRe" > - It contains one patch from Sean with an additional hunk for the > python version. > > Mauro Carvalho Chehab (32): > scripts/kernel-doc: rename it to scripts/kernel-doc.pl > scripts/kernel-doc: add a symlink to the Perl version of kernel-doc > scripts/kernel-doc.py: add a Python parser > scripts/kernel-doc.py: output warnings the same way as kerneldoc > scripts/kernel-doc.py: better handle empty sections > scripts/kernel-doc.py: properly handle struct_group macros > scripts/kernel-doc.py: move regex methods to a separate file > scripts/kernel-doc.py: move KernelDoc class to a separate file > scripts/kernel-doc.py: move KernelFiles class to a separate file > scripts/kernel-doc.py: move output classes to a separate file > scripts/kernel-doc.py: convert message output to an interactor > scripts/kernel-doc.py: move file lists to the parser function > scripts/kernel-doc.py: implement support for -no-doc-sections > scripts/kernel-doc.py: fix line number output > scripts/kernel-doc.py: fix handling of doc output check > scripts/kernel-doc.py: properly handle out_section for ReST > scripts/kernel-doc.py: postpone warnings to the output plugin > docs: add a .pylintrc file with sys path for docs scripts > docs: sphinx: kerneldoc: verbose kernel-doc command if V=1 > docs: sphinx: kerneldoc: ignore "\" characters from options > docs: sphinx: kerneldoc: use kernel-doc.py script > scripts/kernel-doc.py: Set an output format for --none > scripts/kernel-doc.py: adjust some coding style issues > scripts/lib/kdoc/kdoc_parser.py: fix Python compat with < v3.13 > scripts/kernel-doc.py: move modulename to man class > scripts/kernel-doc.py: properly handle KBUILD_BUILD_TIMESTAMP > scripts/lib/kdoc/kdoc_parser.py: remove a python 3.9 dependency > scripts/kernel-doc.py: Properly handle Werror and exit codes > scripts/kernel-doc: switch to use kernel-doc.py > scripts/lib/kdoc/kdoc_files.py: allow filtering output per fname > scripts/kernel_doc.py: better handle exported symbols > scripts/kernel-doc.py: Rename the kernel doc Re class to KernRe > > Sean Anderson (1): > scripts: kernel-doc: fix parsing function-like typedefs (again) > > .pylintrc | 2 + > Documentation/Makefile | 2 +- > Documentation/conf.py | 2 +- > Documentation/sphinx/kerneldoc.py | 46 + > scripts/kernel-doc | 2440 +---------------------------- > scripts/kernel-doc.pl | 2439 ++++++++++++++++++++++++++++ > scripts/kernel-doc.py | 315 ++++ > scripts/lib/kdoc/kdoc_files.py | 282 ++++ > scripts/lib/kdoc/kdoc_output.py | 793 ++++++++++ > scripts/lib/kdoc/kdoc_parser.py | 1715 ++++++++++++++++++++ > scripts/lib/kdoc/kdoc_re.py | 273 ++++ > 11 files changed, 5868 insertions(+), 2441 deletions(-) > create mode 100644 .pylintrc > mode change 100755 => 120000 scripts/kernel-doc > create mode 100755 scripts/kernel-doc.pl > create mode 100755 scripts/kernel-doc.py > create mode 100644 scripts/lib/kdoc/kdoc_files.py > create mode 100755 scripts/lib/kdoc/kdoc_output.py > create mode 100755 scripts/lib/kdoc/kdoc_parser.py > create mode 100755 scripts/lib/kdoc/kdoc_re.py
Em Wed, 09 Apr 2025 13:16:06 +0300 Jani Nikula <jani.nikula@linux.intel.com> escreveu: > On Tue, 08 Apr 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote: > > Hi Jon, > > > > This changeset contains the kernel-doc.py script to replace the verable > > kernel-doc originally written in Perl. It replaces the first version and the > > second series I sent on the top of it. > > Yay! Thanks for doing this. I believe this will make contributing to > kernel-doc more accessible in the long run. > > > I tried to stay as close as possible of the original Perl implementation > > on the first patch introducing kernel-doc.py, as it helps to double check > > if each function was properly translated to Python. This have been > > helpful debugging troubles that happened during the conversion. > > > > I worked hard to make it bug-compatible with the original one. Still, its > > output has a couple of differences from the original one: > > > > - The tab expansion works better with the Python script. With that, some > > outputs that contain tabs at kernel-doc markups are now different; > > > > - The new script works better stripping blank lines. So, there are a couple > > of empty new lines that are now stripped with this version; > > > > - There is a buggy logic at kernel-doc to strip empty description and > > return sections. I was not able to replicate the exact behavior. So, I ended > > adding an extra logic to strip empty sections with a different algorithm. > > > > Yet, on my tests, the results are compatible with the venerable script > > output for all .. kernel-doc tags found in Documentation/. I double-checked > > this by adding support to output the kernel-doc commands when V=1, and > > then I ran a diff between kernel-doc.pl and kernel-doc.py for the same > > command lines. > > > > The only patch that doesn't belong to this series is a patch dropping > > kernel-doc.pl. I opted to keep it for now, as it can help to better > > test the new tools. > > > > With such changes, if one wants to build docs with the old script, > > all it is needed is to use KERNELDOC parameter, e.g.: > > > > $ make KERNELDOC=scripts/kernel-doc.pl htmldocs > > I guess that's good for double checking that the python version > reproduces the output of the old version, warts and all. And it could be > used standalone for comparing the output for .[ch] files directly > instead of going through Sphinx. > > But once we're reasonably sure the new one works fine, I think the > natural follow-up will be to import the kernel-doc python module from > the kernel-doc Sphinx extension instead of running it with > subprocess.Popen(). It'll bypass an absolutely insane amount of forks, > python interpreter launches and module imports. > > It'll also open the door for passing the results in python native > structures instead of text, also making it possible to cache parse > results instead of parsing the source files for every kernel-doc > directive in rst. Yes, this is on my plan. I have already a patch series for that, but it still requires some care to ensure that the results will be identical. > Another idea regarding code organization, again for future. Maybe we > should have a scripts/python/ directory structure, so we can point > python path there, and be able to import stuff from there? And > reasonably share code between modules. And have linters handle it > recursively, etc, etc. Sounds like a plan. I did some code reorg already, but surely there are spaces for improvements. > Anyway, I applaud the work, and I regret that I don't have time to > review it in detail. Regardless, I think the matching output is the most > important part. I did several tests here to check the output, making it similar to the output from the Perl version. > > > BR, > Jani. > > > --- > > > > v3: > > - rebased on the top of v6.15-rc1; > > - Removed patches that weren't touching kernel-doc and its Sphinx extension; > > - The "Re" class was renamed to "KernRe" > > - It contains one patch from Sean with an additional hunk for the > > python version. > > > > Mauro Carvalho Chehab (32): > > scripts/kernel-doc: rename it to scripts/kernel-doc.pl > > scripts/kernel-doc: add a symlink to the Perl version of kernel-doc > > scripts/kernel-doc.py: add a Python parser > > scripts/kernel-doc.py: output warnings the same way as kerneldoc > > scripts/kernel-doc.py: better handle empty sections > > scripts/kernel-doc.py: properly handle struct_group macros > > scripts/kernel-doc.py: move regex methods to a separate file > > scripts/kernel-doc.py: move KernelDoc class to a separate file > > scripts/kernel-doc.py: move KernelFiles class to a separate file > > scripts/kernel-doc.py: move output classes to a separate file > > scripts/kernel-doc.py: convert message output to an interactor > > scripts/kernel-doc.py: move file lists to the parser function > > scripts/kernel-doc.py: implement support for -no-doc-sections > > scripts/kernel-doc.py: fix line number output > > scripts/kernel-doc.py: fix handling of doc output check > > scripts/kernel-doc.py: properly handle out_section for ReST > > scripts/kernel-doc.py: postpone warnings to the output plugin > > docs: add a .pylintrc file with sys path for docs scripts > > docs: sphinx: kerneldoc: verbose kernel-doc command if V=1 > > docs: sphinx: kerneldoc: ignore "\" characters from options > > docs: sphinx: kerneldoc: use kernel-doc.py script > > scripts/kernel-doc.py: Set an output format for --none > > scripts/kernel-doc.py: adjust some coding style issues > > scripts/lib/kdoc/kdoc_parser.py: fix Python compat with < v3.13 > > scripts/kernel-doc.py: move modulename to man class > > scripts/kernel-doc.py: properly handle KBUILD_BUILD_TIMESTAMP > > scripts/lib/kdoc/kdoc_parser.py: remove a python 3.9 dependency > > scripts/kernel-doc.py: Properly handle Werror and exit codes > > scripts/kernel-doc: switch to use kernel-doc.py > > scripts/lib/kdoc/kdoc_files.py: allow filtering output per fname > > scripts/kernel_doc.py: better handle exported symbols > > scripts/kernel-doc.py: Rename the kernel doc Re class to KernRe > > > > Sean Anderson (1): > > scripts: kernel-doc: fix parsing function-like typedefs (again) > > > > .pylintrc | 2 + > > Documentation/Makefile | 2 +- > > Documentation/conf.py | 2 +- > > Documentation/sphinx/kerneldoc.py | 46 + > > scripts/kernel-doc | 2440 +---------------------------- > > scripts/kernel-doc.pl | 2439 ++++++++++++++++++++++++++++ > > scripts/kernel-doc.py | 315 ++++ > > scripts/lib/kdoc/kdoc_files.py | 282 ++++ > > scripts/lib/kdoc/kdoc_output.py | 793 ++++++++++ > > scripts/lib/kdoc/kdoc_parser.py | 1715 ++++++++++++++++++++ > > scripts/lib/kdoc/kdoc_re.py | 273 ++++ > > 11 files changed, 5868 insertions(+), 2441 deletions(-) > > create mode 100644 .pylintrc > > mode change 100755 => 120000 scripts/kernel-doc > > create mode 100755 scripts/kernel-doc.pl > > create mode 100755 scripts/kernel-doc.py > > create mode 100644 scripts/lib/kdoc/kdoc_files.py > > create mode 100755 scripts/lib/kdoc/kdoc_output.py > > create mode 100755 scripts/lib/kdoc/kdoc_parser.py > > create mode 100755 scripts/lib/kdoc/kdoc_re.py >
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > This changeset contains the kernel-doc.py script to replace the verable > kernel-doc originally written in Perl. It replaces the first version and the > second series I sent on the top of it. OK, I've applied it, looked at the (minimal) changes in output, and concluded that it's good - all this stuff is now in docs-next. Many thanks for doing this! I'm going to hold off on other documentation patches for a day or two just in case anything turns up. But it looks awfully good. Thanks, jon
On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > This changeset contains the kernel-doc.py script to replace the verable > > kernel-doc originally written in Perl. It replaces the first version and the > > second series I sent on the top of it. > > OK, I've applied it, looked at the (minimal) changes in output, and > concluded that it's good - all this stuff is now in docs-next. Many > thanks for doing this! > > I'm going to hold off on other documentation patches for a day or two > just in case anything turns up. But it looks awfully good. This started well, until it becomes a scripts/lib/kdoc. So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave "disgusting turd" )as said by Linus) in the clean tree. *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution.
Andy Shevchenko <andriy.shevchenko@intel.com> writes: > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: >> >> > This changeset contains the kernel-doc.py script to replace the verable >> > kernel-doc originally written in Perl. It replaces the first version and the >> > second series I sent on the top of it. >> >> OK, I've applied it, looked at the (minimal) changes in output, and >> concluded that it's good - all this stuff is now in docs-next. Many >> thanks for doing this! >> >> I'm going to hold off on other documentation patches for a day or two >> just in case anything turns up. But it looks awfully good. > > This started well, until it becomes a scripts/lib/kdoc. > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > "disgusting turd" )as said by Linus) in the clean tree. > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. If nothing else, "make cleandocs" should clean it up, certainly. We can also tell CPython to not create that directory at all. I'll run some tests to see what the effect is on the documentation build times; I'm guessing it will not be huge... Thanks, jon
> Andy Shevchenko <andriy.shevchenko@intel.com> writes: > >> On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: >>> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: >>> >>> > This changeset contains the kernel-doc.py script to replace the verable >>> > kernel-doc originally written in Perl. It replaces the first version and the >>> > second series I sent on the top of it. >>> >>> OK, I've applied it, looked at the (minimal) changes in output, and >>> concluded that it's good - all this stuff is now in docs-next. Many >>> thanks for doing this! >>> >>> I'm going to hold off on other documentation patches for a day or two >>> just in case anything turns up. But it looks awfully good. >> >> This started well, until it becomes a scripts/lib/kdoc. >> So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave >> "disgusting turd" )as said by Linus) in the clean tree. >> >> *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. Actually, I find myself unable to reproduce this; can you tell me how you get Python to create that directory on your system? Which version of Python? Thanks, jon
On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > >> > kernel-doc originally written in Perl. It replaces the first version and the > >> > second series I sent on the top of it. > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > >> concluded that it's good - all this stuff is now in docs-next. Many > >> thanks for doing this! > >> > >> I'm going to hold off on other documentation patches for a day or two > >> just in case anything turns up. But it looks awfully good. > > > > This started well, until it becomes a scripts/lib/kdoc. > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > "disgusting turd" )as said by Linus) in the clean tree. > > > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > If nothing else, "make cleandocs" should clean it up, certainly. > > We can also tell CPython to not create that directory at all. I'll run > some tests to see what the effect is on the documentation build times; > I'm guessing it will not be huge... I do not build documentation at all, it's just a regular code build that leaves tree dirty. $ python3 --version Python 3.13.2 It's standard Debian testing distribution, no customisation in the code. To reproduce. 1) I have just done a new build to reduce the churn, so, running make again does nothing; 2) The following snippet in shell shows the issue $ git clean -xdf $ git status --ignored On branch ... nothing to commit, working tree clean $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 make[1]: Entering directory '...' GEN Makefile DESCEND objtool CALL .../scripts/checksyscalls.sh INSTALL libsubcmd_headers .pylintrc: warning: ignored by one of the .gitignore files Kernel: arch/x86/boot/bzImage is ready (#23) make[1]: Leaving directory '...' $ touch drivers/gpio/gpiolib-acpi.c $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 make[1]: Entering directory '...' GEN Makefile DESCEND objtool CALL .../scripts/checksyscalls.sh INSTALL libsubcmd_headers ... OBJCOPY arch/x86/boot/setup.bin BUILD arch/x86/boot/bzImage Kernel: arch/x86/boot/bzImage is ready (#24) make[1]: Leaving directory '...' $ git status --ignored On branch ... Untracked files: (use "git add <file>..." to include in what will be committed) scripts/lib/kdoc/__pycache__/ nothing added to commit but untracked files present (use "git add" to track) It's 100% reproducible on my side. I am happy to test any patches to fix this. It's really annoying "feature" for `make O=...` builds. Also note that theoretically the Git worktree may be located on read-only storage / media and this can induce subtle issues.
On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > >> > > >> > This changeset contains the kernel-doc.py script to replace the verable > > >> > kernel-doc originally written in Perl. It replaces the first version and the > > >> > second series I sent on the top of it. > > >> > > >> OK, I've applied it, looked at the (minimal) changes in output, and > > >> concluded that it's good - all this stuff is now in docs-next. Many > > >> thanks for doing this! > > >> > > >> I'm going to hold off on other documentation patches for a day or two > > >> just in case anything turns up. But it looks awfully good. > > > > > > This started well, until it becomes a scripts/lib/kdoc. > > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > "disgusting turd" )as said by Linus) in the clean tree. > > > > > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > If nothing else, "make cleandocs" should clean it up, certainly. > > > > We can also tell CPython to not create that directory at all. I'll run > > some tests to see what the effect is on the documentation build times; > > I'm guessing it will not be huge... > > I do not build documentation at all, it's just a regular code build that leaves > tree dirty. > > $ python3 --version > Python 3.13.2 > > It's standard Debian testing distribution, no customisation in the code. > > To reproduce. > 1) I have just done a new build to reduce the churn, so, running make again does nothing; > 2) The following snippet in shell shows the issue > > $ git clean -xdf > $ git status --ignored > On branch ... > nothing to commit, working tree clean > > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > make[1]: Entering directory '...' > GEN Makefile > DESCEND objtool > CALL .../scripts/checksyscalls.sh > INSTALL libsubcmd_headers > .pylintrc: warning: ignored by one of the .gitignore files > Kernel: arch/x86/boot/bzImage is ready (#23) > make[1]: Leaving directory '...' > > $ touch drivers/gpio/gpiolib-acpi.c > > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > make[1]: Entering directory '...' > GEN Makefile > DESCEND objtool > CALL .../scripts/checksyscalls.sh > INSTALL libsubcmd_headers > ... > OBJCOPY arch/x86/boot/setup.bin > BUILD arch/x86/boot/bzImage > Kernel: arch/x86/boot/bzImage is ready (#24) > make[1]: Leaving directory '...' > > $ git status --ignored > On branch ... > Untracked files: > (use "git add <file>..." to include in what will be committed) > scripts/lib/kdoc/__pycache__/ > > nothing added to commit but untracked files present (use "git add" to track) FWIW, I repeated this with removing the O=.../out folder completely, so it's fully clean build. Still the same issue. And it appears at the very beginning of the build. You don't need to wait to have the kernel to be built actually. > It's 100% reproducible on my side. I am happy to test any patches to fix this. > It's really annoying "feature" for `make O=...` builds. Also note that > theoretically the Git worktree may be located on read-only storage / media > and this can induce subtle issues.
On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: >> > >> >> > >> > This changeset contains the kernel-doc.py script to replace the verable >> > >> > kernel-doc originally written in Perl. It replaces the first version and the >> > >> > second series I sent on the top of it. >> > >> >> > >> OK, I've applied it, looked at the (minimal) changes in output, and >> > >> concluded that it's good - all this stuff is now in docs-next. Many >> > >> thanks for doing this! >> > >> >> > >> I'm going to hold off on other documentation patches for a day or two >> > >> just in case anything turns up. But it looks awfully good. >> > > >> > > This started well, until it becomes a scripts/lib/kdoc. >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave >> > > "disgusting turd" )as said by Linus) in the clean tree. >> > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. >> > >> > If nothing else, "make cleandocs" should clean it up, certainly. >> > >> > We can also tell CPython to not create that directory at all. I'll run >> > some tests to see what the effect is on the documentation build times; >> > I'm guessing it will not be huge... >> >> I do not build documentation at all, it's just a regular code build that leaves >> tree dirty. >> >> $ python3 --version >> Python 3.13.2 >> >> It's standard Debian testing distribution, no customisation in the code. >> >> To reproduce. >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; >> 2) The following snippet in shell shows the issue >> >> $ git clean -xdf >> $ git status --ignored >> On branch ... >> nothing to commit, working tree clean >> >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 >> make[1]: Entering directory '...' >> GEN Makefile >> DESCEND objtool >> CALL .../scripts/checksyscalls.sh >> INSTALL libsubcmd_headers >> .pylintrc: warning: ignored by one of the .gitignore files >> Kernel: arch/x86/boot/bzImage is ready (#23) >> make[1]: Leaving directory '...' >> >> $ touch drivers/gpio/gpiolib-acpi.c >> >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 >> make[1]: Entering directory '...' >> GEN Makefile >> DESCEND objtool >> CALL .../scripts/checksyscalls.sh >> INSTALL libsubcmd_headers >> ... >> OBJCOPY arch/x86/boot/setup.bin >> BUILD arch/x86/boot/bzImage >> Kernel: arch/x86/boot/bzImage is ready (#24) >> make[1]: Leaving directory '...' >> >> $ git status --ignored >> On branch ... >> Untracked files: >> (use "git add <file>..." to include in what will be committed) >> scripts/lib/kdoc/__pycache__/ >> >> nothing added to commit but untracked files present (use "git add" to track) > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > fully clean build. Still the same issue. > > And it appears at the very beginning of the build. You don't need to wait to > have the kernel to be built actually. kernel-doc gets run on source files for W=1 builds. See Makefile.build. BR, Jani. > >> It's 100% reproducible on my side. I am happy to test any patches to fix this. >> It's really annoying "feature" for `make O=...` builds. Also note that >> theoretically the Git worktree may be located on read-only storage / media >> and this can induce subtle issues.
On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > >> > >> > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > >> > >> > second series I sent on the top of it. > >> > >> > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > >> > >> thanks for doing this! > >> > >> > >> > >> I'm going to hold off on other documentation patches for a day or two > >> > >> just in case anything turns up. But it looks awfully good. > >> > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > >> > > "disgusting turd" )as said by Linus) in the clean tree. > >> > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > >> > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > >> > > >> > We can also tell CPython to not create that directory at all. I'll run > >> > some tests to see what the effect is on the documentation build times; > >> > I'm guessing it will not be huge... > >> > >> I do not build documentation at all, it's just a regular code build that leaves > >> tree dirty. > >> > >> $ python3 --version > >> Python 3.13.2 > >> > >> It's standard Debian testing distribution, no customisation in the code. > >> > >> To reproduce. > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > >> 2) The following snippet in shell shows the issue > >> > >> $ git clean -xdf > >> $ git status --ignored > >> On branch ... > >> nothing to commit, working tree clean > >> > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > >> make[1]: Entering directory '...' > >> GEN Makefile > >> DESCEND objtool > >> CALL .../scripts/checksyscalls.sh > >> INSTALL libsubcmd_headers > >> .pylintrc: warning: ignored by one of the .gitignore files > >> Kernel: arch/x86/boot/bzImage is ready (#23) > >> make[1]: Leaving directory '...' > >> > >> $ touch drivers/gpio/gpiolib-acpi.c > >> > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > >> make[1]: Entering directory '...' > >> GEN Makefile > >> DESCEND objtool > >> CALL .../scripts/checksyscalls.sh > >> INSTALL libsubcmd_headers > >> ... > >> OBJCOPY arch/x86/boot/setup.bin > >> BUILD arch/x86/boot/bzImage > >> Kernel: arch/x86/boot/bzImage is ready (#24) > >> make[1]: Leaving directory '...' > >> > >> $ git status --ignored > >> On branch ... > >> Untracked files: > >> (use "git add <file>..." to include in what will be committed) > >> scripts/lib/kdoc/__pycache__/ > >> > >> nothing added to commit but untracked files present (use "git add" to track) > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > fully clean build. Still the same issue. > > > > And it appears at the very beginning of the build. You don't need to wait to > > have the kernel to be built actually. > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. Thanks for the clarification, so we know that it runs and we know that it has an issue. > >> It's 100% reproducible on my side. I am happy to test any patches to fix this. > >> It's really annoying "feature" for `make O=...` builds. Also note that > >> theoretically the Git worktree may be located on read-only storage / media > >> and this can induce subtle issues.
On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > >> > >> > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > >> > >> > second series I sent on the top of it. > > >> > >> > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > >> > >> thanks for doing this! > > >> > >> > > >> > >> I'm going to hold off on other documentation patches for a day or two > > >> > >> just in case anything turns up. But it looks awfully good. > > >> > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > >> > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > >> > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > >> > > > >> > We can also tell CPython to not create that directory at all. I'll run > > >> > some tests to see what the effect is on the documentation build times; > > >> > I'm guessing it will not be huge... > > >> > > >> I do not build documentation at all, it's just a regular code build that leaves > > >> tree dirty. > > >> > > >> $ python3 --version > > >> Python 3.13.2 > > >> > > >> It's standard Debian testing distribution, no customisation in the code. > > >> > > >> To reproduce. > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > >> 2) The following snippet in shell shows the issue > > >> > > >> $ git clean -xdf > > >> $ git status --ignored > > >> On branch ... > > >> nothing to commit, working tree clean > > >> > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > >> make[1]: Entering directory '...' > > >> GEN Makefile > > >> DESCEND objtool > > >> CALL .../scripts/checksyscalls.sh > > >> INSTALL libsubcmd_headers > > >> .pylintrc: warning: ignored by one of the .gitignore files > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > >> make[1]: Leaving directory '...' > > >> > > >> $ touch drivers/gpio/gpiolib-acpi.c > > >> > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > >> make[1]: Entering directory '...' > > >> GEN Makefile > > >> DESCEND objtool > > >> CALL .../scripts/checksyscalls.sh > > >> INSTALL libsubcmd_headers > > >> ... > > >> OBJCOPY arch/x86/boot/setup.bin > > >> BUILD arch/x86/boot/bzImage > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > >> make[1]: Leaving directory '...' > > >> > > >> $ git status --ignored > > >> On branch ... > > >> Untracked files: > > >> (use "git add <file>..." to include in what will be committed) > > >> scripts/lib/kdoc/__pycache__/ > > >> > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > fully clean build. Still the same issue. > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > have the kernel to be built actually. > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > Thanks for the clarification, so we know that it runs and we know that it has > an issue. Ideal solution what would I expect is that the cache folder should respect the given O=... argument, or disabled at all (but I don't think the latter is what we want as it may slow down the build). > > >> It's 100% reproducible on my side. I am happy to test any patches to fix this. > > >> It's really annoying "feature" for `make O=...` builds. Also note that > > >> theoretically the Git worktree may be located on read-only storage / media > > >> and this can induce subtle issues.
Em Tue, 15 Apr 2025 10:03:47 +0300 Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > >> > > > >> > This changeset contains the kernel-doc.py script to replace the verable > > > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > >> > second series I sent on the top of it. > > > >> > > > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > >> concluded that it's good - all this stuff is now in docs-next. Many > > > >> thanks for doing this! > > > >> > > > >> I'm going to hold off on other documentation patches for a day or two > > > >> just in case anything turns up. But it looks awfully good. > > > > > > > > This started well, until it becomes a scripts/lib/kdoc. > > > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > > "disgusting turd" )as said by Linus) in the clean tree. > > > > > > > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > > > If nothing else, "make cleandocs" should clean it up, certainly. Not sure about that, as __pycache__ is completely managed by Python: it will not only create it for scripts/lib, but also for all Python libraries, including the Sphinx ones. IMO, it makes more sense, instead, to ensure that __pycache__ won't be created at the sourcedir if O= is used, but ignore it if this is created. Btw, the same problem should already happen with get_abi.py, as it also uses "import" from scripts/lib. So, we need a more generic solution. See below. > > > > > > We can also tell CPython to not create that directory at all. I'll run > > > some tests to see what the effect is on the documentation build times; > > > I'm guessing it will not be huge... I doubt it would have much impact for kernel-doc, but it can have some impact for Sphinx, as disabling Python JIT to store bytecode would affect it too. - Andy, Could you please remove __pycache__ and set this env: PYTHONDONTWRITEBYTECODE=1 before building the Kernel? If this works, one alternative would be to set it when O= is used. > > I do not build documentation at all, it's just a regular code build that leaves > > tree dirty. > > > > $ python3 --version > > Python 3.13.2 > > > > It's standard Debian testing distribution, no customisation in the code. > > > > To reproduce. > > 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > 2) The following snippet in shell shows the issue > > > > $ git clean -xdf > > $ git status --ignored > > On branch ... > > nothing to commit, working tree clean > > > > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > make[1]: Entering directory '...' > > GEN Makefile > > DESCEND objtool > > CALL .../scripts/checksyscalls.sh > > INSTALL libsubcmd_headers > > .pylintrc: warning: ignored by one of the .gitignore files > > Kernel: arch/x86/boot/bzImage is ready (#23) > > make[1]: Leaving directory '...' > > > > $ touch drivers/gpio/gpiolib-acpi.c > > > > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > make[1]: Entering directory '...' > > GEN Makefile > > DESCEND objtool > > CALL .../scripts/checksyscalls.sh > > INSTALL libsubcmd_headers > > ... > > OBJCOPY arch/x86/boot/setup.bin > > BUILD arch/x86/boot/bzImage > > Kernel: arch/x86/boot/bzImage is ready (#24) > > make[1]: Leaving directory '...' > > > > $ git status --ignored > > On branch ... > > Untracked files: > > (use "git add <file>..." to include in what will be committed) > > scripts/lib/kdoc/__pycache__/ > > > > nothing added to commit but untracked files present (use "git add" to track) > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > fully clean build. Still the same issue. > > And it appears at the very beginning of the build. You don't need to wait to > have the kernel to be built actually. Depending on your .config, kernel-doc will be called even without building documentation to check for some problems at kernel-doc tags. > > > It's 100% reproducible on my side. I am happy to test any patches to fix this. > > It's really annoying "feature" for `make O=...` builds. Also note that > > theoretically the Git worktree may be located on read-only storage / media > > and this can induce subtle issues. Python's JIT compiler automatically creates __pycache__ whenever it encounters an "import" and the *.pyc is older than the script (or doesn't exist). This happens with external libraries, and also with the internal ones, like the ones we now have at the Kernel. I dunno what happens if the FS is read-only. I would expect that the JIT compiler would just work as if bytecode creation is disabled. That's said, I never played myself with __pycache__. Yet, I have some raw ideas about how to deal with that. This requires more tests, though. I can see some possible solutions for that: 1. Assuming that PYTHONDONTWRITEBYTECODE=1 works, the build system could set it if O= is used. This would have some performance impact for both Kernel compilation (because kernel-doc is called to check doc issues), and for Kernel compilation itself. I dunno how much it would impact, but this is probably the quickest solution to implement; 2. when O=<targetdir> is used, copy scripts/lib/*/*.py to the target directory and change kernel-doc.py to use <targetdir> for library search on such case. This way, the __pycache__ would be created at the <targetdir>. This might work as well with symlinks. The performance impact here would be minimal, but it will require an extra step for O= to copy data (or to create symlinks); 3. eventually there is a way to teach Python to place the __pycache__ at <targetdir>, instead of <sourcedir>. If supported, this would be the cleanest solution. Regards, Mauro
Em Tue, 15 Apr 2025 11:19:26 +0300 Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > >> > >> > > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > >> > >> > second series I sent on the top of it. > > > >> > >> > > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > > >> > >> thanks for doing this! > > > >> > >> > > > >> > >> I'm going to hold off on other documentation patches for a day or two > > > >> > >> just in case anything turns up. But it looks awfully good. > > > >> > > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > > >> > > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > >> > > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > > >> > > > > >> > We can also tell CPython to not create that directory at all. I'll run > > > >> > some tests to see what the effect is on the documentation build times; > > > >> > I'm guessing it will not be huge... > > > >> > > > >> I do not build documentation at all, it's just a regular code build that leaves > > > >> tree dirty. > > > >> > > > >> $ python3 --version > > > >> Python 3.13.2 > > > >> > > > >> It's standard Debian testing distribution, no customisation in the code. > > > >> > > > >> To reproduce. > > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > > >> 2) The following snippet in shell shows the issue > > > >> > > > >> $ git clean -xdf > > > >> $ git status --ignored > > > >> On branch ... > > > >> nothing to commit, working tree clean > > > >> > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > >> make[1]: Entering directory '...' > > > >> GEN Makefile > > > >> DESCEND objtool > > > >> CALL .../scripts/checksyscalls.sh > > > >> INSTALL libsubcmd_headers > > > >> .pylintrc: warning: ignored by one of the .gitignore files > > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > > >> make[1]: Leaving directory '...' > > > >> > > > >> $ touch drivers/gpio/gpiolib-acpi.c > > > >> > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > >> make[1]: Entering directory '...' > > > >> GEN Makefile > > > >> DESCEND objtool > > > >> CALL .../scripts/checksyscalls.sh > > > >> INSTALL libsubcmd_headers > > > >> ... > > > >> OBJCOPY arch/x86/boot/setup.bin > > > >> BUILD arch/x86/boot/bzImage > > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > > >> make[1]: Leaving directory '...' > > > >> > > > >> $ git status --ignored > > > >> On branch ... > > > >> Untracked files: > > > >> (use "git add <file>..." to include in what will be committed) > > > >> scripts/lib/kdoc/__pycache__/ > > > >> > > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > > fully clean build. Still the same issue. > > > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > > have the kernel to be built actually. > > > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > > > Thanks for the clarification, so we know that it runs and we know that it has > > an issue. > > Ideal solution what would I expect is that the cache folder should respect > the given O=... argument, or disabled at all (but I don't think the latter > is what we want as it may slow down the build). From: https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6 and: https://peps.python.org/pep-3147/ It sounds that Python 3.8 and above have a way to specify the cache location, via PYTHONPYCACHEPREFIX env var, and via "-X pycache_prefix=path". As the current minimal Python version is 3.9, we can safely use it. So, maybe this would work: make O="../out" PYTHONPYCACHEPREFIX="../out" or a variant of it: PYTHONPYCACHEPREFIX="../out" make O="../out" If this works, we can adjust the building system to fill PYTHONPYCACHEPREFIX env var when O= is used. Regards, Mauro
Em Tue, 15 Apr 2025 16:40:34 +0800 Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu: > Em Tue, 15 Apr 2025 11:19:26 +0300 > Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > > > On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > > > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > > >> > >> > > > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > > >> > >> > second series I sent on the top of it. > > > > >> > >> > > > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > > > >> > >> thanks for doing this! > > > > >> > >> > > > > >> > >> I'm going to hold off on other documentation patches for a day or two > > > > >> > >> just in case anything turns up. But it looks awfully good. > > > > >> > > > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > > > >> > > > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > >> > > > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > > > >> > > > > > >> > We can also tell CPython to not create that directory at all. I'll run > > > > >> > some tests to see what the effect is on the documentation build times; > > > > >> > I'm guessing it will not be huge... > > > > >> > > > > >> I do not build documentation at all, it's just a regular code build that leaves > > > > >> tree dirty. > > > > >> > > > > >> $ python3 --version > > > > >> Python 3.13.2 > > > > >> > > > > >> It's standard Debian testing distribution, no customisation in the code. > > > > >> > > > > >> To reproduce. > > > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > > > >> 2) The following snippet in shell shows the issue > > > > >> > > > > >> $ git clean -xdf > > > > >> $ git status --ignored > > > > >> On branch ... > > > > >> nothing to commit, working tree clean > > > > >> > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > >> make[1]: Entering directory '...' > > > > >> GEN Makefile > > > > >> DESCEND objtool > > > > >> CALL .../scripts/checksyscalls.sh > > > > >> INSTALL libsubcmd_headers > > > > >> .pylintrc: warning: ignored by one of the .gitignore files > > > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > > > >> make[1]: Leaving directory '...' > > > > >> > > > > >> $ touch drivers/gpio/gpiolib-acpi.c > > > > >> > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > >> make[1]: Entering directory '...' > > > > >> GEN Makefile > > > > >> DESCEND objtool > > > > >> CALL .../scripts/checksyscalls.sh > > > > >> INSTALL libsubcmd_headers > > > > >> ... > > > > >> OBJCOPY arch/x86/boot/setup.bin > > > > >> BUILD arch/x86/boot/bzImage > > > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > > > >> make[1]: Leaving directory '...' > > > > >> > > > > >> $ git status --ignored > > > > >> On branch ... > > > > >> Untracked files: > > > > >> (use "git add <file>..." to include in what will be committed) > > > > >> scripts/lib/kdoc/__pycache__/ > > > > >> > > > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > > > fully clean build. Still the same issue. > > > > > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > > > have the kernel to be built actually. > > > > > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > > > > > Thanks for the clarification, so we know that it runs and we know that it has > > > an issue. > > > > Ideal solution what would I expect is that the cache folder should respect > > the given O=... argument, or disabled at all (but I don't think the latter > > is what we want as it may slow down the build). > > From: > https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6 > and: > https://peps.python.org/pep-3147/ > > It sounds that Python 3.8 and above have a way to specify the cache > location, via PYTHONPYCACHEPREFIX env var, and via "-X pycache_prefix=path". > > As the current minimal Python version is 3.9, we can safely use it. > > So, maybe this would work: > > make O="../out" PYTHONPYCACHEPREFIX="../out" > > or a variant of it: > > PYTHONPYCACHEPREFIX="../out" make O="../out" > > If this works, we can adjust the building system to fill PYTHONPYCACHEPREFIX > env var when O= is used. That's interesting... Sphinx is already called with PYTHONDONTWRITEBYTECODE. From Documentation/Makefile: quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media $2 && \ PYTHONDONTWRITEBYTECODE=1 \ ... It seems that the issue happens only when W=1 is used and kernel-doc is called outside Sphinx. Anyway, IMHO, the best would be to change the above to: PYTHONPYCACHEPREFIX=$(abspath $(BUILDDIR)) And do the same for the other places where kernel-doc is called: include/drm/Makefile: $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ scripts/Makefile.build: cmd_checkdoc = $(srctree)/scripts/kernel-doc -none $(KDOCFLAGS) \ scripts/find-unused-docs.sh: str=$(scripts/kernel-doc -export "$file" 2>/dev/null) Comments? Regards, Mauro
On Tue, Apr 15, 2025 at 04:40:34PM +0800, Mauro Carvalho Chehab wrote: > Em Tue, 15 Apr 2025 11:19:26 +0300 > Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > > On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > > > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > > >> > >> > > > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > > >> > >> > second series I sent on the top of it. > > > > >> > >> > > > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > > > >> > >> thanks for doing this! > > > > >> > >> > > > > >> > >> I'm going to hold off on other documentation patches for a day or two > > > > >> > >> just in case anything turns up. But it looks awfully good. > > > > >> > > > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > > > >> > > > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > >> > > > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > > > >> > > > > > >> > We can also tell CPython to not create that directory at all. I'll run > > > > >> > some tests to see what the effect is on the documentation build times; > > > > >> > I'm guessing it will not be huge... > > > > >> > > > > >> I do not build documentation at all, it's just a regular code build that leaves > > > > >> tree dirty. > > > > >> > > > > >> $ python3 --version > > > > >> Python 3.13.2 > > > > >> > > > > >> It's standard Debian testing distribution, no customisation in the code. > > > > >> > > > > >> To reproduce. > > > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > > > >> 2) The following snippet in shell shows the issue > > > > >> > > > > >> $ git clean -xdf > > > > >> $ git status --ignored > > > > >> On branch ... > > > > >> nothing to commit, working tree clean > > > > >> > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > >> make[1]: Entering directory '...' > > > > >> GEN Makefile > > > > >> DESCEND objtool > > > > >> CALL .../scripts/checksyscalls.sh > > > > >> INSTALL libsubcmd_headers > > > > >> .pylintrc: warning: ignored by one of the .gitignore files > > > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > > > >> make[1]: Leaving directory '...' > > > > >> > > > > >> $ touch drivers/gpio/gpiolib-acpi.c > > > > >> > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > >> make[1]: Entering directory '...' > > > > >> GEN Makefile > > > > >> DESCEND objtool > > > > >> CALL .../scripts/checksyscalls.sh > > > > >> INSTALL libsubcmd_headers > > > > >> ... > > > > >> OBJCOPY arch/x86/boot/setup.bin > > > > >> BUILD arch/x86/boot/bzImage > > > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > > > >> make[1]: Leaving directory '...' > > > > >> > > > > >> $ git status --ignored > > > > >> On branch ... > > > > >> Untracked files: > > > > >> (use "git add <file>..." to include in what will be committed) > > > > >> scripts/lib/kdoc/__pycache__/ > > > > >> > > > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > > > fully clean build. Still the same issue. > > > > > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > > > have the kernel to be built actually. > > > > > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > > > > > Thanks for the clarification, so we know that it runs and we know that it has > > > an issue. > > > > Ideal solution what would I expect is that the cache folder should respect > > the given O=... argument, or disabled at all (but I don't think the latter > > is what we want as it may slow down the build). > > From: > https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6 > and: > https://peps.python.org/pep-3147/ > > It sounds that Python 3.8 and above have a way to specify the cache > location, via PYTHONPYCACHEPREFIX env var, and via "-X pycache_prefix=path". > > As the current minimal Python version is 3.9, we can safely use it. > > So, maybe this would work: > > make O="../out" PYTHONPYCACHEPREFIX="../out" > > or a variant of it: > > PYTHONPYCACHEPREFIX="../out" make O="../out" > > If this works, we can adjust the building system to fill PYTHONPYCACHEPREFIX > env var when O= is used. It works, the problem is that it should be automatically assigned to the respective folder, so when compiling kdoc, it should be actually $O/scripts/lib/kdoc/__pycache__ and so on for _each_ of the python code.
On Tue, Apr 15, 2025 at 04:51:02PM +0800, Mauro Carvalho Chehab wrote: > Em Tue, 15 Apr 2025 16:40:34 +0800 > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu: > > Em Tue, 15 Apr 2025 11:19:26 +0300 > > Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > > > On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > > > > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > > > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > > > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > > > >> > >> > > > > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > > > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > > > >> > >> > second series I sent on the top of it. > > > > > >> > >> > > > > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > > > > >> > >> thanks for doing this! > > > > > >> > >> > > > > > >> > >> I'm going to hold off on other documentation patches for a day or two > > > > > >> > >> just in case anything turns up. But it looks awfully good. > > > > > >> > > > > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > > > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > > > > >> > > > > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > > >> > > > > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > > > > >> > > > > > > >> > We can also tell CPython to not create that directory at all. I'll run > > > > > >> > some tests to see what the effect is on the documentation build times; > > > > > >> > I'm guessing it will not be huge... > > > > > >> > > > > > >> I do not build documentation at all, it's just a regular code build that leaves > > > > > >> tree dirty. > > > > > >> > > > > > >> $ python3 --version > > > > > >> Python 3.13.2 > > > > > >> > > > > > >> It's standard Debian testing distribution, no customisation in the code. > > > > > >> > > > > > >> To reproduce. > > > > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > > > > >> 2) The following snippet in shell shows the issue > > > > > >> > > > > > >> $ git clean -xdf > > > > > >> $ git status --ignored > > > > > >> On branch ... > > > > > >> nothing to commit, working tree clean > > > > > >> > > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > > >> make[1]: Entering directory '...' > > > > > >> GEN Makefile > > > > > >> DESCEND objtool > > > > > >> CALL .../scripts/checksyscalls.sh > > > > > >> INSTALL libsubcmd_headers > > > > > >> .pylintrc: warning: ignored by one of the .gitignore files > > > > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > > > > >> make[1]: Leaving directory '...' > > > > > >> > > > > > >> $ touch drivers/gpio/gpiolib-acpi.c > > > > > >> > > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > > >> make[1]: Entering directory '...' > > > > > >> GEN Makefile > > > > > >> DESCEND objtool > > > > > >> CALL .../scripts/checksyscalls.sh > > > > > >> INSTALL libsubcmd_headers > > > > > >> ... > > > > > >> OBJCOPY arch/x86/boot/setup.bin > > > > > >> BUILD arch/x86/boot/bzImage > > > > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > > > > >> make[1]: Leaving directory '...' > > > > > >> > > > > > >> $ git status --ignored > > > > > >> On branch ... > > > > > >> Untracked files: > > > > > >> (use "git add <file>..." to include in what will be committed) > > > > > >> scripts/lib/kdoc/__pycache__/ > > > > > >> > > > > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > > > > fully clean build. Still the same issue. > > > > > > > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > > > > have the kernel to be built actually. > > > > > > > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > > > > > > > Thanks for the clarification, so we know that it runs and we know that it has > > > > an issue. > > > > > > Ideal solution what would I expect is that the cache folder should respect > > > the given O=... argument, or disabled at all (but I don't think the latter > > > is what we want as it may slow down the build). > > > > From: > > https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6 > > and: > > https://peps.python.org/pep-3147/ > > > > It sounds that Python 3.8 and above have a way to specify the cache > > location, via PYTHONPYCACHEPREFIX env var, and via "-X pycache_prefix=path". > > > > As the current minimal Python version is 3.9, we can safely use it. > > > > So, maybe this would work: > > > > make O="../out" PYTHONPYCACHEPREFIX="../out" > > > > or a variant of it: > > > > PYTHONPYCACHEPREFIX="../out" make O="../out" > > > > If this works, we can adjust the building system to fill PYTHONPYCACHEPREFIX > > env var when O= is used. > > That's interesting... Sphinx is already called with PYTHONDONTWRITEBYTECODE. > From Documentation/Makefile: > > quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) > cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media $2 && \ > PYTHONDONTWRITEBYTECODE=1 \ > ... > > It seems that the issue happens only when W=1 is used and kernel-doc > is called outside Sphinx. > > Anyway, IMHO, the best would be to change the above to: > > PYTHONPYCACHEPREFIX=$(abspath $(BUILDDIR)) > > And do the same for the other places where kernel-doc is called: > > include/drm/Makefile: $(srctree)/scripts/kernel-doc -none $(if $(CONFIG_WERROR)$(CONFIG_DRM_WERROR),-Werror) $<; \ > scripts/Makefile.build: cmd_checkdoc = $(srctree)/scripts/kernel-doc -none $(KDOCFLAGS) \ > scripts/find-unused-docs.sh: str=$(scripts/kernel-doc -export "$file" 2>/dev/null) > > Comments? I would like that, but it should be properly formed, because somewhere we drop the path in the source tree and if $O is used as is, it becomes _the_ pycache folder!
On Tue, Apr 15, 2025 at 12:51:38PM +0300, Andy Shevchenko wrote: > On Tue, Apr 15, 2025 at 04:40:34PM +0800, Mauro Carvalho Chehab wrote: > > Em Tue, 15 Apr 2025 11:19:26 +0300 > > Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > > > On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > > > > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > > > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > > > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > > > >> > >> > > > > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > > > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > > > >> > >> > second series I sent on the top of it. > > > > > >> > >> > > > > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > > > > >> > >> thanks for doing this! > > > > > >> > >> > > > > > >> > >> I'm going to hold off on other documentation patches for a day or two > > > > > >> > >> just in case anything turns up. But it looks awfully good. > > > > > >> > > > > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > > > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > > > > >> > > > > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > > >> > > > > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > > > > >> > > > > > > >> > We can also tell CPython to not create that directory at all. I'll run > > > > > >> > some tests to see what the effect is on the documentation build times; > > > > > >> > I'm guessing it will not be huge... > > > > > >> > > > > > >> I do not build documentation at all, it's just a regular code build that leaves > > > > > >> tree dirty. > > > > > >> > > > > > >> $ python3 --version > > > > > >> Python 3.13.2 > > > > > >> > > > > > >> It's standard Debian testing distribution, no customisation in the code. > > > > > >> > > > > > >> To reproduce. > > > > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > > > > >> 2) The following snippet in shell shows the issue > > > > > >> > > > > > >> $ git clean -xdf > > > > > >> $ git status --ignored > > > > > >> On branch ... > > > > > >> nothing to commit, working tree clean > > > > > >> > > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > > >> make[1]: Entering directory '...' > > > > > >> GEN Makefile > > > > > >> DESCEND objtool > > > > > >> CALL .../scripts/checksyscalls.sh > > > > > >> INSTALL libsubcmd_headers > > > > > >> .pylintrc: warning: ignored by one of the .gitignore files > > > > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > > > > >> make[1]: Leaving directory '...' > > > > > >> > > > > > >> $ touch drivers/gpio/gpiolib-acpi.c > > > > > >> > > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > > >> make[1]: Entering directory '...' > > > > > >> GEN Makefile > > > > > >> DESCEND objtool > > > > > >> CALL .../scripts/checksyscalls.sh > > > > > >> INSTALL libsubcmd_headers > > > > > >> ... > > > > > >> OBJCOPY arch/x86/boot/setup.bin > > > > > >> BUILD arch/x86/boot/bzImage > > > > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > > > > >> make[1]: Leaving directory '...' > > > > > >> > > > > > >> $ git status --ignored > > > > > >> On branch ... > > > > > >> Untracked files: > > > > > >> (use "git add <file>..." to include in what will be committed) > > > > > >> scripts/lib/kdoc/__pycache__/ > > > > > >> > > > > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > > > > fully clean build. Still the same issue. > > > > > > > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > > > > have the kernel to be built actually. > > > > > > > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > > > > > > > Thanks for the clarification, so we know that it runs and we know that it has > > > > an issue. > > > > > > Ideal solution what would I expect is that the cache folder should respect > > > the given O=... argument, or disabled at all (but I don't think the latter > > > is what we want as it may slow down the build). > > > > From: > > https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6 > > and: > > https://peps.python.org/pep-3147/ > > > > It sounds that Python 3.8 and above have a way to specify the cache > > location, via PYTHONPYCACHEPREFIX env var, and via "-X pycache_prefix=path". > > > > As the current minimal Python version is 3.9, we can safely use it. > > > > So, maybe this would work: > > > > make O="../out" PYTHONPYCACHEPREFIX="../out" > > > > or a variant of it: > > > > PYTHONPYCACHEPREFIX="../out" make O="../out" > > > > If this works, we can adjust the building system to fill PYTHONPYCACHEPREFIX > > env var when O= is used. > > It works, the problem is that it should be automatically assigned to the > respective folder, so when compiling kdoc, it should be actually > > $O/scripts/lib/kdoc/__pycache__ > > and so on for _each_ of the python code. So, the bottom line, can we just disable it for a quick fix and when a proper solution comes, it will redo that?
Em Tue, 15 Apr 2025 12:54:12 +0300 Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > On Tue, Apr 15, 2025 at 12:51:38PM +0300, Andy Shevchenko wrote: > > On Tue, Apr 15, 2025 at 04:40:34PM +0800, Mauro Carvalho Chehab wrote: > > > Em Tue, 15 Apr 2025 11:19:26 +0300 > > > Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: > > > > On Tue, Apr 15, 2025 at 11:17:12AM +0300, Andy Shevchenko wrote: > > > > > On Tue, Apr 15, 2025 at 10:49:29AM +0300, Jani Nikula wrote: > > > > > > On Tue, 15 Apr 2025, Andy Shevchenko <andriy.shevchenko@intel.com> wrote: > > > > > > > On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote: > > > > > > >> On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote: > > > > > > >> > Andy Shevchenko <andriy.shevchenko@intel.com> writes: > > > > > > >> > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote: > > > > > > >> > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > > > > > > >> > >> > > > > > > >> > >> > This changeset contains the kernel-doc.py script to replace the verable > > > > > > >> > >> > kernel-doc originally written in Perl. It replaces the first version and the > > > > > > >> > >> > second series I sent on the top of it. > > > > > > >> > >> > > > > > > >> > >> OK, I've applied it, looked at the (minimal) changes in output, and > > > > > > >> > >> concluded that it's good - all this stuff is now in docs-next. Many > > > > > > >> > >> thanks for doing this! > > > > > > >> > >> > > > > > > >> > >> I'm going to hold off on other documentation patches for a day or two > > > > > > >> > >> just in case anything turns up. But it looks awfully good. > > > > > > >> > > > > > > > > >> > > This started well, until it becomes a scripts/lib/kdoc. > > > > > > >> > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave > > > > > > >> > > "disgusting turd" )as said by Linus) in the clean tree. > > > > > > >> > > > > > > > > >> > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution. > > > > > > >> > > > > > > > >> > If nothing else, "make cleandocs" should clean it up, certainly. > > > > > > >> > > > > > > > >> > We can also tell CPython to not create that directory at all. I'll run > > > > > > >> > some tests to see what the effect is on the documentation build times; > > > > > > >> > I'm guessing it will not be huge... > > > > > > >> > > > > > > >> I do not build documentation at all, it's just a regular code build that leaves > > > > > > >> tree dirty. > > > > > > >> > > > > > > >> $ python3 --version > > > > > > >> Python 3.13.2 > > > > > > >> > > > > > > >> It's standard Debian testing distribution, no customisation in the code. > > > > > > >> > > > > > > >> To reproduce. > > > > > > >> 1) I have just done a new build to reduce the churn, so, running make again does nothing; > > > > > > >> 2) The following snippet in shell shows the issue > > > > > > >> > > > > > > >> $ git clean -xdf > > > > > > >> $ git status --ignored > > > > > > >> On branch ... > > > > > > >> nothing to commit, working tree clean > > > > > > >> > > > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > > > >> make[1]: Entering directory '...' > > > > > > >> GEN Makefile > > > > > > >> DESCEND objtool > > > > > > >> CALL .../scripts/checksyscalls.sh > > > > > > >> INSTALL libsubcmd_headers > > > > > > >> .pylintrc: warning: ignored by one of the .gitignore files > > > > > > >> Kernel: arch/x86/boot/bzImage is ready (#23) > > > > > > >> make[1]: Leaving directory '...' > > > > > > >> > > > > > > >> $ touch drivers/gpio/gpiolib-acpi.c > > > > > > >> > > > > > > >> $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64 > > > > > > >> make[1]: Entering directory '...' > > > > > > >> GEN Makefile > > > > > > >> DESCEND objtool > > > > > > >> CALL .../scripts/checksyscalls.sh > > > > > > >> INSTALL libsubcmd_headers > > > > > > >> ... > > > > > > >> OBJCOPY arch/x86/boot/setup.bin > > > > > > >> BUILD arch/x86/boot/bzImage > > > > > > >> Kernel: arch/x86/boot/bzImage is ready (#24) > > > > > > >> make[1]: Leaving directory '...' > > > > > > >> > > > > > > >> $ git status --ignored > > > > > > >> On branch ... > > > > > > >> Untracked files: > > > > > > >> (use "git add <file>..." to include in what will be committed) > > > > > > >> scripts/lib/kdoc/__pycache__/ > > > > > > >> > > > > > > >> nothing added to commit but untracked files present (use "git add" to track) > > > > > > > > > > > > > > FWIW, I repeated this with removing the O=.../out folder completely, so it's > > > > > > > fully clean build. Still the same issue. > > > > > > > > > > > > > > And it appears at the very beginning of the build. You don't need to wait to > > > > > > > have the kernel to be built actually. > > > > > > > > > > > > kernel-doc gets run on source files for W=1 builds. See Makefile.build. > > > > > > > > > > Thanks for the clarification, so we know that it runs and we know that it has > > > > > an issue. > > > > > > > > Ideal solution what would I expect is that the cache folder should respect > > > > the given O=... argument, or disabled at all (but I don't think the latter > > > > is what we want as it may slow down the build). > > > > > > From: > > > https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6 > > > and: > > > https://peps.python.org/pep-3147/ > > > > > > It sounds that Python 3.8 and above have a way to specify the cache > > > location, via PYTHONPYCACHEPREFIX env var, and via "-X pycache_prefix=path". > > > > > > As the current minimal Python version is 3.9, we can safely use it. > > > > > > So, maybe this would work: > > > > > > make O="../out" PYTHONPYCACHEPREFIX="../out" > > > > > > or a variant of it: > > > > > > PYTHONPYCACHEPREFIX="../out" make O="../out" > > > > > > If this works, we can adjust the building system to fill PYTHONPYCACHEPREFIX > > > env var when O= is used. > > > > It works, Good! > > the problem is that it should be automatically assigned to the > > respective folder, so when compiling kdoc, it should be actually > > > > $O/scripts/lib/kdoc/__pycache__ > > > > and so on for _each_ of the python code. Yeah, agreed. We need to think on a more generic solution though, as we also may have scripts/lib/abi/__pycache__ if one runs get_abi.pl, and, in the future, we may have more. Not sure how hard/easy would be to do that, though. > So, the bottom line, can we just disable it for a quick fix and when a proper > solution comes, it will redo that? Agreed, this sounds to be the best approach. I'll try to craft a patch along the week to add PYTHONDONTWRITEBYTECODE=1 to the places where kernel-doc is called. Regards, Mauro
On Tue, Apr 15, 2025 at 06:06:31PM +0800, Mauro Carvalho Chehab wrote: > Em Tue, 15 Apr 2025 12:54:12 +0300 > Andy Shevchenko <andriy.shevchenko@intel.com> escreveu: ... > I'll try to craft a patch along the week to add > PYTHONDONTWRITEBYTECODE=1 to the places where kernel-doc > is called. Cc me and I'll be happy to test. Thank you!
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes: > I'll try to craft a patch along the week to add > PYTHONDONTWRITEBYTECODE=1 to the places where kernel-doc > is called. This may really be all we need. It will be interesting to do some build-time tests; I don't really see this as making much of a difference. Thanks, jon