| Message ID | 20250404-kconfig-docs-v1-0-4c3155d4ba44@collabora.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add Kconfig pages and cross-references to Documentation | expand |
Nícolas F. R. A. Prado <nfraprado@collabora.com> writes: > This series adds Kconfig pages (patch 1) to the Documentation, and > automarkups CONFIG_* text as cross-references to those pages (patch 2). > > There is a huge change in build time with this series, so we'd either > have to so some optimization and/or put this behind a flag in make so it > is only generated when desired (for instance for the online > documentation): > > (On an XPS 13 9300) > > Before: > > real 6m43.576s > user 23m32.611s > sys 1m48.220s > > After: > > real 11m56.845s > user 47m40.528s > sys 2m27.382s > > There are also some issues that were solved in ad-hoc ways (eg the > sphinx warnings due to repeated Kconfigs, by embedding the list of > repeated configs in the script). Hence the RFC. I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong. I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case. I'll try to look closer, but I'll remain a bit distracted for a little while yet. Thanks, jon
On Fri, Apr 04, 2025 at 08:31:35AM -0600, Jonathan Corbet wrote: > Nícolas F. R. A. Prado <nfraprado@collabora.com> writes: > > > This series adds Kconfig pages (patch 1) to the Documentation, and > > automarkups CONFIG_* text as cross-references to those pages (patch 2). > > > > There is a huge change in build time with this series, so we'd either > > have to so some optimization and/or put this behind a flag in make so it > > is only generated when desired (for instance for the online > > documentation): > > > > (On an XPS 13 9300) > > > > Before: > > > > real 6m43.576s > > user 23m32.611s > > sys 1m48.220s > > > > After: > > > > real 11m56.845s > > user 47m40.528s > > sys 2m27.382s > > > > There are also some issues that were solved in ad-hoc ways (eg the > > sphinx warnings due to repeated Kconfigs, by embedding the list of > > repeated configs in the script). Hence the RFC. > > I'm still digging out from LSFMM, so have only glanced at this ... I can > see the appeal of doing this, but nearly doubling the docs build time > really isn't going to fly. Have you looked to see what is taking all of > that time? The idea that it takes as long to process KConfig entries as > it does to build the entire rest of the docs seems ... a bit wrong. I have not yet. Thought I'd get some feedback before looking into the performance. But I agree with the sentiment. > > I wonder what it would take to create a Sphinx extension that would > simply walk the source tree and slurp up the KConfig entries directly? > That would be nicer than adding a separate script in any case. That is what is currently done for the ABI, AFAIK, so definitely seems doable. The key difference between the ABI approach and this here, is that my goal was to reflect the Kconfig file hierarchy in the Documentation. So each Kconfig file gets its own documentation page, while the ABI approach collects the contents of all ABI files into just a few documentation pages (stable, testing, etc). (So there's a non-constant number of .rst files, which means they have to be generated and can't be a sphinx plugin in this approach). I went for this approach because the filesystem hierarchy seemed the most logical way to group the Kconfig symbols. Also Kconfig files have directives like 'menu' that should be present in the documentation in the same order they appear in the file to fully describe dependencies of the symbols, and having all of that in the same page seems like it would be confusing. But given the potential benefits it's worth a try for sure. Now that I think about it, seems quite likely that a lot of the time spent comes from creating a subshell and running the script for every Kconfig file. So making a single script or sphinx extension that itself handles iterating over all the files would likely greatly reduce the run time. I'll test that. Thanks, Nícolas > > I'll try to look closer, but I'll remain a bit distracted for a little > while yet. > > Thanks, > > jon
Em Fri, 4 Apr 2025 12:24:27 -0400 Nícolas F. R. A. Prado <nfraprado@collabora.com> escreveu: > On Fri, Apr 04, 2025 at 08:31:35AM -0600, Jonathan Corbet wrote: > > Nícolas F. R. A. Prado <nfraprado@collabora.com> writes: > > > > > This series adds Kconfig pages (patch 1) to the Documentation, and > > > automarkups CONFIG_* text as cross-references to those pages (patch 2). > > > > > > There is a huge change in build time with this series, so we'd either > > > have to so some optimization and/or put this behind a flag in make so it > > > is only generated when desired (for instance for the online > > > documentation): > > > > > > (On an XPS 13 9300) > > > > > > Before: > > > > > > real 6m43.576s > > > user 23m32.611s > > > sys 1m48.220s > > > > > > After: > > > > > > real 11m56.845s > > > user 47m40.528s > > > sys 2m27.382s > > > > > > There are also some issues that were solved in ad-hoc ways (eg the > > > sphinx warnings due to repeated Kconfigs, by embedding the list of > > > repeated configs in the script). Hence the RFC. > > > > I'm still digging out from LSFMM, so have only glanced at this ... I can > > see the appeal of doing this, but nearly doubling the docs build time > > really isn't going to fly. Have you looked to see what is taking all of > > that time? The idea that it takes as long to process KConfig entries as > > it does to build the entire rest of the docs seems ... a bit wrong. > > I have not yet. Thought I'd get some feedback before looking into the > performance. But I agree with the sentiment. My feeling is that the issue is using :glob" and a lot of wildcards inside Sphinx. Instead, you should use something similar to what I've done to get *.[ch] for the new kernel-doc.py implementation. Placing it as an extension on a similar way to what i did with get_abi.py would likely help as well. > > I wonder what it would take to create a Sphinx extension that would > > simply walk the source tree and slurp up the KConfig entries directly? > > That would be nicer than adding a separate script in any case. > > That is what is currently done for the ABI, AFAIK, so definitely seems doable. Yes, doing that via an extension is doable. If done right, it can also be fast. > The key difference between the ABI approach and this here, is that my goal was > to reflect the Kconfig file hierarchy in the Documentation. So each Kconfig > file gets its own documentation page, while the ABI approach collects the > contents of all ABI files into just a few documentation pages (stable, testing, > etc). (So there's a non-constant number of .rst files, which means they have to > be generated and can't be a sphinx plugin in this approach). Actually, get-api.py (the new version, merged for 6.15) generates a dict just once. Then, Sphinx rst files filters part of the doc, but I see your point: for every entry, we would need a .rst file if we follow the same approach. That's said, it may have a way to tell Sphinx to threat Kconfig files on a similar way it handles ".txt" and ".rst" files. Something like the extension to handle markdown works: https://www.sphinx-doc.org/en/master/usage/markdown.html Another alternative would be to use: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-include_patterns but this would require Sphinx 5.1, which is above our current minimal version. That's said, nothing prevents to only enable generating such documentatation if the Sphinx version supports it. > I went for this approach because the filesystem hierarchy seemed the most > logical way to group the Kconfig symbols. Also Kconfig files have directives like > 'menu' that should be present in the documentation in the same order they appear > in the file to fully describe dependencies of the symbols, and having all of > that in the same page seems like it would be confusing. But given the potential > benefits it's worth a try for sure. > > Now that I think about it, seems quite likely that a lot of the time spent comes > from creating a subshell and running the script for every Kconfig file. So > making a single script or sphinx extension that itself handles iterating over > all the files would likely greatly reduce the run time. I'll test that. > > Thanks, > Nícolas > > > > > I'll try to look closer, but I'll remain a bit distracted for a little > > while yet. > > > > Thanks, > > > > jon
This series adds Kconfig pages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2). There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation): (On an XPS 13 9300) Before: real 6m43.576s user 23m32.611s sys 1m48.220s After: real 11m56.845s user 47m40.528s sys 2m27.382s There are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC. Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com> --- Nícolas F. R. A. Prado (2): docs: Add documentation generation for Kconfig symbols docs: automarkup: Cross-reference CONFIG_ symbols Documentation/.gitignore | 2 + Documentation/Config/index.rst | 17 ++ Documentation/Makefile | 12 +- Documentation/kbuild/index.rst | 2 + Documentation/sphinx/automarkup.py | 36 +++- scripts/kconfig2rst.py | 336 +++++++++++++++++++++++++++++++++++++ 6 files changed, 403 insertions(+), 2 deletions(-) --- base-commit: 405e2241def89c88f008dcb899eb5b6d4be8b43c change-id: 20250403-kconfig-docs-19d90ba266dd Best regards,