| Message ID | 1435331337-22924-1-git-send-email-danilo.cesar@collabora.co.uk (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Fri, Jun 26, 2015 at 12:08:57PM -0300, Danilo Cesar Lemes de Paula wrote: > Functions, Structs and Parameters definitions on kernel documentation > are pure cosmetic, it only highlights the element. > > To ease the navigation in the documentation we should use <links> inside > those tags so readers can easily jump between methods directly. > > This was discussed in 2014[1] and is implemented by getting a list > of <refentries> from the DocBook XML to generate a database. Then it looks > for <function>,<structnames> and <paramdef> tags that matches the ones in > the database. As it only links existent references, no broken links are > added. > > [1] - lists.freedesktop.org/archives/dri-devel/2014-August/065404.html > > Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> > Cc: Randy Dunlap <rdunlap@infradead.org> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch> > Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com> > Cc: Jonathan Corbet <corbet@lwn.net> > Cc: Herbert Xu <herbert@gondor.apana.org.au> > Cc: Stephan Mueller <smueller@chronox.de> > Cc: Michal Marek <mmarek@suse.cz> > Cc: linux-kernel@vger.kernel.org > Cc: linux-doc@vger.kernel.org > Cc: intel-gfx <intel-gfx@lists.freedesktop.org> > Cc: dri-devel <dri-devel@lists.freedesktop.org> I'm using this now to generate the drm docbook and I think it's a great improvement for them. No r-b since I can't really perl. Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch> > --- > To understand a bit more of what this patch is trying to acomplish you can find > two examples of the old and new htmldocs outputs: > OLD: https://people.collabora.com/~danilo/intel/Documentation.old/DocBook/drm/API-drm-crtc-vblank-on.html > NEW: https://people.collabora.com/~danilo/intel/Documentation.new/DocBook/drm/API-drm-crtc-vblank-on.html > > Documentation/DocBook/Makefile | 34 +++++--- > scripts/kernel-doc-xml-ref | 181 +++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 202 insertions(+), 13 deletions(-) > create mode 100755 scripts/kernel-doc-xml-ref > > diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile > index b6a6a2e..8aea45a 100644 > --- a/Documentation/DocBook/Makefile > +++ b/Documentation/DocBook/Makefile > @@ -64,8 +64,9 @@ installmandocs: mandocs > > ### > #External programs used > -KERNELDOC = $(srctree)/scripts/kernel-doc > -DOCPROC = $(objtree)/scripts/docproc > +KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref > +KERNELDOC = $(srctree)/scripts/kernel-doc > +DOCPROC = $(objtree)/scripts/docproc > > XMLTOFLAGS = -m $(srctree)/$(src)/stylesheet.xsl > XMLTOFLAGS += --skip-validation > @@ -89,7 +90,7 @@ define rule_docproc > ) > $(dir $@).$(notdir $@).cmd > endef > > -%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) FORCE > +%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE > $(call if_changed_rule,docproc) > > # Tell kbuild to always build the programs > @@ -139,8 +140,12 @@ quiet_cmd_db2html = HTML $@ > cmd_db2html = xmlto html $(XMLTOFLAGS) -o $(patsubst %.html,%,$@) $< && \ > echo '<a HREF="$(patsubst %.html,%,$(notdir $@))/index.html"> \ > $(patsubst %.html,%,$(notdir $@))</a><p>' > $@ > +%.aux.xml: %.xml > + @rm -rf $@ > + (cat $< | egrep "^<refentry id" | egrep -o "\".*\"" | cut -f 2 -d \" > $<.db) > + $(KERNELDOCXMLREF) -db $<.db $< > $@ > > -%.html: %.xml > +%.html: %.aux.xml > @(which xmlto > /dev/null 2>&1) || \ > (echo "*** You need to install xmlto ***"; \ > exit 1) > @@ -209,15 +214,18 @@ dochelp: > ### > # Temporary files left by various tools > clean-files := $(DOCBOOKS) \ > - $(patsubst %.xml, %.dvi, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.aux, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.tex, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.log, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.out, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.ps, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.pdf, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.html, $(DOCBOOKS)) \ > - $(patsubst %.xml, %.9, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.dvi, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.aux, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.tex, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.log, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.out, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.ps, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.pdf, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.html, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.9, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.aux.xml, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.xml.db, $(DOCBOOKS)) \ > + $(patsubst %.xml, %.xml, $(DOCBOOKS)) \ > $(index) > > clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man > diff --git a/scripts/kernel-doc-xml-ref b/scripts/kernel-doc-xml-ref > new file mode 100755 > index 0000000..c61cdad > --- /dev/null > +++ b/scripts/kernel-doc-xml-ref > @@ -0,0 +1,181 @@ > +#!/usr/bin/perl -w > + > +use strict; > + > +## Copyright (C) 2015 Intel Corporation ## > +# ## > +## This software falls under the GNU General Public License. ## > +## Please read the COPYING file for more information ## > +# > +# > +# This software reads a XML file and a list of valid interal > +# references to replace Docbook tags with links. > +# > +# usage: > +# kernel-doc-xml-ref -db filename > +# xml filename > outputfile > + > +# read arguments > +if ($#ARGV != 2) { > + usage(); > +} > + > +#Holds the database filename > +my $databasefile; > +my @database; > + > +#holds the inputfile > +my $inputfile; > +my $errors = 0; > + > +my %highlights = ( > + "<function>(.*?)</function>", > + "\"<function>\" . convert_function(\$1, \$line) . \"</function>\"", > + "<structname>(.*?)</structname>", > + "\"<structname>\" . convert_struct(\$1) . \"</structname>\"", > + "<funcdef>(.*?)<function>(.*?)</function></funcdef>", > + "\"<funcdef>\" . convert_param(\$1) . \"<function>\$2</function></funcdef>\"", > + "<paramdef>(.*?)<parameter>(.*?)</parameter></paramdef>", > + "\"<paramdef>\" . convert_param(\$1) . \"<parameter>\$2</parameter></paramdef>\""); > + > +while($ARGV[0] =~ m/^-(.*)/) { > + my $cmd = shift @ARGV; > + if ($cmd eq "-db") { > + $databasefile = shift @ARGV > + } > +} > +$inputfile = shift @ARGV; > + > +sub open_database { > + open my $handle, '<', $databasefile; > + chomp(my @lines = <$handle>); > + close $handle; > + > + @database = @lines; > +} > + > +sub process_file { > + open_database(); > + > + my $dohighlight; > + foreach my $pattern (keys %highlights) { > + $dohighlight .= "\$line =~ s:$pattern:$highlights{$pattern}:eg;\n"; > + } > + > + open(FILE, $inputfile) or die("Could not open $inputfile"); > + foreach my $line (<FILE>) { > + eval $dohighlight; > + print $line; > + } > +} > + > +sub trim($_) > +{ > + my $str = $_[0]; > + $str =~ s/^\s+|\s+$//g; > + return $str > +} > + > +sub has_key_defined($_) > +{ > + if ( grep( /^$_[0]$/, @database)) { > + return 1; > + } > + return 0; > +} > + > +sub convert_function($_) > +{ > + my $arg = $_[0]; > + my $key = $_[0]; > + > + my $line = $_[1]; > + > + $key = trim($key); > + > + $key =~ s/[^A-Za-z0-9]/-/g; > + $key = "API-" . $key; > + > + # We shouldn't add links to <funcdef> prototype > + if (!has_key_defined($key) || $line =~ m/\s+<funcdef/i) { > + return $arg; > + } > + > + my $head = $arg; > + my $tail = ""; > + if ($arg =~ /(.*?)( ?)$/) { > + $head = $1; > + $tail = $2; > + } > + return "<link linkend=\"$key\">$head</link>$tail"; > +} > +sub convert_struct($_) > +{ > + my $arg = $_[0]; > + my $key = $_[0]; > + $key =~ s/(struct )?(\w)/$2/g; > + $key =~ s/[^A-Za-z0-9]/-/g; > + $key = "API-struct-" . $key; > + > + if (!has_key_defined($key)) { > + return $arg; > + } > + > + my ($head, $tail) = split_pointer($arg); > + return "<link linkend=\"$key\">$head</link>$tail"; > +} > + > +sub split_pointer($_) > +{ > + my $arg = $_[0]; > + if ($arg =~ /(.*?)( ?\* ?)/) { > + return ($1, $2); > + } > + return ($arg, ""); > +} > + > +sub convert_param($_) > +{ > + my $type = $_[0]; > + my $keyname = convert_key_name($type); > + > + if (!has_key_defined($keyname)) { > + return $type; > + } > + > + my ($head, $tail) = split_pointer($type); > + return "<link linkend=\"$keyname\">$head</link>$tail"; > + > +} > + > +sub convert_key_name($_) > +{ > + #Pattern $2 is optional and might be uninitialized > + no warnings 'uninitialized'; > + > + my $str = $_[0]; > + $str =~ s/(const|static)? ?(struct)? ?([a-zA-Z0-9_]+) ?(\*|&)?/$2 $3/g ; > + > + # trim > + $str =~ s/^\s+|\s+$//g; > + > + # spaces and _ to - > + $str =~ s/[^A-Za-z0-9]/-/g; > + > + return "API-" . $str; > +} > + > +sub usage { > + print "Usage: $0 -db database filename ]\n"; > + print " xml source file(s) > outputfile\n"; > + exit 1; > +} > + > +# starting point > +process_file(); > + > +if ($errors) { > + print STDERR "$errors errors\n"; > +} > + > +exit($errors); > -- > 2.1.4 >
On Fri, 26 Jun 2015 12:08:57 -0300 Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote: > To ease the navigation in the documentation we should use <links> inside > those tags so readers can easily jump between methods directly. > > This was discussed in 2014[1] and is implemented by getting a list > of <refentries> from the DocBook XML to generate a database. Then it looks > for <function>,<structnames> and <paramdef> tags that matches the ones in > the database. As it only links existent references, no broken links are > added. So I put a lot more time into this today than I really had available. I think it's cool stuff, and we definitely want it. But can I ask for one more pass? In particular: - It makes the docs build a lot more noisy, that would be nice to fix. - A bit more documentation in the script would be nice. It also is happy to run with silly arguments; a detail since nobody will run it directly, but still... - Most importantly, it breaks "make htmldocs"; in particular, vast amounts of error spew results when it gets around to media_api.html. I spent a while trying to figure out what was going on but didn't come up with anything conclusive; my suspicion is that it has to do with the separate makefile in Documentation/DocBook/media/. Thanks, jon
On 07/09/2015 08:56 PM, Jonathan Corbet wrote: > On Fri, 26 Jun 2015 12:08:57 -0300 > Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote: > >> To ease the navigation in the documentation we should use <links> inside >> those tags so readers can easily jump between methods directly. >> >> This was discussed in 2014[1] and is implemented by getting a list >> of <refentries> from the DocBook XML to generate a database. Then it looks >> for <function>,<structnames> and <paramdef> tags that matches the ones in >> the database. As it only links existent references, no broken links are >> added. > > So I put a lot more time into this today than I really had available. I Thanks, I really appreciate that. > think it's cool stuff, and we definitely want it. But can I ask for one > more pass? In particular: > > - It makes the docs build a lot more noisy, that would be nice to fix. Fair enough. It was showing all the commands. I did change that to a fancy "XMLREF Documentation/DocBook/FooBar.xml" message. > > - A bit more documentation in the script would be nice. It also is happy > to run with silly arguments; a detail since nobody will run it > directly, but still... I did improve the documentation and also fixed the silly argument thing. > > - Most importantly, it breaks "make htmldocs"; in particular, vast > amounts of error spew results when it gets around to media_api.html. I > spent a while trying to figure out what was going on but didn't come up > with anything conclusive; my suspicion is that it has to do with the > separate makefile in Documentation/DocBook/media/. I'm not sure about this. media-api is spitting lots of warnings with or without the documentation patch. I compared the number of files and they're the same (excepting those auxiliary db files). For me, both builds actually spits 2825 lines on STDERR... I also did a smoke check in the media-api html documentation and it looks fine. Would you mind to check again if it happens with the v3 of my patch I'm sending next? Danilo > > Thanks, > > jon >
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index b6a6a2e..8aea45a 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -64,8 +64,9 @@ installmandocs: mandocs ### #External programs used -KERNELDOC = $(srctree)/scripts/kernel-doc -DOCPROC = $(objtree)/scripts/docproc +KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref +KERNELDOC = $(srctree)/scripts/kernel-doc +DOCPROC = $(objtree)/scripts/docproc XMLTOFLAGS = -m $(srctree)/$(src)/stylesheet.xsl XMLTOFLAGS += --skip-validation @@ -89,7 +90,7 @@ define rule_docproc ) > $(dir $@).$(notdir $@).cmd endef -%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) FORCE +%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE $(call if_changed_rule,docproc) # Tell kbuild to always build the programs @@ -139,8 +140,12 @@ quiet_cmd_db2html = HTML $@ cmd_db2html = xmlto html $(XMLTOFLAGS) -o $(patsubst %.html,%,$@) $< && \ echo '<a HREF="$(patsubst %.html,%,$(notdir $@))/index.html"> \ $(patsubst %.html,%,$(notdir $@))</a><p>' > $@ +%.aux.xml: %.xml + @rm -rf $@ + (cat $< | egrep "^<refentry id" | egrep -o "\".*\"" | cut -f 2 -d \" > $<.db) + $(KERNELDOCXMLREF) -db $<.db $< > $@ -%.html: %.xml +%.html: %.aux.xml @(which xmlto > /dev/null 2>&1) || \ (echo "*** You need to install xmlto ***"; \ exit 1) @@ -209,15 +214,18 @@ dochelp: ### # Temporary files left by various tools clean-files := $(DOCBOOKS) \ - $(patsubst %.xml, %.dvi, $(DOCBOOKS)) \ - $(patsubst %.xml, %.aux, $(DOCBOOKS)) \ - $(patsubst %.xml, %.tex, $(DOCBOOKS)) \ - $(patsubst %.xml, %.log, $(DOCBOOKS)) \ - $(patsubst %.xml, %.out, $(DOCBOOKS)) \ - $(patsubst %.xml, %.ps, $(DOCBOOKS)) \ - $(patsubst %.xml, %.pdf, $(DOCBOOKS)) \ - $(patsubst %.xml, %.html, $(DOCBOOKS)) \ - $(patsubst %.xml, %.9, $(DOCBOOKS)) \ + $(patsubst %.xml, %.dvi, $(DOCBOOKS)) \ + $(patsubst %.xml, %.aux, $(DOCBOOKS)) \ + $(patsubst %.xml, %.tex, $(DOCBOOKS)) \ + $(patsubst %.xml, %.log, $(DOCBOOKS)) \ + $(patsubst %.xml, %.out, $(DOCBOOKS)) \ + $(patsubst %.xml, %.ps, $(DOCBOOKS)) \ + $(patsubst %.xml, %.pdf, $(DOCBOOKS)) \ + $(patsubst %.xml, %.html, $(DOCBOOKS)) \ + $(patsubst %.xml, %.9, $(DOCBOOKS)) \ + $(patsubst %.xml, %.aux.xml, $(DOCBOOKS)) \ + $(patsubst %.xml, %.xml.db, $(DOCBOOKS)) \ + $(patsubst %.xml, %.xml, $(DOCBOOKS)) \ $(index) clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man diff --git a/scripts/kernel-doc-xml-ref b/scripts/kernel-doc-xml-ref new file mode 100755 index 0000000..c61cdad --- /dev/null +++ b/scripts/kernel-doc-xml-ref @@ -0,0 +1,181 @@ +#!/usr/bin/perl -w + +use strict; + +## Copyright (C) 2015 Intel Corporation ## +# ## +## This software falls under the GNU General Public License. ## +## Please read the COPYING file for more information ## +# +# +# This software reads a XML file and a list of valid interal +# references to replace Docbook tags with links. +# +# usage: +# kernel-doc-xml-ref -db filename +# xml filename > outputfile + +# read arguments +if ($#ARGV != 2) { + usage(); +} + +#Holds the database filename +my $databasefile; +my @database; + +#holds the inputfile +my $inputfile; +my $errors = 0; + +my %highlights = ( + "<function>(.*?)</function>", + "\"<function>\" . convert_function(\$1, \$line) . \"</function>\"", + "<structname>(.*?)</structname>", + "\"<structname>\" . convert_struct(\$1) . \"</structname>\"", + "<funcdef>(.*?)<function>(.*?)</function></funcdef>", + "\"<funcdef>\" . convert_param(\$1) . \"<function>\$2</function></funcdef>\"", + "<paramdef>(.*?)<parameter>(.*?)</parameter></paramdef>", + "\"<paramdef>\" . convert_param(\$1) . \"<parameter>\$2</parameter></paramdef>\""); + +while($ARGV[0] =~ m/^-(.*)/) { + my $cmd = shift @ARGV; + if ($cmd eq "-db") { + $databasefile = shift @ARGV + } +} +$inputfile = shift @ARGV; + +sub open_database { + open my $handle, '<', $databasefile; + chomp(my @lines = <$handle>); + close $handle; + + @database = @lines; +} + +sub process_file { + open_database(); + + my $dohighlight; + foreach my $pattern (keys %highlights) { + $dohighlight .= "\$line =~ s:$pattern:$highlights{$pattern}:eg;\n"; + } + + open(FILE, $inputfile) or die("Could not open $inputfile"); + foreach my $line (<FILE>) { + eval $dohighlight; + print $line; + } +} + +sub trim($_) +{ + my $str = $_[0]; + $str =~ s/^\s+|\s+$//g; + return $str +} + +sub has_key_defined($_) +{ + if ( grep( /^$_[0]$/, @database)) { + return 1; + } + return 0; +} + +sub convert_function($_) +{ + my $arg = $_[0]; + my $key = $_[0]; + + my $line = $_[1]; + + $key = trim($key); + + $key =~ s/[^A-Za-z0-9]/-/g; + $key = "API-" . $key; + + # We shouldn't add links to <funcdef> prototype + if (!has_key_defined($key) || $line =~ m/\s+<funcdef/i) { + return $arg; + } + + my $head = $arg; + my $tail = ""; + if ($arg =~ /(.*?)( ?)$/) { + $head = $1; + $tail = $2; + } + return "<link linkend=\"$key\">$head</link>$tail"; +} +sub convert_struct($_) +{ + my $arg = $_[0]; + my $key = $_[0]; + $key =~ s/(struct )?(\w)/$2/g; + $key =~ s/[^A-Za-z0-9]/-/g; + $key = "API-struct-" . $key; + + if (!has_key_defined($key)) { + return $arg; + } + + my ($head, $tail) = split_pointer($arg); + return "<link linkend=\"$key\">$head</link>$tail"; +} + +sub split_pointer($_) +{ + my $arg = $_[0]; + if ($arg =~ /(.*?)( ?\* ?)/) { + return ($1, $2); + } + return ($arg, ""); +} + +sub convert_param($_) +{ + my $type = $_[0]; + my $keyname = convert_key_name($type); + + if (!has_key_defined($keyname)) { + return $type; + } + + my ($head, $tail) = split_pointer($type); + return "<link linkend=\"$keyname\">$head</link>$tail"; + +} + +sub convert_key_name($_) +{ + #Pattern $2 is optional and might be uninitialized + no warnings 'uninitialized'; + + my $str = $_[0]; + $str =~ s/(const|static)? ?(struct)? ?([a-zA-Z0-9_]+) ?(\*|&)?/$2 $3/g ; + + # trim + $str =~ s/^\s+|\s+$//g; + + # spaces and _ to - + $str =~ s/[^A-Za-z0-9]/-/g; + + return "API-" . $str; +} + +sub usage { + print "Usage: $0 -db database filename ]\n"; + print " xml source file(s) > outputfile\n"; + exit 1; +} + +# starting point +process_file(); + +if ($errors) { + print STDERR "$errors errors\n"; +} + +exit($errors);
Functions, Structs and Parameters definitions on kernel documentation are pure cosmetic, it only highlights the element. To ease the navigation in the documentation we should use <links> inside those tags so readers can easily jump between methods directly. This was discussed in 2014[1] and is implemented by getting a list of <refentries> from the DocBook XML to generate a database. Then it looks for <function>,<structnames> and <paramdef> tags that matches the ones in the database. As it only links existent references, no broken links are added. [1] - lists.freedesktop.org/archives/dri-devel/2014-August/065404.html Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> Cc: Randy Dunlap <rdunlap@infradead.org> Cc: Daniel Vetter <daniel.vetter@ffwll.ch> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Stephan Mueller <smueller@chronox.de> Cc: Michal Marek <mmarek@suse.cz> Cc: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: intel-gfx <intel-gfx@lists.freedesktop.org> Cc: dri-devel <dri-devel@lists.freedesktop.org> --- To understand a bit more of what this patch is trying to acomplish you can find two examples of the old and new htmldocs outputs: OLD: https://people.collabora.com/~danilo/intel/Documentation.old/DocBook/drm/API-drm-crtc-vblank-on.html NEW: https://people.collabora.com/~danilo/intel/Documentation.new/DocBook/drm/API-drm-crtc-vblank-on.html Documentation/DocBook/Makefile | 34 +++++--- scripts/kernel-doc-xml-ref | 181 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 202 insertions(+), 13 deletions(-) create mode 100755 scripts/kernel-doc-xml-ref