diff mbox

[v2,1/4] scripts/kernel-doc: Adding cross-reference links to html documentation.

Message ID 1438112718-12168-2-git-send-email-danilo.cesar@collabora.co.uk
State New, archived
Headers show

Commit Message

Danilo Cesar Lemes de Paula July 28, 2015, 7:45 p.m. UTC
Functions, Structs and Parameters definitions on kernel documentation
are pure cosmetic, it only highlights the element.

To ease the navigation in the documentation we should use <links> inside
those tags so readers can easily jump between methods directly.

This was discussed in 2014[1] and is implemented by getting a list
of <refentries> from the DocBook XML to generate a database. Then it looks
for <function>,<structnames> and <paramdef> tags that matches the ones in
the database. As it only links existent references, no broken links are
added.

[1] - lists.freedesktop.org/archives/dri-devel/2014-August/065404.html

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Stephan Mueller <smueller@chronox.de>
Cc: Michal Marek <mmarek@suse.cz>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
---
 Changelog:
  v1 (missing changlog from v1):
     * Add a "XMLREF FooBar.xml" as build output to reduce build noise.
     * Make kernel-doc-xml-ref ignore invalid arguments.
     * Improve kernel-doc-xml-ref documentation.

 Documentation/DocBook/Makefile |  43 ++++++---
 scripts/kernel-doc-xml-ref     | 198 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 228 insertions(+), 13 deletions(-)
 create mode 100755 scripts/kernel-doc-xml-ref

Comments

Jonathan Corbet Aug. 17, 2015, 4:10 a.m. UTC | #1
On Tue, 28 Jul 2015 16:45:15 -0300
Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:

> Functions, Structs and Parameters definitions on kernel documentation
> are pure cosmetic, it only highlights the element.
> 
> To ease the navigation in the documentation we should use <links> inside
> those tags so readers can easily jump between methods directly.
> 
> This was discussed in 2014[1] and is implemented by getting a list
> of <refentries> from the DocBook XML to generate a database. Then it looks
> for <function>,<structnames> and <paramdef> tags that matches the ones in
> the database. As it only links existent references, no broken links are
> added.

So I had some airplane time today and was able to mess with this some.  I
can't make it break anymore, and it clearly improves the resulting
documentation, so I've applied it to the docs tree for 4.3.

I want to look at the rest of the stuff a bit more and play with it, but
it's hard to imagine why we wouldn't want that as well.  I'm a bit more
leery just because it adds another dependency to the build, even if it's
an optional dependency.  My thinking at the moment is to apply it shortly
after the merge window so it can have a long soak in linux-next before a
4.4 merge; hope that sounds good.

Thanks for doing this work,

jon
Danilo Cesar Lemes de Paula Aug. 17, 2015, 1:06 p.m. UTC | #2
On 08/17/2015 01:10 AM, Jonathan Corbet wrote:
> On Tue, 28 Jul 2015 16:45:15 -0300
> Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:
> 
>> Functions, Structs and Parameters definitions on kernel documentation
>> are pure cosmetic, it only highlights the element.
>>
>> To ease the navigation in the documentation we should use <links> inside
>> those tags so readers can easily jump between methods directly.
>>
>> This was discussed in 2014[1] and is implemented by getting a list
>> of <refentries> from the DocBook XML to generate a database. Then it looks
>> for <function>,<structnames> and <paramdef> tags that matches the ones in
>> the database. As it only links existent references, no broken links are
>> added.
> 
> So I had some airplane time today and was able to mess with this some.  I
> can't make it break anymore, and it clearly improves the resulting
> documentation, so I've applied it to the docs tree for 4.3.
> 
> I want to look at the rest of the stuff a bit more and play with it, but
> it's hard to imagine why we wouldn't want that as well.  I'm a bit more
> leery just because it adds another dependency to the build, even if it's
> an optional dependency.  My thinking at the moment is to apply it shortly

I totally agree about the dependency stuff. I even discussed it with
Daniel Vetter a bit. I started by writing my-very-own-markup-parser to
put alongside kernel-doc to avoid external dependencies, but it gets too
complex too quickly (specially when dealing with tables and multi-line
stuff). It would be a pain to maintain a something like that, and the
world probably doesn't need yet-another-markup-parser, so I decided to
use another tool.

> after the merge window so it can have a long soak in linux-next before a
> 4.4 merge; hope that sounds good.

It does sound good. Thanks!

> 
> Thanks for doing this work,

Glad I could help.


Danilo
diff mbox

Patch

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index b6a6a2e..322255b 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -64,8 +64,9 @@  installmandocs: mandocs
 
 ###
 #External programs used
-KERNELDOC = $(srctree)/scripts/kernel-doc
-DOCPROC   = $(objtree)/scripts/docproc
+KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref
+KERNELDOC       = $(srctree)/scripts/kernel-doc
+DOCPROC         = $(objtree)/scripts/docproc
 
 XMLTOFLAGS = -m $(srctree)/$(src)/stylesheet.xsl
 XMLTOFLAGS += --skip-validation
@@ -89,7 +90,7 @@  define rule_docproc
         ) > $(dir $@).$(notdir $@).cmd
 endef
 
-%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) FORCE
+%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
 	$(call if_changed_rule,docproc)
 
 # Tell kbuild to always build the programs
@@ -140,7 +141,20 @@  quiet_cmd_db2html = HTML    $@
 		echo '<a HREF="$(patsubst %.html,%,$(notdir $@))/index.html"> \
 		$(patsubst %.html,%,$(notdir $@))</a><p>' > $@
 
-%.html:	%.xml
+###
+# Rules to create an aux XML and .db, and use them to re-process the DocBook XML
+# to fill internal hyperlinks
+       gen_aux_xml = :
+ quiet_gen_aux_xml = echo '  XMLREF  $@'
+silent_gen_aux_xml = :
+%.aux.xml: %.xml
+	@$($(quiet)gen_aux_xml)
+	@rm -rf $@
+	@(cat $< | egrep "^<refentry id" | egrep -o "\".*\"" | cut -f 2 -d \" > $<.db)
+	@$(KERNELDOCXMLREF) -db $<.db $< > $@
+.PRECIOUS: %.aux.xml
+
+%.html:	%.aux.xml
 	@(which xmlto > /dev/null 2>&1) || \
 	 (echo "*** You need to install xmlto ***"; \
 	  exit 1)
@@ -209,15 +223,18 @@  dochelp:
 ###
 # Temporary files left by various tools
 clean-files := $(DOCBOOKS) \
-	$(patsubst %.xml, %.dvi,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.aux,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.tex,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.log,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.out,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.ps,   $(DOCBOOKS)) \
-	$(patsubst %.xml, %.pdf,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.html, $(DOCBOOKS)) \
-	$(patsubst %.xml, %.9,    $(DOCBOOKS)) \
+	$(patsubst %.xml, %.dvi,     $(DOCBOOKS)) \
+	$(patsubst %.xml, %.aux,     $(DOCBOOKS)) \
+	$(patsubst %.xml, %.tex,     $(DOCBOOKS)) \
+	$(patsubst %.xml, %.log,     $(DOCBOOKS)) \
+	$(patsubst %.xml, %.out,     $(DOCBOOKS)) \
+	$(patsubst %.xml, %.ps,      $(DOCBOOKS)) \
+	$(patsubst %.xml, %.pdf,     $(DOCBOOKS)) \
+	$(patsubst %.xml, %.html,    $(DOCBOOKS)) \
+	$(patsubst %.xml, %.9,       $(DOCBOOKS)) \
+	$(patsubst %.xml, %.aux.xml, $(DOCBOOKS)) \
+	$(patsubst %.xml, %.xml.db,  $(DOCBOOKS)) \
+	$(patsubst %.xml, %.xml,     $(DOCBOOKS)) \
 	$(index)
 
 clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man
diff --git a/scripts/kernel-doc-xml-ref b/scripts/kernel-doc-xml-ref
new file mode 100755
index 0000000..104a5a5
--- /dev/null
+++ b/scripts/kernel-doc-xml-ref
@@ -0,0 +1,198 @@ 
+#!/usr/bin/perl -w
+
+use strict;
+
+## Copyright (C) 2015  Intel Corporation                         ##
+#                                                                ##
+## This software falls under the GNU General Public License.     ##
+## Please read the COPYING file for more information             ##
+#
+#
+# This software reads a XML file and a list of valid interal
+# references to replace Docbook tags with links.
+#
+# The list of "valid internal references" must be one-per-line in the following format:
+#      API-struct-foo
+#      API-enum-bar
+#      API-my-function
+#
+# The software walks over the XML file looking for xml tags representing possible references
+# to the Document. Each reference will be cross checked against the "Valid Internal Reference" list. If
+# the referece is found it replaces its content by a <link> tag.
+#
+# usage:
+# kernel-doc-xml-ref -db filename
+#		     xml filename > outputfile
+
+# read arguments
+if ($#ARGV != 2) {
+	usage();
+}
+
+#Holds the database filename
+my $databasefile;
+my @database;
+
+#holds the inputfile
+my $inputfile;
+my $errors = 0;
+
+my %highlights = (
+	"<function>(.*?)</function>",
+	    "\"<function>\" . convert_function(\$1, \$line) . \"</function>\"",
+	"<structname>(.*?)</structname>",
+	    "\"<structname>\" . convert_struct(\$1) . \"</structname>\"",
+	"<funcdef>(.*?)<function>(.*?)</function></funcdef>",
+	    "\"<funcdef>\" . convert_param(\$1) . \"<function>\$2</function></funcdef>\"",
+	"<paramdef>(.*?)<parameter>(.*?)</parameter></paramdef>",
+	    "\"<paramdef>\" . convert_param(\$1) . \"<parameter>\$2</parameter></paramdef>\"");
+
+while($ARGV[0] =~ m/^-(.*)/) {
+	my $cmd = shift @ARGV;
+	if ($cmd eq "-db") {
+		$databasefile = shift @ARGV
+	} else {
+		usage();
+	}
+}
+$inputfile = shift @ARGV;
+
+sub open_database {
+	open (my $handle, '<', $databasefile) or die "Cannot open $databasefile";
+	chomp(my @lines = <$handle>);
+	close $handle;
+
+	@database = @lines;
+}
+
+sub process_file {
+	open_database();
+
+	my $dohighlight;
+	foreach my $pattern (keys %highlights) {
+		$dohighlight .=  "\$line =~ s:$pattern:$highlights{$pattern}:eg;\n";
+	}
+
+	open(FILE, $inputfile) or die("Could not open $inputfile") or die ("Cannot open $inputfile");
+	foreach my $line (<FILE>)  {
+		eval $dohighlight;
+		print $line;
+	}
+}
+
+sub trim($_)
+{
+	my $str = $_[0];
+	$str =~ s/^\s+|\s+$//g;
+	return $str
+}
+
+sub has_key_defined($_)
+{
+	if ( grep( /^$_[0]$/, @database)) {
+		return 1;
+	}
+	return 0;
+}
+
+# Gets a <function> content and add it a hyperlink if possible.
+sub convert_function($_)
+{
+	my $arg = $_[0];
+	my $key = $_[0];
+
+	my $line = $_[1];
+
+	$key = trim($key);
+
+	$key =~ s/[^A-Za-z0-9]/-/g;
+	$key = "API-" . $key;
+
+	# We shouldn't add links to <funcdef> prototype
+	if (!has_key_defined($key) || $line =~ m/\s+<funcdef/i) {
+		return $arg;
+	}
+
+	my $head = $arg;
+	my $tail = "";
+	if ($arg =~ /(.*?)( ?)$/) {
+		$head = $1;
+		$tail = $2;
+	}
+	return "<link linkend=\"$key\">$head</link>$tail";
+}
+
+# Converting a struct text to link
+sub convert_struct($_)
+{
+	my $arg = $_[0];
+	my $key = $_[0];
+	$key =~ s/(struct )?(\w)/$2/g;
+	$key =~ s/[^A-Za-z0-9]/-/g;
+	$key = "API-struct-" . $key;
+
+	if (!has_key_defined($key)) {
+		return $arg;
+	}
+
+	my ($head, $tail) = split_pointer($arg);
+	return "<link linkend=\"$key\">$head</link>$tail";
+}
+
+# Identify "object *" elements
+sub split_pointer($_)
+{
+	my $arg = $_[0];
+	if ($arg =~ /(.*?)( ?\* ?)/) {
+		return ($1, $2);
+	}
+	return ($arg, "");
+}
+
+sub convert_param($_)
+{
+	my $type = $_[0];
+	my $keyname = convert_key_name($type);
+
+	if (!has_key_defined($keyname)) {
+		return $type;
+	}
+
+	my ($head, $tail) = split_pointer($type);
+	return "<link linkend=\"$keyname\">$head</link>$tail";
+
+}
+
+# DocBook links are in the API-<TYPE>-<STRUCT-NAME> format
+# This method gets an element and returns a valid DocBook reference for it.
+sub convert_key_name($_)
+{
+	#Pattern $2 is optional and might be uninitialized
+	no warnings 'uninitialized';
+
+	my $str = $_[0];
+	$str =~ s/(const|static)? ?(struct)? ?([a-zA-Z0-9_]+) ?(\*|&)?/$2 $3/g ;
+
+	# trim
+	$str =~ s/^\s+|\s+$//g;
+
+	# spaces and _ to -
+	$str =~ s/[^A-Za-z0-9]/-/g;
+
+	return "API-" . $str;
+}
+
+sub usage {
+	print "Usage: $0 -db database filename\n";
+	print "         xml source file(s) > outputfile\n";
+	exit 1;
+}
+
+# starting point
+process_file();
+
+if ($errors) {
+	print STDERR "$errors errors\n";
+}
+
+exit($errors);