From patchwork Wed Nov 25 17:07:56 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 7701561 Return-Path: X-Original-To: patchwork-intel-gfx@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 97786BF90C for ; Wed, 25 Nov 2015 17:08:19 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 913412085A for ; Wed, 25 Nov 2015 17:08:15 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) by mail.kernel.org (Postfix) with ESMTP id 31F5620842 for ; Wed, 25 Nov 2015 17:08:14 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BEF3E6EAD4; Wed, 25 Nov 2015 09:08:12 -0800 (PST) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mail-wm0-f51.google.com (mail-wm0-f51.google.com [74.125.82.51]) by gabe.freedesktop.org (Postfix) with ESMTPS id C53296E72A for ; Wed, 25 Nov 2015 09:08:10 -0800 (PST) Received: by wmvv187 with SMTP id v187so266806565wmv.1 for ; Wed, 25 Nov 2015 09:08:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Es4tJTIASEsS9ZTkgofnRZutzAhuPtwb/mYM8nHxvPc=; b=kT5SkYudkGWkEuAQEAl14y3pqcevHzXTWBSup3iltXt6E8kWqwSZoh3hBQcoVu/yTf o73iDqAm4EobLq0rB56YFJR3welBnLV5vXCz/iYzIV1S+B5yxbK6R3qSrqIN+oglyu3m u8BoSD9ZIF7dsRCiJbbXq7ZHn0/5m9lKXPtsA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Es4tJTIASEsS9ZTkgofnRZutzAhuPtwb/mYM8nHxvPc=; b=Atd8ugp9vYkZ5XEQv1WOrv8V6QE6wCeg+OXuZQ0mhU2dwIygWa5HGgIlh6/NxUMBv5 VeKgHlgwvNKDzmJFhSyzZLEdjx0Re31/E/uE4NUwjX5AxqaaQDDVpugZpb8eLmSpOJd/ SNmcuI3JlTOKsxkqo+zk6rfmoWNt60kxX2TMXtWMzEGg6MkShbCUfWlqtivCAJkGuymA YC04nSaQNM9NPUSqhQ1JfE7OShITrvVpy0+R+se47xRCTvUNN0E7yVGnOSykQkER3Vc3 8C5FEJmjXGiyH5CShh152VsTTR4YvmuNOmiY2MHO4w8m5bu/0GV9vAp3OBBfKw0E3928 1N+Q== X-Gm-Message-State: ALoCoQmcBEvZxh6pG9iAzX/7o/dNk4l0piCns9Vmshy7c0ZFDJKb7adbU+ob8/Uzerqd9hFipcJc X-Received: by 10.28.180.9 with SMTP id d9mr6156497wmf.62.1448471289523; Wed, 25 Nov 2015 09:08:09 -0800 (PST) Received: from phenom.ffwll.local (212-51-149-109.fiber7.init7.net. [212.51.149.109]) by smtp.gmail.com with ESMTPSA id h4sm24070603wjx.41.2015.11.25.09.08.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 25 Nov 2015 09:08:08 -0800 (PST) From: Daniel Vetter To: DRI Development Date: Wed, 25 Nov 2015 18:07:56 +0100 Message-Id: <1448471279-19748-3-git-send-email-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.5.1 In-Reply-To: <1448471279-19748-1-git-send-email-daniel.vetter@ffwll.ch> References: <1448471279-19748-1-git-send-email-daniel.vetter@ffwll.ch> Cc: Michal Marek , Herbert Xu , Danilo Cesar Lemes de Paula , Jonathan Corbet , Stephan Mueller , Daniel Vetter , Intel Graphics Development , Randy Dunlap , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Laurent Pinchart Subject: [Intel-gfx] [PATCH 2/5] scripts/kernel-doc: Adding infrastructure for markdown support X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Danilo Cesar Lemes de Paula Markdown support is given by calling an external tool, pandoc, for all highlighted text on kernel-doc. Pandoc converts Markdown text to proper Docbook tags, which will be later translated to pdf, html or other targets. This adds the capability of adding human-readle text highlight (bold, underline, etc), bullet and numbered lists, simple tables, fixed-width text (including asciiart), requiring minimal changes to current documentation. At this moment, pandoc is totally optional. Docbooks ready for markdown should be added to the MARKDOWNREADY variable inside the Makefile. In case the developer doesn't have pandoc installed, Make will throw a warning and the documentation build will continue, generating simple Documentation without the features brought by pandoc. Signed-off-by: Danilo Cesar Lemes de Paula Cc: Randy Dunlap Cc: Daniel Vetter Cc: Laurent Pinchart Cc: Jonathan Corbet Cc: Herbert Xu Cc: Stephan Mueller Cc: Michal Marek Cc: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: intel-gfx Cc: dri-devel --- Documentation/DocBook/Makefile | 25 +++++++++++----- scripts/docproc.c | 54 ++++++++++++++++++++++++---------- scripts/kernel-doc | 66 ++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 119 insertions(+), 26 deletions(-) diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 91f6d89bb19f..246ad38550e5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -17,6 +17,8 @@ DOCBOOKS := z8530book.xml device-drivers.xml \ tracepoint.xml gpu.xml media_api.xml w1.xml \ writing_musb_glue_layer.xml crypto-API.xml iio.xml +MARKDOWNREADY := + include Documentation/DocBook/media/Makefile ### @@ -87,18 +89,23 @@ XMLTOFLAGS += --skip-validation # The following rules are used to generate the .xml documentation # required to generate the final targets. (ps, pdf, html). quiet_cmd_docproc = DOCPROC $@ - cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $< >$@ + cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $< define rule_docproc - set -e; \ - $(if $($(quiet)cmd_$(1)),echo ' $($(quiet)cmd_$(1))';) \ - $(cmd_$(1)); \ - ( \ - echo 'cmd_$@ := $(cmd_$(1))'; \ - echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`; \ + set -e; \ + USEMARKDOWN=""; \ + FILE=`basename $@`; \ + [[ "$(MARKDOWNREADY)" =~ "$${FILE}" ]] && USEMARKDOWN="-use-markdown"; \ + $(if $($(quiet)cmd_$(1)),echo ' $($(quiet)cmd_$(1))';) \ + $(cmd_$(1)) $$USEMARKDOWN >$@; \ + ( \ + echo 'cmd_$@ := $(cmd_$(1))'; \ + echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`; \ ) > $(dir $@).$(notdir $@).cmd endef %.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE + @(which pandoc > /dev/null 2>&1) || \ + (echo "*** To get propper documentation you need to install pandoc ***";) $(call if_changed_rule,docproc) # Tell kbuild to always build the programs @@ -109,6 +116,10 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $< +ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) + MARKDOWNREADY := ""; +endif + # determine which methods are available ifeq ($(shell which db2ps >/dev/null 2>&1 && echo found),found) use-db2x = db2x diff --git a/scripts/docproc.c b/scripts/docproc.c index e267e621431a..7c6b225a6a50 100644 --- a/scripts/docproc.c +++ b/scripts/docproc.c @@ -73,12 +73,15 @@ FILELINE * docsection; #define NOFUNCTION "-nofunction" #define NODOCSECTIONS "-no-doc-sections" #define SHOWNOTFOUND "-show-not-found" +#define USEMARKDOWN "-use-markdown" static char *srctree, *kernsrctree; static char **all_list = NULL; static int all_list_len = 0; +static int use_markdown = 0; + static void consume_symbol(const char *sym) { int i; @@ -95,10 +98,11 @@ static void consume_symbol(const char *sym) static void usage (void) { - fprintf(stderr, "Usage: docproc {doc|depend} file\n"); + fprintf(stderr, "Usage: docproc {doc|depend} [-use-markdown] file\n"); fprintf(stderr, "Input is read from file.tmpl. Output is sent to stdout\n"); fprintf(stderr, "doc: frontend when generating kernel documentation\n"); fprintf(stderr, "depend: generate list of files referenced within file\n"); + fprintf(stderr, "-use-markdown: pass -use-markdown to kernel-doc\n"); fprintf(stderr, "Environment variable SRCTREE: absolute path to sources.\n"); fprintf(stderr, " KBUILD_SRC: absolute path to kernel source tree.\n"); } @@ -257,12 +261,15 @@ static void docfunctions(char * filename, char * type) for (i=0; i <= symfilecnt; i++) symcnt += symfilelist[i].symbolcnt; - vec = malloc((2 + 2 * symcnt + 3) * sizeof(char *)); + vec = malloc((2 + 2 * symcnt + 4) * sizeof(char *)); if (vec == NULL) { perror("docproc: "); exit(1); } vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; + vec[idx++] = DOCBOOK; vec[idx++] = NODOCSECTIONS; for (i=0; i < symfilecnt; i++) { @@ -294,6 +301,9 @@ static void singfunc(char * filename, char * line) int i, idx = 0; int startofsym = 1; vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; + vec[idx++] = DOCBOOK; vec[idx++] = SHOWNOTFOUND; @@ -328,8 +338,9 @@ static void singfunc(char * filename, char * line) static void docsect(char *filename, char *line) { /* kerneldoc -docbook -show-not-found -function "section" file NULL */ - char *vec[7]; + char *vec[8]; char *s; + int idx = 0; for (s = line; *s; s++) if (*s == '\n') @@ -342,30 +353,37 @@ static void docsect(char *filename, char *line) consume_symbol(s); free(s); - vec[0] = KERNELDOC; - vec[1] = DOCBOOK; - vec[2] = SHOWNOTFOUND; - vec[3] = FUNCTION; - vec[4] = line; - vec[5] = filename; - vec[6] = NULL; + vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; + + vec[idx++] = DOCBOOK; + vec[idx++] = SHOWNOTFOUND; + vec[idx++] = FUNCTION; + vec[idx++] = line; + vec[idx++] = filename; + vec[idx] = NULL; exec_kernel_doc(vec); } static void find_all_symbols(char *filename) { - char *vec[4]; /* kerneldoc -list file NULL */ + char *vec[5]; /* kerneldoc -list file NULL */ pid_t pid; int ret, i, count, start; char real_filename[PATH_MAX + 1]; int pipefd[2]; char *data, *str; size_t data_len = 0; + int idx = 0; + + vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; - vec[0] = KERNELDOC; - vec[1] = LIST; - vec[2] = filename; - vec[3] = NULL; + vec[idx++] = LIST; + vec[idx++] = filename; + vec[idx] = NULL; if (pipe(pipefd)) { perror("pipe"); @@ -509,7 +527,7 @@ int main(int argc, char *argv[]) kernsrctree = getenv("KBUILD_SRC"); if (!kernsrctree || !*kernsrctree) kernsrctree = srctree; - if (argc != 3) { + if (argc < 3 || argc > 4) { usage(); exit(1); } @@ -521,6 +539,10 @@ int main(int argc, char *argv[]) exit(2); } + if (argc == 4 && strcmp("-use-markdown", argv[3]) == 0) { + use_markdown = 1; + } + if (strcmp("doc", argv[1]) == 0) { /* Need to do this in two passes. * First pass is used to collect all symbols exported diff --git a/scripts/kernel-doc b/scripts/kernel-doc index 638a38e1b419..28737053395a 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -1,6 +1,7 @@ #!/usr/bin/perl -w use strict; +use IPC::Open3; ## Copyright (c) 1998 Michael Zucchi, All Rights Reserved ## ## Copyright (C) 2000, 1 Tim Waugh ## @@ -282,6 +283,7 @@ if ($#ARGV == -1) { my $kernelversion; my $dohighlight = ""; +my $use_markdown = 0; my $verbose = 0; my $output_mode = "man"; @@ -424,6 +426,8 @@ while ($ARGV[0] =~ m/^-(.*)/) { $function_only = 2; $function = shift @ARGV; $function_table{$function} = 1; + } elsif ($cmd eq "-use-markdown") { + $use_markdown = 1; } elsif ($cmd eq "-v") { $verbose = 1; } elsif (($cmd eq "-h") || ($cmd eq "--help")) { @@ -442,6 +446,7 @@ sub usage { print " [ -no-doc-sections ]\n"; print " [ -function funcname [ -function funcname ...] ]\n"; print " [ -nofunction funcname [ -nofunction funcname ...] ]\n"; + print " [ -use-markdown ]\n"; print " [ -v ]\n"; print " c source file(s) > outputfile\n"; print " -v : verbose output, more warnings & other info listed\n"; @@ -515,6 +520,49 @@ sub dump_doc_section { } } +sub markdown_to_docbook { + my $orig_content = $_[0]; + + my $pid = open3( \*CHLD_IN, \*CHLD_OUT, \*CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" ); + + print CHLD_IN "$orig_content"; + close(CHLD_IN); + + waitpid($pid, 0); + + my $content = ""; + chomp(my @lines = ); + foreach my $line (@lines) { + $content .= $line . "\n"; + } + close(CHLD_OUT); + close(CHLD_ERR); + + # pandoc insists in adding Main , we should remove them. + $content =~ s:\A\s*\s*\n(.*)\n\Z$:$1:egsm; + + return $content; +} + +# Markdown->Docbook conversion by pandoc requires unescaped text +# Kernel-doc converts every & to "&", we need to convert it back. +sub markdown_unescape +{ + my $text = shift; + my @lines = split /\n/, $text; + + my @result; + foreach my $line (@lines) { + if ( $line =~ m /^ /s ) { + $line =~ s:\&:\&:gs + } + push @result, $line; + } + + return join "\n",@result; + +} + ## # output function # @@ -541,11 +589,19 @@ sub output_highlight { $contents = local_unescape($contents); # convert data read & converted thru xml_escape() into &xyz; format: $contents =~ s/\\\\\\/\&/g; + + if ($use_markdown) { + $contents = markdown_unescape($contents); + } } + # print STDERR "contents b4:$contents\n"; eval $dohighlight; die $@ if $@; # print STDERR "contents af:$contents\n"; + if ($use_markdown) { + $contents = markdown_to_docbook($contents); + } # strip whitespaces when generating html5 if ($output_mode eq "html5") { @@ -553,7 +609,8 @@ sub output_highlight { $contents =~ s/\s+$//; } foreach $line (split "\n", $contents) { - if (! $output_preformatted) { + if (! $output_preformatted && + !($use_markdown && $line =~ m /^ /s)) { $line =~ s/^\s*//; } if ($line eq ""){ @@ -974,7 +1031,9 @@ sub output_section_xml(%) { print "\n"; print "$section\n"; if ($section =~ m/EXAMPLE/i) { - print "\n"; + print "\n"; + # programlisting is already included by pandoc + print "\n" unless $use_markdown; $output_preformatted = 1; } else { print "\n"; @@ -982,7 +1041,8 @@ sub output_section_xml(%) { output_highlight($args{'sections'}{$section}); $output_preformatted = 0; if ($section =~ m/EXAMPLE/i) { - print "\n"; + print "\n" unless $use_markdown; + print "\n"; } else { print "\n"; }