| Message ID | 1448471279-19748-6-git-send-email-daniel.vetter@ffwll.ch (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, 25 Nov 2015 18:07:59 +0100 Daniel Vetter <daniel.vetter@ffwll.ch> wrote: > Unfortunately the entire improved docbook project died at KS in a > massive bikeshed. So we need to carry this around in drm private trees > forever :( I don't think that's an entirely helpful way to look at things, honestly. Changing how the docs system works affects a lot of people, they're going to have input on the matter. And I sure hope it hasn't "died". The posting of these patches suggests that perhaps it has not. Anyway, I wanted to say that, my silence notwithstanding, I haven't just dropped these. I had some hassles to deal with (replacing the entire LWN server infrastructure, for example) but those are done; I hope to be able to mess with this stuff a bit in the very near future. Have the table-rendering issues that Graham talked about before gotten any better with the newer scheme? Thanks, jon
On Fri, Dec 11, 2015 at 03:12:26PM -0700, Jonathan Corbet wrote: > On Wed, 25 Nov 2015 18:07:59 +0100 > Daniel Vetter <daniel.vetter@ffwll.ch> wrote: > > > Unfortunately the entire improved docbook project died at KS in a > > massive bikeshed. So we need to carry this around in drm private trees > > forever :( > > I don't think that's an entirely helpful way to look at things, honestly. > Changing how the docs system works affects a lot of people, they're going > to have input on the matter. > > And I sure hope it hasn't "died". The posting of these patches suggests > that perhaps it has not. Hm, I think I fumbled the cc for the cover letter: http://www.spinics.net/lists/intel-gfx/msg81351.html In short it hasn't died, and 4.5 will have a few thousand more lines of doc already employing asciidoc (and ofc also the other stuff that already landed in 4.4). I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful. > Anyway, I wanted to say that, my silence notwithstanding, I haven't just > dropped these. I had some hassles to deal with (replacing the entire LWN > server infrastructure, for example) but those are done; I hope to be able > to mess with this stuff a bit in the very near future. > > Have the table-rendering issues that Graham talked about before gotten any > better with the newer scheme? I haven't tackled the table conversion story yet, but one reason for me to switch to asciidoc (besides that Dave thought it was a good idea) was the better table support. I haven't tried, but it /should/ be powerful enough. Another bonus is in-line figures as asciiart that get rendered to png. But haven't done the integration work on that yet either. Anyway things seem to work and I'm happy. Thanks, Daniel
On Sat, 12 Dec 2015 12:13:45 +0100 Daniel Vetter <daniel@ffwll.ch> wrote: > I just figured there's no way this could get it, and I'd > much rather improve the docs themselves than trying to convince core > kernel folks that this might be useful. So I'm not quite sure why you figured that; I never said it, certainly. I've been messing with it a bit, seems to work. I do still wish we could consider alternatives, especially those that might simplify the toolchain rather than complicating it. But it's clear that I'm not succeeding in finding time to actually explore that idea; the contents of $EXCUSES are good, but the end result is the same. And the patch fairy just isn't coming through for me on this one. In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams. That gives a couple of weeks for an updated patch set, should you have one. The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that? - How many of the comments actually use asciidoc features? Might there be some possibility of detecting those in kernel-doc and skipping the callout to asciidoc when it's not needed? - Pandoc seems to do asciidoc. I still don't like the idea of depending on it for this to work, but having the *option* to use it is fine. If it's really that much faster (yes, Python startup is painful) then maybe providing the option is worth it. - All over the kernel we've seen that batching improves performance. It would take a bit of work, but I bet kernel-doc could put together all the snippets from one file, pass them through a single asciidoc invocation, then split the results back apart. That would probably eliminate the performance hit entirely. None of that is a condition for pulling this stuff in, but can it be looked into? Thanks, jon
On Tue, 12 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote: > In my mind, there's clearly no good that can come from (further) delaying > something that works in favor of an "it would be nice" that may never > even exist. So I'm currently thinking that I'll pull this into the docs > tree once the merge window is done, with the plan to push it for 4.6. > Then we can see if anybody screams. Must... resist... urge to bikeshed about the choice of markup... > The build-time increase is painful in the extreme - about a factor of > three for a -j1 build, and that's with only one file using the feature. > It feels wrong, somehow, for the docs build to take longer than building > the kernel itself. Can we do something about that? "Holy big-O, batman. Asciidoc appears to be quadractically slow." [1] Fortunately the same quote lead me to asciidoctor [2], which was maybe twice as fast as asciidoc. An improvement, but could be much better. BR, Jani. [1] https://twitter.com/marijnjh/status/473935469676216321 [2] http://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/
On Mon, Jan 11, 2016 at 06:12:12PM -0700, Jonathan Corbet wrote: > On Sat, 12 Dec 2015 12:13:45 +0100 > Daniel Vetter <daniel@ffwll.ch> wrote: > > > I just figured there's no way this could get it, and I'd > > much rather improve the docs themselves than trying to convince core > > kernel folks that this might be useful. > > So I'm not quite sure why you figured that; I never said it, certainly. To clarify this wasn't really my impression of your stance, but of the overall room opinion when we had the discussion at KS. And then my main goal here is to write great docs for drm (we have about 3k lines more docs in 4.5 already), so that's why I dropped the ball on upstreaming. It seemed unlikely to succeed, at least without some really seriuos effort at convincing everyone, all while the drm docs for atomic haven't been in good shape yet. Since then we had a few contributors of new atomic drivers note on irc already that "oh cool, this is documented now". Overall really just boils down to what I see as the most important things for drm ;-) > I've been messing with it a bit, seems to work. I do still wish we could > consider alternatives, especially those that might simplify the toolchain > rather than complicating it. But it's clear that I'm not succeeding in > finding time to actually explore that idea; the contents of $EXCUSES are > good, but the end result is the same. And the patch fairy just isn't > coming through for me on this one. > > In my mind, there's clearly no good that can come from (further) delaying > something that works in favor of an "it would be nice" that may never > even exist. So I'm currently thinking that I'll pull this into the docs > tree once the merge window is done, with the plan to push it for 4.6. > Then we can see if anybody screams. > > That gives a couple of weeks for an updated patch set, should you have > one. > > The build-time increase is painful in the extreme - about a factor of > three for a -j1 build, and that's with only one file using the feature. > It feels wrong, somehow, for the docs build to take longer than building > the kernel itself. Can we do something about that? > > - How many of the comments actually use asciidoc features? Might there > be some possibility of detecting those in kernel-doc and skipping the > callout to asciidoc when it's not needed? I think that amounts to writing a partial parser (we use asciidoc for tables, lists, links, formatting, code snippets by now already, someone even thought of using the asciiart->png feature it has but it's not yet wired up). I don't think it's feasible. > - Pandoc seems to do asciidoc. I still don't like the idea of depending > on it for this to work, but having the *option* to use it is fine. If > it's really that much faster (yes, Python startup is painful) then > maybe providing the option is worth it. Hm, Dave asked me to convert to use python-based asciidoc insted of haskell-based pandoc. > - All over the kernel we've seen that batching improves performance. It > would take a bit of work, but I bet kernel-doc could put together all > the snippets from one file, pass them through a single asciidoc > invocation, then split the results back apart. That would probably > eliminate the performance hit entirely. > > None of that is a condition for pulling this stuff in, but can it be > looked into? Besides what Jani mention that asciidoctor should be a drop-in replacement if installed it also seems possible to parallelize the call-out to kernel-doc from docproc.c without too much effort. I hoped Jani would get around to implement the asciidoctor support, and I'm hoping I can snipe away some free sometimes the next few months to look at docproc.c more seriously. This would kinda be a cool intern project, but atm we throw them all at improving testing infrastructure ... Anyway I'm of course still open to get this upstream, and I think a few things should be polished (like the speed-up). But right now bandwidth on my side isn't too plentiful. Maybe we should aim to have a few better ideas (perhaps even for all of the docs stuff) for next KS and respin that discussion? Thanks, Daniel
On Tue, 2016-01-12 at 09:34 +0100, Daniel Vetter wrote: > On Mon, Jan 11, 2016 at 06:12:12PM -0700, Jonathan Corbet wrote: > > On Sat, 12 Dec 2015 12:13:45 +0100 > > Daniel Vetter <daniel@ffwll.ch> wrote: > > > > > I just figured there's no way this could get it, and I'd > > > much rather improve the docs themselves than trying to convince > > > core > > > kernel folks that this might be useful. > > > > So I'm not quite sure why you figured that; I never said it, > > certainly. > > To clarify this wasn't really my impression of your stance, but of > the > overall room opinion when we had the discussion at KS. And then my > main > goal here is to write great docs for drm (we have about 3k lines more > docs > in 4.5 already), so that's why I dropped the ball on upstreaming. It > seemed unlikely to succeed, at least without some really seriuos > effort at > convincing everyone, all while the drm docs for atomic haven't been > in > good shape yet. Since then we had a few contributors of new atomic > drivers > note on irc already that "oh cool, this is documented now". Overall > really > just boils down to what I see as the most important things for drm ; > -) > > > I've been messing with it a bit, seems to work. I do still wish we > > could > > consider alternatives, especially those that might simplify the > > toolchain > > rather than complicating it. But it's clear that I'm not > > succeeding in > > finding time to actually explore that idea; the contents of > > $EXCUSES are > > good, but the end result is the same. And the patch fairy just > > isn't > > coming through for me on this one. > > > > In my mind, there's clearly no good that can come from (further) > > delaying > > something that works in favor of an "it would be nice" that may > > never > > even exist. So I'm currently thinking that I'll pull this into the > > docs > > tree once the merge window is done, with the plan to push it for > > 4.6. > > Then we can see if anybody screams. > > > > That gives a couple of weeks for an updated patch set, should you > > have > > one. > > > > The build-time increase is painful in the extreme - about a factor > > of > > three for a -j1 build, and that's with only one file using the > > feature. > > It feels wrong, somehow, for the docs build to take longer than > > building > > the kernel itself. Can we do something about that? > > > > - How many of the comments actually use asciidoc features? Might > > there > > be some possibility of detecting those in kernel-doc and > > skipping the > > callout to asciidoc when it's not needed? > > I think that amounts to writing a partial parser (we use asciidoc for > tables, lists, links, formatting, code snippets by now already, > someone > even thought of using the asciiart->png feature it has but it's not > yet > wired up). I don't think it's feasible. > > > - Pandoc seems to do asciidoc. I still don't like the idea of > > depending > > on it for this to work, but having the *option* to use it is > > fine. If > > it's really that much faster (yes, Python startup is painful) > > then > > maybe providing the option is worth it. > > Hm, Dave asked me to convert to use python-based asciidoc insted of > haskell-based pandoc. > > > - All over the kernel we've seen that batching improves > > performance. It > > would take a bit of work, but I bet kernel-doc could put > > together all > > the snippets from one file, pass them through a single asciidoc > > invocation, then split the results back apart. That would > > probably > > eliminate the performance hit entirely. > > > > None of that is a condition for pulling this stuff in, but can it > > be > > looked into? > > Besides what Jani mention that asciidoctor should be a drop-in > replacement > if installed it also seems possible to parallelize the call-out to > kernel-doc from docproc.c without too much effort. I hoped Jani would > get > around to implement the asciidoctor support, and I'm hoping I can > snipe > away some free sometimes the next few months to look at docproc.c > more > seriously. This would kinda be a cool intern project, but atm we > throw > them all at improving testing infrastructure ... > > Anyway I'm of course still open to get this upstream, and I think a > few > things should be polished (like the speed-up). But right now > bandwidth on > my side isn't too plentiful. Maybe we should aim to have a few better > ideas (perhaps even for all of the docs stuff) for next KS and respin > that > discussion? I was just about to reply to the thread.... looking at the linux.conf.au schedule it would seem that you are both attending and presenting, and there appears to be some sort of Documentation mini -summit on the Monday as well (not sure if that is the place for a discussion though). I will be at LCA for the Wed-Fri. You may not have to wait until the next KS? Graham > > Thanks, Daniel
On Tue, Jan 12, 2016 at 11:06:17AM +0000, Graham Whaley wrote: > On Tue, 2016-01-12 at 09:34 +0100, Daniel Vetter wrote: > > On Mon, Jan 11, 2016 at 06:12:12PM -0700, Jonathan Corbet wrote: > > > On Sat, 12 Dec 2015 12:13:45 +0100 > > > Daniel Vetter <daniel@ffwll.ch> wrote: > > > > > > > I just figured there's no way this could get it, and I'd > > > > much rather improve the docs themselves than trying to convince > > > > core > > > > kernel folks that this might be useful. > > > > > > So I'm not quite sure why you figured that; I never said it, > > > certainly. > > > > To clarify this wasn't really my impression of your stance, but of > > the > > overall room opinion when we had the discussion at KS. And then my > > main > > goal here is to write great docs for drm (we have about 3k lines more > > docs > > in 4.5 already), so that's why I dropped the ball on upstreaming. It > > seemed unlikely to succeed, at least without some really seriuos > > effort at > > convincing everyone, all while the drm docs for atomic haven't been > > in > > good shape yet. Since then we had a few contributors of new atomic > > drivers > > note on irc already that "oh cool, this is documented now". Overall > > really > > just boils down to what I see as the most important things for drm ; > > -) > > > > > I've been messing with it a bit, seems to work. I do still wish we > > > could > > > consider alternatives, especially those that might simplify the > > > toolchain > > > rather than complicating it. But it's clear that I'm not > > > succeeding in > > > finding time to actually explore that idea; the contents of > > > $EXCUSES are > > > good, but the end result is the same. And the patch fairy just > > > isn't > > > coming through for me on this one. > > > > > > In my mind, there's clearly no good that can come from (further) > > > delaying > > > something that works in favor of an "it would be nice" that may > > > never > > > even exist. So I'm currently thinking that I'll pull this into the > > > docs > > > tree once the merge window is done, with the plan to push it for > > > 4.6. > > > Then we can see if anybody screams. > > > > > > That gives a couple of weeks for an updated patch set, should you > > > have > > > one. > > > > > > The build-time increase is painful in the extreme - about a factor > > > of > > > three for a -j1 build, and that's with only one file using the > > > feature. > > > It feels wrong, somehow, for the docs build to take longer than > > > building > > > the kernel itself. Can we do something about that? > > > > > > - How many of the comments actually use asciidoc features? Might > > > there > > > be some possibility of detecting those in kernel-doc and > > > skipping the > > > callout to asciidoc when it's not needed? > > > > I think that amounts to writing a partial parser (we use asciidoc for > > tables, lists, links, formatting, code snippets by now already, > > someone > > even thought of using the asciiart->png feature it has but it's not > > yet > > wired up). I don't think it's feasible. > > > > > - Pandoc seems to do asciidoc. I still don't like the idea of > > > depending > > > on it for this to work, but having the *option* to use it is > > > fine. If > > > it's really that much faster (yes, Python startup is painful) > > > then > > > maybe providing the option is worth it. > > > > Hm, Dave asked me to convert to use python-based asciidoc insted of > > haskell-based pandoc. > > > > > - All over the kernel we've seen that batching improves > > > performance. It > > > would take a bit of work, but I bet kernel-doc could put > > > together all > > > the snippets from one file, pass them through a single asciidoc > > > invocation, then split the results back apart. That would > > > probably > > > eliminate the performance hit entirely. > > > > > > None of that is a condition for pulling this stuff in, but can it > > > be > > > looked into? > > > > Besides what Jani mention that asciidoctor should be a drop-in > > replacement > > if installed it also seems possible to parallelize the call-out to > > kernel-doc from docproc.c without too much effort. I hoped Jani would > > get > > around to implement the asciidoctor support, and I'm hoping I can > > snipe > > away some free sometimes the next few months to look at docproc.c > > more > > seriously. This would kinda be a cool intern project, but atm we > > throw > > them all at improving testing infrastructure ... > > > > Anyway I'm of course still open to get this upstream, and I think a > > few > > things should be polished (like the speed-up). But right now > > bandwidth on > > my side isn't too plentiful. Maybe we should aim to have a few better > > ideas (perhaps even for all of the docs stuff) for next KS and respin > > that > > discussion? > > I was just about to reply to the thread.... looking at the > linux.conf.au schedule it would seem that you are both attending and > presenting, and there appears to be some sort of Documentation mini > -summit on the Monday as well (not sure if that is the place for a > discussion though). I will be at LCA for the Wed-Fri. You may not have > to wait until the next KS? Sounds like a great idea to pick this up at lca and toss around for some. -Daniel
On Tue, 12 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote: > On Sat, 12 Dec 2015 12:13:45 +0100 > Daniel Vetter <daniel@ffwll.ch> wrote: > >> I just figured there's no way this could get it, and I'd >> much rather improve the docs themselves than trying to convince core >> kernel folks that this might be useful. > > So I'm not quite sure why you figured that; I never said it, certainly. > > I've been messing with it a bit, seems to work. I do still wish we could > consider alternatives, especially those that might simplify the toolchain > rather than complicating it. But it's clear that I'm not succeeding in > finding time to actually explore that idea; the contents of $EXCUSES are > good, but the end result is the same. And the patch fairy just isn't > coming through for me on this one. > > In my mind, there's clearly no good that can come from (further) delaying > something that works in favor of an "it would be nice" that may never > even exist. So I'm currently thinking that I'll pull this into the docs > tree once the merge window is done, with the plan to push it for 4.6. > Then we can see if anybody screams. > > That gives a couple of weeks for an updated patch set, should you have > one. > > The build-time increase is painful in the extreme - about a factor of > three for a -j1 build, and that's with only one file using the feature. > It feels wrong, somehow, for the docs build to take longer than building > the kernel itself. Can we do something about that? > > - How many of the comments actually use asciidoc features? Might there > be some possibility of detecting those in kernel-doc and skipping the > callout to asciidoc when it's not needed? > > - Pandoc seems to do asciidoc. I still don't like the idea of depending > on it for this to work, but having the *option* to use it is fine. If > it's really that much faster (yes, Python startup is painful) then > maybe providing the option is worth it. > > - All over the kernel we've seen that batching improves performance. It > would take a bit of work, but I bet kernel-doc could put together all > the snippets from one file, pass them through a single asciidoc > invocation, then split the results back apart. That would probably > eliminate the performance hit entirely. I had another look at the whole thing, inspired by your lwn article [1]. I am guessing the whole design of calling the markup tool (asciidoc or pandoc or whatever) from kernel-doc for every documentation comment comes from trying really hard to minimize changes to the rest of the documentation system. From trying not to touch the DocBook parts so much. And thus the documentation build works really hard to convert thousands of markup snippets to DocBook and then again back to a bunch of other formats like html or pdf. What if we added support for some markup language as an alternative to DocBook for the high level documentation? What if we taught kernel-doc to output said markup natively, and included those generated pieces into the high level documentation using the markup's own include directives? At least AsciiDoc and reStructuredText support this. kernel-doc already supports several output formats, DocBook, HTML, plain text, man, whatnot. It should not be a big deal to add another. Even though I don't do perl (and that's the main reason I've generally steered clear anything kernel-doc) I toyed with adding reStructuredText support, and I got some sensible output surprisingly quickly. This would mean *one* kernel-doc invocation per source file included, and *one* markup tool invocation per high level document. Additionally there would have to be dependency generation from the high level document to the included files. As a bonus, one could write (and *read*!) the high level documentation in a markup that's meant for humans. If needed, at least AsciiDoc supports generating DocBook, which I suppose could be fed back to the existing documentation pipeline. > None of that is a condition for pulling this stuff in, but can it be > looked into? To be clear, I also don't want this idea to block the code that we have now from being merged. This is a longer term thing anyway. But I'd like to explore the alternatives. Thoughts? BR, Jani. [1] https://lwn.net/Articles/671496/
On Thu, 14 Jan 2016 22:03:26 +0200 Jani Nikula <jani.nikula@linux.intel.com> wrote: > What if we added support for some markup language as an alternative to > DocBook for the high level documentation? What if we taught kernel-doc > to output said markup natively, and included those generated pieces into > the high level documentation using the markup's own include directives? > At least AsciiDoc and reStructuredText support this. That is kind of what I've been thinking. I think we could dispense with docbook entirely, simplifying the toolchain and making the template files easier to read and edit. Something like that would also make it easy to keep the regular .txt documents in a structured form and to integrate them into the "formatted" documents if it makes sense. Jani, want to join us at LCA and talk about it? :) jon
On Thu, 14 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote: > On Thu, 14 Jan 2016 22:03:26 +0200 > Jani Nikula <jani.nikula@linux.intel.com> wrote: > >> What if we added support for some markup language as an alternative to >> DocBook for the high level documentation? What if we taught kernel-doc >> to output said markup natively, and included those generated pieces into >> the high level documentation using the markup's own include directives? >> At least AsciiDoc and reStructuredText support this. > > That is kind of what I've been thinking. I think we could dispense with > docbook entirely, simplifying the toolchain and making the template files > easier to read and edit. Something like that would also make it easy to > keep the regular .txt documents in a structured form and to integrate them > into the "formatted" documents if it makes sense. Thanks, this is enough encouragement that maybe I'll toy around with this a little more in my, uh, copious free time. > Jani, want to join us at LCA and talk about it? :) Heh, sure I'd *want* to... BR, Jani.
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 5335955c0de5..b655343631cf 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -17,7 +17,7 @@ DOCBOOKS := z8530book.xml device-drivers.xml \ tracepoint.xml gpu.xml media_api.xml w1.xml \ writing_musb_glue_layer.xml crypto-API.xml iio.xml -MARKDOWNREADY := +MARKDOWNREADY := gpu.xml include Documentation/DocBook/media/Makefile