Add AsciiDoc Reader / AsciiDoc input support #1456
Comments
|
I have started working on this, it might not be ready for a few months. |
|
Have you tried going AsciiDoc -> DocBook -> Pandoc? Can you describe the shortcomings of doing this if you have? |
|
Thanks for your positive comment! I personally have not tried the conversion chain as you mentioned. The current choices are bulky with regards to its dependencies, while not being and not aspiring to be universal markup translators, like pandoc. Going through intermediate formats instead of one pandoc invocation seems hacky to me. Recently, I read about two publishing houses switching away from LaTeX and moving to AsciiDoc as their source format, so I gather that it fulfills the needs of technical writing well, also regarding referencing and I find it useful to have another capable plain-text format for documents, if software support is good and can easily convert between different formats and offers multiple output choices. Which is where this feature would come in ;-) I personally would also like to write articles and possibly a book rather in AsciiDoc than LaTeX, which is - at least my personal - motivation for this feature. |
|
👍 I would love to see support for reading asciidoc files in pandoc. It was actually my primary reason to use pandoc. At the moment i have a bit of a hacky solution converting my asciidoc into html and then feeding that into pandoc. I would love to eliminate the extra project dependency and simplify my build chain. |
|
I can testify that this would be a very useful addition to pandoc, thanks in advance mr. mpickering! |
|
PR #2100 contributes a basic AsciiDoc reader (with many features not yet implemented). |
|
I commented on the #2100 |
|
I agree to that asciidoc is getting popular. I've just tried to convert the book Pro Git 2 into epub using pandoc and soon noticed that the book was coded in asciidoc, and pandoc was unable to read it. |
|
I’ve tried an asciidoc > html (via https://github.com/asciidoctor/asciidoctor) > epub (via pandoc) conversion chain and it works extremely well except for the following issue. Asciidoctor wraps all HTML elements in divs with additional classes. This stops pandoc from splitting the epub automatically at headings, because it will never see a 'naked' h1 etc. Also mentioned this issue here: asciidoctor/asciidoctor#184 The ePub file I created using this method did not pass epubcheck 3.0.1 because it contained one duplicate ID (something I could have avoided). More seriously, the way footnotes are handled is not compliant, and raised several errors like this: (paths truncated with ellipsis at start) At the risk of stating the obvious, it's important that pandoc-generated epubs from any source format avoid epubcheck validation errors, as authors and publishers may need to submit these epubs to storefronts that will require epubcheck compliance (i.e. Smashwords, iBooks). Many of the Github-hosted CLI epub generators I've tried (e.g. https://github.com/avdgaag/rpub) omit consideration of epubcheck compliance, so this may not be an obvious point after all. The most common point of failure seems to be the manifest, which pandoc does correctly, which is great. But it could be yet more robust, as the above errors indicate. |
|
+++ Ben Hourigan [Jul 05 15 08:56 ]:
You could handle this easily with a filter that strips out the
When I convert pandoc's README to epub3, I see no errors My guess is that the HTML footnotes produced by asciidoctor If you attach a short sample file (of HTML produced by Unfortunately, there's no standard way of doing footnotes |
|
Hope this is a sufficient sample: <div class="paragraph">
<p>… the kind of politics that the liberal economist F. A. Hayek called “socialist.” <span class="footnote">[<a id="_footnoteref_1" class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</span></p>
</div> |
|
Could you attach or link to the generated (noncompliant) epub itself? |
|
By the way, here's a simple filter ( import Text.Pandoc.JSON
main = toJSONFilter undiv
where undiv (Div (ident, ["content"], kvs) bs) = bs
undiv b = [b] |
|
Depending on how asciidoc formats the notes, you may be able to get the HTML reader to parse them as notes. If you use |
Sorry, this is a bit misleading. Since a filter is applied only after the HTML reader, this wouldn't work unless you first filtered, then piped the resulting HTML into another invocation of pandoc. Anyway, there are numerous tools you could use to insert the type attribute where it's needed in the HTML, before passing to pandoc. |
|
Or maybe asciidoctor could be persuaded to insert the needed |
|
Actually, rather than the epub itself, it would be most useful for me to have the HTML from which it was generated. |
|
Thanks for the filter. Will try this out. You can get the HTML file from https://www.dropbox.com/s/c2ror63pz16hc3w/2015-07-06-BH-STG-adoc-test.html?dl=0 |
|
I tried: So I edited adoc-test.html and changed one of the duplicate |
|
PS. You might have more success using asciidoc to produce DocBook, then converting that with pandoc. Have you tried that route? |
|
Ah, damn---I didn't think to do a version check. Sorry for being such a novice. I'd been using 1.13.2, which is the latest version on homebrew. Will install 1.15 and try again. Your results sound promising. |
|
BTW, as of pandoc 1.13.2, when I tried the asciidoctor docbook > pandoc epub route the output from docbook was inferior to the output from HTML. One particular thing that I noticed was that admonition blocks came in to the epub as blockquotes without an additional class, and so couldn't be styled specifically with CSS. |
|
Currently DocBook elements like and convert that to +++ Ben Hourigan [Jul 06 15 10:34 ]:
|
|
With the duplicate ID sorted in the .adoc file, from the pandoc 1.15 produces from the asciidoctor.html an epub that passes ePubcheck! :) Inability to split the file at chapter headings remains an issue. Now trying the undiv filter. At first I got this error: Then after installing ghc (7.8.4) and cabal-install (1.22.0.0) (not sure if latter was necessary) from homebrew, I got this error. Oddly, if I run All files used during generation here: https://www.dropbox.com/s/40uv3f4ad2fwuco/bh-undiv.hs-test.zip?dl=0 I'm running a script called generate.sh, which is just the following command: |
|
Is undiv.hs in your working directory? What OS are you on? +++ Ben Hourigan [Jul 06 15 18:13 ]:
|
|
@mb21 Thank you, I understood that. As I said, using line blocks syntax for verse would not be compatible with our guideline of keeping original text as clean and intact as possible. That is even more so for poetry. Thank you for pointer to rewriting divs in LaTeX. That is exactly what we need. I hope that ticket will be completed too, it seems nice to have same possibility in LaTeX as in HTML writers. Apology for creating noise in this ticket. After all, that is about asciidoc. Thank you so much to John MacFarlane and developers for the wonderful Pandoc! |
|
Note that content in fenced divs will be parsed as Markdown before it gets to your filter. So, to get line breaks between lines of verse, you'd have to insert them manually (two space or backslash at end of line) or enable the If you have further discussion on this topic, though, please move it to pandoc-discuss. |
|
Thank you so much for the valuable advice.
Thank you, I will. |
|
An asciidoc reader would also be very useful to me (to convert from asciidoc to rst) |
|
Regarding the DIVs problem mentioned above, I wanted to add that there's a new third party semantic HTML backend for Asciidoctor now (if I remember correctly, it was added after this thread began): https://github.com/jirutka/asciidoctor-html5s So this might be a viable workaround for the problems mentioned above concerning ePubs creation. |
2014 ... months? ;-) |
|
@gmarpons just posted asciidoc-hs on pandoc-discuss:
|
|
@ciampix Open Source Months are a fairly flexible unit of measure, each month being roughly equivalent to one promise on the more commercial 6-8 weeks scale. |
|
What feature branch is this issue in the github code please? I can't see any branch related to 1456 or asciidoc. |
|
There's this library being developed independently: |
|
I'm confused. https://pandoc.org/demos.html 28.
Why it supports only Also if as output I set |
|
Pandoc assumes that files with I'm not sure what you are asking in the second paragraph. Please raise this on the pandoc-discuss mailing list. We want to reserve this tracker for bug reports and feature requests, questions should go to the mailing list. |
|
|
We don't have to be convinced that it would be good to have an asciidoc reader. What is needed is just for someone to contribute one. asciidoc-hs noted above might be a good starting point if someone wants to take this on. In the mean time, you can probably get good results using asciidoctor to convert to HTML or DocBook and then converting the result using pandoc to your target format. |
Is It Viable?Since I use both pandoc and AsciiDoc (Asciidoctor Ruby) on a daily basis I'd really love to see a native AsciiDoc reader in pandoc, but ... having given it quite a lot of though, I'm just wondering whether it's viable. The AsciiDoc syntax is much richer and broader than pandoc, so it stands to reason that except for simple documents it will tend to be a lossy conversion. Which makes me wonder if it wouldn't make sense to approach the issue the other way round, and create a dedicated pandoc AsciiDoc backend instead. There are currently various AsciiDoc incarnations (Python, Ruby, JS, Java) and some of them might offer an easier approach to this task — indeed, the Haskell library mentioned by @jgm (gmarpons/asciidoc-hs) would pave the path to possibly including a full-fledged AsciiDoc parser into pandoc. I did look into this for Asciidoctor Ruby, but at the time the AST wasn't documented enough to make the task easy. I'm not sure if the situation has changed, but once you have access the AsciiDoc document in AST format, converting it to a pandoc AST should be fairly straightforward (but still be a lossy conversion, e.g. in case of book Parts, callouts, and other features not present in pandoc native elements). So, ideally, the creation of an AsciiDoc backend capable of emitting a pandoc AST would solve the problem, even though it might not be as practical as having pandoc natively support AsciiDoc as an input format. What I'm trying to say is that, given that in many cases the conversion could be lossy, it might make more sense to handle it on the AsciiDoc side, which might make it easier to control how unsupported elements are to be rendered (or ignored) during conversion. Having to control such settings from within pandoc might be rather complicated, and surely not something that could be easily handled via command line switches (maybe a YAML settings file? or a dedicated defaults file?). The Haskell AsciiDoc library could indeed be a game changer in this respect, since it would allow inclusion of a real AsciiDoc parser into pandoc. But even so, the question of how to handle unsupported elements would have to be faced. I've used various intermediate formats as a workaround for AsciiDoc to pandoc conversion, but the issues with unsupported features (and what to do with/to them) remains. DocBook is probably the closest syntax to AsciiDoc, and for simple documents it works pretty well out of the box. |
|
Yes, exactly. It would certainly be lossy. I just assumed that this was
what explained our lack of it. Are there any other lossy inputs? How they
deal with the problem?
…On Fri, Dec 6, 2024 at 6:10 PM Tristano Ajmone ***@***.***> wrote:
Is It Viable?
Since I use both pandoc and AsciiDoc (Asciidoctor Ruby) on a daily basis
I'd really love to see a native AsciiDoc reader in pandoc, but ... having
given it quite a lot of though, I'm just wondering whether it's viable.
The AsciiDoc syntax is much richer and broader than pandoc, so it stands
to reason that except for simple documents it will tend to be a lossy
conversion. Which makes me wonder if it wouldn't make sense to approach the
issue the other way round, and create a dedicated pandoc AsciiDoc backend
instead. There are currently various AsciiDoc incarnations (Python, Ruby,
JS, Java) and some of them might offer an easier approach to this task —
indeed, the Haskell library mentioned bu @jgm <https://github.com/jgm>
(gmarpons/asciidoc-hs) would pave the path to possibly including a
full-fledged AsciiDoc parser into pandoc.
I did look into this for Asciidoctor Ruby, but at the time the AST wasn't
documented enough to make the task easy. I'm not sure if the situation has
changed, but once you have access the AsciiDoc document in AST format,
converting it to a pandoc AST should be fairly straightforward (but still
be a lossy conversion, e.g. in case of book Parts, callouts, and other
features not present in pandoc native elements).
So, ideally, the creation of an AsciiDoc backend capable of emitting a
pandoc AST would solve the problem, even though it might not be as
practical as having pandoc natively support AsciiDoc as an input format.
What I'm trying to say is that, given that in many cases the conversion
could be lossy, it might make more sense to handle it on the AsciiDoc side,
which might make it easier to control how unsupported elements are to be
rendered (or ignored) during conversion. Having to control such settings
from within pandoc might be rather complicated, and surely not something
that could be easily handled via command line switches (maybe a YAML
settings file? or a dedicated defaults file?).
The Haskell AsciiDoc library could indeed be a game changer in this
respect, since it would allow inclusion of a real AsciiDoc parser into
pandoc. But even so, the question of how to handle unsupported elements
would have to be faced.
I've used various intermediate formats as a workaround for AsciiDoc to
pandoc conversion, but the issues with unsupported features (and what to do
with/to them) remains. DocBook is probably the closest syntax to AsciiDoc,
and for simple documents it works pretty well out of the box.
—
Reply to this email directly, view it on GitHub
<#1456 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABUYZYP7YX5CKULV7UBM3D2EIVHBAVCNFSM6AAAAABTE3ZLO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRUGU4DONBRGM>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
--
"I have delusions of grandeur. I believe myself to be Steve Jones"
|
|
Sure, many formats are lossy, as explained at the beginning of the manual. Pandoc just does its best. And we can do a lot with Div and Span elements in the AST; with these, we can probably capture most or perhaps all of the semantic information in an asciidoc document. |
|
@tajmone thanks for the information! Is there a comparison between their respective features? |
I can't think of a single document with a direct comparison of features, especially the pandoc Markdown flavor vs AsciiDoc. The AsciiDoc Language Documentation is a good source of information when it comes to AsciiDoc features, and contains various Appendixes and summaries which come in handy. A main obstacle that comes to mind is the level of book partitioning supported by AsciiDoc, which includes book parts and special sections, for which pandoc doesn't have any equivalent native elements, so I don't see how such documents could be converted without loosing the book structure. In theory, it should be possible to use Lua filters to emulate book parts, e.g. by using Level 1 headings as Part titles and shifting down one level all chapter titles, etc., e.g. in chunked documents, but this would also require to manually rebuild the ToC to keep into account how headings are shifted within book parts, compared to sections which are not inside parts (Introduction, Appendices, etc.). But then, since there are many possible ways to organize such book partitioning schemes, it might be hard to find a unified solution. But even for simple documents, elements like admonitions, the various block types supported by AsciiDoc, callouts in code listings, etc., will be problematic to render in pandoc without agreeing on some non-native ways to represent them, according to format (e.g. in HTML admonitions could be represented by divs with agreed-upon classes) and elements like callouts might require dedicated Lua filters in ordered to be recreated in pandoc markdown and be usable in different output formats. Similar choices would inevitable be arbitrary and somewhat opinionated, unless the list of native elements is expanded to officially accommodate them as first class citizens of the pandoc AST. AsciiDoc also relies quite heavily on attributes (native or custom) which allow conditional behaviors via preprocessor directives, e.g. to include or exclude certain blocks of contents, or to dynamically include external files (recursively). It's somewhat similar to how pandoc templates work, except that it takes place directly inside the source document, whereas with pandoc similar functionality only applies to templates, not the body text. A native pandoc reader would have to resolve all the preprocessing before conversion, which seems quite a lot of work and its the reason why I think it might make more sense to be handling it from within AsciiDoc itself, since a same source project might produce different results depending on the output format (chunked HTML, EPub, DocBook, etc.), so preserving the various preprocessor directives allows better control over different editions. The real problem here is that in order to be able to properly support AsciiDoc, the list of pandoc native elements would have to be expanded to include book parts, special sections, and possibly support admonitions. These are not the only elements available in AsciiDoc and absent in pandoc, but they are definitely good examples of commonly used elements which are going to be problematic to handle. |
|
As I understand it, AsciiDoc's document model is a subset of DocBook's. We support DocBook as an input format, so there is no reason we couldn't support AsciiDoc -- and the same sorts of tradeoffs and workarounds would be needed. If someone is interested in doing this, the best way forward would be to develop a separate Haskell library with an AST that is tailored to AsciiDoc. Pandoc could then add an AsciiDoc reader that calls this library's parser and then "downshifts" from its AST to the Pandoc AST. The library could also be used independently for people who want the Asciidoc-tailored AST. In the mean time, I think you should get decent results by using asciidoctor to produce docbook and then converting that with pandoc. |
|
The following example generates Markdown with headings shifted down one level (e.g., asciidoctor -b docbook5 -o - README.adoc | pandoc -f docbook -t markdownTo generate Markdown with the same heading level as AsciiDoc, you need to specify asciidoctor -b docbook5 -o - -a leveloffset=+1 README.adoc | pandoc -f docbook -t markdownI think this is a bit inconvenient. |
and others
Greetings,
I would like to hereby suggest the addition of AsciiDoc input resp. an AsciiDoc Reader.
Besides Markdown, this format is growing in popularity, also in use inside a publishing toolchain (asciidoc -> docbook -> pdf/epub/html). Currently the only other viable implementation is asciidoctor, which uses Ruby or JRuby, but it is AsciiDoc-only in its input format and not a universal markup converter, like pandoc.
I am aware of only one relevant discussion thread regarding this, which showed positive echo for this feature. Someone actually had some basic code there, but I am not sure if kuznero resp. Roman Kuznetsov still has his code from back then to start from, but anyway, it would certainly make sense to have AsciiDoc input in the feature set.
The text was updated successfully, but these errors were encountered: