diff --git a/LICENSE b/LICENSE index bb8c36f5..ebf05087 100644 --- a/LICENSE +++ b/LICENSE @@ -28,3 +28,33 @@ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +--- + +bench/samples/*.md: + +With the exception of `bench/samples/README.md`, the samples in +`bench/samples` are taken from https://github.com/markdown-it/markdown-it. + +Copyright (c) 2014 Vitaly Puzrin, Alex Kocharin. + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/Makefile b/Makefile index 74ce5621..d76d073c 100644 --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ BENCHINP?=README.md JSMODULES=$(wildcard lib/*.js) VERSION?=$(SPECVERSION) -.PHONY: dingus dist test bench npm lint clean update-spec +.PHONY: dingus dist test bench bench-detailed npm lint clean update-spec dist: dist/commonmark.js dist/commonmark.min.js @@ -27,7 +27,7 @@ lint: bench: sudo renice 99 $$$$; node bench/bench.js ${BENCHINP} -bench-detailed: bench/samples +bench-detailed: sudo renice 99 $$$$; \ for x in bench/samples/*.md; do echo $$x; node bench/bench.js $$x; done | \ tee out | awk -f bench/format_benchmarks.awk diff --git a/bench/samples/README.md b/bench/samples/README.md new file mode 100644 index 00000000..4bbac88e --- /dev/null +++ b/bench/samples/README.md @@ -0,0 +1,285 @@ +CommonMark +========== + +CommonMark is a rationalized version of Markdown syntax, +with a [spec][the spec] and BSD3-licensed reference +implementations in C and JavaScript. + +[Try it now!](http://spec.commonmark.org/dingus.html) + +The implementations +------------------- + +The C implementation provides both a shared library (`libcmark`) and a +standalone program `cmark` that converts CommonMark to HTML. It is +written in standard C99 and has no library dependencies. The parser is +very fast (see [benchmarks](benchmarks.md)). + +It is easy to use `libcmark` in python, lua, ruby, and other dynamic +languages: see the `wrappers/` subdirectory for some simple examples. + +The JavaScript implementation provides both an NPM package and a +single JavaScript file, with no dependencies, that can be linked into +an HTML page. For further information, see the +[README in the js directory](js/README.md). + +**A note on security:** +Neither implementation attempts to sanitize link attributes or +raw HTML. If you use these libraries in applications that accept +untrusted user input, you must run the output through an HTML +sanitizer to protect against +[XSS attacks](http://en.wikipedia.org/wiki/Cross-site_scripting). + +Installing (C) +-------------- + +Building the C program (`cmark`) and shared library (`libcmark`) +requires [cmake]. If you modify `scanners.re`, then you will also +need [re2c], which is used to generate `scanners.c` from +`scanners.re`. We have included a pre-generated `scanners.c` in +the repository to reduce build dependencies. + +If you have GNU make, you can simply `make`, `make test`, and `make +install`. This calls [cmake] to create a `Makefile` in the `build` +directory, then uses that `Makefile` to create the executable and +library. The binaries can be found in `build/src`. + +For a more portable method, you can use [cmake] manually. [cmake] knows +how to create build environments for many build systems. For example, +on FreeBSD: + + mkdir build + cd build + cmake .. # optionally: -DCMAKE_INSTALL_PREFIX=path + make # executable will be created as build/src/cmark + make test + make install + +Or, to create Xcode project files on OSX: + + mkdir build + cd build + cmake -G Xcode .. + make + make test + make install + +The GNU Makefile also provides a few other targets for developers. +To run a benchmark: + + make bench + +To run a "fuzz test" against ten long randomly generated inputs: + + make fuzztest + +To run a test for memory leaks using `valgrind`: + + make leakcheck + +To reformat source code using `astyle`: + + make astyle + +To make a release tarball and zip archive: + + make archive + + +Compiling for Windows +--------------------- + +To compile with MSVC and NMAKE: + + nmake + +You can cross-compile a Windows binary and dll on linux if you have the +`mingw32` compiler: + + make mingw + +The binaries will be in `build-mingw/windows/bin`. + +Installing (JavaScript) +----------------------- + +The JavaScript library can be installed through `npm`: + + npm install commonmark + +This includes a command-line converter called `commonmark`. + +If you want to use it in a client application, you can fetch +a pre-built copy of `commonmark.js` from +. + +For further information, see the +[README in the js directory](js/README.md). + +The spec +-------- + +[The spec] contains over 500 embedded examples which serve as conformance +tests. To run the tests using an executable `$PROG`: + + python3 test/spec_tests.py --program $PROG + +If you want to extract the raw test data from the spec without +actually running the tests, you can do: + + python3 test/spec_tests.py --dump-tests + +and you'll get all the tests in JSON format. + +[The spec]: http://spec.commonmark.org/0.13/ + +The source of [the spec] is `spec.txt`. This is basically a Markdown +file, with code examples written in a shorthand form: + + . + Markdown source + . + expected HTML output + . + +To build an HTML version of the spec, do `make spec.html`. To build a +PDF version, do `make spec.pdf`. (Creating a PDF requires [pandoc] +and a LaTeX installation. Creating the HTML version requires only +`libcmark` and `python3`.) + +The spec is written from the point of view of the human writer, not +the computer reader. It is not an algorithm---an English translation of +a computer program---but a declarative description of what counts as a block +quote, a code block, and each of the other structural elements that can +make up a Markdown document. + +Because John Gruber's [canonical syntax +description](http://daringfireball.net/projects/markdown/syntax) leaves +many aspects of the syntax undetermined, writing a precise spec requires +making a large number of decisions, many of them somewhat arbitrary. +In making them, we have appealed to existing conventions and +considerations of simplicity, readability, expressive power, and +consistency. We have tried to ensure that "normal" documents in the many +incompatible existing implementations of Markdown will render, as far as +possible, as their authors intended. And we have tried to make the rules +for different elements work together harmoniously. In places where +different decisions could have been made (for example, the rules +governing list indentation), we have explained the rationale for +my choices. In a few cases, we have departed slightly from the canonical +syntax description, in ways that we think further the goals of Markdown +as stated in that description. + +For the most part, we have limited ourselves to the basic elements +described in Gruber's canonical syntax description, eschewing extensions +like footnotes and definition lists. It is important to get the core +right before considering such things. However, we have included a visible +syntax for line breaks and fenced code blocks. + +Differences from original Markdown +---------------------------------- + +There are only a few places where this spec says things that contradict +the canonical syntax description: + +- It allows all punctuation symbols to be backslash-escaped, + not just the symbols with special meanings in Markdown. We found + that it was just too hard to remember which symbols could be + escaped. + +- It introduces an alternative syntax for hard line + breaks, a backslash at the end of the line, supplementing the + two-spaces-at-the-end-of-line rule. This is motivated by persistent + complaints about the “invisible” nature of the two-space rule. + +- Link syntax has been made a bit more predictable (in a + backwards-compatible way). For example, `Markdown.pl` allows single + quotes around a title in inline links, but not in reference links. + This kind of difference is really hard for users to remember, so the + spec allows single quotes in both contexts. + +- The rule for HTML blocks differs, though in most real cases it + shouldn't make a difference. (See the section on HTML Blocks + for details.) The spec's proposal makes it easy to include Markdown + inside HTML block-level tags, if you want to, but also allows you to + exclude this. It is also makes parsing much easier, avoiding + expensive backtracking. + +- It does not collapse adjacent bird-track blocks into a single + blockquote: + + > this is two + + > blockquotes + + > this is a single + > + > blockquote with two paragraphs + +- Rules for content in lists differ in a few respects, though (as with + HTML blocks), most lists in existing documents should render as + intended. There is some discussion of the choice points and + differences in the subsection of List Items entitled Motivation. + We think that the spec's proposal does better than any existing + implementation in rendering lists the way a human writer or reader + would intuitively understand them. (We could give numerous examples + of perfectly natural looking lists that nearly every existing + implementation flubs up.) + +- The spec stipulates that two blank lines break out of all list + contexts. This is an attempt to deal with issues that often come up + when someone wants to have two adjacent lists, or a list followed by + an indented code block. + +- Changing bullet characters, or changing from bullets to numbers or + vice versa, starts a new list. We think that is almost always going + to be the writer's intent. + +- The number that begins an ordered list item may be followed by + either `.` or `)`. Changing the delimiter style starts a new + list. + +- The start number of an ordered list is significant. + +- Fenced code blocks are supported, delimited by either + backticks (```` ``` ```` or tildes (` ~~~ `). + +Contributing +------------ + +There is a [forum for discussing +CommonMark](http://talk.commonmark.org); you should use it instead of +github issues for questions and possibly open-ended discussions. +Use the [github issue tracker](http://github.com/jgm/CommonMark/issues) +only for simple, clear, actionable issues. + +Authors +------- + +The spec was written by John MacFarlane, drawing on + +- his experience writing and maintaining Markdown implementations in several + languages, including the first Markdown parser not based on regular + expression substitutions ([pandoc](http://github.com/jgm/pandoc)) and + the first markdown parsers based on PEG grammars + ([peg-markdown](http://github.com/jgm/peg-markdown), + [lunamark](http://github.com/jgm/lunamark)) +- a detailed examination of the differences between existing Markdown + implementations using [BabelMark 2](http://johnmacfarlane.net/babelmark2/), + and +- extensive discussions with David Greenspan, Jeff Atwood, Vicent + Marti, Neil Williams, and Benjamin Dumke-von der Ehe. + +John MacFarlane was also responsible for the original versions of the +C and JavaScript implementations. The block parsing algorithm was +worked out together with David Greenspan. Vicent Marti +optimized the C implementation for performance, increasing its speed +tenfold. Kārlis Gaņģis helped work out a better parsing algorithm +for links and emphasis, eliminating several worst-case performance +issues. Nick Wellnhofer contributed many improvements, including +most of the C library's API and its test harness. Vitaly Puzrin +has offered much good advice about the JavaScript implementation. + +[cmake]: http://www.cmake.org/download/ +[pandoc]: http://johnmacfarlane.net/pandoc/ +[re2c]: http://re2c.org + diff --git a/bench/samples/block-bq-flat.md b/bench/samples/block-bq-flat.md new file mode 100644 index 00000000..33e382af --- /dev/null +++ b/bench/samples/block-bq-flat.md @@ -0,0 +1,16 @@ +> the simple example of a blockquote +> the simple example of a blockquote +> the simple example of a blockquote +> the simple example of a blockquote +... continuation +... continuation +... continuation +... continuation + +empty blockquote: + +> +> +> +> + diff --git a/bench/samples/block-bq-nested.md b/bench/samples/block-bq-nested.md new file mode 100644 index 00000000..7ddcffa4 --- /dev/null +++ b/bench/samples/block-bq-nested.md @@ -0,0 +1,13 @@ +>>>>>> deeply nested blockquote +>>>>> deeply nested blockquote +>>>> deeply nested blockquote +>>> deeply nested blockquote +>> deeply nested blockquote +> deeply nested blockquote + +> deeply nested blockquote +>> deeply nested blockquote +>>> deeply nested blockquote +>>>> deeply nested blockquote +>>>>> deeply nested blockquote +>>>>>> deeply nested blockquote diff --git a/bench/samples/block-code.md b/bench/samples/block-code.md new file mode 100644 index 00000000..2b1554f4 --- /dev/null +++ b/bench/samples/block-code.md @@ -0,0 +1,11 @@ + + an + example + + of + + + + a code + block + diff --git a/bench/samples/block-fences.md b/bench/samples/block-fences.md new file mode 100644 index 00000000..5fe8b3c3 --- /dev/null +++ b/bench/samples/block-fences.md @@ -0,0 +1,14 @@ + +``````````text +an +example +``` +of + + +a fenced +``` +code +block +`````````` + diff --git a/bench/samples/block-heading.md b/bench/samples/block-heading.md new file mode 100644 index 00000000..fd985587 --- /dev/null +++ b/bench/samples/block-heading.md @@ -0,0 +1,9 @@ +# heading +### heading +##### heading + +# heading # +### heading ### +##### heading \#\#\#\#\###### + +############ not a heading diff --git a/bench/samples/block-hr.md b/bench/samples/block-hr.md new file mode 100644 index 00000000..e1bad6f4 --- /dev/null +++ b/bench/samples/block-hr.md @@ -0,0 +1,10 @@ + + * * * * * + + - - - - - + + ________ + + + ************************* text + diff --git a/bench/samples/block-html.md b/bench/samples/block-html.md new file mode 100644 index 00000000..ff7f8fa6 --- /dev/null +++ b/bench/samples/block-html.md @@ -0,0 +1,32 @@ +
+ +blah blah + +
+ + + + + +
+ **test** +
+ + + + + + + + + +
+ + test + +
+ + + diff --git a/bench/samples/block-lheading.md b/bench/samples/block-lheading.md new file mode 100644 index 00000000..e5c0d99d --- /dev/null +++ b/bench/samples/block-lheading.md @@ -0,0 +1,8 @@ +heading +--- + +heading +=================================== + +not a heading +----------------------------------- text diff --git a/bench/samples/block-list-flat.md b/bench/samples/block-list-flat.md new file mode 100644 index 00000000..14149dbf --- /dev/null +++ b/bench/samples/block-list-flat.md @@ -0,0 +1,67 @@ + - tidy + - bullet + - list + + + - loose + + - bullet + + - list + + + 0. ordered + 1. list + 2. example + + + - + - + - + - + + + 1. + 2. + 3. + + + - an example +of a list item + with a continuation + + this part is inside the list + + this part is just a paragraph + + + 1. test + - test + 1. test + - test + + +111111111111111111111111111111111111111111. is this a valid bullet? + + - _________________________ + + - this + - is + + a + + long + - loose + - list + + - with + - some + + tidy + + - list + - items + - in + + - between + - _________________________ diff --git a/bench/samples/block-list-nested.md b/bench/samples/block-list-nested.md new file mode 100644 index 00000000..d30aed3b --- /dev/null +++ b/bench/samples/block-list-nested.md @@ -0,0 +1,36 @@ + + - this + - is + - a + - deeply + - nested + - bullet + - list + + + 1. this + 2. is + 3. a + 4. deeply + 5. nested + 6. unordered + 7. list + + + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + + + - - - - - - - - - deeply-nested one-element item + diff --git a/bench/samples/block-ref-flat.md b/bench/samples/block-ref-flat.md new file mode 100644 index 00000000..c83dccb6 --- /dev/null +++ b/bench/samples/block-ref-flat.md @@ -0,0 +1,15 @@ +[1] [2] [3] [1] [2] [3] + +[looooooooooooooooooooooooooooooooooooooooooooooooooong label] + + [1]: + [2]: http://something.example.com/foo/bar 'test' + [3]: + http://foo/bar + [ looooooooooooooooooooooooooooooooooooooooooooooooooong label ]: + 111 + 'test' + [[[[[[[[[[[[[[[[[[[[ this should not slow down anything ]]]]]]]]]]]]]]]]]]]]: q + (as long as it is not referenced anywhere) + + [[[[[[[[[[[[[[[[[[[[]: this is not a valid reference diff --git a/bench/samples/block-ref-nested.md b/bench/samples/block-ref-nested.md new file mode 100644 index 00000000..1e10a8c2 --- /dev/null +++ b/bench/samples/block-ref-nested.md @@ -0,0 +1,17 @@ +[[[[[[[foo]]]]]]] + +[[[[[[[foo]]]]]]]: bar +[[[[[[foo]]]]]]: bar +[[[[[foo]]]]]: bar +[[[[foo]]]]: bar +[[[foo]]]: bar +[[foo]]: bar +[foo]: bar + +[*[*[*[*[foo]*]*]*]*] + +[*[*[*[*[foo]*]*]*]*]: bar +[*[*[*[foo]*]*]*]: bar +[*[*[foo]*]*]: bar +[*[foo]*]: bar +[foo]: bar diff --git a/bench/samples/inline-autolink.md b/bench/samples/inline-autolink.md new file mode 100644 index 00000000..0f71482d --- /dev/null +++ b/bench/samples/inline-autolink.md @@ -0,0 +1,14 @@ +closed (valid) autolinks: + + + + + + +these are not autolinks: + + diff --git a/bench/samples/inline-backticks.md b/bench/samples/inline-backticks.md new file mode 100644 index 00000000..a6ec6e1f --- /dev/null +++ b/bench/samples/inline-backticks.md @@ -0,0 +1,3 @@ +`lots`of`backticks` + +``i``wonder``how``this``will``be``parsed`` diff --git a/bench/samples/inline-em-flat.md b/bench/samples/inline-em-flat.md new file mode 100644 index 00000000..b7668a54 --- /dev/null +++ b/bench/samples/inline-em-flat.md @@ -0,0 +1,5 @@ +*this* *is* *your* *basic* *boring* *emphasis* + +_this_ _is_ _your_ _basic_ _boring_ _emphasis_ + +**this** **is** **your** **basic** **boring** **emphasis** diff --git a/bench/samples/inline-em-nested.md b/bench/samples/inline-em-nested.md new file mode 100644 index 00000000..6bb0a0d3 --- /dev/null +++ b/bench/samples/inline-em-nested.md @@ -0,0 +1,5 @@ +*this *is *a *bunch* of* nested* emphases* + +__this __is __a __bunch__ of__ nested__ emphases__ + +***this ***is ***a ***bunch*** of*** nested*** emphases*** diff --git a/bench/samples/inline-em-worst.md b/bench/samples/inline-em-worst.md new file mode 100644 index 00000000..b6d21da9 --- /dev/null +++ b/bench/samples/inline-em-worst.md @@ -0,0 +1,5 @@ +*this *is *a *worst *case *for *em *backtracking + +__this __is __a __worst __case __for __em __backtracking + +***this ***is ***a ***worst ***case ***for ***em ***backtracking diff --git a/bench/samples/inline-entity.md b/bench/samples/inline-entity.md new file mode 100644 index 00000000..da095edb --- /dev/null +++ b/bench/samples/inline-entity.md @@ -0,0 +1,11 @@ +entities: + +  & © Æ Ď ¾ ℋ ⅆ ∲ + +# Ӓ Ϡ � + +non-entities: + +&18900987654321234567890; &1234567890098765432123456789009876543212345678987654; + +&qwertyuioppoiuytrewqwer; &oiuytrewqwertyuioiuytrewqwertyuioytrewqwertyuiiuytri; diff --git a/bench/samples/inline-escape.md b/bench/samples/inline-escape.md new file mode 100644 index 00000000..4e1bb396 --- /dev/null +++ b/bench/samples/inline-escape.md @@ -0,0 +1,15 @@ + +\t\e\s\t\i\n\g \e\s\c\a\p\e \s\e\q\u\e\n\c\e\s + +\!\\\"\#\$\%\&\'\(\)\*\+\,\.\/\:\;\<\=\>\? + +\@ \[ \] \^ \_ \` \{ \| \} \~ \- \' + +\ +\\ +\\\ +\\\\ +\\\\\ + +\ \ \ \ + diff --git a/bench/samples/inline-html.md b/bench/samples/inline-html.md new file mode 100644 index 00000000..f6e63419 --- /dev/null +++ b/bench/samples/inline-html.md @@ -0,0 +1,44 @@ +Taking commonmark tests from the spec for benchmarking here: + + + + + + + + + +<33> <__> + + + + + +foo + +foo + +foo + +foo + +foo &<]]> + + + + + + diff --git a/bench/samples/inline-links-flat.md b/bench/samples/inline-links-flat.md new file mode 100644 index 00000000..5117db87 --- /dev/null +++ b/bench/samples/inline-links-flat.md @@ -0,0 +1,23 @@ +Valid links: + + [this is a link]() + [this is a link]() + [this is a link](http://something.example.com/foo/bar 'test') + ![this is an image]() + ![this is an image]() + ![this is an image](http://something.example.com/foo/bar 'test') + + [escape test](<\>\>\>\>\>\>\>\>\>\>\>\>\>\>> '\'\'\'\'\'\'\'\'\'\'\'\'\'\'') + [escape test \]\]\]\]\]\]\]\]\]\]\]\]\]\]\]\]](\)\)\)\)\)\)\)\)\)\)\)\)\)\)) + +Invalid links: + + [this is not a link + + [this is not a link]( + + [this is not a link](http://something.example.com/foo/bar 'test' + + [this is not a link]((((((((((((((((((((((((((((((((((((((((((((((( + + [this is not a link]((((((((((()))))))))) (((((((((())))))))))) diff --git a/bench/samples/inline-links-nested.md b/bench/samples/inline-links-nested.md new file mode 100644 index 00000000..4e7dc857 --- /dev/null +++ b/bench/samples/inline-links-nested.md @@ -0,0 +1,13 @@ +Valid links: + +[[[[[[[[](test)](test)](test)](test)](test)](test)](test)] + +[ [[[[[[[[[[[[[[[[[[ [](test) ]]]]]]]]]]]]]]]]]] ](test) + +Invalid links: + +[[[[[[[[[ + +[ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ + +![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![![ diff --git a/bench/samples/inline-newlines.md b/bench/samples/inline-newlines.md new file mode 100644 index 00000000..068a807d --- /dev/null +++ b/bench/samples/inline-newlines.md @@ -0,0 +1,24 @@ + +this\ +should\ +be\ +separated\ +by\ +newlines + +this +should +be +separated +by +newlines +too + +this +should +not +be +separated +by +newlines + diff --git a/bench/samples/rawtabs.md b/bench/samples/rawtabs.md new file mode 100644 index 00000000..dc989ea7 --- /dev/null +++ b/bench/samples/rawtabs.md @@ -0,0 +1,18 @@ + +this is a test for tab expansion, be careful not to replace them with spaces + +1 4444 +22 333 +333 22 +4444 1 + + + tab-indented line + space-indented line + tab-indented line + + +a lot of spaces in between here + +a lot of tabs in between here +