From 19868a284deeeb30906ebfe1aa8312fcbf5ec51f Mon Sep 17 00:00:00 2001
From: Robin Schneider <ypid@riseup.net>
Date: Sat, 28 May 2016 17:54:23 +0200
Subject: [PATCH 1/5] Make examples HTML build deterministic by removing the
 version number.

---
 Makefile           | 2 +-
 examples/main.html | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index deafea9..5ccd027 100644
--- a/Makefile
+++ b/Makefile
@@ -54,7 +54,7 @@ examples: PYTHONPATH = .
 examples:
 	PYTHONPATH=.
 	bin/yaml2rst examples/main.yml examples/main.rst
-	rst2html --stylesheet=examples/demo.css examples/main.rst > examples/main.html
+	rst2html --stylesheet=examples/demo.css examples/main.rst | grep --invert-match --fixed-strings '<meta name="generator"' > examples/main.html
 	rst2html --stylesheet=examples/demo.css tests/patternsTest.rst > tests/patternsTest.html
 
 #-- interaction with PyPI
diff --git a/examples/main.html b/examples/main.html
index 35e89be..9363a0d 100644
--- a/examples/main.html
+++ b/examples/main.html
@@ -3,7 +3,6 @@
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
 <title>collectd</title>
 <style type="text/css">
 

From 3702f9b9c92a5ec98d3b6f4b8be2fb3ec130052b Mon Sep 17 00:00:00 2001
From: Robin Schneider <ypid@riseup.net>
Date: Sat, 28 May 2016 18:09:50 +0200
Subject: [PATCH 2/5] Filter out Fold markers.

It started out when I introduced more and more fold markers to the DebOps project. I used `(((` as marker because it was not highlighted by my editor as being a Jinja2 control sequence. The fold marker as turned out to be not-optimal because it also appears in complex Jinja2 conditions quite often. So @drybjed and me settled on `[[[`. The problem was, however, that we always needed to introduce additional redundancy to the files. Example:

```YAML
---

```

@drybjed and I discussed this and agreed that the best thing would be to patch up `yaml2rst` to filter out the fold marker which would allow it to be directly appended after headings for instance. Example:

```YAML
---

```

With this patch, this input gets converted into:

```reStructuredText
.. vim: foldmarker=[[[,]]]:foldmethod=marker

Default variables
=================

.. contents:: Sections
   :local:

----------------------
  Role configuration
----------------------
```

Can we make this the default behavior or should it be made optional? I limited the regex to only match a very defined set of occurrences.
---
 examples/main.yml | 2 +-
 yaml2rst.py       | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/examples/main.yml b/examples/main.yml
index 6eb1d57..d71f913 100644
--- a/examples/main.yml
+++ b/examples/main.yml
@@ -88,7 +88,7 @@ collectd_plugins_default_config:
     IgnoreSelected true
 
 #
-# Indention of Literal Blocks behind Lists
+# Indention of Literal Blocks behind Lists [[[1
 # -----------------------------------------
 #
 # Code will properly be indented after lists:
diff --git a/yaml2rst.py b/yaml2rst.py
index 6b4c429..9e04b0e 100644
--- a/yaml2rst.py
+++ b/yaml2rst.py
@@ -104,6 +104,9 @@ def convert(lines):
         elif line.startswith('# ') or line == '#':
             if state != STATE_TEXT:
                 yield ''
+            # Filter out [[[\d and ]]] at the end of a documentation line.
+            # Those sequences can be used for folding sections.
+            line = re.sub(r"\s*(:?\[{3}|\]{3})\d?$", "", line)
             line = last_text_line = line[2:]
             yield line
             last_indent = get_indent(line) * ' '

From 63e020dd513dc283c513c0364ac1cb177dd5ad7d Mon Sep 17 00:00:00 2001
From: Robin Schneider <ypid@riseup.net>
Date: Thu, 2 Jun 2016 23:56:41 +0200
Subject: [PATCH 3/5] Made fold marker strip configurable via `--strip-regex`.

---
 Makefile              |  2 +-
 README.rst            | 17 +++++++++++++----
 bin/yaml2rst          |  9 +++++++--
 tests/test_convert.py | 25 ++++++++++++++++++++++---
 yaml2rst.py           | 12 ++++++++----
 5 files changed, 51 insertions(+), 14 deletions(-)

diff --git a/Makefile b/Makefile
index 5ccd027..f298351 100644
--- a/Makefile
+++ b/Makefile
@@ -53,7 +53,7 @@ dist:
 examples: PYTHONPATH = .
 examples:
 	PYTHONPATH=.
-	bin/yaml2rst examples/main.yml examples/main.rst
+	bin/yaml2rst examples/main.yml examples/main.rst --strip-regex '\s*(:?\[{3}|\]{3})\d?$$'
 	rst2html --stylesheet=examples/demo.css examples/main.rst | grep --invert-match --fixed-strings '<meta name="generator"' > examples/main.html
 	rst2html --stylesheet=examples/demo.css tests/patternsTest.rst > tests/patternsTest.html
 
diff --git a/README.rst b/README.rst
index e344f3a..3974832 100644
--- a/README.rst
+++ b/README.rst
@@ -21,14 +21,19 @@ allows to process the YAML file directly without any pre-processing.
 
 Usage::
 
-  yaml2rst [-h] infile outfile
+  yaml2rst [-h] [--strip-regex regex] infile outfile
 
   positional arguments:
-    infile      YAML-file to read (`-` for stdin)
-    outfile     rst-file to write (`-` for stdout)
+    infile               YAML-file to read (`-` for stdin)
+    outfile              rst-file to write (`-` for stdout)
 
   optional arguments:
-    -h, --help  show this help message and exit
+    -h, --help           show this help message and exit
+    --strip-regex regex  Regex which will remove everything it matches. Can be
+                         used to remove fold markers from headings for example.
+                         Example to strip out [[[,]]] fold markers:
+                         '\s*(:?\[{3}|\]{3})\d?$'. Check the README for more
+                         details.
 
 
 How it works
@@ -43,6 +48,10 @@ Additionally at the start and at the end of a "code"-block, lines are
 added as required by reStructuredText. Also at the begin of a
 "code"-block, a ``::`` is added if required.
 
+``--strip-regex`` can be used to remove matching characters from text-lines
+when needed. Refer to documentation about
+`Folding marks support <docs/fold-markers.rst>`_ for details.
+
 
 Examples
 -------------
diff --git a/bin/yaml2rst b/bin/yaml2rst
index ff669c9..16344fb 100755
--- a/bin/yaml2rst
+++ b/bin/yaml2rst
@@ -32,7 +32,7 @@ import argparse
 import sys
 
 
-def main(infilename, outfilename):
+def main(infilename, outfilename, strip_regex):
     if infilename == '-':
         infh = sys.stdin
     else:
@@ -41,7 +41,7 @@ def main(infilename, outfilename):
         outfh = sys.stdout
     else:
         outfh = open(outfilename, "w")
-    for l in yaml2rst.convert(infh.readlines()):
+    for l in yaml2rst.convert(infh.readlines(), strip_regex):
         print(l.rstrip(), file=outfh)
 
 
@@ -50,5 +50,10 @@ parser.add_argument('infilename', metavar='infile',
                     help="YAML-file to read (`-` for stdin)")
 parser.add_argument('outfilename', metavar='outfile',
                     help="rst-file to write (`-` for stdout)")
+parser.add_argument('--strip-regex', metavar='regex',
+                    help="Regex which will remove everything it matches."
+                    " Can be used to remove fold markers from headings for example."
+                    " Example to strip out [[[,]]] fold markers: '\s*(:?\[{3}|\]{3})\d?$'."
+                    " Check the README for more details.")
 args = parser.parse_args()
 main(**vars(args))
diff --git a/tests/test_convert.py b/tests/test_convert.py
index 799ce01..3795b32 100644
--- a/tests/test_convert.py
+++ b/tests/test_convert.py
@@ -53,14 +53,14 @@ def _write_pattern(self, *lines):
             print(line, file=self._outfile)
         print(file=self._outfile)
 
-    def _test(self, text, expected):
+    def _test(self, text, expected, strip_regex=None):
         text = textwrap.dedent(text)
         if isinstance(expected, basestring):
             expected = textwrap.dedent(expected).splitlines()
         self._write_pattern(*expected)
-        res = list(yaml2rst.convert(text.splitlines()))
+        res = list(yaml2rst.convert(text.splitlines(), strip_regex))
         self.assertListEqual(expected, res)
-    
+
     def test_no_text_at_all(self):
         text = """\
         ---
@@ -265,3 +265,22 @@ def test_nested_enumeration(self):
              Some code under list-entry 2
         """
         self._test(text, expected)
+
+    def test_strip_regex(self):
+        text = """\
+        ---
+        # Heading [[[1
+        # =======
+        key: value
+        # ]]]
+        """
+        expected = """\
+        Heading
+        =======
+        ::
+
+          key: value
+
+
+        """
+        self._test(text, expected, '\s*(:?\[{3}|\]{3})\d?$')
diff --git a/yaml2rst.py b/yaml2rst.py
index 9e04b0e..1d6fc05 100644
--- a/yaml2rst.py
+++ b/yaml2rst.py
@@ -92,7 +92,13 @@ def get_indent(line):
     return indent
 
 
-def convert(lines):
+def get_stripped_line(line, strip_regex):
+    if strip_regex:
+        line = re.sub(strip_regex, "", line)
+    return line
+
+
+def convert(lines, strip_regex):
     state = STATE_TEXT
     last_text_line = ''
     last_indent = ''
@@ -104,9 +110,7 @@ def convert(lines):
         elif line.startswith('# ') or line == '#':
             if state != STATE_TEXT:
                 yield ''
-            # Filter out [[[\d and ]]] at the end of a documentation line.
-            # Those sequences can be used for folding sections.
-            line = re.sub(r"\s*(:?\[{3}|\]{3})\d?$", "", line)
+            line = get_stripped_line(line, strip_regex)
             line = last_text_line = line[2:]
             yield line
             last_indent = get_indent(line) * ' '

From 267c7c2c24c0575d3b535d7221f462f89f329090 Mon Sep 17 00:00:00 2001
From: Robin Schneider <ypid@riseup.net>
Date: Fri, 3 Jun 2016 00:00:21 +0200
Subject: [PATCH 4/5] Wrote documentation about using fold markers.

---
 docs/fold-markers.rst | 103 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)
 create mode 100644 docs/fold-markers.rst

diff --git a/docs/fold-markers.rst b/docs/fold-markers.rst
new file mode 100644
index 0000000..286ecfe
--- /dev/null
+++ b/docs/fold-markers.rst
@@ -0,0 +1,103 @@
+==========================
+Folding marks support
+==========================
+
+Folding marks in text editors like Vim can be quite helpful when you are used to
+them. One common way to fold different sections of your file is to use a fold
+marker. The thing is that you probably don’t want your fold markers ending up
+in the rendered documentation.
+
+So to use folds in your files, you basally have two options:
+
+#. Use reStructuredText comments for fold markers. Example:
+
+   .. code:: yaml
+
+      ---
+      # .. vim: foldmarker=[[[,]]]:foldmethod=marker
+
+      # .. contents:: Sections
+      #    :local:
+
+      # .. First heading [[[
+      #
+      # -----------------
+      #   First heading
+      # -----------------
+
+      # .. envvar:: example__list
+      #
+      # Some text describing this list.
+      example__list:
+        - 'Something'
+        - 'Something'
+
+      # .. ]]]
+
+      # .. Second heading [[[1
+      #
+      # ------------------
+      #   Second heading
+      # ------------------
+
+#. Append your fold markers directly after reStructuredText headings. Example:
+
+   .. code:: yaml
+
+      ---
+      # .. vim: foldmarker=[[[,]]]:foldmethod=marker
+
+      # .. contents:: Sections
+      #    :local:
+
+      # First heading [[[
+      # -------------
+
+      # .. envvar:: example__list
+      #
+      # Some text describing this list.
+      example__list:
+        - 'Something'
+        - 'Something'
+
+      # .. ]]]
+
+      # Second heading [[[1
+      # --------------
+      #
+
+Both variants have their pros and cons and both are supported by `yaml2rst`, so
+you can choose.
+
+Advantages of using reStructuredText comments for fold markers:
+
+* It works out of the box. You will not need to do anything extra as it only
+  depends on reStructuredText comments for the fold markers.
+  (For this, the ``--strip-regex`` command line option is not required.)
+
+* Section "overlines" don’t come in the way when moving folds around. This can
+  be an advantage when you are already using them.
+
+Advantages of appending your fold markers directly after reStructuredText headings:
+
+* No redundancy. (Don't repeat yourself)
+
+Because the "Don't repeat yourself" argument is actually quantifiable and has
+an impact on your ability to maintain your files it should be preferred when
+starting freshly. The pro arguments for fold markers in reStructuredText
+comments are just style questions.
+
+Which fold marker to use
+------------------------
+
+Now a short hash up regarding which fold markers to use.
+
+* ``{{{`` is the Vim default. Has the disadvantage that it might gets in the way
+  with your syntax highlighting. However, '{{{' is not expected to be usually
+  found anywhere else in Ansible/YAML/Jinja.
+
+* ``(((`` is more likely to appear in Ansible/YAML/Jinja.
+
+* ``[[[`` is not expected to appear in Ansible/YAML/Jinja. Current recommendation.
+
+Refer to the `PR about fold markers support <https://github.com/htgoebel/yaml2rst/pull/3>`_ for more details.

From e1245bec52e09c682e12353b71c317992f68d719 Mon Sep 17 00:00:00 2001
From: Robin Schneider <ypid@riseup.net>
Date: Fri, 3 Jun 2016 18:03:31 +0200
Subject: [PATCH 5/5] Removed ref to PR in docs as requested by @htgoebel.

---
 docs/fold-markers.rst | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/docs/fold-markers.rst b/docs/fold-markers.rst
index 286ecfe..f258744 100644
--- a/docs/fold-markers.rst
+++ b/docs/fold-markers.rst
@@ -92,12 +92,10 @@ Which fold marker to use
 
 Now a short hash up regarding which fold markers to use.
 
-* ``{{{`` is the Vim default. Has the disadvantage that it might gets in the way
+* ``{{{`` is the Vim default. Has the disadvantage that it might get in the way
   with your syntax highlighting. However, '{{{' is not expected to be usually
   found anywhere else in Ansible/YAML/Jinja.
 
 * ``(((`` is more likely to appear in Ansible/YAML/Jinja.
 
 * ``[[[`` is not expected to appear in Ansible/YAML/Jinja. Current recommendation.
-
-Refer to the `PR about fold markers support <https://github.com/htgoebel/yaml2rst/pull/3>`_ for more details.
