Minor tweaks

CONTRIBUTING.rst

@@ -47,11 +47,11 @@ Coding guidelines
 
 It is recommended to check that your contribution complies with the following rules before submitting a pull request:
 
-* CLI applications (i.e. Python modules that have an entrypoint + manpage in `pyproject.toml <./pyproject.toml>`__) should have informative docstrings and arguments, with usage examples presented as argparse epilogues. All CLIs and plugins should be described in the Sphinx RTD documentation.
-* Docstrings are formatted in `Sphinx style <https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html>`__
-* New modules or functionality should be accompanied with `type hints <https://docs.python.org/3/library/typing.html>`__ and new (py)tests
-* All tests performed with `tox <https://tox.wiki>`__ must pass (python environments can be skipped, if at least one of them is checked)
+* CLI applications (i.e. Python modules that have an entrypoint + manpage in `pyproject.toml <./pyproject.toml>`__) should have informative docstrings and arguments, with usage examples presented as argparse epilogues. All CLIs and plugins must be described in the Sphinx RTD documentation
+* Docstrings should be formatted in `Sphinx style <https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html>`__
+* New modules, classes or functionality should be accompanied with `type hints <https://docs.python.org/3/library/typing.html>`__ and new (py)tests
+* All tests performed with `tox <https://tox.wiki>`__ must pass (python environments may be skipped, if at least one of them is checked)
 * Screens are wide nowadays, so the PEP directives for short code lines is considered outdated and does not have to be respected
-* To improve code readability, minor comments can (should) be appended at the end of the code lines they apply to
+* To improve code readability, minor comments can (should) be appended at the end of the code lines they apply to (even if that means right scrolling)
 * Horizontal space is not limited, so multi-line readability is preferred, e.g. the vertical alignment of ``=`` operators (i.e. padded horizontally with whitespaces)
-* Vertical space should not be readily wasted to promote better overviews and minimize the need for scrolling
+* Vertical space should not be readily wasted to promote better overviews and minimize the need for vertical scrolling

README.rst

@@ -51,11 +51,7 @@ Features
 How to contribute
 -----------------
 
-Are you a Python programmer with an interest in BIDS who knows about GE and/or Philips data?
-Are you experienced with parsing stimulus presentation log-files? Or do you have ideas to improve
-BIDScoin or its documentation? Have you come across bugs? In any case, you are more than welcome
-to provide feedback and to `contribute <https://github.com/Donders-Institute/bidscoin/blob/master/CONTRIBUTING.rst>`__
-to this project.
+Are you a Python programmer with an interest in BIDS who knows about GE and/or Philips data? Are you experienced with parsing stimulus presentation log-files? Or do you have ideas to improve BIDScoin or its documentation? Have you come across bugs? In any case, you are more than welcome to provide feedback and to `contribute to this project <https://github.com/Donders-Institute/bidscoin/blob/master/CONTRIBUTING.rst>`__.
 
 .. |PyPI version| image:: https://img.shields.io/pypi/v/bidscoin?color=success
    :target: https://pypi.org/project/bidscoin

docs/CHANGELOG.md

@@ -1,6 +1,6 @@
 # Changelog
 
-All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
+*All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*
 
 ## [dev]
 

docs/bidsmap.rst

@@ -45,25 +45,32 @@ Source property and attribute values of run-items in a bidsmap are interpreted a
 
 Dynamic values
 ^^^^^^^^^^^^^^
-Dictionary values can be static, in which case the value is just a normal string, or dynamic, when the string is enclosed with single or double pointy brackets. In case of single pointy brackets the bids value will be replaced during bidsmapper, bidseditor and bidscoiner runtime by the value of the source attribute or property of the data sample at hand. It is also possible to then extract a substring from the source string by adding a colon-separated regular expression to the bids value. For instance the two dynamic values in ``{acq: <MRAcquisitionType>Demo<SeriesDescription:t1_(.*?)_sag>}`` will be replaced by ``{acq: 3DDemoMPRAGE}`` if the 'MRAcquisitionType' of the data sample is '3D' and 'SeriesDescription' is 't1_MPRAGE_sag_p2_iso_1.0'. More precisely, the Python expression that is evaluated for the second  dynamic 'SeriesDescription' value is: ``substring = re.findall('t1_(.*?)_sag', 't1_mprage_sag_p2_iso_1.0')``. If dynamic values are enclosed with double pointy brackets, the only difference is that they will be replaced only during bidscoiner runtime -- this is useful for bids values that are subject/session dependent. Double bracket dynamic values can for instance be used to add DICOM meta data that is not saved by default in the json sidecar files, such as <<ImageComments>> or <<RadionuclideTotalDose>>. Another example is the extraction of the subject and session label. For instance, you can use ``<<filepath:/sub-(.*?)/>>`` to extract '003' (i.e. the shortest string between ``/sub-`` and ``/``) if the data for that subject is in ``/data/raw/sub-003/ses-01``. Alternatively, if the subject label is encoded in the DICOM ``PatientName`` as e.g. ``ID_003_anon``, then ``<<PatientName:ID_(.*?)_>>`` would likewise extract '003'. To test out dynamic values (either with or without appended regular expressions), you can handily enter them in the bidseditor within single brackets to instantly obtain their resulting value. Dynamic values can handle many use cases and can be used throughout BIDScoin.
+Dictionary values can be static, in which case the value is just a normal string, or dynamic, when the string is enclosed with single ``<>`` or double ``<<>>`` pointy brackets. The enclosed string is an attribute or property that will be extracted from the data source. In case of single pointy brackets the bids value will be replaced during bidsmapper, bidseditor and bidscoiner runtime by the value of the source attribute or property of the data sample at hand. Single brackets are typically used in template bidsmaps, meaning that you will not see them at all in the bidseditor (or anywhere after that), but instead you will see the actual values from the data. If the values are enclosed with double pointy brackets, then they won't be replaced until bidscoiner runtime. This means that they will be moved over to the study bidsmap unmodified, and that you can see, edit or add them yourself anywhere in the bidseditor. In the final BIDS output data, they will be replaced, just like the single bracket values. The rationale for using double bracket values is that certain properties or attributes vary from subject to subject, or even from acquisition to acquisition, and therefore they cannot directly be represented in the study bidsmap, i.e. their extraction needs to be postponed until bidscoiner runtime. For instance, suppose you want to include the scan operator's comments or the PET dose in the BIDS sidecar files, then you could add ``<<ImageComments>>`` and ``<<RadionuclideTotalDose>>`` as meta data values in your bidsmap.
+
+It is useful that dynamic values can extract source properties and attributes, but sometimes you may want to extract just a part of the value. That is where regular expressions come in. You can simply append a semi-colon to the property or attribute, followed by a `findall <https://docs.python.org/3/library/re.html#re.findall>`__ regexp pattern, which is then applied to the extracted value. For instance, say you want to extract the subject and session label from the filepath of your source data in ``/data/raw/sub-003/ses-01``. In your bidsmap you could then use ``{subject: <<filepath:/sub-(.*?)/>>}`` to evaluate ``re.findall('/sub-(.*?)/', '/data/raw/sub-003/ses-01')`` under the hood, and get ``{subject: 003}`` (i.e. the shortest string between ``/sub-`` and ``/``) during bidscoiner runtime. Alternatively, if the subject label is encoded in the DICOM ``PatientName`` attribute as e.g. ``ID_003_anon``, then ``{subject: <<PatientName:ID_(.*?)_>>}`` in your bidsmap would likewise evaluate ``re.findall('ID_(.*?)_', 'ID_003_anon')`` and give you ``{subject: 003}`` at bidscoiner runtime.
+
+As may have become clear from the above, dynamic values are BIDScoin's hidden powerhouse. But you can take it even one step further and make combinations of static and dynamic values. For instance the static and dynamic values in ``<MRAcquisitionType>Demo<SeriesDescription:t1_(.*?)_sag>`` will result in ``3DDemoMPRAGE`` if the 'MRAcquisitionType' of the data sample is '3D' and 'SeriesDescription' is 't1_MPRAGE_sag_p2_iso_1.0' (hint: the second dynamic value will evaluates ``re.findall('t1_(.*?)_sag', 't1_mprage_sag_p2_iso_1.0')`` to give ``MPRAGE``).
+
+.. tip::
+   Dynamic values with (or without) regular expressions can be hard to grasp and predict their outcome. To easily test out their working, you can just enter dynamic values in the bidseditor (in any value field) using single brackets and instantly obtain their resulting value.
 
 Run-index
 ^^^^^^^^^
 If the run index is encoded in the header or filename, then the index numer can be normally extracted using dynamic values. For instance, using ``{run: <<ProtocolName:run_nr-(.*?)_>>}`` will give ``{run: 3}`` if the DICOM ProtocolName is ``t1_mprage_sag_run_nr-3_iso_1.0``. Yet, if the index information is not available in the header or filename, then it needs to be determined from the presence of other files in the output directory. To handle that, the run-index in the bids output dictionary can be a 'dynamic number'. This dynamic number (e.g. ``{run: <<1>>}``) will be incremented during bidscoiner runtime if an output file with that run-index already exists (e.g. to become ``{run: 2}``). If the dynamic number if left empty (``{run: <<>>}``), then the run-index is omitted if only one run was acquired. If multiple runs were acquired, then ``{run: <<>>}`` will behave the same as ``{run: <<1>>}``.
 
 Fieldmaps: IntendedFor
 ^^^^^^^^^^^^^^^^^^^^^^
-According to the BIDS specification, the IntendedFor value of fieldmaps must be a list of relative pathnames of associated output files. However, these output files may vary from session to session, i.e. the 'IntendedFor' value is dependent on the presence of other files in the output folder. To handle that, the dynamic `IntendedFor` value of the meta dictionary can be specified using Unix shell-style wildcard search strings. In that way, during bidscoiner runtime, the exact paths of these images on disk will be looked up using the Python `glob <https://docs.python.org/3.8/library/pathlib.html#pathlib.Path.glob>`__(\*dynamic_value\*) expression. For instance, using a simple ``{IntendedFor: <<task>>}`` value will lookup all functional runs in the BIDS subject[/session] folder (since in BIDS these runs always have 'task' in their filename), whereas a more specific ``{IntendedFor: <<func/*Stop*Go_bold><func/*Reward*_bold>>}`` value will select all 'Stop1Go'-, 'Stop2Go'- and 'Reward' bold-runs in the func sub-folder. In case duplicated field maps are acquired (e.g. when a scan failed or a session was interrupted) you can limit the search scope by appending a colon-separated "bounding" term to the search pattern. E.g. ``{IntendedFor: <<task:[]>>}`` will bound the wildcard search to files that are 'uninterruptedly connected' to the current field map, i.e. without there being another run of the field map in between. The bounded search can be further constrained by limiting the maximum number of matches, indicated with lower and upper limits. For instance ``{IntendedFor: <<task:[-3:0]>>}`` will limit the bounded search to maximally three runs preceding the field map. Similarly, ``{IntendedFor: <<task:[-2:2]>>}`` will limit the bounded search to maximally two preceding and two subsequent runs, and ``{IntendedFor: <<task:[0:]>>}`` will limit the bounded search to all matches acquired after the field map. In this latter case, for the first field map, only ``task-Stop_run-1`` and ``task-Stop_run-2`` will match the bounded search if the 5 collected runs were named: 1) ``fieldmap_run-1``, 2) ``task-Stop_run-1``, 3) ``task-Stop_run-2``, 4) ``fieldmap_run-2``, 5) ``task-Stop_run-3``. The second run of the field map will match with ``task-Stop_run-3`` only (note that the second field map would have matched all task runs if the bounding term would have been ``[]``, ``[:]`` or ``[-2:2]``).
+According to the BIDS specification, the IntendedFor value of fieldmaps must be a list of relative pathnames of associated output files. However, these output files may vary from session to session, i.e. the 'IntendedFor' value is dependent on the presence of other files in the output folder. To handle that, the dynamic `IntendedFor` value of the meta dictionary can be specified using Unix shell-style wildcard search strings. In that way, during bidscoiner runtime, the exact paths of these images on disk will be looked up using the Python ``glob(*value*)`` expression (see `here <https://docs.python.org/3.8/library/pathlib.html#pathlib.Path.glob>`__ for the exact syntax). For instance, using a simple ``{IntendedFor: <<task>>}`` value will use ``glob(*task*)`` to lookup all functional runs in the BIDS subject[/session] folder (since in BIDS these runs always have 'task' in their filename), whereas a more advanced ``{IntendedFor: <<func/*Stop*Go_bold><func/*Reward*_bold>>}`` value will select all 'Stop1Go'-, 'Stop2Go'- and 'Reward' bold-runs in the func sub-folder.
+
+In case duplicated field maps are acquired (e.g. when a scan failed or a session was interrupted) you can limit the search scope by appending a colon-separated "bounding" term to the search pattern. E.g. ``{IntendedFor: <<task:[]>>}`` will bound the wildcard search to files that are 'uninterruptedly connected' to the current field map, i.e. without there being another run of the field map in between. The bounded search can be further constrained by limiting the maximum number of matches, indicated with lower and upper limits. For instance ``{IntendedFor: <<task:[-3:0]>>}`` will limit the bounded search to maximally three runs preceding the field map. Similarly, ``{IntendedFor: <<task:[-2:2]>>}`` will limit the bounded search to maximally two preceding and two subsequent runs, and ``{IntendedFor: <<task:[0:]>>}`` will limit the bounded search to all matches acquired after the field map. In this latter case, for the first field map, only ``task-Stop_run-1`` and ``task-Stop_run-2`` will match the bounded search if the 5 collected runs were named: 1) ``fieldmap_run-1``, 2) ``task-Stop_run-1``, 3) ``task-Stop_run-2``, 4) ``fieldmap_run-2``, 5) ``task-Stop_run-3``. The second run of the field map will match with ``task-Stop_run-3`` only (note that the second field map would have matched all task runs if the bounding term would have been ``[]``, ``[:]`` or ``[-2:2]``).
 
 .. note::
 
    The ``IntendedFor`` field is a legacy way to deal with field maps. Instead, it is recommended to use the ``B0FieldIdentifier`` and ``B0FieldSource`` fields that were `introduced with BIDS 1.7 <https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#using-b0fieldidentifier-metadata>`__
 
-* **BIDS value lists**. Instead of a normal string, a bids dictionary value can also be a list of strings, with the last list item being the (zero-based) list index that selects the actual value from the list. For instance the list ``{part: ['', 'mag', 'phase', 'real', 'imag', 2]}`` would select 'phase' as the value belonging to 'part'. A bids value list is made visible in the bidseditor as a drop-down menu in which the user can select the value (i.e. set the list index).
-
-.. tip::
-
-   In addition to DICOM attribute names, the more advanced / unambiguous pydicom-style `tag numbers <https://pydicom.github.io/pydicom/stable/old/base_element.html#tag>`__ can also be used for indexing a DICOM header. For instance, the ``PatientName``, ``0x00100010``, ``0x10,0x10``, ``(0x10, 0x10)``, and ``(0010, 0010)`` index keys are all equivalent.
+BIDS value lists
+^^^^^^^^^^^^^^^^
+Instead of a normal string, a bids dictionary value can also be a list of strings, with the last list item being the (zero-based) list index that selects the actual value from the list. For instance the list ``{part: ['', 'mag', 'phase', 'real', 'imag', 2]}`` would select 'phase' as the value belonging to 'part'. A bids value list is made visible in the bidseditor not as a list, but as as a user-friendly drop-down menu in which the user can choose between the values (i.e. set the list index).
 
 Building your own template bidsmap
 ----------------------------------
@@ -84,6 +91,7 @@ The run-items in the default 'bidsmap_dccn' template bidsmap have values that ar
    - Make a copy of the DCCN template (``[home]/.bidscoin/[version]/templates/bidsmap_dccn.yaml``) as a starting point for your own template bidsmap, and adapt it to your needs. You can set your copy as the new default template by editing the ``[home]/.bidscoin/config.toml`` file. Default templates and config file are automatically recreated from source when deleted
    - The power of regular expressions is nearly unlimited, you can e.g. use `negative look aheads <https://docs.python.org/3/howto/regex.html#lookahead-assertions>`__ to **not** match (exclude) certain strings
    - When creating new run-items, make sure to adhere to the YAML format and to the definitions in the BIDS schema files (``[path_to_bidscoin]/bidscoin/schema/datatypes``). You can test your YAML syntax using an online `YAML-validator <https://www.yamllint.com>`__ and your compliance with the BIDS standard with ``bidscoin -t your_template_bidsmap``. If all seems well you can install it using ``bidscoin -i your_template_bidsmap``.
+   - In addition to DICOM attribute names, the more advanced / unambiguous pydicom-style `tag numbers <https://pydicom.github.io/pydicom/stable/old/base_element.html#tag>`__ can also be used for indexing a DICOM header. For instance, the ``PatientName``, ``0x00100010``, ``0x10,0x10``, ``(0x10, 0x10)``, and ``(0010, 0010)`` index keys are all equivalent.
 
 Editing the template bidsmap
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

pyproject.toml

@@ -38,7 +38,7 @@ deface           = ['pydeface', 'drmaa']
 pet2bids         = ['pypet2bids >= 1.0.12']
 # phys2bidscoin    = ['bioread >= 1.0.5', 'pymatreader >= 0.0.24', 'duecredit', 'phys2bids >= 2.0.0, < 3.0.0']
 all              = ['bidscoin[dcm2niix2bids,spec2nii2bids,deface,pet2bids]']    # + phys2bidscoin
-dev              = ['bidscoin[spec2nii2bids,deface,pet2bids]', 'tox', 'pytest'] # + phys2bidscoin
+dev              = ['bidscoin[spec2nii2bids,deface,pet2bids]', 'tox', 'pytest', 'sphinx-rtd-theme', 'myst-parser'] # + phys2bidscoin
 
 [project.urls]
 documentation    = 'https://bidscoin.readthedocs.io'