WIP Rework mne bids path match

[waldie11]

PR Description

The workflow of using pathlib.Path.rglob and subsequently filtering i.e. by each path starting with root/sub-* is rather non-performant, especially on polluted storage trees.

For starters, by using glob.glob recursively directly, it is possible to apply this filter equivalent immediately.

In an extension, I am suggesting to introduce the option of include_match to get_entity_vals.

Open to suggestions, or stripping this down.

Remark: I failed to join the CI as my email adress is considered work mail for some reason by MS.

Merge checklist

Maintainer, please confirm the following before merging.
If applicable:

• All comments are resolved
• This is not your own PR
• All CIs are happy
• PR title starts with [MRG]
• whats_new.rst is updated
• New contributors have been added to CITATION.cff
• PR description includes phrase "closes <#issue-number>"

[welcome]

Hello! 👋 Thanks for opening your first pull request here!
Please read the contributor guide, and please follow the steps outlined in the "Instructions for first-time contributors" section therein. ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[hoechenberger]

it's part of the standard library and doesn't need to be listed here

[waldie11]

thx, this comes up though 🤔

/home/circleci/project/mne_bids/path.py:docstring of mne_bids.BIDSPath.root:1: WARNING: py:class reference target not found: pathlib._local.Path [ref.class]

[drammock]

See #1358 (comment) . #1353 includes a work around for this but hasn't been merged yet

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.46%. Comparing base (ccfbfee) to head (970751c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1355      +/-   ##
==========================================
+ Coverage   97.43%   97.46%   +0.03%     
==========================================
  Files          40       40              
  Lines        8966     9009      +43     
==========================================
+ Hits         8736     8781      +45     
+ Misses        230      228       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sappelhoff]

thanks @waldie11! we have a bit of a backlog right now so please don't worry if we take a bit longer to get back to you.

[waldie11]

@sappelhoff happy new year!
no worries, as i had seen some of the api changes in python-mne bringing down the test suite, i already expected priorities to be elsewhere.

[sappelhoff]

Hey @waldie11 could you please add an example for the feature that you introduce? Either by modifying an actual example that already exists, or by adding a numpy docstr example below the ones that already exist.

Could you please also resolve the conflicts AND follow the steps for first time contributors here? -->#1354

[waldie11]

Hi @sappelhoff ,

I think the true meat is rather a rework of _return_root_paths. This is not exactly a API function, so it feels a bit weird to write an example.

get_entity_vals seems to undergo some changes within upcoming 0.17 anyhow. I extended the docstring a bit on my feature contribution. Do you think this is sufficient? Neither the ignore_ parameters have an example. It benefits filtering a bids_path, which has deep directory trees for sourcedata, derivatives, subject or who-knows-what, where one wants to select prehand in which directories to look.

my_dataset/
  derivatives/
    downsampled/
      sub-01/
        micr/
          sub-01_sample-01_res-4x_TEM.png
          sub-01_sample-01_res-4x_TEM.json
  sub-01/
    micr/
      sub-01_sample-01_TEM.png
      sub-01_sample-01_TEM.json

(BIDS v1.8.0-1 p. 214)

[waldie11]

Idk how far you are interested in diving into this:

in test_path_benchmark I created a dummy bids compliant tree. If I browse this artificial tree with get_entity_vals, I extract a performance gain in about one order of magnitude by using include_match="sub-*/" in comparison to ignore_dirs=["derivatives", "sourcedata"]

This benchmark is eating up some CI runtime though. I tried keeping it lightweight.