[ENH] microelectrode electrophysiology specification (BEP032) #1705

yarikoptic · 2024-02-14T15:24:13Z

Replaces #1352 submitted from a fork outside of bids-specification.

Add specification for microelectrode electrohpysiology datasets based on the BEP032 proposal

Note

We meet regularly and everyone is welcome

Next meeting: insert date on URL to join

Communication channel: https://framalistes.org/sympa/info/neuroscience-data-structure

Tip

HTML preview of this BEP

To use this WiP schema on sample datasets, see https://deno.land/x/[email protected]#modifying-and-building-a-new-schema on how to use "stock" bids-validator with a custom schema. (attn @TheChymera)

TODOs

Issues this PR would likely to address

Fixes run node tests in github actions legacy-validator#1375

Issues to see being addressed while working on this BEP (likely to move above) or not (moved below):

Other issues which relate but not in scope here and provided for reference/backreference

… _acq-

src/schema/rules/files/raw/channels.yaml

TheChymera · 2024-03-06T16:39:46Z

src/schema/objects/columns.yaml

@@ -42,6 +42,38 @@ age:
    for privacy purposes.
  type: number
  unit: year
+alpha_rotation:


I'd really recommend renaming the rotation axes to yaw, roll, and pitch (that would be the analogous angle order). There was no consensus either way on the google docs discussion. Someone said both are confusing, which I guess might be expected, but alpha, beta, gamma, are just more confusing...

I agree -- very adhoc. Let's discuss in google doc

https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

Could you chime in? I think the other guy commenting might be amenable to accepting this as well.

I agree that both are confusing, but at least alpha, beta and gamma are SO confusing that everyone realizes that additional specification is needed to define them properly. With roll, yaw and pitch it seems at first that all is clear, until you have a number of different people go through different use cases. See the more challenging examples that I posted on the google doc under https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

@robertoostenveld sorry, I only now saw the update, reply there. I don't think confusion on purpose is good in this case, because the documentation is very eclectic so we might be sending people down rabbit holes. Wiki, where people will invariably go first, does a particularly poor job explaining both euler and TB angles for the casual non-mathematics-versed user. The only thing that wiki has going for it here are the aircraft animations on the TB page. Yaw, Pitch, Roll, will be intuited correctly as long as we specify the starting postion. That we can do (1) as (I think, it's pretty vague) is currently proposed, aligning the implant with the world coordinate system (meaning most implants will be at yaw 0 pitch -90 roll 0) or (2) relative to the implantation stereotax normal (meaning most implants will be at 0 0 0).

For comparison, Euler commonly has the normal pointing up so most implants will be.... 0 -180 0 🤔

in the last BEP032 meeting we discussed further and wanted to follow this approach now (based on the Allen Inst. standard and IBL standard):

Assumption: x,y,z is posterior, ventral, right (unit needs to be specified).

Translational origin (0,0,0) needs to be defined (typically Bregma for rodents).

Rotational origin (0,0,0) is the probe facing up with the tip facing forward. Rotations are all clockwise in degrees and around the tip. For multi-shank probes, the “tip” of the probe is defined as the end of the left shank if you are looking at the electrodes.

yaw: clockwise when looking down

pitch: In the direction of the electrode face

roll: clockwise when looking down at the probe

The depth (unit needs to be specified) is a translation in the direction that the tip is pointing.

We need to add a “probe_model”, which references probeinterface_library

versioning of probe library files (raised here)

annotation of “tip” position (raised here)

NOTE: We need to change the electrode x,y,z.
X,y,z in BIDS refers to location in brain, not on probe.

TheChymera · 2024-03-06T16:46:27Z

src/schema/objects/columns.yaml

@@ -391,6 +462,12 @@ reference__ieeg:
    - type: string
      enum:
        - n/a
+reference_atlas:


The “reference atlas” if you visually verify the area is basically the atlas you looked at before you nod and said “eh, good enough”. This seems at once an overly detailed (does this really matter? the coordinates are commonly given with sub-mm precision) and underdetermined (what coordinates did you look at in the atlas?). I think it's best just to drop this.

TheChymera · 2024-03-12T01:33:27Z

Also relevant if you'd like to comment. → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIzHGpUU

TheChymera · 2024-03-12T01:57:39Z

Also I forgot to link to this when I wrote it → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIGPAMOw

src/schema/rules/files/raw/ephys.yaml

…t was missing

src/schema/objects/datatypes.yaml

src/schema/objects/modalities.yaml

src/schema/objects/suffixes.yaml

Remi-Gau · 2024-04-19T07:56:45Z

Did a few minor fix to pacify pre-commit and make sure the HTML page rendered.

Also check the top message of this PR:
#1705 (comment)

I added some github admonitions that need tweaking so people know where this BEP is discussed.

…ext menu

I think this is what might have freaked out `schemacode_ci / windows-latest with Python 3 (pull_request) ` (windows only!) fails with a bunch of ``` 2024-04-19T19:13:44.2815607Z self = <encodings.cp1252.IncrementalDecoder object at 0x0000028D6A14D850> 2024-04-19T19:13:44.2818292Z input = b'---\nHED:\n name: HED\n display_name: HED Tag\n description: |\n Hierarchical Event Descriptor (HED) Tag.\n ...ed or ideal position along the z axis.\n anyOf:\n - type: number\n - type: string\n enum:\n - n/a\n' 2024-04-19T19:13:44.2820386Z final = True 2024-04-19T19:13:44.2820609Z 2024-04-19T19:13:44.2820821Z def decode(self, input, final=False): 2024-04-19T19:13:44.2821664Z > return codecs.charmap_decode(input,self.errors,decoding_table)[0] 2024-04-19T19:13:44.2823260Z E UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2105: character maps to <undefined> 2024-04-19T19:13:44.2824229Z 2024-04-19T19:13:44.2824768Z C:\hostedtoolcache\windows\Python\3.12.3\x64\Lib\encodings\cp1252.py:23: UnicodeDecodeError ```

…w others

codecov · 2024-04-19T22:39:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.92%. Comparing base (1cb92eb) to head (73012b3).
Report is 36 commits behind head on master.

❗ Current head 73012b3 differs from pull request most recent head a46e439

Please upload reports for the commit a46e439 to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    bids-standard/legacy-validator#1705      +/-   ##
==========================================
+ Coverage   87.79%   87.92%   +0.13%     
==========================================
  Files          16       16              
  Lines        1360     1375      +15     
==========================================
+ Hits         1194     1209      +15     
  Misses        166      166

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…es (#1806) * RF: to have "microephys" (Microelectrode physiology) for modality and icephys and ecephys for suffixes and datatypes * Reflecting decision of having two separate datatypes under the Microelectrode Electrophysiology #1800 (comment) Consensus reached during working group meeting on 2024-05-15: - modality = "Microelectrode Electrophysiology" - datatypes = "icephys" and "ecephys" - suffixes = "_icephys" and "_ecephys" * Adjust wording to Horea's recommendation * Various fixups and tune ups to wording from code review

src/modality-specific-files/microelectrode-electrophysiology.md

…ctrode Electrophysiology

bendichter · 2024-11-20T15:10:19Z

src/modality-specific-files/microelectrode-electrophysiology.md

+| **Format**                                                                          | **Extension(s)** | **Description**                                                                                                                                                                                                      |
+--------------------------------------------------------------------------------------|------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Neuroscience Information Exchange Format](https://nixio.readthedocs.io/en/latest/) | `.nix`           | A generic and open  framework with an hdf5 backend and a defined interface to many ephys formats via the [Neo library](https://neo.readthedocs.io/en/latest/). The `.nix` file has to contain a valid Neo structure. |
+| [Neurodata Without Borders](https://www.nwb.org)                                    | `.nwb`           | BRAIN Initiative Data Standard based on an hdf5 backend ...                                                                                                                                                          |


Is the of the ellipse that you are looking for someone to fill in the details?

* test * MACROS___make_suffix_table * markdown * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * undoing changes to yaml files * adding chanell * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * Update src/modality-specific-files/microelectrode-electrophysiology.md Co-authored-by: Yaroslav Halchenko <[email protected]> * adding General ephys metadata * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Examples of real datasets * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor tuneups to formatting --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yaroslav Halchenko <[email protected]>

* [FIX] Minor YAML formatting * [ENH] Added additional birthdate column for participants.tsv * [ENH] Added age_category, age_type and corresponding enums * [ENH] Added surgery_date column * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reverted changes that better suit in a PR against master * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [ENH] Added tabular_data file and columns for _probes.tsv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [ENH] Added tabular_data rules and columns for _electrodes.tsv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [ENH] Added tabular_data rules for _electrodes.tsv * [ENH] Added tabular_data rules and columns for _channels.tsv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [FIX] Fixed pre-commit errors * [ENH] Added additional ephys columns for events.tsv * [ENH] Added tabular data rules for events.tsv to include ephys related columns * [ENH] Added metadata entities for ephys setup metadata * [ENH] Created sidecar rule file for ephys metadata and added data origin and setup fields * Made schema changes to match microephys data type * Renamed rule files from ephys to microephys * Made schema changes to match microephys data type * [ENH] Added objects and rules for coordinate system sidecar file * [ENH] Added processing microephys metadata field in rules * [ENH] Added pharmaceuticals microephys metadata field in rules * [ENH] Added supplementary microephys metadata field and rules * [ENH] Added sample microephys metadata rules * [ENH] Added task microephys metadata rules * Fixed example language convention * Added microephys in schema rule modality * Corrected SampleThickness to SliceThickness * Add yaml document separator for microephys rules file It is optional, and adding does not solve anything but makes it consistent with the other files in the folder * BF: should be a dict, not a list of dicts --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yaroslav Halchenko <[email protected]>

bendichter · 2024-11-20T15:18:06Z

src/schema/rules/tabular_data/microephys.yaml

+    alpha_rotation: recommended
+    beta_rotation: recommended
+    gamma_rotation: recommended


todo: switch out for yaw, pitch and roll

* origin/master: (288 commits) chore(deps): bump codecov/codecov-action from 4 to 5 (#1989) chore: Bump schema post-dev version schema-0.11.3.post3 chore(ci): Fix pytest call for make_archive chore: Remove excess test data from bidsschematools installation directories (#1985) feat(cli): Add tool for filename validation for use in pre-receive hooks (#1986) rm COC (#1979) chore: Bump schema post-dev version schema-0.11.3.post2 Update CONTRIBUTING.md (#1978) fix(schema): Check SliceTiming length against SliceEncodingDirection fix(schema): Do not warn about missing events for task-noise fix(schema): Check for existence of stim_files in beh.tsv chore: Use GITHUB_REF(_NAME) correctly chore: Debug environment chore: Bump schema post-dev version schema-0.11.3.post1 chore: Output version, set correct env var chore: Update publish_schema to publish post-releases fix(schema): Include changes to schema that do not modify spec ... Conflicts: mkdocs.yml src/schema/objects/columns.yaml src/schema/objects/metadata.yaml src/schema/objects/modalities.yaml src/schema/objects/suffixes.yaml src/schema/rules/modalities.yaml

yarikoptic · 2024-11-20T16:01:15Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+
+```
+samples.tsv


use MACROS___make_filetree_example

yarikoptic · 2024-11-20T16:05:14Z

src/modality-specific-files/microelectrode-electrophysiology.md

@@ -0,0 +1,488 @@
+# Microelectrode Electrophysiology
+
+Support for Microelectrode Electrophysiology was developed as a [BIDS Extension Proposal](../extensions.md#bids-extension-proposals) [BEP032: Animal electrophysiology (ephys)](https://bids.neuroimaging.io/bep032).


I found "official" https://github.com/bids-standard/bids-specification/blob/HEAD/CONTRIBUTING.md#soft-rules

Similarly try to use "hard word wrapping": if a sentence gets long and extends a line length beyond 80-100 characters, continue the sentence on the next line.

and below there the advise seems to still keep sentence end ending the line, e.g.

Unprocessed MEG data MUST be stored in the native file format of the MEG instrument
with which the data was collected.
With the MEG specification of BIDS, we wish to promote the adoption of good practices
in the management of scientific data.

@Peyman-N could you please as a first step do such reformatting so to ease adding suggestions and minimize number of commits

Add first BEP032 objects

7ec5d5e

yarikoptic requested review from tsalo and erdalkaraca as code owners February 14, 2024 15:24

yarikoptic mentioned this pull request Feb 14, 2024

[ENH] Add BEP032 (microelectrode electrophysiology) specification #1352

Closed

3 tasks

Add ephys modality, start adding _channels but ran into a question on…

2f393ca

… _acq-

yarikoptic commented Feb 14, 2024

View reviewed changes

src/schema/rules/files/raw/channels.yaml Show resolved Hide resolved

tsalo added the BEP label Feb 15, 2024

yarikoptic mentioned this pull request Feb 29, 2024

Allow for using an arbitrary suffix and/or arbitrary entity bids-standard/bids-2-devel#63

Open

yarikoptic and others added 2 commits March 5, 2024 19:56

Add _acq- to ephys, add _channels and _coords

648781a

Draft for probes TSV file

d3c9996

TheChymera reviewed Mar 6, 2024

View reviewed changes

TheChymera closed this Mar 7, 2024

TheChymera deleted the bep032 branch March 7, 2024 22:13

TheChymera restored the bep032 branch March 7, 2024 22:15

TheChymera reopened this Mar 11, 2024

Fix preferably typo

08c92cd

yarikoptic mentioned this pull request Mar 15, 2024

[SCHEMA] Remove stale test comment -- test no longer fails! #1727

Merged

yarikoptic commented Mar 15, 2024

View reviewed changes

src/schema/rules/files/raw/ephys.yaml Outdated Show resolved Hide resolved

robertoostenveld mentioned this pull request Mar 21, 2024

Draft examples for BEP032 on animal electrophysiology bids-standard/bids-examples#430

Draft

yarikoptic mentioned this pull request Apr 2, 2024

Automatic generation of extra-fields template forms from yaml-based schema (BIDS standard) elabftw/elabftw#5023

Open

Comment out not yet defined iephys and add ephys to datatypes where i…

e2030ac

…t was missing

yarikoptic commented Apr 17, 2024

View reviewed changes

src/schema/objects/datatypes.yaml Outdated Show resolved Hide resolved

dorahermes reviewed Apr 17, 2024

View reviewed changes

src/schema/objects/modalities.yaml Outdated Show resolved Hide resolved

dorahermes reviewed Apr 17, 2024

View reviewed changes

src/schema/objects/suffixes.yaml Outdated Show resolved Hide resolved

TheChymera force-pushed the bep032 branch from cde80a6 to e2030ac Compare April 17, 2024 21:26

Added probes file

53f1087

Remi-Gau changed the title ~~[ENH] Add BEP032 (microelectrode electrophysiology) specification~~ [ENH] microelectrode electrophysiology specification (BEP032) Apr 18, 2024

Remi-Gau added 3 commits April 19, 2024 09:25

Merge remote-tracking branch 'upstream/master' into bep032

a6ae5e2

pacify pre-commit

9a20e4b

add to TOC

7609276

Remi-Gau added the ephys label Apr 19, 2024

semantic line break and fix

9be41f9

yarikoptic added 3 commits April 19, 2024 15:01

Addressed minor left over lint complains

983975a

Add "Microelectrode" specialization for Electrophysiology in the cont…

1afdc40

…ext menu

Fix up for bad latin and adjust wording a bit

6b39d64

yarikoptic force-pushed the bep032 branch from 4e42b6c to 6b39d64 Compare April 19, 2024 19:11

yarikoptic force-pushed the bep032 branch 2 times, most recently from a0068b2 to f03084b Compare April 19, 2024 22:16

Use redirected to www.datalad.org and www.nwb.org website URLs and fe…

73012b3

…w others

yarikoptic force-pushed the bep032 branch from f03084b to 73012b3 Compare April 19, 2024 22:26

TheChymera mentioned this pull request Apr 22, 2024

BEP032 Modality, Datatype, and Suffix Name(s) #1800

Closed

ree-gupta mentioned this pull request May 17, 2024

[ENH] Schema changes related to BEP032 #1826

Merged

17 tasks

ree-gupta mentioned this pull request May 27, 2024

[ENH] Additional age related columns for participants.tsv file to cover animal data #1839

Open

yarikoptic commented Oct 16, 2024

View reviewed changes

src/modality-specific-files/microelectrode-electrophysiology.md Outdated Show resolved Hide resolved

Clarify wording and do mention microephys as shorthand for Microele…

05e9611

…ctrode Electrophysiology

bendichter reviewed Nov 20, 2024

View reviewed changes

Peyman-N and others added 2 commits November 20, 2024 10:11

bendichter reviewed Nov 20, 2024

View reviewed changes

yarikoptic commented Nov 20, 2024

View reviewed changes

src/modality-specific-files/microelectrode-electrophysiology.md

```

samples.tsv

Copy link

Collaborator Author

yarikoptic Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use MACROS___make_filetree_example

yarikoptic commented Nov 20, 2024

View reviewed changes

bendichter mentioned this pull request Nov 30, 2024

BEP032 support SpikeInterface/probeinterface#309

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] microelectrode electrophysiology specification (BEP032) #1705

[ENH] microelectrode electrophysiology specification (BEP032) #1705

yarikoptic commented Feb 14, 2024 •

edited

Loading

TheChymera Mar 6, 2024

yarikoptic Mar 7, 2024

TheChymera Mar 11, 2024

robertoostenveld Mar 12, 2024

TheChymera Apr 17, 2024 •

edited

Loading

lzehl Oct 16, 2024

TheChymera Mar 6, 2024

TheChymera commented Mar 12, 2024 •

edited

Loading

TheChymera commented Mar 12, 2024

Remi-Gau commented Apr 19, 2024

codecov bot commented Apr 19, 2024 •

edited

Loading

bendichter Nov 20, 2024

bendichter Nov 20, 2024

yarikoptic Nov 20, 2024

yarikoptic Nov 20, 2024

		@@ -0,0 +1,488 @@
		# Microelectrode Electrophysiology

		Support for Microelectrode Electrophysiology was developed as a [BIDS Extension Proposal](../extensions.md#bids-extension-proposals) [BEP032: Animal electrophysiology (ephys)](https://bids.neuroimaging.io/bep032).

[ENH] microelectrode electrophysiology specification (BEP032) #1705

Are you sure you want to change the base?

[ENH] microelectrode electrophysiology specification (BEP032) #1705

Conversation

yarikoptic commented Feb 14, 2024 • edited Loading

TODOs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheChymera Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheChymera commented Mar 12, 2024 • edited Loading

TheChymera commented Mar 12, 2024

Remi-Gau commented Apr 19, 2024

codecov bot commented Apr 19, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic commented Feb 14, 2024 •

edited

Loading

TheChymera Apr 17, 2024 •

edited

Loading

TheChymera commented Mar 12, 2024 •

edited

Loading

codecov bot commented Apr 19, 2024 •

edited

Loading