Merge pull request #201 from ttngu207/dev_new_main

Major refactor! Remove additional variation of the "ephys" modules! Support using SpikeInterface for spike sorting

.github/workflows/test.yaml

@@ -1,7 +1,11 @@
 name: Test
 on:
   push:
+    branches:
+      - main
   pull_request:
+    branches:
+      - main
   workflow_dispatch:
 jobs:
   devcontainer-build:
@@ -11,12 +15,7 @@ jobs:
     strategy:
       matrix:
         py_ver: ["3.9", "3.10"]
-        mysql_ver: ["8.0", "5.7"]
-        include:
-          - py_ver: "3.8"
-            mysql_ver: "5.7"
-          - py_ver: "3.7"
-            mysql_ver: "5.7"
+        mysql_ver: ["8.0"]
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{matrix.py_ver}}
@@ -31,4 +30,3 @@ jobs:
         run: |
           python_version=${{matrix.py_ver}}
           black element_array_ephys --check --verbose --target-version py${python_version//.}
-

.pre-commit-config.yaml

@@ -3,7 +3,7 @@ exclude: (^.github/|^docs/|^images/)
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v4.5.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
@@ -16,7 +16,7 @@ repos:
 
   # black
   - repo: https://github.com/psf/black
-    rev: 22.12.0
+    rev: 24.2.0
     hooks:
       - id: black
       - id: black-jupyter
@@ -25,15 +25,15 @@ repos:
 
   # isort
   - repo: https://github.com/pycqa/isort
-    rev: 5.11.2
+    rev: 5.13.2
     hooks:
       - id: isort
         args: ["--profile", "black"]
         description: Sorts imports in an alphabetical order
 
   # flake8
   - repo: https://github.com/pycqa/flake8
-    rev: 4.0.1
+    rev: 7.0.0
     hooks:
       - id: flake8
         args: # arguments to configure flake8

CHANGELOG.md

@@ -3,6 +3,14 @@
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
  [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
+## [0.4.0] - 2025-01-28
+
++ Update - No longer support multiple variation of ephys module, keep only `ephys_no_curation` module, renamed to `ephys`
++ Update - Remove other ephys modules (e.g. `ephys_acute`, `ephys_chronic`) (moved to different branches)
++ Feat - Add support for `SpikeInterface`
++ Update - Remove support for `ecephys_spike_sorting` (moved to a different branch)
++ Update - Simplify the "activate" mechanism
+
 ## [0.3.8] - 2025-01-16
 
 * Fix - Revert GHA Semantic Release caller and update changelog

docs/docker-compose.yaml

@@ -30,12 +30,6 @@ services:
         export ELEMENT_UNDERSCORE=$$(echo $${PACKAGE} | sed 's/element_//g')
         export ELEMENT_HYPHEN=$$(echo $${ELEMENT_UNDERSCORE} | sed 's/_/-/g')
         export PATCH_VERSION=$$(cat /main/$${PACKAGE}/version.py | grep -oE '\d+\.\d+\.[a-z0-9]+')
-        git clone https://github.com/datajoint/workflow-$${ELEMENT_HYPHEN}.git /main/delete || true
-        if [ -d /main/delete/ ]; then
-          mv /main/delete/workflow_$${ELEMENT_UNDERSCORE} /main/
-          mv /main/delete/notebooks/*ipynb /main/docs/src/tutorials/
-          rm -fR /main/delete
-        fi
         if echo "$${MODE}" | grep -i live &>/dev/null; then
             mkdocs serve --config-file ./docs/mkdocs.yaml -a 0.0.0.0:80 2>&1 | tee docs/temp_mkdocs.log
         elif echo "$${MODE}" | grep -iE "qa|push" &>/dev/null; then

docs/mkdocs.yaml

@@ -9,18 +9,7 @@ nav:
   - Concepts: concepts.md
   - Tutorials:
       - Overview: tutorials/index.md
-      - Data Download: tutorials/00-data-download-optional.ipynb
-      - Configure: tutorials/01-configure.ipynb
-      - Workflow Structure: tutorials/02-workflow-structure-optional.ipynb
-      - Process: tutorials/03-process.ipynb
-      - Automate: tutorials/04-automate-optional.ipynb
-      - Explore: tutorials/05-explore.ipynb
-      - Drop: tutorials/06-drop-optional.ipynb
-      - Downstream Analysis: tutorials/07-downstream-analysis.ipynb
-      - Visualizations: tutorials/10-data_visualization.ipynb
-      - Electrode Localization: tutorials/08-electrode-localization.ipynb
-      - NWB Export: tutorials/09-NWB-export.ipynb
-      - Quality Metrics: tutorials/quality_metrics.ipynb
+      - Tutorial: tutorials/tutorial.ipynb
   - Citation: citation.md
   - API: api/ # defer to gen-files + literate-nav
   - Changelog: changelog.md

docs/src/concepts.md

@@ -59,12 +59,16 @@ significant community uptake:
 Kilosort provides most automation and has gained significant popularity, being adopted
 as one of the key spike sorting methods in the majority of the teams/collaborations we
 have worked with. As part of our Year-1 NIH U24 effort, we provide support for data
-ingestion of spike sorting results from Kilosort. Further effort will be devoted for the
+ingestion of spike sorting results from Kilosort. 
+
+Further effort has been devoted for the
 ingestion support of other spike sorting methods. On this end, a framework for unifying
 existing spike sorting methods, named
 [SpikeInterface](https://github.com/SpikeInterface/spikeinterface), has been developed
 by Alessio Buccino, et al. SpikeInterface provides a convenient Python-based wrapper to
-invoke, extract, compare spike sorting results from different sorting algorithms.
+invoke, extract, compare spike sorting results from different sorting algorithms. 
+SpikeInterface is the primary tool supported by Element Array Electrophysiology for 
+spike sorting as of version `0.4.0`.
 
 ## Key Partnerships
 
@@ -95,22 +99,10 @@ Each of the DataJoint Elements creates a set of tables for common neuroscience d
 modalities to organize, preprocess, and analyze data. Each node in the following diagram
 is a table within the Element or a table connected to the Element.
 
-### `ephys_acute` module
+### `ephys` module
 
 ![diagram](https://raw.githubusercontent.com/datajoint/element-array-ephys/main/images/attached_array_ephys_element_acute.svg)
 
-### `ephys_chronic` module
-
-![diagram](https://raw.githubusercontent.com/datajoint/element-array-ephys/main/images/attached_array_ephys_element_chronic.svg)
-
-### `ephys_precluster` module
-
-![diagram](https://raw.githubusercontent.com/datajoint/element-array-ephys/main/images/attached_array_ephys_element_precluster.svg)
-
-### `ephys_no_curation` module
-
-![diagram](https://raw.githubusercontent.com/datajoint/element-array-ephys/main/images/attached_array_ephys_element_no_curation.svg)
-
 ### `subject` schema ([API docs](https://datajoint.com/docs/elements/element-animal/api/element_animal/subject))
 
 Although not required, most choose to connect the `Session` table to a `Subject` table.
@@ -181,12 +173,11 @@ Major features of the Array Electrophysiology Element include:
     + Probe-insertion, ephys-recordings, LFP extraction, clusterings, curations, sorted
       units and the associated data (e.g. spikes, waveforms, etc.).
 
-    + Store/track/manage different curations of the spike sorting results - supporting
-      both curated clustering and kilosort triggered clustering (i.e., `no_curation`).
+    + Store/track/manage the spike sorting results.
 
 + Ingestion support for data acquired with SpikeGLX and OpenEphys acquisition systems. 
-+ Ingestion support for spike sorting outputs from Kilosort.
-+ Triggering support for workflow integrated Kilosort processing.
++ Ingestion support for spike sorting outputs from SpikeInterface.
++ Triggering support for workflow integrated SpikeInterface processing.
 + Sample data and complete test suite for quality assurance.
 
 ## Data Export and Publishing
@@ -208,8 +199,7 @@ pip install element-array-ephys[nwb]
 
 ## Roadmap
 
-Incorporation of SpikeInterface into the Array Electrophysiology Element will be
-on DataJoint Elements development roadmap. Dr. Loren Frank has led a development
+Dr. Loren Frank has led a development
 effort of a DataJoint pipeline with SpikeInterface framework and
 NeurodataWithoutBorders format integrated
 [https://github.com/LorenFrankLab/nwb_datajoint](https://github.com/LorenFrankLab/nwb_datajoint).

docs/src/index.md

@@ -1,29 +1,23 @@
 # Element Array Electrophysiology
 
 This Element features DataJoint schemas for analyzing extracellular array
-electrophysiology data acquired with Neuropixels probes and spike sorted using Kilosort
-spike sorter. Each Element is a modular pipeline for data storage and processing with
+electrophysiology data acquired with Neuropixels probes and spike sorted using [SpikeInterface](https://github.com/SpikeInterface/spikeinterface).
+Each Element is a modular pipeline for data storage and processing with
 corresponding database tables that can be combined with other Elements to assemble a
 fully functional pipeline.
 
 ![diagram](https://raw.githubusercontent.com/datajoint/element-array-ephys/main/images/diagram_flowchart.svg)
 
-The Element is comprised of `probe` and `ephys` schemas. Several `ephys` schemas are
-developed to handle various use cases of this pipeline and workflow:
-
-+ `ephys_acute`: A probe is inserted into a new location during each session.
-
-+ `ephys_chronic`: A probe is inserted once and used to record across multiple
-  sessions.
-
-+ `ephys_precluster`: A probe is inserted into a new location during each session.
-  Pre-clustering steps are performed on the data from each probe prior to Kilosort
-  analysis.
-
-+ `ephys_no_curation`: A probe is inserted into a new location during each session and
-  Kilosort-triggered clustering is performed without the option to manually curate the
-  results.
-
-Visit the [Concepts page](./concepts.md) for more information about the use cases of
+The Element is comprised of `probe` and `ephys` schemas. Visit the 
+[Concepts page](./concepts.md) for more information about the `probe` and
 `ephys` schemas and an explanation of the tables. To get started with building your own
 data pipeline, visit the [Tutorials page](./tutorials/index.md).
+
+Prior to version `0.4.0` , several `ephys` schemas were
+developed and supported to handle various use cases of this pipeline and workflow. These
+ are now deprecated but still available on their own branch within the repository:
+
+* [`ephys_acute`](https://github.com/datajoint/element-array-ephys/tree/main_ephys_acute)
+* [`ephys_chronic`](https://github.com/datajoint/element-array-ephys/tree/main_ephys_chronic)
+* [`ephys_precluster`](https://github.com/datajoint/element-array-ephys/tree/main_ephys_precluster)
+* [`ephys_no_curation`](https://github.com/datajoint/element-array-ephys/tree/main_ephys_no_curation)

docs/src/tutorials/index.md

@@ -1,14 +1,18 @@
 # Tutorials
 
+## Executing the Tutorial Notebooks
+
+The tutorials are set up to run using GitHub Codespaces. To run the tutorials, click on 
+the "Open in Codespaces" button from the GitHub repository. This will open a 
+pre-configured environment with a VSCode IDE in your browser. THe environment contains 
+all the necessary dependencies and sample data to run the tutorials.
+
 ## Installation
 
 Installation of the Element requires an integrated development environment and database.
 Instructions to setup each of the components can be found on the
-[User Instructions](https://datajoint.com/docs/elements/user-guide/) page.  These
-instructions use the example
-[workflow for Element Array Ephys](https://github.com/datajoint/workflow-array-ephys),
-which can be modified for a user's specific experimental requirements.  This example
-workflow uses several Elements (Lab, Animal, Session, Event, and Electrophysiology) to construct
+[User Instructions](https://datajoint.com/docs/elements/user-guide/) page. The example
+tutorial uses several Elements (Lab, Animal, Session, Event, and Electrophysiology) to construct
 a complete pipeline, and is able to ingest experimental metadata and run model training
 and inference.
 
@@ -23,32 +27,10 @@ Electrophysiology.
 ### Notebooks
 
 Each of the notebooks in the workflow
-([download here](https://github.com/datajoint/workflow-array-ephys/tree/main/notebooks)
+([download here](https://github.com/datajoint/workflow-array-ephys/tree/main/notebooks))
 steps through ways to interact with the Element itself. For convenience, these notebooks
 are also rendered as part of this site. To try out the Elements notebooks in an online
 Jupyter environment with access to example data, visit
 [CodeBook](https://codebook.datajoint.io/). (Electrophysiology notebooks coming soon!)
 
-- [Data Download](./00-data-download-optional.ipynb) highlights how to use DataJoint
-  tools to download a sample model for trying out the Element.
-- [Configure](./01-configure.ipynb) helps configure your local DataJoint installation to
-  point to the correct database.
-- [Workflow Structure](./02-workflow-structure-optional.ipynb) demonstrates the table
-   architecture of the Element and key DataJoint basics for interacting with these
-   tables.
-- [Process](./03-process.ipynb) steps through adding data to these tables and launching
-   key Electrophysiology features, like model training.
-- [Automate](./04-automate-optional.ipynb) highlights the same steps as above, but
-  utilizing all built-in automation tools.
-- [Explore](./05-explore.ipynb) demonstrates how to fetch data from the Element.
-- [Drop schemas](./06-drop-optional.ipynb) provides the steps for dropping all the
-  tables to start fresh.
-- [Downstream Analysis](./07-downstream-analysis.ipynb) highlights how to link
-   this Element to Element Event for event-based analyses.
-- [Visualizations](./10-data_visualization.ipynb) highlights how to use a built-in module
-   for visualizing units, probes and quality metrics.
-- [Electrode Localization](./08-electrode-localization.ipynb) demonstrates how to link
-  this Element to
-  [Element Electrode Localization](https://datajoint.com/docs/elements/element-electrode-localization/).
-- [NWB Export](./09-NWB-export.ipynb) highlights the export functionality available for the
-  `no-curation` schema.
+- [Tutorial](../../../notebooks/tutorial.ipynb)

element_array_ephys/__init__.py

@@ -1 +1,3 @@
-from . import ephys_acute as ephys
+from . import ephys
+
+ephys_no_curation = ephys  # alias for backward compatibility