- Fixes and better logging for EphysPostDLC task
- Remove atlas instantiation from import of histology module
- Write electodeSites dataset at end of histology and spikesorting pipelines if insertion resolved
- Removal of native test train support and improved documentation / tests for GLMs
- LFP destriping functions
- Version number is now obtained from ibllib/init.py
- New ReportSnapshot class
- DLC QC plots, as part of EphysPostDLC task
- BadChannelsAP plots for ephys QC
- Fix typo in camera extractor
- SpikeSorting task overwrites old tar file on rerun
- Snapshot class to register images as notes in Alyx
- Ephys pipeline: RawEphysQC: outputs channels labels as part of pipeline
- Ephys pipeline: spike sorting
- dsp.voltage: destripe detects bad channels and interpolate them
- synchronisation 3B: gives a 10% leeway to throw exception
- spikes sorting task outputs the RMS plot
- Setting tasks to Waiting if they encountered lock (status -2)
- Setting tasks to Incomplete if they return status -3
- Completed tasks set held dependent tasks to waiting
- Adding PostDLC task to compute pupil diameters, licks and DLC QC
- Trial wheel extraction: use alternative sync method when first on fails
- bugfix: timer for tasks was returning None
- Add setup method in tasks.py into try except to catch globus download errors
- Add input and output signatures to all ephys tasks
- Add datahandler to task to download and upload data based on location where task is run
- Spike sorting and EphysVideoSyncQc download data on local servers if not available
- brainbox.io.one load_spike_sorting_fast: bugfix returns acronyms
- creates sequence files for spikesorting
- GPU tasks have a lock - local data handler doesn't instantiate one
- allows more than 2 probes in ephys computer probe creation
- brainbox.io.one load_spike_sorting fast: merge channel info in clusters
- brainbox.io.one generic function to interpolate channels after alignment
- dsp: spike detection by voltage thresholding and cadzow filtering docs
- ibllib.io.spikeglx: get geometry from metadata
- RawEphysQC outputs basic detection spike rates
- ibllib.atlas.regions: add new mapping cosmos and revise Beryl
- destriping as pykilosort internal pre-processing
- NP2 probe framework for splitting shanks and LFP band
- Extension of task module to rerun from different locations
- RawEphysQC tasks computes median RMS from samples for .ap data (stored in _iblqc_ephysChannels.RMS)
- New EphysQC class
- Fix issue with RawEphysQC that was not looking in local Subjects folder for data
- Fix ensure_required_data in DlcQc
- Split jobs.py run function in two, one running large tasks (video compression, dlc, spike sorting), one the rest
- Ensure RawEphysQC runs both probes if one fails
- pykilosort error handling
- passive extraction robust to frame2ttl flickers
- pykilosort bugfix after low yield results
- ephys trials extraction when audio FPGA starts on up state
- pykilosort pipeline: output correct version number / fix log file name
- extraction fixes: passive extraction spacers and camera times format
- small bugfix for the sequential selection
- ephys trials extraction: audio from FPGA: up-state and TTLs cleanup from camera wiring
- passive extraction: improved spacers detection
- ephyscellsqc: bugfix on probe creation for pykilosort subfolder
- task version 6.2.5 specific audio extractor
- camera times: index by video length when GPIO unstable
- extraction fixes: passive extraction spacers and camera times format
- small bugfix for the sequential selection
- pykilosort pipeline: output correct version number / fix log file name
- ephys trials extraction when audio FPGA starts on up state
- pykilosort bugfix after low yield results
- passive extraction robust to frame2ttl flickers
- pykilosort error handling
- ONE2 released on the master branch
- Pykilosort is the new default spike sorter in the pipeline
- oneibl and alf now deprecated; moved to separate repository
- other ibllib miscellanea moved to iblutil repository
- test database is now configurable with an env var
- brainbox.modeling: New API for using linear and poisson models, doing sequential feature selection.
- bugfix: spike sorting continues even if probe information not entered (personal projects)
- TaskQC iteration: aggregation excludes iti delays, frame2ttl clean signal used to compute QC instead of raw
- optogenetics for UCL task variant
- check Nvidia status before launching spike sorting
- fix public one params
- fix ks2 logging output
- set default one params to public credentials
- TaskQC: exclude stim_freeze from overall session qc
- Get modality from task type to include personal projects
- Bugfix Atlas ccf coordinates order
- Tool to label Critical sessions / insertions reasons
- Successful ONE setup on instantiation when params file doesn't exist
- Video QC optimization
- New task types for widefield imaging
- Add revision and default dataset to register dataset
- Fix for camera extraction failure for novel protocols
- CameraQC wheel alignment check more robust to short videos and large alignment failures
- Parameters raises FileNotFound error if no defaults are provided
- opto-ephys tasks
- DLC QC
- Do not block session creation if error
- Removed deprecated numpy dtypes
- Fix trajectory query bug
- GPIO pin states loaded as bool array
- Machine info and log append in Task class
- Only delete session flag file after processing
- Extractor support for widefield imaging protocol
- hotfix: fix test_modelling error after removing pytorch
- hotfix: maintain compatibility with Python 3.7
- ibllib.atlas: backend region mapping is a lateralized version of the Allen atlas "Allen-lr" The default behaviour in the Atlas is to remap according to the original mapping "Allen"
- camera timestamps: extraction of the exact camera time stamps by cross-examination of bonsai timestamps, audio pulses and camera pulses
- ephys extraction: remove short TTL pulses of frame2ttl in task extraction
- ibllib.atlas.regions remapping option
- optogenetics pybpod: extraction of laser probabilities
- MANIFEST.in fix
- Added fixtures file for passive protocol extraction
- Amplitudes fix:
- sync_probes doesn't require raw binary file and looks for meta-data files instead
- tar file of intermediate spike sorting results gets uploaded on flatiron
- Ephys Task extraction: ephys extraction doesn't fail when bpod started before ephys
- histology: brain atlas can handle insertion on the sides
- optogenetics dataset type _ibl_trials.laser_stimulation for training sessions
- spikeglx analog sync thresholding removes DC offset option
- ephys FPGA behaviour extraction: bpod sync with FPGA assumes possible missing FPGA fronts
- Ephys pipeline:
- passive extractor
- units QC
- add the available space of system and raid volumes in local servers reports.
- following flatiron server change, add auto-redirect to https
- add widefieldChoiceworld tasks for pipeline
- add opto laser tasks for pipeline
- register ch when mtscomp runs properly
- probes_description runs even if .cbin doesn't exist
- allows dataset registration on errored task
- ks2task import conflict fix
- Ks2 task does not depend on ephys pulses
- Ephys mtscomp
- ks2 task registers first probe even if one failing
- mtscomp task register .ch and .meta even if .cbin doesn't exist
- move ibllib tests to tests_ibllib
- brainbox atlas plot functions
- Ephys extraction:
- synchronisation between probes computed in the ephysPulses job
- spike sorting resync done directly after KS2 output
- unit-based metrics have their own task
- create local server tasks only on raw_session.flag
- ephys alignment QC
- hotfix: ibl errors inherit Exception, not BaseException
- hotfix: partial qc task extractor keeps FPGA stim times
- create tasks looks for create_me.flags
- add Karolina's optogenetics tasks for extractions
- histology probe QC pipeline and final locations dataset export
- numpy needs upgrading >= 1.18
- session creation skips alyx procedure for unknown task protocol (custom projects)
- task extraction:
- Habituation QC
- ephys extraction StimOffTimes fix
- ibllib.atlas
- allen csv Atlas part of package is not installed in dev mode
- improved slicing performance
- ephys extraction: mtscomp registers ch file on run and re-runs bis
- mtscomp registers ch file on run and re-runs
- ffmpeg nostdin option as jobs were stopped in background on a server
- QC base class
- Support for task QC on FPGA data
- TaskQC run during task extraction
- local server: catches error when subject is not registered in Alyx
- ibllib.atlas.AllenAtlas
- re-ordered the volumes in c-order contiguous ml-ap-dv efficient coronal shapes
- top/bottom surface extraction
- Ephys extraction SyncSpikeSorting: specify different dir for ks2 ouput and raw ephys data
- Ephys extraction ks2: mkdir for scratch more robust
Ephys extraction bugfixes:
- RawEphysQC: No object "ephysTimeRmsAP" found
- EphysMtsComp,RawEphysQC, EphysPulses : ValueError: mmap length is greater than file size
- Ks2: ks2 job cleans-up temp dir
- ONE offline mode and cache dataset table to speed up reloading of large datasets (Olivier)
- ALF io naming conventions on loading objects (Miles)
- KS2 Matlab ephys pipeline tasks (Olivier)
- Support for running QC on biased and training sessions (Nico)
- metrics_df and passed_df properties in BpodQC obj for qcplots (Nico)
- Added missing unittest to stim_move_before_goCue metric (Nico)
- ibllib.pipes.training_preprocessing.TrainingAudio
- ibllib.pipes.training_preprocessing.TrainingAudio: returns files for registration and proper status. (Olivier)
- ibllib.atlas: compute nearest region from probe trajectory (Mayo)
- Local server jobs:
- fix wheel moves size mismatch extractor error
- only look for raw_session.flag for ephys extraction to avoid race conditions
- Ephys extraction:
- spike amplitudes in Volts
- added waveforms samples dataset to use Phy from Flatiron datasets
- ONE performance:
- Metaclass implementation of UniqueSingletons for AlyxClient
- Multi-threaded downloads
- Added JSON fields methods to AlyxClient
- QCs: Bpod and ONE QC features, basic plotting, examples
- Added permutation test, comparing a metric on two sets of datasets, by shuffling labels and seeing how plausible the observed actual difference is Sped up calculation of firing_rate
- ephys extraction: updated extracted metrics, including a new contamination estimate and drift metrics.
- ibllib.io.spikeglx
- Hotfix: rig transfer - create probes. One variable used before assignation.
- ONE.load overwrites local file if filesizes different or hash mismatch
- ephys extraction:
- registration sets the session.procedure field to acute recording
- bugfix synchronization on re-extraction: always recompute spike.times from spike.samples
- ephys registration sets the session.procedure field to acute recording
- training extraction:
- added biasedVisOffChoiceWorld as training extractor
- wheel data
- dropping support for wheel velocity, not extracted anymore
- bugfix: Include sessions data files for ephys mock
- bugfix: Single probe 3B gets synchronized
- bugfix: Include sessions data files in pip package
- Big brainbox merge and release
- bugfix: clusters.metrics spiking rates accurate
- probability left for ephys choice world contain generative probabilities, not outcomes
- ONE Light Windows fixes
- Installation documentation separates conda and virtualenv options
- Conda yaml environement file for ibllib
- ONE Light for behaviour paper data release
- ONE() standard syntax matching the one light examples
- Alyx registration: add md5, version and filesize to the pipeline registration
- Data Patcher: allows to register data from anywhere through FTP/SSH/GLobus
- ONE Light for behaviour paper data release
- Ephys extraction: left probability bug when sequence was 0 - fixed
- Ephys extraction:
- left probability extracted properly
- add robustness to audio fronts extraction in FPGA
- Passive stimulus: raw data registered in pipeline
- Training extraction: microphone extraction for habituation sessions
- ALF: specific to_dataframe method for Bunch
- Ephys extraction: handle fringe case where recording is interrupted in the middle
- Wheel extraction: if rotary encoder version is outdated and stores data in the wrong unit, auto-detect and output in seconds even for new versions
- FPGA/bpod events synchronization performed even when their counts do not match
- Updated requirement versions for mtscomp and phylib
- wheel extraction outputs a timestamps attribute, not times
- make the wheel extraction more robust
- Ephys extraction:
- un-synchronized spike sortings not uploaded on flat-iron
- reaction times extracted
- valve-open times bugfix
- Wheel extraction:
- training wheel position and timing are now correct
- ephys & training: units: radians mathematical convention
- ONE:
- Alyx client handles pagination seamlessly
- cross-platform get of session folder for rig computer copy to server
- spikeglx.verify_hash() method to check file integrity after transfers/manipulations
- create wirings settings files on ephys computer transfer to server
- Ephys Extraction:
- duplicate probe.trajectories bugfix
- extraction works with unoperational fram2ttl at beginning of ephys session
- clusters.metrics.csv has consistent size with npy cluster objects
- ephys transfer: create ephys extraction flags after the transfer is complete
- Rename spike.times on failed sync to reflect the clock as per ALF convention
- sync 3A fails if first cam event whithin 200ms of start
- compress ephys goes through a tempfile to not interfere with transfers/globbing
- Ephys Extraction (phylib)
- convert ks2 amplitudes to volts for spikes.amps, clusters.amps. templates.waveforms, clusters.waveforms to get uV
- generates Cluster UUIDs file
- individual spike depths computed from PC features
- Ephys Synchronization
- use frame2TTL split for 3A by default, if not found look for right_camera
- output individual probe sync in ALF timestamps format
- registration ignores ks2alf probes subfolders
- fix typo in raw qc dataset types
- Alyx registration adds the relative path to session root as dataset.subcollection
- Ephys extraction:
- split probe folders output alf/probe00 and alf/probe01 instead of merge
- outputs templates.waveforms and clusters.waveforms in sparse arrays
- outputs probes.description and probes.trajectory
- renamed the raw ephys QC output
- outputs clusters.metrics
- Bugfixes:
- 3B raw ephys QC output ap.file not found on nidq object
- 3A sync probe threshold set to 2.1 samples
- transfer scripts from ephys/video/rig computers to local servers
- bugfix spigeglx.glob_ephys_files when metadata file without ap.bin file
- Transfer rig data takes into account session type (ephys/training) to create flags
- Ephys video compression in pipeline
- Ephys audio compression in pipeline
- Ephys extraction: provide full 3B default wirings if files do not exist.
- Ephys extraction: merge sync ephys in the pipeline overwrites ks2_alf directory if it already exists
- Added
biasedScanningChoiceWorldtask to biased extractor for Zador lab
spikeglx.Readersupports mtscomp ephys binaries- server pipeline for compression of ephys files
brainbox.singlecell.pethswith testsbrainbox.processing.bincount2Dsupports aggregation on fixed scale- simple examples script and notebook
- examples/brainbox/plot_peths.py: by Matt. W.
- examples/brainbox/rasters by Michaël S.
- Allen Atlas framework and probe registration base functions
- server pipeline for audio extraction of training sessions