SML fMRI Preprocessing Template

This repo is a work in progress intended to transform the Stanford Memory Lab's (SML) internal fMRI preprocessing scripts into a generalizable workflow for consistency within and across lab projects.

As such, this repo is intended to be used as a GitHub template for setting up fMRI preprocessing pipelines that handle:

• 1. automated transfer of scanner acquisitions from FlyWheel -> BIDS format + dcm2niix DICOM to NIfTI converter,
• 2. dummy scan removal,
• 3. fieldmap-based susceptibility distortion correction,
• 4. fMRIPrep,
• 5. spatial smoothing (SUSAN) post-fMRIPrep processing,
• 6. plug-and-play fMRIQC reports,
• 7. mask generation,
• 8. model spec generation (level 1, level 2, and single trial GLM), and
• 9. automated tools for HDF5 file management and compression out of the box (i.e., to limit lab inode usage on OAK storage)

Note

• indicates workflows that have been finished and validated
• indicates workflows that are still under active development

Using this Template

1. Click the "Use this template" button at the top of this repository
2. Select "Create a new repository"
3. Choose a name for your repository
4. Select whether you want it to be public or private
5. Click "Create repository from template"

This will create a new repository with all the files from this template, allowing you to customize it for your specific preprocessing needs while maintaining the core functionality for handling:

• Fieldmap-based distortion correction
• Dummy scan removal
• BIDS-compliance
• JSON metadata management
• Quality control checks

The template provides a standardized structure and validated scripts that you can build upon, while keeping your specific study parameters and paths separate in configuration files.

What's Included

• Preprocessing scripts for handling fieldmaps and dummy scans
• Configuration templates and examples
• Documentation and usage guides
• Quality control utilities
• BIDS metadata management tools

Getting Started

After creating your repository from this template:

1. Clone your new repository
2. Copy settings.template.sh to settings.sh and customize parameters
3. Modify paths and scan parameters for your study
4. Follow the configuration guide in the detailed documentation below

---

SML fMRI Configuration Guide

Overview

The preprocessing pipeline requires proper configuration of several parameters to handle your study's specific requirements. This guide explains how to set up the settings.sh file that controls the pipeline's behavior.

Important

Submitting Jobs to Slurm Workload Manager

Each core step directory (e.g., 01-prepare) has an associated sidecar executable (e.g., 01-run.sbatch). Thus, from the root of your project scripts directory, you can call:

# example: running step 1
./01-run.sbatch

# example: running step 2
# here, --anat-only is an optional flag that is passed directly to fMRIPrep
# use this if you only want to run anatomical workflows:
./02-run.sbatch --anat-only
#
# otherwise, to run both anatomical and functional workflows, use this:
./02-run.sbatch

Configuration Steps

1. Copy Settings Template

cp settings.template.sh settings.sh

2. Modify Paths

• Set BASE_DIR to your study's root directory
• Ensure RAW_DIR points to your BIDS-formatted data
• Verify TRIM_DIR location for trimmed BIDS-compliant outputs that will later be used for fmriprep
• Set WORKFLOW_LOG_DIR for fMRIPrep workflow logs
• Set TEMPLATEFLOW_HOST_HOME for templateflow local cache
• Set FMRIPREP_HOST_CACHE for fmriprep local cache
• Set FREESURFER_LICENSE to the location of your freesurfer license

3. Set Study Parameters

• Update task_id to match your BIDS task name
• Set new_task_id if task renaming is needed
• Modify run_numbers to match your scan sequence / number of task runs
• Adjust n_dummy based on your scanning protocol

4. Configure Validation Values

• Set EXPECTED_FMAP_VOLS to match your fieldmap acquisition
• Set EXPECTED_BOLD_VOLS to match your BOLD acquisition

5. Map Fieldmaps

• Update fmap_mapping to reflect your fieldmap/BOLD correspondence
• Ensure each BOLD run has a corresponding fieldmap entry

6. Specify Subject IDs

• Copy all-subjects.template.txt to all-subjects.txt and list all subject ids (just the numbers, not the "sub-" part)

7. Set Permissions

• Adjust DIR_PERMISSIONS and FILE_PERMISSIONS based on your system requirements

8. Setup General Slurm Job Manager Parameters

9. Setup fMRIPrep Pipeline Paths

10. Setup fMRIPrep-specific Slurm Parameters

11. Setup fMRIPrep Command Prompt

12. Miscellaneous Settings

• Enable DEBUG mode (for testing)

---

Required Settings

Path Configuration

# ============================================================================
# (1) SETUP DIRECTORIES
# ============================================================================
BASE_DIR="/my/project/dir"           # ROOT DIR FOR THE STUDY
SCRIPTS_DIR="${BASE_DIR}/scripts"    # PATH OF CLONED FMRI REPO
RAW_DIR="${BASE_DIR}/bids"           # RAW BIDS-COMPLIANT DATA LOCATION
TRIM_DIR="${BASE_DIR}/bids_trimmed"  # DESIRED DESTINATION FOR PROCESSED DATA
WORKFLOW_LOG_DIR="${BASE_DIR}/logs/workflows"
TEMPLATEFLOW_HOST_HOME="${HOME}/.cache/templateflow"
FMRIPREP_HOST_CACHE="${HOME}/.cache/fmriprep"
FREESURFER_LICENSE="${HOME}/freesurfer.txt"

Email Update Preference

# ============================================================================
# (2) USER EMAIL (for slurm report updates)
# ============================================================================
USER_EMAIL="[email protected]"

Study Parameters

# ============================================================================
# (3) TASK/SCAN PARAMETERS
# ============================================================================
task_id="SomeTaskName"   # ORIGINAL TASK NAME IN BIDS FORMAT
new_task_id="cleanname"  # NEW TASK NAME (IF RENAMING IS NEEDED), OTHERWISE SET SAME VALUE AS $task_id
n_dummy=5                # NUMBER OF "DUMMY" TRs to remove
run_numbers=("01" "02" "03" "04" "05" "06" "07" "08")  # ALL TASK BOLD RUN NUMBERS

Data Validation

# ============================================================================
# (4) DATA VALIDATION VALUES FOR UNIT TESTS
# ============================================================================
EXPECTED_FMAP_VOLS=12   # EXPECTED NUMBER OF VOLUMES IN ORIGINAL FIELDMAP SCANS
EXPECTED_BOLD_VOLS=220  # EXPECTED NUMBER OF VOLUMES IN BOLD SCANS

Fieldmap (fmap) Mapping

# ============================================================================
# (5) FIELDMAP <-> TASK BOLD MAPPING
# ============================================================================
# example: here, each fmap covers two runs, 
#  so define the mapping as such:
declare -A fmap_mapping=(
    ["01"]="01"  # TASK BOLD RUN 01 USES FMAP 01
    ["02"]="01"  # TASK BOLD RUN 02 USES FMAP 01
    ["03"]="02"  # TASK BOLD RUN 03 USES FMAP 02
    ["04"]="02"  # TASK BOLD RUN 04 USES FMAP 02
    ["05"]="03"  # ...
    ["06"]="03"
    ["07"]="04"
    ["08"]="04"
)

Specifying Subject IDs

# ============================================================================
# (6) SUBJECT IDS <-> PER PREPROC STEP MAPPING
# ============================================================================
# by default, subjects will be pulled from the master `all-subjects.txt` file
# however, if you want to specify different subject lists per pipeline step,
# you may do so here by following this general template:
#
# declare -A subjects_mapping=(
#     ["01-prepare"]="01-subjects.txt"  # PREPROC STEP 01 USES "01-subjects.txt"
#     ["02-fmriprep"]="02-subjects.txt"
# )
#
# note: keep in mind that we've built in checks at the beginning of each pipeline
# step that skip a subject if there's already a record of them being preprocessed;
# thus, you shouldn't necessarily need separate 0x-subjects.txt files per step
# unless this extra layer of control is useful for your needs.

Permissions

# ============================================================================
# (7) DEFAULT PERMISSIONS
# ============================================================================
DIR_PERMISSIONS=775   # DIRECTORY LEVEL
FILE_PERMISSIONS=775  # FILE LEVEL

Slurm Job Header Configurator

# ============================================================================
# (8) SLURM JOB HEADER CONFIGURATOR (FOR GENERAL TASKS)
# ============================================================================
num_subjects=$(wc -l < "all-subjects.txt")  # count number of subjects
echo "($(date)) [INFO] Found ${num_subjects} total subjects in dataset"
array_range="0-$((num_subjects-1))"  # compute array size (0 to num_subjects-1 since array indices start at 0)
export SLURM_EMAIL="${USER_EMAIL}"
export SLURM_TIME="2:00:00"
export SLURM_MEM="8G"  # memory alloc per cpu
export SLURM_CPUS="8"
export SLURM_ARRAY_SIZE="${array_range}"  # use computed range
export SLURM_ARRAY_THROTTLE="10"  # number of subjects to run concurrently
export SLURM_LOG_DIR="${BASE_DIR}/logs/slurm"  # use BASE_DIR from main settings file
export SLURM_PARTITION="hns,normal"  # compute resource preferences order

fMRIPrep Settings

# ============================================================================
# (9) PIPELINE SETTINGS
# ============================================================================
FMRIPREP_VERSION="24.0.1"
DERIVS_DIR="${TRIM_DIR}/derivatives/fmriprep-${FMRIPREP_VERSION}"
SINGULARITY_IMAGE_DIR="${BASE_DIR}/singularity_images"
SINGULARITY_IMAGE="fmriprep-${FMRIPREP_VERSION}.simg"
#
# ============================================================================
# (10) FMRIPREP SPECIFIC SLURM SETTINGS
# ============================================================================
FMRIPREP_SLURM_JOB_NAME="fmriprep${FMRIPREP_VERSION//.}_${new_task_id}"
FMRIPREP_SLURM_ARRAY_SIZE=1
FMRIPREP_SLURM_TIME="12:00:00"
FMRIPREP_SLURM_CPUS_PER_TASK="16"
FMRIPREP_SLURM_MEM_PER_CPU="4G"
#
# ============================================================================
# (11) FMRIPREP SETTINGS 
# ============================================================================
FMRIPREP_OMP_THREADS=8
FMRIPREP_NTHREADS=12
FMRIPREP_MEM_MB=30000
FMRIPREP_FD_SPIKE_THRESHOLD=0.9
FMRIPREP_DVARS_SPIKE_THRESHOLD=3.0
FMRIPREP_OUTPUT_SPACES="MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5"

Miscellaneous

# ============================================================================
# (12) MISC SETTINGS 
# ============================================================================
# Debug mode (0=off, 1=on)
DEBUG=0

---

Tip

Before running the pipeline:

1. Verify all paths exist and are accessible
2. Confirm volume counts match your acquisition protocol
3. Test the configuration on a single subject
4. Review logs for any configuration warnings

Caution

Common Issues

• Incorrect path specifications
• Mismatched volume counts
• Incorrect fieldmap mappings
• Permission issues

Note

Comments, suggestions, questions, issues?

Please use the issues tab (https://github.com/shawntz/fmri/issues) to make note of any bugs, comments, suggestions, feedback, etc… all are welcomed and appreciated, thanks!

cheers, shawn

---

SML fMRI Dev Team

	Team Member	Role
	Shawn Schwartz, M.S., M.A. (Ph.D. Candidate)	Lead Developer Maintainer Project Conception
	Jintao Sheng, Ph.D. (Postdoc)	Core Developer Project Conception Technical Reviewer
	Haopei Yang, Ph.D. (Postdoc)	Core Developer Project Conception Technical Reviewer
	Douglas Miller, B.A. (Ph.D. Candidate)	Core Contributor Code Reviewer Technical Reviewer
	Subbulakshmi S, Ph.D. (Postdoc)	Core Contributor Code Reviewer Technical Reviewer
	Mingjian (Alex) He, Ph.D. (Postdoc)	Core Contributor Code Reviewer Technical Reviewer
	Ali Trelle, Ph.D. (Instructor, SoM)	Core Contributor
	Anthony Wagner, Ph.D. (PI)	Lab Director Conceptual Reviewer

Want to Be Listed?

Make significant contributions to the project and get listed here!
See our Contributing Guidelines for how to get involved.