CREST fmriprep: FITK subjects take 5+ days to run

[ariveradompenciel]

Script/Job File

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M [email protected]
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg \
/mnt/Projects/CREST/ /mnt/Projects/CREST/derivatives \
participant --participant_label FITK314 \
-w /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

Email Message

Job 5768491 (sub-FITK314.job) Complete
User             = ariveradompenciel
Queue            = [email protected]
Host             = argon-lc-g11-15.hpc
**Start Time       = 03/16/2020 14:31:41
End Time         = 03/21/2020 05:45:32**
User Time        = 08:06:37
System Time      = 00:01:50
Wallclock Time   = 111:13:51
CPU              = 194:13:52
Max vmem         = 20.288G
Exit Status      = 0

Output File

The following is an excerpt from output indicating how long it took to run this sub (this example shows it took a day to complete the left hemisphere alone):

200317-07:49:44,322 nipype.interface INFO:
	 resume recon-all : recon-all -autorecon-hemi lh -noparcstats -nocortparc2 -noparcstats2 -nocortparc3 -noparcstats3 -nopctsurfcon -nohyporelabel -noaparc2aseg -noapas2aseg -nosegstats -nowmparc -nobalabels -openmp 10 -subjid sub-FITK314 -sd /mnt/Projects/CREST/derivatives/freesurfer 
200318-09:14:32,797 nipype.workflow INFO:
	 [Node] Finished "_autorecon_surfs0".

Error (stderr file)

N/A

Potential Solution

Increase ram, memory and/or threads assigned to job.

[jdkent]

you could try changing the top of your job submission script to look like this:

#$ -pe smp 20
#$ -l mem_256G=true
#$ -q UI
#$ -m bea
#$ -M [email protected]
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=16

to see if that helps at all.

As we talked, I'm guessing this data is relatively low quality (contaminated with movement).