Bring code to this repository for better reproducibility #16

yw7 · 2023-05-24T12:27:21Z

Add code for registering MPRAGE and spine-generic datasets to the PAM50 template.
Closes #15

…slices + fix naming

- Refactored seg_manual_fix_3d_slicer.py script - Added support for opening seg-manual file if it exists - Renamed existing segmentation file to .bkp.nii.gz if it already exists - Added functionality to create json sidecar with meta information - Renamed existing json sidecar file to .bkp.json if it already exists

accept an optional seg_suffix parameter

and PAM50 segmentation file

fixes skip when no T12-L1 IVD.

integration - Separated dataset preparation and training instructions in the README for clarity, specifying use of nnUNetv2 structure. - Clarified the prerequisite of having a trained model before running inferences in the README. - Fixed the output directory variable assignment in inference script to correctly use a second parameter. - Enhanced the inference script to handle missing `_0000` suffixes and to support new postprocessing steps. - Added a new dataset preparation script (`prepare_nnunet_datasets.sh`) to set up data structure for nnUNetv2. - Removed dataset preparation steps from the training script (`train_nnunet.sh`), focusing it solely on model training as per the new separation of concerns. The changes improve the accuracy, usability, and maintainability of TotalSegMRI's implementation with nnUNetv2, facilitating better segmentation results and a smoother experience for users following the updated instructions.

Updated the argument flags across various MRI utility scripts to standardize input directory flags as '-s' instead of the previous '-i'. This change enhances the consistency of script interfaces, making it easier to understand and use the tools for MRI data preparation and processing. Adjusted scripts include those for generating sequential labels, mapping labels, and fixing CSF labels. Compatibility with existing conventions in subject subdirectory flags has been maintained by switching from '-s' to '-u'.

and retains only the largest connected component for each label.

valosekj · 2024-02-06T16:10:29Z

run/train_nnunet.sh

+# RAM requirement in GB
+RAM_REQUIREMENT=8
+# Get the number of CPUs, subtract 1, and ensure the value is at least 1
+JOBS_FOR_CPUS=$(( $(($(nproc) - 1 < 1 ? 1 : $(nproc) - 1 )) ))


Isn't the training done on GPU?

@valosekj Yes you're right but still some work done on CPU so this way it is mor efficient

valosekj · 2024-02-06T16:17:16Z

run/prepare_bids_datasets.sh

+    for f in data/bids/spider/*t2_SPACE.nii.gz; do mv $f ${f/t2_SPACE/T2Sw}; done
+    for f in data/bids/spider/derivatives/labels/*t2_SPACE.nii.gz; do mv $f ${f/t2_SPACE/T2Sw}; done


Just a nitpick, but we usually do not use the T2Sw suffix. For spine-generic, we used T2star.nii.gz; see here. BIDS standard then recommends T2starw.

It is T2 SPACE so I just use it.. Do you have a better idea?
Thanks!!!

Sorry for the delay, but I had to deal with this same issue when adding the data to our servers. What I decided, because the only difference is a change in resolution, is to use:

sub-XXX_acq-lowresSag_T2w for the standard T2w images with a resolution ranged from 3.3 x 0.33 x 0.33 mm to 4.8 x 0.90 x 0.90 mm.

sub-XXX_acq-highresSag_T2w for the T2w images obtained with the SPACE sequence, with a more isotropic resolution of 0.90 x 0.47 x 0.47 mm

valosekj · 2024-02-06T16:21:09Z

src/totalsegmri/resources/datasets/dataset_210.json

+        "disc_O": 2,
+        "disc_E": 3,


What do disc_O and disc_E stand for?

odd and even

valosekj · 2024-02-06T16:24:35Z

src/totalsegmri/utils/fix_csf_label.py

+    # Parse command line arguments
+    parser = argparse.ArgumentParser(
+        description=textwrap.dedent(f'''
+            Fix csf label to include all non cord spinal canal, this will put the spinal canal label in all the voxels (labeled as a backgroupn) between the spinal canal and the spinal cord.


This is slightly hard to understand. Maybe you could consider rephrasing.

Could you maybe help me with that? Thank you!!!
Sometimes there are voxels between the spinal cord and CSF that are maped by the model to background. This code make them labeled as spinal canal.

Refactored the dataset preparation steps, now sourcing the new `get_spine_generic_datasets.sh` script to fetch specific datasets from the Spine Generic Project repository. This update clarifies the preparation of SPIDER datasets in the BIDS structure. Revised training and inference instructions in `README.md` to correspond with the new dataset structure and included clear directives for model training and running inference. Removed unnecessary code from the `prepare_spider_bids_datasets.sh` script, further streamlining the process. These changes make the data setup more intuitive and maintainable, enabling easier replication of the research environment. Related to issue #4567.

…_PAM50_seg.

Removed redundant `pairs_dict` function from both generate_labels_sequential.py and generate_largest_labels.py as it was no longer used in the current codebase. Updated generate_largest_labels.py to enhance clarity: renamed the function generate_labels to generate_largest_labels and updated its references to match the new name, ensuring consistency with the module's purpose. Removed unused imports to streamline dependencies and maintain clean and efficient code.

jcohenadad

these big binaries should not be pushed in the main repository:

instead, upload them as assets (eg: in issues, or in releases), and link to these images.

git should ONLY be used for code, not binaries

jcohenadad · 2024-02-20T19:11:29Z

because of #16 (review) this PR should be squashed on merging-- tag @jcohenadad when is time to do the merge and i'll do it

yw7 · 2024-02-20T20:04:29Z

I've created an issue for Images here: #21 and deleted all the images from the repository.
@jcohenadad We're ready to merge now.

Ensure the creation of nested directories for the SPIDER dataset by adding the '-p' flag to the 'mkdir' command in the README instructions. This prevents potential errors when users attempt to create subdirectories in a non-existent path.

yw7 added 6 commits April 19, 2023 02:00

support n_neutral_labels and n_neutral_labels

a2cb10f

Rester all subjects to PAM50

cb755be

fix bugs related to subject subfolders

cdf67f5

⚙️chore(reg2pam50) resample image and segmentation before processing …

e0337df

…slices + fix naming

add script to get spine-generic data via git-annex

5abf8dc

copy -u instead of -n+check for warp_template2anat

7a92770

yw7 requested a review from jcohenadad May 24, 2023 12:27

yw7 self-assigned this May 24, 2023

yw7 added 22 commits June 17, 2023 19:57

add parallel processing

1877662

add seg_manual_fix_3d_slicer.py script

32d254c

Multiple updates

39d62da

Add color map for 3D Slicer

e5ce67a

modify load_files function

c6c8551

accept an optional seg_suffix parameter

add remove_warp_outliers script

5da5304

add script to convert .mha to .nii.gz

3be8220

add script to map segmentation labels

d376394

Add spider_labels_map.json

13807af

add generate_seg_jpg_nnunet.py

5f1fc85

update default paths and folder names

ba280ec

update git-annex installation command

7f8e2a4

add make_nnunet_dataset.py

405e23d

change default values of data folder

5769bcb

and PAM50 segmentation file

add mrspineseg_labels_map.json

f38a0fc

add nnunet_labels_map.json

d3648c6

add support for customizing output suffix

5d83c78

fix T12-L1 IVD from 207 to 42

12a4c57

remove default compression level

7c17b09

rename output directory

62a8e81

skip when the output files already exist.

530be4c

fixes skip when no T12-L1 IVD.

yw7 added 11 commits January 31, 2024 02:01

Update SPIDER dataset path in preparation script

f17f3ef

Enhance verbose logging with script names

dbbe5cd

Enhance training script output verbosity

6f8652e

convert crlf to lf

7300008

Added argparse dependency to dirpath utility

71a69a6

Balanced dataset by duplicating instances

9e487d0

Update README.md

26ae73f

Added script for processingNIfTI segmentation files

168b2cb

and retains only the largest connected component for each label.

Ignore non-critical warnings

ddd044b

valosekj reviewed Feb 6, 2024

View reviewed changes

yw7 added 2 commits February 7, 2024 00:31

Rename multi-subject and single-subject dataset zip files to included…

91f34d2

…_PAM50_seg.

jcohenadad requested changes Feb 20, 2024

View reviewed changes

yw7 mentioned this pull request Feb 20, 2024

Images storage for README #21

Open

Remove big binaries from main repository

a3069d5

Update README with mkdir flag enhancement

3c22070

Ensure the creation of nested directories for the SPIDER dataset by adding the '-p' flag to the 'mkdir' command in the README instructions. This prevents potential errors when users attempt to create subdirectories in a non-existent path.

jcohenadad merged commit 0300574 into main Feb 20, 2024

jcohenadad deleted the reg2pam50 branch February 20, 2024 21:13

yw7 mentioned this pull request Feb 20, 2024

Put all DirPath in a single utils file #8

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bring code to this repository for better reproducibility #16

Bring code to this repository for better reproducibility #16

yw7 commented May 24, 2023

valosekj Feb 6, 2024

yw7 Feb 6, 2024 •

edited

Loading

valosekj Feb 6, 2024

yw7 Feb 6, 2024

NathanMolinier Feb 20, 2024 •

edited

Loading

valosekj Feb 6, 2024

yw7 Feb 6, 2024

valosekj Feb 6, 2024

yw7 Feb 6, 2024 •

edited

Loading

jcohenadad left a comment

jcohenadad commented Feb 20, 2024 •

edited

Loading

yw7 commented Feb 20, 2024 •

edited

Loading

		for f in data/bids/spider/*t2_SPACE.nii.gz; do mv $f ${f/t2_SPACE/T2Sw}; done
		for f in data/bids/spider/derivatives/labels/*t2_SPACE.nii.gz; do mv $f ${f/t2_SPACE/T2Sw}; done

Bring code to this repository for better reproducibility #16

Bring code to this repository for better reproducibility #16

Conversation

yw7 commented May 24, 2023

valosekj Feb 6, 2024

Choose a reason for hiding this comment

yw7 Feb 6, 2024 • edited Loading

Choose a reason for hiding this comment

valosekj Feb 6, 2024

Choose a reason for hiding this comment

yw7 Feb 6, 2024

Choose a reason for hiding this comment

NathanMolinier Feb 20, 2024 • edited Loading

Choose a reason for hiding this comment

valosekj Feb 6, 2024

Choose a reason for hiding this comment

yw7 Feb 6, 2024

Choose a reason for hiding this comment

valosekj Feb 6, 2024

Choose a reason for hiding this comment

yw7 Feb 6, 2024 • edited Loading

Choose a reason for hiding this comment

jcohenadad left a comment

Choose a reason for hiding this comment

jcohenadad commented Feb 20, 2024 • edited Loading

yw7 commented Feb 20, 2024 • edited Loading

yw7 Feb 6, 2024 •

edited

Loading

NathanMolinier Feb 20, 2024 •

edited

Loading

yw7 Feb 6, 2024 •

edited

Loading

jcohenadad commented Feb 20, 2024 •

edited

Loading

yw7 commented Feb 20, 2024 •

edited

Loading