Auto3dSeg CUDA Error when running on custom dataset

[soumen02]

Hi all,

I'm facing the below error when I try to run AutoRunner on a MS Lesions dataset. I'm able to train dints and segresnet on the autorunner tutorials notebook but I'm not able to get past this on my own dataset, which makes me think the error comes from my dataset/configuration.

Also, pasting in my datastats.yaml.

Would appreciate any insight.

Thanks,

2024-06-02 13:05:43,329 - INFO - AutoRunner using work directory ./work_dir
2024-06-02 13:05:43,331 - INFO - Loading input config /home/mri/laminate/auto3Dseg_laminate/input.yaml
2024-06-02 13:05:43,332 - INFO - Datalist was copied to work_dir: /home/mri/laminate/auto3Dseg_laminate/work_dir/ms_lesion_segmentation_folds.json
2024-06-02 13:05:43,332 - INFO - Setting num_fold 5 based on the input datalist /home/mri/laminate/auto3Dseg_laminate/work_dir/ms_lesion_segmentation_folds.json.
2024-06-02 13:05:43,334 - INFO - Using user defined command running prefix , will override other settings
2024-06-02 13:05:43,334 - INFO - Running data analysis...
2024-06-02 13:05:43,334 - INFO - Found 4 GPUs for data analyzing!
100%|██████████| 5/5 [00:07<00:00,  1.52s/it]
2024-06-02 13:05:59,389 - INFO - Writing data stats to /home/mri/laminate/auto3Dseg_laminate/work_dir/datastats.yaml.
2024-06-02 13:05:59,400 - INFO - Writing by-case data stats to /home/mri/laminate/auto3Dseg_laminate/work_dir/datastats_by_case.yaml, this may take a while.
2024-06-02 13:05:59,480 - INFO - BundleGen from https://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/249bf4b.tar.gz
algo_templates.tar.gz: 104kB [00:01, 63.3kB/s]                             2024-06-02 13:06:01,167 - INFO - Downloaded: /tmp/tmp010zxzk_/algo_templates.tar.gz
2024-06-02 13:06:01,167 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmp010zxzk_/algo_templates.tar.gz.
2024-06-02 13:06:01,167 - INFO - Writing into directory: /home/mri/laminate/auto3Dseg_laminate/work_dir.
2024-06-02 13:06:01,304 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0

2024-06-02 13:06:01,369 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_1
2024-06-02 13:06:01,432 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_2
2024-06-02 13:06:01,495 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_3
2024-06-02 13:06:01,559 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_4
2024-06-02 13:06:01,669 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/segresnet_0
2024-06-02 13:06:01,710 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/segresnet_1
2024-06-02 13:06:01,746 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/segresnet_2
2024-06-02 13:06:01,781 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/segresnet_3
2024-06-02 13:06:01,817 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/segresnet_4
2024-06-02 13:06:01,840 - INFO - segresnet2d_0 is skipped! SegresNet2D is skipped due to median spacing of [0.699999988079071, 0.699999988079071, 0.699999988079071],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 13:06:01,855 - INFO - segresnet2d_1 is skipped! SegresNet2D is skipped due to median spacing of [0.699999988079071, 0.699999988079071, 0.699999988079071],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 13:06:01,871 - INFO - segresnet2d_2 is skipped! SegresNet2D is skipped due to median spacing of [0.699999988079071, 0.699999988079071, 0.699999988079071],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 13:06:01,886 - INFO - segresnet2d_3 is skipped! SegresNet2D is skipped due to median spacing of [0.699999988079071, 0.699999988079071, 0.699999988079071],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 13:06:01,901 - INFO - segresnet2d_4 is skipped! SegresNet2D is skipped due to median spacing of [0.699999988079071, 0.699999988079071, 0.699999988079071],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 13:06:02,744 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/swinunetr_0
2024-06-02 13:06:02,791 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/swinunetr_1
2024-06-02 13:06:02,838 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/swinunetr_2
2024-06-02 13:06:02,884 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/swinunetr_3
2024-06-02 13:06:02,932 - INFO - Generated:/home/mri/laminate/auto3Dseg_laminate/work_dir/swinunetr_4
2024-06-02 13:06:03,056 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/hyper_parameters.yaml,/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/hyper_parameters_search.yaml,/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/network.yaml,/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/network_search.yaml,/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/transforms_infer.yaml,/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/transforms_train.yaml,/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/configs/transforms_validate.yaml'"]
W0602 13:06:04.012023 140484227236864 torch/distributed/run.py:757] 
W0602 13:06:04.012023 140484227236864 torch/distributed/run.py:757] *****************************************
W0602 13:06:04.012023 140484227236864 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 13:06:04.012023 140484227236864 torch/distributed/run.py:757] *****************************************
dints_0 - training ...:   0%|          | 0/28 [00:07<?, ?round/s]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/scripts/train.py", line 984, in <module>
[rank0]:     fire.Fire()
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 143, in Fire
[rank0]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 477, in _Fire
[rank0]:     component, remaining_args = _CallAndUpdateTrace(
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
[rank0]:     component = fn(*varargs, **kwargs)
[rank0]:   File "/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/scripts/train.py", line 594, in run
[rank0]:     outputs = model(inputs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1593, in forward
[rank0]:     else self._run_ddp_forward(*inputs, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1411, in _run_ddp_forward
[rank0]:     return self.module(*inputs, **kwargs)  # type: ignore[index]
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/monai/networks/nets/dints.py", line 498, in forward
[rank0]:     outputs = self.dints_space(inputs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/monai/networks/nets/dints.py", line 678, in forward
[rank0]:     if activation:
[rank0]: RuntimeError: CUDA error: uncorrectable ECC error encountered
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

W0602 13:07:04.097122 140484227236864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 52542 closing signal SIGTERM
W0602 13:07:04.098500 140484227236864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 52543 closing signal SIGTERM
W0602 13:07:04.098790 140484227236864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 52544 closing signal SIGTERM
E0602 13:07:05.865423 140484227236864 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 52541) of binary: /opt/miniconda3/envs/monailabel/bin/python
Traceback (most recent call last):
  File "/opt/miniconda3/envs/monailabel/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/home/mri/laminate/auto3Dseg_laminate/work_dir/dints_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-06-02_13:07:04
  host      : b141-l-oa22-monai
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 52541)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

stats_summary:
  image_stats:
    shape:
      max: [260, 311, 260]
      mean: [260.0, 311.0, 260.0]
      median: [260.0, 311.0, 260.0]
      min: [260, 311, 260]
      stdev: [0.0, 0.0, 0.0]
      percentile:
      - [260, 311, 260]
      - [260, 311, 260]
      - [260, 311, 260]
      - [260, 311, 260]
      percentile_00_5: [260, 311, 260]
      percentile_10_0: [260, 311, 260]
      percentile_90_0: [260, 311, 260]
      percentile_99_5: [260, 311, 260]
    channels:
      max: 1
      mean: 1.0
      median: 1.0
      min: 1
      stdev: 0.0
      percentile: [1, 1, 1, 1]
      percentile_00_5: 1
      percentile_10_0: 1
      percentile_90_0: 1
      percentile_99_5: 1
    cropped_shape:
      max: [260, 304, 235]
      mean: [260.0, 288.75, 226.1]
      median: [260.0, 289.0, 225.5]
      min: [260, 249, 212]
      stdev: [0.0, 11.669940016983807, 6.7520367297579185]
      percentile:
      - [260, 251, 212]
      - [260, 278, 220]
      - [260, 298, 235]
      - [260, 304, 235]
      percentile_00_5: [260, 251, 212]
      percentile_10_0: [260, 278, 220]
      percentile_90_0: [260, 298, 235]
      percentile_99_5: [260, 304, 235]
    spacing:
      max: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      mean: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      median: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      min: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      stdev: [0.0, 0.0, 0.0]
      percentile:
      - [0.699999988079071, 0.699999988079071, 0.699999988079071]
      - [0.699999988079071, 0.699999988079071, 0.699999988079071]
      - [0.699999988079071, 0.699999988079071, 0.699999988079071]
      - [0.699999988079071, 0.699999988079071, 0.699999988079071]
      percentile_00_5: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      percentile_10_0: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      percentile_90_0: [0.699999988079071, 0.699999988079071, 0.699999988079071]
      percentile_99_5: [0.699999988079071, 0.699999988079071, 0.699999988079071]
    sizemm:
      max: [181, 217, 181]
      mean: [181.0, 217.0, 181.0]
      median: [181.0, 217.0, 181.0]
      min: [181, 217, 181]
      stdev: [0.0, 0.0, 0.0]
      percentile:
      - [181, 217, 181]
      - [181, 217, 181]
      - [181, 217, 181]
      - [181, 217, 181]
      percentile_00_5: [181, 217, 181]
      percentile_10_0: [181, 217, 181]
      percentile_90_0: [181, 217, 181]
      percentile_99_5: [181, 217, 181]
    intensity: {max: 1474.814697265625, mean: 37.600435733795166, median: 4.529735326766968,
      min: 0.0, stdev: 51.64284496307373, percentile_00_5: 0.0, percentile_10_0: 0.0,
      percentile_90_0: 127.4942840576172, percentile_99_5: 169.97248077392578}
  image_foreground_stats:
    intensity: {max: 290.2734069824219, mean: 167.3603515625, median: 166.04261627197266,
      min: 52.88066101074219, stdev: 23.246277141571046, percentile_00_5: 114.68076934814454,
      percentile_10_0: 138.15342750549317, percentile_90_0: 198.8703872680664, percentile_99_5: 228.04590377807617}
  label_stats:
    labels: [0, 1]
    image_intensity: {max: 290.2734069824219, mean: 167.3603515625, median: 166.04261627197266,
      min: 52.88066101074219, stdev: 23.246277141571046, percentile_00_5: 114.68076934814454,
      percentile_10_0: 138.15342750549317, percentile_90_0: 198.8703872680664, percentile_99_5: 228.04590377807617}
    label:
    - foreground_percentage:
        max: 0.9999701380729675
        mean: 0.9978718221187591
        median: 0.9993385672569275
        min: 0.9926437139511108
        stdev: 0.0024380633887526176
        percentile: [0.992706198990345, 0.9934401214122772, 0.9997470676898956, 0.9999692037701606]
        percentile_00_5: 0.992706198990345
        percentile_10_0: 0.9934401214122772
        percentile_90_0: 0.9997470676898956
        percentile_99_5: 0.9999692037701606
      image_intensity: {max: 1474.814697265625, mean: 30.14405059814453, median: 0.0,
        min: 0.0, stdev: 48.30650043487549, percentile_00_5: 0.0, percentile_10_0: 0.0,
        percentile_90_0: 118.1228744506836, percentile_99_5: 166.20281219482422}
    - foreground_percentage:
        max: 0.007356304209679365
        mean: 0.002128170261312334
        median: 0.0006614233425352722
        min: 2.982362639158964e-05
        stdev: 0.0024380580496643958
        percentile: [3.075900369367446e-05, 0.0002529015055188211, 0.006559823453426361,
          0.0072938192146830266]
        percentile_00_5: 3.075900369367446e-05
        percentile_10_0: 0.0002529015055188211
        percentile_90_0: 0.006559823453426361
        percentile_99_5: 0.0072938192146830266
      image_intensity: {max: 290.2734069824219, mean: 167.3603515625, median: 166.04261627197266,
        min: 52.88066101074219, stdev: 23.246277141571046, percentile_00_5: 114.68076934814454,
        percentile_10_0: 138.15342750549317, percentile_90_0: 198.8703872680664, percentile_99_5: 228.04590377807617}
  n_cases: 20
stats_by_cases: []

[ericspod]

The complaint I see is about ECC errors, this is usually a hardware fault. If you get the same result with multiple attempts on this one model and not others it's unlikely, but it sometimes works then it probably really is bad memory.

[soumen02]

Thanks for your reply,

That makes sense, I remember running into these errors while running the tutorial notebook but they got fixed unexpectedly without any changes.
Unfortunately, it hasn't gotten fixed on my FLAIR Scans Dataset even after multiple tries

[ericspod]

Unfortunately I don't know what more to say here other than it looks like a hardware issue or something in CUDA/Pytorch. Using fewer cores for OMP or only one GPU may avoid the issue for a time but if it's actually bad memory this will need more investigation on your end.