Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with Singularity #11

Open
PedroRASB opened this issue Jul 11, 2024 · 1 comment
Open

Problem with Singularity #11

PedroRASB opened this issue Jul 11, 2024 · 1 comment

Comments

@PedroRASB
Copy link

Hi, I am having some problems inferencing (producing segmentation masks) the Singularity file for DiffTumor, difftumor_final.sif. I am following the procedure in the HuggingFace readme (https://huggingface.co/qicq1c/DiffTumor), and I got a few bugs:

1- When I run the singularity file (singularity run), it tries to install a package, which causes a "Read-only file system" error. The ideal solution for this problem would be to avoid installing any package during singularity run, and intall them only when creating the .sif file (singularity build). My alternative solution was creating an overlay.img file and using it to make the singularity file system not read-only:

sudo singularity overlay create --size 1024 overlay.img
sudo chown user_name:user_name overlay.img
chmod 644 overlay.img
SINGULARITYENV_CUDA_VISIBLE_DEVICES=0 singularity run --fakeroot --overlay overlay.img --nv -B $inputs_data:/workspace/inputs -B $outputs_data:/workspace/outputs difftumor_final.sif

2- After solving problem 1, I got an error when trying to run DiffTumor with multiple CT scans in my input folder ($inputs_data), even after organizing them in the format specified in the readme. The bug is reported in:
BugMultipleFiles.txt

My non-ideal solution to this problem was using a temporary directory ($dir) with a single CT scan on it to perform inference:

for ct_file in "$inputs_data"/*.nii.gz; do
  if [ -f "$ct_file" ]; then
    cp "$ct_file" "$dir/$(basename "$ct_file")"
    SINGULARITYENV_CUDA_VISIBLE_DEVICES=0 singularity run --fakeroot --overlay overlay.img --nv -B $dir:/workspace/inputs -B $outputs_data:/workspace/outputs difftumor_final.sif>> "$output_file" 2>&1
    rm "$dir/$(basename "$ct_file")"
  fi
done

3- The code above worked, and I could produce segmentations. However, I still got an error:

Traceback (most recent call last):
File "organ_mask_access/organ_test.py", line 135, in
main()
File "organ_mask_access/organ_test.py", line 132, in main
validation(model, test_loader, val_transforms, args)
File "organ_mask_access/organ_test.py", line 30, in validation
for index, batch in enumerate(tqdm(ValLoader)):
File "organ_mask_access/organ_test.py", line 30, in validation
for index, batch in enumerate(tqdm(ValLoader)):
File "/workspace/anaconda3/lib/python3.8/bdb.py", line 88, in trace_dispatch
return self.dispatch_line(frame)
File "/workspace/anaconda3/lib/python3.8/bdb.py", line 113, in dispatch_line
if self.quitting: raise BdbQuit
bdb.BdbQuit

@qic999
Copy link
Collaborator

qic999 commented Jul 12, 2024

Hi @PedroRASB. Sorry for these errors. For the "Read-only file system" error, you can convert the singularity container to sandbox. Additionally, I have updated the singularity container on HuggingFace. You can download the latest file and try again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants