Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix memory issue on Compute Canada #43

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Fix memory issue on Compute Canada #43

wants to merge 11 commits into from

Conversation

CHrlS98
Copy link
Contributor

@CHrlS98 CHrlS98 commented Dec 18, 2024 •
edited
Loading

Closes #22 .

The bug came from the process initialization in nextflow which will first copy the input files to the working directory. Because the input work directory is very heavy, copying it to the working directory fills up the RAM. This behaviour can be configured in the nextflow.config file with process.stageInMode. I changed it from mode copy to mode symlink and everything works perfectly now.

Works with nextflow and apptainer both locally and on a cluster.

Bonus: Refactor multiprocessing in scripts, when n_cpus is 1, implementation won't use a multiprocessing pool.

@CHrlS98
Copy link
Contributor Author

CHrlS98 commented Dec 19, 2024 •
edited
Loading

@joe-from-mtl Problem was absolutely not what I thought it was. Not a problem with the scripts or multiprocessing, the OUT OF MEMORY error is caused by nextflow process setup, where the input data is copied to the nextflow working directory. Because the input work directory is very heavy, copying it to the working directory fills up the RAM. This behaviour can be configured in the nextflow.config file with process.stageInMode. I changed it from mode copy to mode symlink and everything works perfectly now.

@CHrlS98 CHrlS98 changed the title [WIP] Fix memory issue in multiprocessed steps Fix memory issue on Compute Canada Dec 19, 2024
@CHrlS98 CHrlS98 marked this pull request as ready for review December 19, 2024 17:13
@CHrlS98
Copy link
Contributor Author

CHrlS98 commented Dec 19, 2024 •
edited
Loading

@joe-from-mtl However, it's a little bit weird that it still takes around 70GB from start to finish.

nextflow run /home/poic2312/source/linumpy/workflows/workflow_soct_3d_slice_reconstruction.nf
  -c /home/poic2312/source/linumpy/workflows/reconstruction_2.5d_beluga.config -with-singularity
  /home/poic2312/containers/linumpy-issue-22.sif --inputDir /home/poic2312/scratch/Alexia-eau-10x/
  --outputDir /home/poic2312/scratch/output-nextflow --slice 16 --processes 20

image

@CHrlS98 CHrlS98 requested a review from joe-from-mtl December 19, 2024 17:21
@joe-from-mtl
Copy link
Contributor

Did you have a look at the nextflow report? You can the resources required by each step of the pipeline.

@joe-from-mtl However, it's a little bit weird that it still takes around 70GB from start to finish.

@CHrlS98 CHrlS98 mentioned this pull request Dec 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joe-from-mtl joe-from-mtl joe-from-mtl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3D reconstruction workflow crashes with OUT OF MEMORY
2 participants
@CHrlS98 @joe-from-mtl