Pipeline completes with unknown error

[raagagrawal]

Describe the issue
Pipeline completes create_input_csv_metapipeli... and create_config_json steps but fails at call_metapipeline_DNA after 30 minutes. Log file does not show error message - just a failed task.

• Pipeline release version: 4.0.0
• Cluster you are using (SGE/Slurm-Dev/Slurm-Test): Slurm
• Node type (F2s (lowmem) / F72s (midmem) / M64s (execute)): F72
• Submission method (interactive/submission script): script
• Actual submission script (python submission script, "nextflow run ...", etc.): /hot/project/disease/ProstateTumor/PRAD-000095-ColibactinDHTDNARNA/scripts/pipelines/metapipeline-dna/submit.sh
• Sbatch or qsub command and logs if applicable
• Config files: /hot/user/ragrawal/Liss-Agrawal-PRAD-COLB/metapipeline-dna/config/LAPRCOLB000001.config
• Path to the working directory: /hot/user/ragrawal/Liss-Agrawal-PRAD-COLB/metapipeline-dna/output
• Any logs produced by the pipeline: /hot/project/disease/ProstateTumor/PRAD-000095-ColibactinDHTDNARNA/scripts/pipelines/metapipeline-dna/LAPRCOLB000001.log

[yashpatel6]

To find the logs for the specific error, you'll want to look at the .command.log file in the common working dir (The specific one can be found by looking at the main log in front of the process; you'll see [48/b348b2], which tells you the directory. The relevant one here: /hot/user/ragrawal/Liss-Agrawal-PRAD-COLB/metapipeline-dna/output/48/b348b27906d06896669ff5847e5215/.command.log

The issue seems to be with /scratch, which is something we've been seeing recently. Can you open a ServiceNOW ticket with the job ID if you have it? You can include the error message in the .command.log file above along with the job ID and the node the job ran on.

[raagagrawal]

I'll open a ticket now. I'm havint the same issue with /scratch on other pipelines too. Thank you for the help!

[raagagrawal]

Ticket opened. Ticket number: [TIX03798282]