Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metapipeline job silently failed during recalibrate BAM - identify_outputs #189

Open
nwiltsie opened this issue Apr 8, 2024 · 2 comments
Open

Metapipeline job silently failed during recalibrate BAM - identify_outputs #189

nwiltsie opened this issue Apr 8, 2024 · 2 comments

Comments

@nwiltsie
Copy link
Member

nwiltsie commented Apr 8, 2024 •
edited
Loading

Describe the issue

@WuSelina's job silently failed while reporting success:

Submission Script output:
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/output/P004_P006_paired_metaDNA.log:

[fa/549b6a] process > create_config_metapipeline_... [100%] 2 of 2 ✔
[0b/b645b4] process > call_metapipeline_DNA (2)      [100%] 2 of 2 ✔
[40/9e021f] process > check_process_status (2)       [100%] 2 of 2 ✔

Completed at: 05-Apr-2024 12:56:33
Duration    : 3d 17h 13m 14s
CPU hours   : 89.2
Succeeded   : 8

Step output:
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a/.command.log:

[16/a88725] process > recalibrate_BAM:create_YAML... [100%] 1 of 1 ✔
[-        ] process > recalibrate_BAM:run_recalib... -
[03/ff7666] process > recalibrate_BAM:run_recalib... [  0%] 0 of 1
[-        ] process > call_gSNP:create_YAML_call_... -
[-        ] process > call_gSNP:run_call_gSNP        -
[-        ] process > call_sSNV:create_YAML_call_... -
[-        ] process > call_sSNV:run_call_sSNV        -
[-        ] process > call_mtSNV:create_YAML_call... -
[-        ] process > call_mtSNV:run_call_mtSNV      -
[-        ] process > call_gSV:create_YAML_call_gSV  -
[-        ] process > call_gSV:run_call_gSV          -
[-        ] process > call_sSV:create_YAML_call_sSV  -
[-        ] process > call_sSV:run_call_sSV          -
[-        ] process > call_sCNA:create_YAML_call_... -
[-        ] process > call_sCNA:run_call_sCNA        -
ERROR ~ Unexpected error [InvocationTargetException]

 -- Check script '/hot/software/pipeline/metapipeline-DNA/Nextflow/release/6.0.0-rc.2/module/recalibrate_BAM/./identify_outputs.nf' at line: 25 or see '.nextflow.log' file for more details

executor >  local (2)
[16/a88725] process > recalibrate_BAM:create_YAML... [100%] 1 of 1 ✔
[-        ] process > recalibrate_BAM:run_recalib... -
[03/ff7666] process > recalibrate_BAM:run_recalib... [100%] 1 of 1 ✔
[-        ] process > call_gSNP:create_YAML_call_... -
[-        ] process > call_gSNP:run_call_gSNP        -
[-        ] process > call_sSNV:create_YAML_call_... -
[-        ] process > call_sSNV:run_call_sSNV        -
[-        ] process > call_mtSNV:create_YAML_call... -
[-        ] process > call_mtSNV:run_call_mtSNV      -
[-        ] process > call_gSV:create_YAML_call_gSV  -
[-        ] process > call_gSV:run_call_gSV          -
[-        ] process > call_sSV:create_YAML_call_sSV  -
[-        ] process > call_sSV:run_call_sSV          -
[-        ] process > call_sCNA:create_YAML_call_... -
[-        ] process > call_sCNA:run_call_sCNA        -
ERROR ~ Unexpected error [InvocationTargetException]

 -- Check script '/hot/software/pipeline/metapipeline-DNA/Nextflow/release/6.0.0-rc.2/module/recalibrate_BAM/./identify_outputs.nf' at line: 25 or see '.nextflow.log' file for more details

Step .nextflow.log file:
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a/.nextflow.log:

~> TaskHandler[id: 2; name: recalibrate_BAM:run_recalibrate_BAM_delete_all (1); status: RUNNING; exit: -; error: -; workDir: /scratch/110525/03/ff76662bfcb03f435899f47ba4b4dd]
Apr-05 12:54:59.923 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: recalibrate_BAM:run_recalibrate_BAM_delete_all (1); status: COMPLETED; exit: 0; error: -; workDir: /scratch/110525/03/ff76662bfcb03f435899f47ba4b4dd]
Apr-05 12:55:00.004 [Actor Thread 71] ERROR nextflow.extension.OperatorImpl - @unknown
java.lang.reflect.InvocationTargetException: null
>---at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>---at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
... snip ...
>---at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>---at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert file_found.size() == 1
       |          |
       []         0
>---at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:432)
>---at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:670)
>---at Script_c0b80539.identify_file(Script_c0b80539:20)
>---... 76 common frames omitted
Apr-05 12:55:00.038 [Actor Thread 71] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.reflect.InvocationTargetException
  • Pipeline release version: 6.0.0-rc.2
  • Cluster you are using: Main cluster
  • Node type: F72
  • Submission method: submission script
  • Actual submission script: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/run_metapipeline-DNA_batch2_paired_P004_P006.sh
  • Sbatch or qsub command and logs if applicable
  • Config files
  • Path to the working directory
  • Any logs produced by the pipeline

/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/output
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/input/batch2_aligned_BAMs_input_P004_P006.csv
@nwiltsie
Copy link
Member Author

nwiltsie commented Apr 8, 2024

So there are really two issues here:

  1. The pipeline died for some unknown reason
  2. The wrapper job reported success despite the failure
$ sacct -j 110520,110525 --parsable --format=JobID,JobName,State,Start,End | xsv table -d'|'
JobID          JobName                                                                     State      Start                End
110520         P004_P006_paired_metaDNA                                                    COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110520.batch   batch                                                                       COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110520.extern  extern                                                                      COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110525         wgs_call_metapipeline_DNA_STGHKGFH000004_bb_ff891798967bb1bbcebd760873cd7a  FAILED     2024-04-01T23:19:59  2024-04-05T12:55:01
110525.batch   batch                                                                       FAILED     2024-04-01T23:19:59  2024-04-05T12:55:01
110525.extern  extern                                                                      COMPLETED  2024-04-01T23:19:59  2024-04-05T12:55:01

@WuSelina
Copy link

WuSelina commented Apr 8, 2024 •
edited
Loading

I ran metapipeline of the same patient (STGHKGFH000004) in 'paired' mode using only the first 1000 lines in the tumor/normal BAMs (excluding the header). I have set the pipeilne_work_dir = leading_work_dir instead of /scratch to save logs/output that might otherwise be lost for this run. Perhaps we can see some errors logs/outputs by setting it up this way while keeping the inputs small.
Update: It ran successfully.

Here is the path to the metapipeline run: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/debug_recalibrate-BAM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nwiltsie @WuSelina