Runtime improvements for deduplication

[atrull314]

We found an issue with the curent PR, where umitools took a very long time to process the transcriptome bams. We've implemented two solutions:

1.) Add picard as an alternative. We note that we have seen some use of this for single cell, however it may not be the best as it does not account for UMIs, and also results in much more filtering

2.) Split the bam up for umitools. This was already implemented for the genome alignment. For transcriptome alignment, we group transcripts by chromosome and then split in order to ensure there are not too many files created.

[nf-core-bot]

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

[github-actions]

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 5ecc97d

+| ✅ 187 tests passed       |+
#| ❔   5 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

• nextflow_config - Config manifest.version should end in dev: 1.1.0
• pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
• pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
• pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

• files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
• files_unchanged - File ignored due to lint config: docs/images/nf-core-scnanoseq_logo_light.png
• files_unchanged - File ignored due to lint config: docs/images/nf-core-scnanoseq_logo_dark.png
• files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
• template_strings - template_strings

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: assets/nf-core-scnanoseq_logo_light.png
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: docs/images/nf-core-scnanoseq_logo_light.png
• files_exist - File found: docs/images/nf-core-scnanoseq_logo_dark.png
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: conf/igenomes.config
• files_exist - File found: conf/igenomes_ignored.config
• files_exist - File found: .github/workflows/awstest.yml
• files_exist - File found: .github/workflows/awsfulltest.yml
• files_exist - File found: modules.json
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: docs/images/nf-core-scnanoseq_logo.png
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/NfcoreTemplate.groovy
• files_exist - File not found check: lib/Utils.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: lib/WorkflowMain.groovy
• files_exist - File not found check: lib/WorkflowScnanoseq.groovy
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: Singularity
• files_exist - File not found check: lib/nfcore_external_java_deps.jar
• files_exist - File not found check: .travis.yml
• nextflow_config - Found nf-schema plugin
• nextflow_config - Config variable found: manifest.name
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: manifest.homePage
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: validation.help.enabled
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable found: validation.help.beforeText
• nextflow_config - Config variable found: validation.help.afterText
• nextflow_config - Config variable found: validation.help.command
• nextflow_config - Config variable found: validation.summary.beforeText
• nextflow_config - Config variable found: validation.summary.afterText
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config variable (correctly) not found: params.max_cpus
• nextflow_config - Config variable (correctly) not found: params.max_memory
• nextflow_config - Config variable (correctly) not found: params.max_time
• nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
• nextflow_config - Config variable (correctly) not found: params.validationLenientMode
• nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
• nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config manifest.name began with nf-core/
• nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• nextflow_config - nextflow.config contains configuration profile test
• nextflow_config - Config default value correct: params.igenomes_ignore= true
• nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
• nextflow_config - Config default value correct: params.split_amount= 0
• nextflow_config - Config default value correct: params.min_length= 1
• nextflow_config - Config default value correct: params.min_q_score= 10
• nextflow_config - Config default value correct: params.kmer_size= 14
• nextflow_config - Config default value correct: params.save_transcript_secondary_alignment= true
• nextflow_config - Config default value correct: params.retain_introns= true
• nextflow_config - Config default value correct: params.custom_config_version= master
• nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Config default value correct: params.publish_dir_mode= copy
• nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
• nextflow_config - Config default value correct: params.validate_params= true
• nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - CODE_OF_CONDUCT.md matches the template
• files_unchanged - LICENSE matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/CONTRIBUTING.md matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
• files_unchanged - .github/workflows/branch.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - assets/email_template.html matches the template
• files_unchanged - assets/email_template.txt matches the template
• files_unchanged - assets/sendmail_template.txt matches the template
• files_unchanged - assets/nf-core-scnanoseq_logo_light.png matches the template
• files_unchanged - docs/README.md matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
• readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
• readme - README Zenodo placeholder was replaced with DOI.
• plugin_includes - No wrong validation plugin imports have been found
• pipeline_name_conventions - Name adheres to nf-core convention
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• schema_params - Schema matched params returned from nextflow config
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: clean-up.yml
• actions_schema_validation - Workflow validation passed: download_pipeline.yml
• actions_schema_validation - Workflow validation passed: fix-linting.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: ci.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: awstest.yml
• actions_schema_validation - Workflow validation passed: awsfulltest.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• actions_schema_validation - Workflow validation passed: template_version_comment.yml
• actions_schema_validation - Workflow validation passed: release-announcements.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• multiqc_config - assets/multiqc_config.yml found and not ignored.
• multiqc_config - assets/multiqc_config.yml contains report_section_order
• multiqc_config - assets/multiqc_config.yml contains export_plots
• multiqc_config - assets/multiqc_config.yml contains report_comment
• multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
• multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
• multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
• base_config - conf/base.config found and not ignored.
• modules_config - conf/modules.config found and not ignored.
• modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
• nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
• nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

• nf-core/tools version 3.0.2
• Run at 2025-02-19 14:35:39

[lianov]

Just adding for the record as @atrull314 stated that this has been reviewed in depth (code and results). This PR addresses the gap in performance in UMI-Tools processing for transcriptome files but also adds Pircard MarkDuplicates as an alternative to deduplication (UMI-Tools still is the default, as it is more sensitive than Pircard since UMI-Tools uses the UMI sequences to track unique molecules instead of just the Cell Barcodes).