Lk pd 2736 doublets

[ekiernan]

This PR adds new dockers for warp-tools and for merge-star-npz in order to accomodate new calculations for doublet scores.

The warp-tools docker has been modified to include a new script for calculating doublet scores using a modified DoubletFinder from the Allen Institute.

It then uses this script to calculate new library-level metrics such as percent doublets and new keeper cell metrics.

The merge-npz-docker has been modified with a new combined_matrix.py script that removes the calculation of the keeper cell metrics so that it can be calculated after doublet scores.

The Optimus WDL no longer calculates keeper metrics in the mergestaroutputs task, but instead does this calculation during h5ad production using h5adUtils.wdl.

This PR also updates documentation.

[github-actions]

Remember to squash merge!

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

Remember to squash merge!

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

Remember to squash merge!

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Optimus.wdl has not had its version updated
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

[github-actions]

Remember to squash merge!

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

[ekiernan]

Smart-test failures are as expected. We added two new columns to the h5ad. Gene metric comparisons fail because we've made gene_names unique now. When we make truth gene_names unique the metrics match.

[ekiernan]

retest this please

[jessicaway]

Looks great! Thanks @ekiernan !

[ekiernan]

First run of smart-tests here: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-smart-tests/8500/

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

[github-actions]

Remember to squash merge!

[github-actions]

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

[github-actions]

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

[ekiernan]

The only smart-test that failed was scATAC - which ran because we modified the VerifyTasks.