Fix join barcode step for Opossum #1397

khajoue2 · 2024-10-21T19:39:55Z

Fix tabix indexing for large chromosomes

Issue

The previous tabix command was failing due to limitations in TBI (tabix) index
format when dealing with large chromosomes (>512 Mbp). The error message was:

"Region 536870658..536870963 cannot be stored in a tbi index. Try using a csi index"

Fix

Modified the tabix command to use CSI (Coordinate-Sorted Index) instead of TBI.
CSI can handle much larger chromosome sizes.

Changed from:
tabix -s 1 -b 2 -e 3 "${atac_fragment_base}.sorted.tsv.gz"

To:
tabix -s 1 -b 2 -e 3 -C "${atac_fragment_base}.sorted.tsv.gz"

The -C flag tells tabix to create a CSI index. This allows indexing of larger
genomic regions while maintaining the specific column structure of our file.

Impact

This change allows successful indexing of ATAC fragment files that contain
genomic regions larger than 512 Mbp, improving compatibility with a wider
range of reference genomes and ensuring the pipeline can handle larger
chromosomes without failing at the indexing step.

…mosomes larger than 512 Mbp.

github-actions · 2024-10-21T19:40:07Z

Remember to squash merge!

github-actions · 2024-10-21T19:40:19Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
PairedTag.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Optimus.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

github-actions · 2024-10-21T19:40:22Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
PairedTag.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Optimus.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

tasks/skylab/H5adUtils.wdl

ekiernan · 2024-10-22T15:20:56Z

pipelines/skylab/multiome/Multiome.changelog.md

@@ -1,3 +1,7 @@
+# 5.7.2
+2024-10-21 (Date of Last Commit)
+* Changed a flag in H5adUtils.wdl to se CSI instead of TBI indexing in tabix command to support chromosomes larger than 512 Mbp.


Suggested change

* Changed a flag in H5adUtils.wdl to se CSI instead of TBI indexing in tabix command to support chromosomes larger than 512 Mbp.

* Changed a flag in H5adUtils.wdl to set CSI instead of TBI indexing in tabix command to support chromosomes larger than 512 Mbp.

github-actions · 2024-11-04T14:48:03Z

Remember to squash merge!

github-actions · 2024-11-04T14:48:12Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
PairedTag.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
Optimus.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

github-actions · 2024-11-04T14:48:13Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
PairedTag.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
Optimus.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

github-actions · 2024-11-04T14:59:26Z

Remember to squash merge!

github-actions · 2024-11-04T14:59:39Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

github-actions · 2024-11-04T14:59:41Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
PairedTag.wdl and PairedTag.changelog.md do not have matching versions
Some WDLs or changelog files need updating. See output for details.
validation_failed

…warp into pd-2747-multiome-opossum

github-actions · 2024-11-05T14:39:04Z

Remember to squash merge!

github-actions · 2024-11-05T14:39:14Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

github-actions · 2024-11-05T14:39:16Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2024-11-05T19:32:57Z

Remember to squash merge!

github-actions · 2024-11-05T19:33:08Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

github-actions · 2024-11-05T19:33:11Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

pipelines/skylab/optimus/Optimus.changelog.md

github-actions · 2024-11-05T19:50:48Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

ekiernan

Thank you!

pipelines/skylab/atac/atac.wdl

ekiernan

We need to grab the correct output

github-actions · 2024-11-05T20:54:22Z

Remember to squash merge!

github-actions · 2024-11-05T20:54:32Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

github-actions · 2024-11-05T20:54:34Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

ekiernan

The h5ad files look correct; I have a ticket in next Sprint to fix the Verify task for comparing them to handle the non-determinism better.

github-actions · 2024-11-06T12:48:47Z

Remember to squash merge!

github-actions · 2024-11-06T12:48:53Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2024-11-06T12:48:58Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

…to index

github-actions · 2024-11-06T15:20:48Z

Remember to squash merge!

github-actions · 2024-11-06T15:20:56Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2024-11-06T15:21:51Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

pipelines/skylab/atac/atac.wdl

changed JoinBarcodes task index file name

github-actions · 2024-11-06T16:17:45Z

Remember to squash merge!

github-actions · 2024-11-06T16:17:51Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2024-11-06T16:17:55Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Co-authored-by: Elizabeth Kiernan <[email protected]>

github-actions · 2024-11-06T16:37:51Z

Remember to squash merge!

github-actions · 2024-11-06T16:37:58Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2024-11-06T16:38:00Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Fix: Use CSI instead of TBI indexing in tabix command to support chro…

6aac6b6

…mosomes larger than 512 Mbp.

Updated pipeline_versions.txt with all pipeline version information

003c7cd

khajoue2 requested review from ekiernan and nikellepetrillo October 21, 2024 19:40

ekiernan reviewed Oct 21, 2024

View reviewed changes

tasks/skylab/H5adUtils.wdl Show resolved Hide resolved

ekiernan reviewed Oct 22, 2024

View reviewed changes

Merge branch 'develop' into pd-2747-multiome-opossum

4180947

Updated pipeline_versions.txt with all pipeline version information

59b58d2

updated changelogs

507dd83

Updated pipeline_versions.txt with all pipeline version information

a61c9db

khajoue2 added 2 commits November 5, 2024 08:37

updated pairedtag version

31993d8

Merge branch 'pd-2747-multiome-opossum' of github.com:broadinstitute/…

df9d7f6

…warp into pd-2747-multiome-opossum

Updated pipeline_versions.txt with all pipeline version information

ad8366a

Merge branch 'develop' into pd-2747-multiome-opossum

e5987b3

ekiernan reviewed Nov 5, 2024

View reviewed changes

pipelines/skylab/optimus/Optimus.changelog.md Outdated Show resolved Hide resolved

nikellepetrillo approved these changes Nov 5, 2024

View reviewed changes

ekiernan approved these changes Nov 5, 2024

View reviewed changes

ekiernan reviewed Nov 5, 2024

View reviewed changes

pipelines/skylab/atac/atac.wdl Show resolved Hide resolved

ekiernan requested changes Nov 5, 2024

View reviewed changes

updating indexing for pairedtag and changing index file name

6a68d53

ekiernan approved these changes Nov 6, 2024

View reviewed changes

Merge branch 'develop' into pd-2747-multiome-opossum

a51e6b1

khajoue2 added 2 commits November 6, 2024 09:18

Updated outputs to reflect the csi index. Renamed the index from tbi …

645883b

…to index

resolved conflict

b66ddd8

ekiernan reviewed Nov 6, 2024

View reviewed changes

pipelines/skylab/atac/atac.wdl Outdated Show resolved Hide resolved

Update Multiome.wdl

ea6c770

changed JoinBarcodes task index file name

Update pipelines/skylab/atac/atac.wdl

0407007

Co-authored-by: Elizabeth Kiernan <[email protected]>

khajoue2 merged commit 88b1e48 into develop Nov 6, 2024
11 of 12 checks passed

khajoue2 deleted the pd-2747-multiome-opossum branch November 6, 2024 17:57

	* Changed a flag in H5adUtils.wdl to se CSI instead of TBI indexing in tabix command to support chromosomes larger than 512 Mbp.
	* Changed a flag in H5adUtils.wdl to set CSI instead of TBI indexing in tabix command to support chromosomes larger than 512 Mbp.

Fix join barcode step for Opossum #1397

Fix join barcode step for Opossum #1397

Conversation

khajoue2 commented Oct 21, 2024

Fix tabix indexing for large chromosomes

Issue

Fix

Impact

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

🔍Changelog Validation Results:

github-actions bot commented Oct 21, 2024

🔍Version Validation Results:

ekiernan Oct 22, 2024

Choose a reason for hiding this comment

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 4, 2024

🔍Version Validation Results:

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 4, 2024

🔍Version Validation Results:

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 5, 2024

🔍Version Validation Results:

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 5, 2024

🔍Version Validation Results:

github-actions bot commented Nov 5, 2024

🔍Version Validation Results:

ekiernan left a comment

Choose a reason for hiding this comment

ekiernan left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 5, 2024

🔍Version Validation Results:

ekiernan left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024

🔍Version Validation Results:

github-actions bot commented Nov 6, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024

🔍Version Validation Results:

github-actions bot commented Nov 6, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024

🔍Version Validation Results:

github-actions bot commented Nov 6, 2024

🔍Changelog Validation Results:

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024

🔍Version Validation Results:

github-actions bot commented Nov 6, 2024

🔍Changelog Validation Results: