Staging -> master #1521
Open
Staging -> master #1521
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added Peak Calling as optional task to the Multiome workflow
…ion (#1502) * use np_group_qcs warp tools docker * warp-tools 2.6.0 -> use np_group_qcs warp tools docker * Updated pipeline_versions.txt with all pipeline version information * use official warp-tools docker image * changelogs * Updated pipeline_versions.txt with all pipeline version information --------- Co-authored-by: GitHub Action <[email protected]>
* return method_config_name * return method_config_name * return method_config_name * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * cat github env * clean up * clean up * more clean up * more clean up * more clean up * more clean up * more clean up * more clean up * more clean up * more clean up * more clean up * trying to port optimus for fun!! * trying to port optimus for fun!! * trying to port optimus for fun!! * trying to port optimus for fun!! * update UpdateTestInputs.py to handle array inputs * update UpdateTestInputs.py to handle array inputs * update UpdateTestInputs.py to handle array inputs * fix branch name * fix branch name * fix branch name * fix branch name * fix branch name * try updating test inputs again * try updating test inputs again * try updating test inputs again * try updating test inputs again * try updating test inputs again * try updating test inputs again * fix setting the test type * fix setting the test type * fix setting the test type * fix setting the test type * fix setting the test type * fix setting the test type * fix the quote values function * fix the quote values function * fix the quote values function * fix the quote values function * pushing a commit to trigger * udate quote values to handle arrays? * udate quote values to handle arrays? * make truth path and results path dynamic * make pipeline_dir dynamic * fix submission url * fix submission url * make both tests the same * make both tests the same * add testsnm3c to dockstore yml * port snm3c * port snm3c * debugging fetch_dockstore_commit.py * debugging fetch_dockstore_commit.py * debugging fetch_dockstore_commit.py * debugging fetch_dockstore_commit.py * remove nested inputs * remove nested inputs * Lk atac test porting (#1457) Added changes for paired-tag tests * try to add testmultiome * add variant call test * additional changes to add variant calling test * add http imports to TestMultiome.wdl and Multiome.wdl * add TestReblockGVCF.wdl to the new testing framework * try to handle stucts * try to handle stucts * try to handle stucts * try to handle stucts * try to handle stucts * try to handle stucts * try to handle stucts * handle nested inputs * handle nested inputs * add wgs germline ss * try testing with removing always * Lk add exome testing (#1461) Added GHA testing for ExomeSingleSample * try testing with removing always * Rc test smartseq 3 (#1466) * smartseq action branch * remove old smartseq2 single nucleus path * gha test ss2sn * change DOCKSTORE_PIPELINE_NAME * update smartseq paths * fix destination cloud path * update the Dockstore pipeline name * readd smartseq2singlenucleus * try testing with removing always * add OVERALL_SUCCESS to all tests * Lk rnawithumis testing (#1467) Added GHA testing for RNAWithUMIsPipeline * add OVERALL_SUCCESS to all tests again * add OVERALL_SUCCESS to all tests again * add joint genotyping * add UltimaGenomicsWholeGenomeGermline * remove https and move generate file summary after all procesing is complete * fix summary tables for all ymls * update run_pipeline * update run_pipeline * update run_pipeline * update run_pipeline * update run_pipeline * Ultima Whole Genome Cram Only testing (#1472) Added Ultima Cram Only * try updating usercomment * try updating usercomment * try updating usercomment * try updating usercomment * try updating usercomment * try updating usercomment * move submission data file * move submission data file * move eof * move eof * move eof * move eof * move eof * move eof * move eof * move eof * input_file instead of test_input_file * input_file instead of test_input_file * input_file instead of test_input_file * input_file instead of test_input_file * add input json name to user comment * Updated pipeline_versions.txt with all pipeline version information * add ultima joint genotyping * add cram to unmapped bams * add exomereprocessing * add wholeGenomeReprocessing * remove truth_branch and replace with master * test out adding authorized_users.yml * test out adding authorized_users.yml * test out adding authorized_users.yml * test out adding authorized_users.yml * update post * update post * update post * update post * echo reponse * use np_jw_test_illumina_genotyping_arrays * use np_jw_test_illumina_genotyping_arrays * make testType not required * make testType not required * make testType not required * make testType not required * make testType not required * make testType not required * make testType not required * make testType not required * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * restructure how we check authed users * test out cram to ubam * dont convert float to scientific notation * Rc test slideseq 3 (#1475) * add testslideseq to dockstore * fix slideseq test wdl path * start slideseq yml with task wdls * update test wdl and yaml * fix testslideseq paths * Updated pipeline_versions.txt with all pipeline version information * update jobs to TestSlideSeq --------- Co-authored-by: GitHub Action <[email protected]> Co-authored-by: Nikelle Petrillo <[email protected]> * For workflow_dispatch, default based on target branch * For workflow_dispatch, default based on target branch * new way of chekcing auth * new way of chekcing auth * new way of chekcing auth * new way of chekcing auth * new way of chekcing auth * new way of chekcing auth * new way of chekcing auth * kick off tests * make terra failure message more visible * Updated pipeline_versions.txt with all pipeline version information * Updated pipeline_versions.txt with all pipeline version information * add new line * remove new line * Updated pipeline_versions.txt with all pipeline version information * remove new line * Updated pipeline_versions.txt with all pipeline version information * fix error message * disable auth check * Updated pipeline_versions.txt with all pipeline version information * remove whitelist auth check * add multiome * add multiome * add dockstore retry mechanism * Np abort jobs from previous commits (#1498) * add abort jobs functionality * add abort jobs functionality * add abort jobs functionality * add abort GHA functionality * add abort GHA functionality * add abort GHA functionality * add abort GHA functionality * add abort GHA functionality * add abort GHA functionality * try to add in test type to method config name * dont default to plumbing * dont default to plumbing * try to add test type to upload_test_inputs * try to add test type to upload_test_inputs * only abort plumbing jobs * only abort plumbing jobs * debugging * add test type to cancel terra runs * add test type to cancel terra runs * logigng info * logigng info * remove print * revert * revert firecloud_api.py * logging info method config name * logging info method config name * add testtpye to cancel and upload * quotes * quotes * bug after else * only abort plumbing gha jobs * install deps * move setup python * uncomment method config name filter * uncomment method config name filter * uncomment method config name filter * uncomment method config name filter * uncomment method config name filter * uncomment method config name filter * dont cancel scientific * dont cancel scientific GHA * dont cancel scientific GHA * debugging dont cancel scientific GHA * debugging dont cancel scientific GHA * look for inputs * fix cancel gha * clean up logging * clean up testtype * clean up testtype * clean up testtype * add abort jobs functionality to all tests * fix bug in upload step * fix bug in upload step * rearrange delete and create steps * rearrange delete and create steps * PD-2854: ATAC tests (#1500) * bug fix?? * remove unused variables * remove some more (hopefully) unused variables * remove the rest of the env blocks * add some env vars * add documentation and clean up env vars * revert nested plumbing inputs * revert nested plumbing inputs * hardcode truth paths for snm3c and multisnss2 * refactor illumina arrays * refactor illumina arrays * refactor illumina arrays * refactor illumina arrays * refactor illumina arrays * use METHOD_CONFIG_NAME * refactor all ymls except snm3c and multisnss2 * fix commenting and logging * stop hardcoding snm3c and multisnss2 * commenting * new inputs --------- Co-authored-by: Elizabeth Kiernan <[email protected]> Co-authored-by: jessicaway <[email protected]> Co-authored-by: Robert Sidney Cox <[email protected]> Co-authored-by: GitHub Action <[email protected]>
Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.31.0...v2.32.2) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert Sidney Cox <[email protected]>
Bumps [serialize-javascript](https://github.com/yahoo/serialize-javascript) from 6.0.0 to 6.0.2. - [Release notes](https://github.com/yahoo/serialize-javascript/releases) - [Commits](yahoo/serialize-javascript@v6.0.0...v6.0.2) --- updated-dependencies: - dependency-name: serialize-javascript dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nikelle Petrillo <[email protected]>
* handle errors during submitting jobs * handle errors during creating method config * put sleep time back to normal * put sleep time back to normal * adding commenting to dependencies block
Added reference cloud path to the bam headers for optimus, multiome, pairedtag
* address intermittent failures * Updated pipeline_versions.txt with all pipeline version information --------- Co-authored-by: GitHub Action <[email protected]>
* remove 15 min timeout from poll_job_status * add a log when submission is done
* wip add beagle imputation stuff * add 2 wdls to dockstore.yml * fix docker gar url * use the right path for jars * wip on imputation wdl * oops use correct jar * missing equals * fix java call again * fix java call * oops match file names * update beagle jar to 01Mar24.d36 * debug GatherVcfs * debug GatherVcfs 2 * try to resolve missing file issue * don't impute over padding * make the index again * supply vcf_index input to SelectVariantsByIds * update Imputation wdl too * newlines * update for hg38 * Revert "update for hg38" This reverts commit 3757137. * update for hg38 * liftover wdl * remove GCP-specific vm commands * use gatk * fix suffix and basename * fix more filenames * remove missing contig stuff for now * fix ref panel path * another chr fix * warn on missign contig * do fail if missing contig * more mem * troubleshooting wld * fixed plink path * add select_first test * cleanup * add if block to test * create and use ref panel interval list * move interval list creation to ref panel wdl * give default values for optional inputs, weird * change CountVariants calls * test * add output to test * next test * more test * another test * update real task * TSPS-226 presplit and prechunk beagle inputs (#1272) *pre splitting and prechunking beagle imputation inputs to lower log numbers and storage account egress --------- Co-authored-by: Jose Soto <[email protected]> * TSPS-221 remove index input and add seed to make beagle tool deterministic (#1285) * remove multi sample vcf index workflow input and add it to the PreSplitVcf task. add seed number so that beagle is always deterministic. add comment to cpu input for PhaseAndImputeBeagle task * change output_callset_name to output_base_name and remove optional outputs * change n_failed_chunks ticket to an int --------- Co-authored-by: Jose Soto <[email protected]> * rename workflow * TSPS-241 Clean up beagle wdl (#1288) * clean up wdl with stuff from TSPS-241 * try to make fail fast work with double nested scatters --------- Co-authored-by: Jose Soto <[email protected]> * add specific gatk_docker * TSPS-142 updates to help creating simulated reference panel and running imputation against it (#1296) * add optional error count override for testing * rename reference base prefix variable and make it more user friendly --------- Co-authored-by: Jose Soto <[email protected]> * add maxRetries 2 to all imputation beagle tasks * add prechunk wdl to dockstore * use acr for default ubuntu image * add preemptible 3 * use acr gatk docker as default * don't use preemptibles on GatherVcfs * basename fix for imputation beagle ref panel generation (#1332) * try auto specifying chr at end of basename * both tasks * add liftovervcfs to dockstore * allow specifying max mem * TSPS-269 Speed up CountVariantsInChunksBeagle by using bedtools (#1335) * try creating bed files * try again * try again again * a different thing * use bedtools and bed ref panel files * oops update the correct task * fix * use the right freaking file name * remove comment * update pipeline version to 0.0.2 * TSPS-293: Fix up streaming imputation beagle (#1347) update ImputationBeagle * add array imputation quota consumed wdl (#1425) * add array imputation quota consumed wdl * add changelogs for imputation array related workflows --------- Co-authored-by: Jose Soto <[email protected]> * TSPS-239 get wdl running on 400k sample ref panel (#1373) * changes to help beagle imputation wdl run on a 400k sample reference panel --------- Co-authored-by: Jose Soto <[email protected]> * remove create imputation ref panel beagle wdl and changelog * PR feedback --------- Co-authored-by: Jose Soto <[email protected]> Co-authored-by: M. Morgan Aster <[email protected]> * add set -e -o pipefail to all relevant imputation tasks (#1434) Co-authored-by: Jose Soto <[email protected]> * TSPS-341 remove tasks for recovering variants not in the reference panel (#1468) * remove tasks for recovering variants not in the reference panel and separate out beagle tasks from imputation tasks * remove prechunk wdl and references to it remove "Beagle" from task names in BeagleTasks.wdl --------- Co-authored-by: Jose Soto <[email protected]> * Updated pipeline_versions.txt with all pipeline version information * [PR to feature branch] Add testing to imputation beagle (#1503) * TSPS-239 get wdl running on 400k sample ref panel (#1373) * changes to help beagle imputation wdl run on a 400k sample reference panel --------- Co-authored-by: Jose Soto <[email protected]> * remove create imputation ref panel beagle wdl and changelog * PR feedback --------- Co-authored-by: Jose Soto <[email protected]> Co-authored-by: M. Morgan Aster <[email protected]> * add new files for testing * add test wdl to .dockstore.yml * add test data json files, other updates * version to 1.0.0, update changelog * update beagle docker * update beagle docker again * fix call phase task * re-deleting ImputationBeaglePreChunk.wdl * temporarily try to run test on feature branch pr * remove vault inputs * update output basename for plumbing test * remove feature branch from gha pr branches * pr comments * add quotes in VerifyTasks.CompareVcfs * update dockers, move CreateVcfIndex to BeagleTasks --------- Co-authored-by: jsotobroad <[email protected]> Co-authored-by: Jose Soto <[email protected]> * Updated pipeline_versions.txt with all pipeline version information * remove newline at end of Utilities.wdl * remove LiftoverVcfs, add README for imputation_beagle * oops this commit adds the README for imputation_beagle * rename test inputs files to reflect contents * PR comments round 1 * Updated pipeline_versions.txt with all pipeline version information * update changelog for BroadInternalImputation * Updated pipeline_versions.txt with all pipeline version information * add back newline to Utilities.wdl with -w flag on changed file check * remove change to Minimac4 task * revert change to tool command in OptionalQCSites * fix fail task dependency, revert attempt to ignore newline in diff, other pr comments * update README for ImputationBeagle * rename test files * Updated pipeline_versions.txt with all pipeline version information * another commit for hashes * Updated pipeline_versions.txt with all pipeline version information * dummy commit * pr comments * Updated pipeline_versions.txt with all pipeline version information * dummy commit * dummy commit --------- Co-authored-by: jsotobroad <[email protected]> Co-authored-by: Jose Soto <[email protected]> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: Nikelle Petrillo <[email protected]> Co-authored-by: npetrill <[email protected]>
dev -> staging
|
DO NOT SQUASH MERGE!!! |
🔍Changelog Validation Results: |
🔍Version Validation Results: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Give your PR a concise yet descriptive title.
Please explain the changes you made here.
Explain the motivation for making this change. What existing problem does the pull request solve?
Mention any issues fixed, addressed, or otherwise related to this pull request, including issue numbers or hard links for issues in other repos.
You can delete these instructions once you have written your PR description.
Checklist
If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP team by tagging @broadinstitute/warp-admins in a comment on this PR.