Added feature icav2 copy batch state machine #143
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 10, 2024
victorskl
approved these changes
Mar 11, 2024
reisingerf
approved these changes
Mar 12, 2024
victorskl
reviewed
Mar 14, 2024
...ad/stateless/icav2_copy_batch_utility/step_functions_templates/copy_batch_state_machine.json
Outdated
Show resolved
Hide resolved
|
Put this branch into draft mode until we finalise |
Ignore secret in Readme.md
alexiswl
force-pushed
the
feature/icav2-copy-batch-utility
branch
from
b7e2984 to
b453395
Compare
|
@victorskl this one is now ready for review |
|
thanks Alexis for update, will review today. |
victorskl
approved these changes
Apr 23, 2024
| ] | ||
| } | ||
| ] | ||
| ``` |
There was a problem hiding this comment.
"Flip Manifest" Very creative concept, Alexis.! 👍
| DEFAULT_WAIT_TIME_SECONDS = 10 | ||
|
|
||
|
|
||
| def handler(event, context): |
There was a problem hiding this comment.
Non blocker comment (due to time constraint).
One fine day down the track, we should unit test this handler to guardrail couple of if branching statements down under. Need to make sure these branching logics maintain throughout the system SDLC and, retain it when refactoring happens.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This ICAv2 Copy Batch Utility wraps an AWS Step Function over the ICAv2 CopyBatch API.
This api is designed to copy a list of illumina file ids into a directory.
We exploit this API by taking in a manifest (a list of key values of source ids with their respective destinations) ( a source may have multiple destinations if it needs to be copied into a few different places ), and then monitor a set of API jobs to completion.
These CopyBatch API jobs have about a 20% fail rate, so we sometimes need to resubmit, this is built into the step function.
A 20% fail rate seems quite high, but when they work, these jobs can transfer 80 Gb of data in under 10 seconds so it's worth persisting with.
When working with the ICAv2 CopyBatch API, we need to provide a unique identifier for the run, and a unique location for the outputs.
Once all jobs are deployed we monitor their status (redeploy if failed - up to five times) and wait for all jobs to complete.
This process will transfer an entire BCLConvert process from one ICAv2 project to another in under 10 minutes!
Note that a current limitation prevents using this API within the same project.
TODO List