Create the mock metadatamanager / workflow run manager #305
Conversation
Generate secrets, ssm parameters, table objects, outside of construct (inside stack instead), use grantRead etc rather than create our own policy statements.
Still a massive WIP, a lot of room for improvement too. At the moment, we have four step functions for each service First three are involved in the launch stage: * one wrapper than handles ready events from workflow manager, then invokes second step function to generate the inputs json * The second step function generates the inputs json from the payload + any other extra things required, in the case of cttso, this might mean copying files around so that they're in the correct format, for most CWL pipelines, this should be a reasonably straightforward configuration * The final step function then launches the icav2 workflow and writes the analysis id to the database * We then have a fourth step function that monitors any workflow state changes from ICAv2 and writes them to the database and then launches an internal event.
Updated handle event change sfn (wrong template was copied) Various sfn fixes Added readmes Updated cttso sfn to just do the copy and samplesheet creation, and generate the inputs json. Let the sfn do the pipeline launching and handling
Not a trivial process since AWS Step functions makes it difficult to evalutate strings in a list. Instead need to make them dictionaries and then collect only the values hence the second step above is required.
Added readmes to constructs and stacks
Refactor bssh and cttso v2 so each service generates its own icav2 copy file service See #293 for more info
Update output directory, to just cache path / run id / portal run id (which is how it is in icav1)
AWS Step Functions template detail type and source should be set in the inputs.
…r-event-subscription Add Events system to bssh fastq copy manager (and deprecate icav2 copy batch utility stack)
* icav2 event translator / now bclconvert manager * Move workflowrunstatechange into its own construct * TypeHint workflow run state change object, need to do a bit more testing on this first * Add samplesheet outputs and run info collection into icav2 copy command
Components: * Event relayer for workflowRunManager to submit ready events * Initialised stacky-mcstackface mock manager * Comprises four main components
|
@victorskl @reisingerf First step function templates available in readme for first step (scotch): https://github.com/umccr/orcabus/tree/feature/stacky-mcstackface/lib/workload/stateless/stacks/stacky-mcstackface/glue-constructs/scotch |
|
One discussion for tomorrow is merging parts 1 and 2 of the 'scotch' events (see readme above). This discussion is also going to be relevant for the Selleys events as well (post bsshfastqcopy). In the prototype setup above we have 'hey the bclconvert manager is complete (p1)', another service subscribing to say 'hey let me add this to the databases (p2)', and other simultaneously saying 'let me trigger this downstream service (copy bssh fastqs) (p3)', hoping that the former event (p2) finishes writing to databases before the downstream services finishes and another event tries to link the bssh fastqs to the library ids in the run. In my opinion, external-to-internal events i.e bclconvertmanager source with 'complete' status, to workflowrunmanager with status 'complete' for bclconvert manager should be more than a simple event 'parse'. When the workflowrunmanager submits an event saying that the bclconvert manager is complete, that should signal to other services subscribing to the workflow run manager event, 'hey the bclconvert manager is complete AND I've successfully imported the required information AND you can query my databases to find out more of that information'! i.e what libraries were on the run, what does the samplesheet comprise etc. These don't need to be part of the event that's raised by the event manager, it can still just parse the event from the bclconvert manager as is albeit with a different source attribute. The second component (selleys) comprises ingesting the fastq list rows from the bsshfastqcopy output event and parsing as a workflowrunmanager event (p1), matching those fastq list rows with the library lane ids in the database (part 2), and launching the bclconvert interop qc (part 3 + 4). I would prefer to have the cttsov2 be triggered from the completion of the bsshfastqcopymanager (p1) rather from a different source altogether (currently triggered from the libraryrunmanager fastqlistrows registsred), but then we have a conundrum since the databases may not contain the information needed for the input makers / glue to work correctly (getting into race-condition territory). I know that this is early days and just prototying, but I think it's good to consider the principles that this system should follow |
|
Step functions for 'selleys' now available too: |
|
Step functions for 'super' now available as a template too: |
Added the following 'global' constructs: * event-workflowmanager-workflowrunstatechange-parse-external-event-detail * event-workflowrunstatechange-internal-to-inputmaker-sfn * lambda-b64gz-translator * lambda-uuid7-generator
Various other fixes in scotch
Connect the bsshfastqcopy outputs to the bclconvert interop qc pipeline
victorskl
added
feature
* Name state-machines, easier to find things * Use ssm parameters with placeholders for paths (all icav2 services use the same analysis output ssm) * Add workflow run name to list of tables
|
Superseded by #337 |
Stacky McStackFace
This CDK stack will imitate the events of the workflow run manager.
There are three main parts of this stack:
Part 1 - Glue the BCLConvertManager to the BSSHFastqCopyManager 🚧
This Construct is known as Scotch. A stock-standard glue.
Construct 1a ✅
Input Event Source:
orcabus.bclconvertmanagerInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
succeededOutput Event source:
orcabus.workflowrunmanagerOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
completeConstruct 1b 🚧
Input Event Source:
orcabus.workflowrunmanagerInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
completeOutput Event source:
orcabus.metadatamanagerOutput Event DetailType:
orcabus.librarystatechangeOutput Event status:
libraryrunidregisteredConstruct 1c 🚧
Input Event Source:
orcabus.workflowrunmanagerInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
completeOutput Event source:
orcabus.bclconvertmanagerinputeventglueOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
completeConstruct 1d ✅
Input Event Source:
orcabus.bsshfastqcopymanagerinputeventglueInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
completeOutput Event source:
orcabus.workflowrunmanagerOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
readyPart 2 - Glue the BSSHFastqCopyManager to BCLConvertInteropQC 🚧
This construct will be known as Selleys.
Construct 2a 🚧
Input Event Source:
orcabus.bsshfastqcopymanagerInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
succeededOutput Event source:
orcabus.workflowrunmanagerOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
completeConstruct 2b 🚧
Input Event Source:
orcabus.workflowrunmanagerInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
completeOutput Event source:
orcabus.metadatamanagerOutput Event DetailType:
orcabus.librarystatechangeOutput Event status:
fastqlistrowregisteredConstruct 2c 🚧
Input Event Source:
orcabus.workflowrunmanagerInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
completeOutput Event source:
orcabus.bclconvertinteropqcinputeventglueOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
completeConstruct 2d 🚧
Input Event Source:
orcabus.bclconvertinteropqcinputeventglueInput Event DetailType:
orcabus.workflowrunstatechangeInput Event status:
completeOutput Event source:
orcabus.workflowrunmanagerOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
readyPart 3 - Glue the FastqListRow Output Event to the ctTSOv2ReadySfn 🚧
The most important glue of them all. Gorilla Glue!
Construct 3a 🚧
Input Event Source:
orcabus.metadatamanagerInput Event DetailType:
orcabus.librarystatechangeInput Event status:
fastqlistrowregisteredOutput Event source:
orcabus.workflowrunmanagerOutput Event DetailType:
orcabus.librarystatechangeOutput Event status:
fastqlistrowsregisteredConstruct 3b 🚧
Input Event source:
orcabus.workflowrunmanagerInput Event DetailType:
orcabus.librarystatechangeInput Event status:
fastqlistrowsregisteredOutput Event source:
orcabus.cttsov2inputeventglueOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
completeConstruct 3c 🚧
Output Event source:
orcabus.cttsov2inputeventglueOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
completeOutput Event source:
orcabus.workflowrunmanagerOutput Event DetailType:
orcabus.workflowrunstatechangeOutput Event status:
readyNotes