Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stacky Glue / BCLConvert InteropQC / cttsoV2 Fixes / Deprecate ICAv2 Copy Batch Utility #337

Merged
merged 16 commits into from
Jun 4, 2024

Conversation

alexiswl
Copy link
Member

Generic Constructs

  • Added python lambda components (810f1f1)

  • Added partition table constructs (9f824cc)

  • Added ICAv2 workflow handlers (d9b6bc4)

  • Added ICAv2 Copy Batch Utility Construct (1cf7f2f)

BSSH Fastq Copy Manager

  • Refactored the bssh fastq copy manager to use the icav2 copy batch utility as a construct (792908a)

BCLConvert InterOp QC PIpeline Stack

  • Initialised the BCLConvert Interop QC Pipeline Stack (cf055f1)

ctTSO500 Pipeline Stack

  • Modernise the ctTSO500 Pipeline Stack (07537fc)

StackyMcStackFace! (The Glue bits)

  • Initialised Stacky for Code Pipeline Integration (6765e77)
  • Added stateful databases.
    • Instrument Run DynamoDb
      • links an instrument run id to a samplesheet (json)
      • links an instrument run id to fastq list rows
    • InputMaker DynamoDb
      • Place to store glue steps - input maker glue ofter contain multiple (nested) step functions and can communicate information to each other over the input maker database.

Scotch (bclconvert manager to bssh fastq copy)

  • Added Scotch glue (89f6b4c)
    • Integration for connecting the bclconvert manager success event to triggering the bssh fastq copy event

Selleys (bssh fastq copy to BCLConvert InterOp QC)

  • Added Selleys glue (1729fba)
    • Integration for connecting the bssh fastq copy event to triggering the bclconvert interop qc pipeline
    • Integration for storing the fastq list rows output from the bssh fastq copy event to the instrument run database

Super (bssh fastq copy to cttsov2)

  • Added Super Glue (6d0a55a)
    • Now that the fastq list rows are registered in the database, we can check the samplesheet for the instrument run id for if there were any tso500 samples on there.
    • For each cttso500 sample we spin off a samplesheet and fastq list rows for a ready launch event

alexiswl added 11 commits May 31, 2024 19:40
…tabases

Generated a dynamodb table with a sort key and one without a sort key
With latest changes to WRSC schema, update the icav2 handler and WRSC ready event handler
Also added a SFN for generating ready events from a prior workflowrunstatechange

Renamed icav2-table stack

Also renamed dynamodb constructs in the bclconvert manager from the previous commit
Removed the icav2 copy batch utility stack
Removed remnants of icav2 copy batch utility stack.

Use the new icav2 copy batch component inside the bssh fastq copy manager
CWL Pipeline that is triggered after the fastqs have landed in our bucket
Updated the inputs to match the new WRSC.  Rely on the input glue to shrink the samplesheet and fastq list rows
Initialised stacky stateful components and stateless integration
Scotch integration for connecting the bclconvert manager success event to starting the bssh fastq event
Connect the completion of the bssh fastq copy manager to the start of the bclconvert interop qc pipeline and register the fastq list rows in the database
Now that the fastq list rows are in the database, find all samples that are tso500L samples and launch the cttso v2 analysis
@victorskl
Copy link
Member

.. will take time & review today

@victorskl victorskl added the feature New feature label Jun 3, 2024
Copy link
Member

@victorskl victorskl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Massive effort with path finding impl!

I tried my best to skim through them all. Couple of convention spotted but we can go by iteration follow up. Perhaps we shall break down smaller pieces too down track...

Let us give it spin to this far.

@victorskl
Copy link
Member

Happy to merge this whenever you ready, Alexis.

(I will take half the pizza 🍕 , if things go south)

@alexiswl alexiswl merged commit a220fae into main Jun 4, 2024
5 checks passed
@alexiswl alexiswl deleted the feature/stacky-glue branch June 4, 2024 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment