Create TES Basespace Upload manager service #186

alexiswl · 2024-03-25T04:06:24Z

TODO

Instead of using portal token secret, get authentication from sscheck service another way
Add event rule to collect SRM PendingAnalysis event to trigger step functions
Add stack to OrcaBus pipeline
Determine if BSSH Secret should be part of this stack or part of infrastructure stack? (This should only be a temp stack anyway)

victorskl · 2024-03-25T06:31:06Z

Made a quick go through, looking good. Will go through full review once ready.

alexiswl · 2024-04-17T08:54:41Z

@victorskl would you mind triggering a srm sequencestatechange in dev and seeing if that triggers this AWS Step function?

Might need to be more specific on the filter too (so just to when the statechange is pending analysis?).

I would expect the dev step function to fail because it might not be able to generate the v2 samplesheet but just want to make sure that the events are talking.

I have deployed stack manually in dev but also added to orcabus pipeline. Will destroy manual dev stack before merging.

victorskl · 2024-04-17T09:01:17Z

will do

alexiswl · 2024-04-18T02:10:51Z

Gah @williamputraintan is #231 the last of the refactoring?

victorskl · 2024-04-18T02:20:02Z

Gah

(I knew it.! Hang in there, Alexis)

williamputraintan · 2024-04-18T02:23:57Z

#231 the last of the refactoring?

Yes, no more refactoring!

victorskl · 2024-04-18T02:24:06Z

Will; let pause the refactor activity at this point. I reckon, we really achieved pretty good refactor this far (thank you, again). We shall let it sit the current code base (what we got up to main this far) as it is for a bit more. We shall later go more rounds refactor later down the track, again.

lib/workload/stateless/statelessStackCollectionClass.ts

victorskl

Looking good, Alexis. Minor comments to attend.

...ad/stateless/stacks/bs-runs-upload-manager/deploy/lib/stacks/bs_runs_upload_manager_stack.ts

config/stacks/bsRunsUploadManager.ts

...rkload/stateless/stacks/bs-runs-upload-manager/deploy/stacks/bs_runs_upload_manager_stack.ts

...ad/stateless/stacks/bs-runs-upload-manager/deploy/constructs/bs_runs_upload_manager_stack.ts

lib/workload/stateless/stacks/bs-runs-upload-manager/deploy/constructs/lambda_layer.ts

...rkload/stateless/stacks/bs-runs-upload-manager/deploy/stacks/bs_runs_upload_manager_stack.ts

lib/workload/stateless/stacks/bs-runs-upload-manager/deploy/README.md

Use jwt portal token from secrets maanger

Remove bs access token from stage, new custom id

Filtered event and updated step function inputs to match sequence. Ran `yarn run prettier --write` to update linting for ts files

…us only Updated orcabus pipeline stack collection class

Import path fixes after refactor Deleted blank deployment readme

ESLint ignore for beforeBinding command Run prettier --write for bssh tools lambda layer

alexiswl · 2024-04-22T01:53:13Z

@victorskl do you want to look over this again, have refactored onto main and cdk synth now working

victorskl · 2024-04-22T01:55:05Z

Go for it, Alexis. Pls merge whenever you feel ready. These are all moving targets. We can readjust down the track. Let us pushing forwards...

alexiswl · 2024-04-22T01:58:59Z

Woo hoo! One down, five PRs to go

victorskl · 2024-04-22T03:57:30Z

five PRs to go

You should rest and enjoy ur holis...

I can/will follow up testing this part meanwhile.

Once we got this part up and running, we should get to fastqs - that is pretty good, already.!

Thanks for all hard work.!

victorskl · 2024-04-29T00:03:34Z

lib/workload/components/python-lambda-layer/index.ts

+  constructor(scope: Construct, id: string, props: PythonLambdaLayerConstructProps) {
+    super(scope, id);
+
+    this.lambda_layer_version_obj = new PythonLayerVersion(this, 'python_lambda_layer', {


@alexiswl Just minor request, pls.

I just notice these as post-mortem review. When you have spare time down the track, could you please follow up refactoring naming convention on these CDK variables. There seems to be mixed use of snake_case and camelCase.

ditto - current guidance on TypeScript
https://github.com/umccr/orcabus/blob/main/README.md#typography

This apply to both bsRunsUploadManager and ICAv2CopyBatchManager services that have merged.

And.. separate PRs for these refactor, pls.

And.. separate PRs for these refactor, pls.

Ah that's going to cause some annoying merge conflicts, the PythonLayer construct example above will rename the attribute lambda_layer_version_obj to lambdaLayerVersionObj, but the property of the bs runs upload manager stack (lambda_layer_obj will be renamed to lambdaLayerObj),

so the layers attribute in any of the python functions will go from layers: [props.lambda_layer_obj.lambda_layer_version_obj] to layers: [props.lambdaLayerObj.lambdaLayerVersionObj] where lambdaLayerObj comes from one branch and lambdaLayerVersionObj comes from another branch.

I think this would be easier to all be in one PR

Right, sure. In that case, that's fine, Alexis. Thanks

I mean, ok to be all in one go, refactor PR, yes.

alexiswl added the feature New feature label Mar 25, 2024

alexiswl requested a review from victorskl March 25, 2024 04:06

alexiswl self-assigned this Mar 25, 2024

alexiswl marked this pull request as draft March 25, 2024 04:06

alexiswl linked an issue Mar 25, 2024 that may be closed by this pull request

Implement sequence run data copy service #140

Closed

alexiswl force-pushed the feature/tes-bs-runs-upload-manager branch from eb1be50 to 5e13dd4 Compare April 17, 2024 08:52

alexiswl marked this pull request as ready for review April 17, 2024 08:52

alexiswl force-pushed the feature/tes-bs-runs-upload-manager branch from 5e13dd4 to 143d38b Compare April 18, 2024 02:37

williamputraintan reviewed Apr 18, 2024

View reviewed changes

lib/workload/stateless/statelessStackCollectionClass.ts Show resolved Hide resolved

victorskl reviewed Apr 18, 2024

View reviewed changes

lib/workload/stateless/statelessStackCollectionClass.ts Outdated Show resolved Hide resolved

victorskl approved these changes Apr 18, 2024

View reviewed changes

...ad/stateless/stacks/bs-runs-upload-manager/deploy/lib/stacks/bs_runs_upload_manager_stack.ts Outdated Show resolved Hide resolved

config/stacks/bsRunsUploadManager.ts Outdated Show resolved Hide resolved

config/stacks/bsRunsUploadManager.ts Outdated Show resolved Hide resolved

alexiswl force-pushed the feature/tes-bs-runs-upload-manager branch from fcf9326 to a3cc3e4 Compare April 18, 2024 10:42

victorskl mentioned this pull request Apr 18, 2024

Feature/cttso icav2 step function branch #145

Merged

7 tasks

victorskl reviewed Apr 18, 2024

View reviewed changes

lib/workload/stateless/stacks/bs-runs-upload-manager/deploy/README.md Outdated Show resolved Hide resolved

alexiswl and others added 7 commits April 22, 2024 11:14

Create TES Basespace Upload manager service

a48e465

Update ReadMe TOC

4902a16

Miscellaneous fixes to bs runs upload lambdas

848f139

Generate stack objects before construct

fbdc220

Use jwt portal token from secrets maanger

Update bs constructs

ba56665

Remove bs access token from stage, new custom id

Added event target being the State Machine

6654e23

Added bsRunsManager uploader to configuration

8507c6c

alexiswl and others added 8 commits April 22, 2024 11:14

Move bsRunsUploadManager to new stacks directory

624155a

Linted and filtered

1bd8db1

Filtered event and updated step function inputs to match sequence. Ran `yarn run prettier --write` to update linting for ts files

Descaled standalone cdk construct, integrate stack be to within orcab…

9a09df9

…us only Updated orcabus pipeline stack collection class

Lint stack file

96bac51

Renamed stateful/stateless configuration from #232

41c7384

Use values in constants to populate stack parameters

726b365

Fix configuration import

53ca728

Refactor python lambda construct

d5ec8db

Import path fixes after refactor Deleted blank deployment readme

alexiswl force-pushed the feature/tes-bs-runs-upload-manager branch from 583507c to d5ec8db Compare April 22, 2024 01:23

Move python-lambda-layer

1ff4353

ESLint ignore for beforeBinding command Run prettier --write for bssh tools lambda layer

alexiswl merged commit 11f77d3 into main Apr 22, 2024
2 checks passed

alexiswl deleted the feature/tes-bs-runs-upload-manager branch April 22, 2024 01:59

victorskl reviewed Apr 29, 2024

View reviewed changes

victorskl mentioned this pull request Apr 29, 2024

Feature/bssh icav2 fastq copy manager #144

Merged

2 tasks

alexiswl mentioned this pull request May 1, 2024

Use camel case for construct and class attributes #259

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create TES Basespace Upload manager service #186

Create TES Basespace Upload manager service #186

alexiswl commented Mar 25, 2024 •

edited

Loading

victorskl commented Mar 25, 2024

alexiswl commented Apr 17, 2024

victorskl commented Apr 17, 2024

alexiswl commented Apr 18, 2024

victorskl commented Apr 18, 2024

williamputraintan commented Apr 18, 2024

victorskl commented Apr 18, 2024

victorskl left a comment

alexiswl commented Apr 22, 2024

victorskl commented Apr 22, 2024

alexiswl commented Apr 22, 2024

victorskl commented Apr 22, 2024

victorskl Apr 29, 2024

victorskl Apr 29, 2024

alexiswl Apr 29, 2024 •

edited

Loading

victorskl Apr 29, 2024

victorskl Apr 29, 2024

Create TES Basespace Upload manager service #186

Create TES Basespace Upload manager service #186

Conversation

alexiswl commented Mar 25, 2024 • edited Loading

TODO

victorskl commented Mar 25, 2024

alexiswl commented Apr 17, 2024

victorskl commented Apr 17, 2024

alexiswl commented Apr 18, 2024

victorskl commented Apr 18, 2024

williamputraintan commented Apr 18, 2024

victorskl commented Apr 18, 2024

victorskl left a comment

Choose a reason for hiding this comment

alexiswl commented Apr 22, 2024

victorskl commented Apr 22, 2024

alexiswl commented Apr 22, 2024

victorskl commented Apr 22, 2024

victorskl Apr 29, 2024

Choose a reason for hiding this comment

victorskl Apr 29, 2024

Choose a reason for hiding this comment

alexiswl Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

victorskl Apr 29, 2024

Choose a reason for hiding this comment

victorskl Apr 29, 2024

Choose a reason for hiding this comment

alexiswl commented Mar 25, 2024 •

edited

Loading

alexiswl Apr 29, 2024 •

edited

Loading