Performance Improvements for Large Projects

[SCA-ZMT]

User Story

GOAL

It is very important for the POs that the user experience of handling (potentially many and ) large projects is smooth.

This considers

• loading/startup time and user experience within the platform
• heavy projects, many smash/simulation projects (in the tree)
• submitting jobs to the comp backend, monitoring jobs
• receiving computed jobs and performing postprocessing

Event Horizon

Beta Give feedback

1. Startup of dynamic sidecar osparc-simcore#6859
   a:dynamic-sidecar a:infra+ops +2
   
2. RClone options tuning osparc-simcore#6858
   a:dynamic-sidecar +1
   
3. Use S3 transfer acceleration osparc-simcore#6247
   a:infra+ops +1
   
4. Autoscaling: When hot buffer are in use the warm buffer are not replacing them osparc-simcore#6929
   a:autoscaling bug +2
   

Tasks

Beta Give feedback

1. Optimize EBS buffering osparc-simcore#6329
   1 of 8
   a:autoscaling a:clusters-keeper a:infra+ops +3
   
2. Filesystem - Large File Handling (Caching) osparc-simcore#6411
   2 of 7
   PO issue +1
   

   
3. Check Ec2 networking tuning (ec2 type dependent?)

MartinKippenberger

Beta Give feedback

1. Use S3 VPC Gateway Endpoint osparc-simcore#5865
   0 of 1
   t:enhancement +1
   

   
2. Use deterministic zipping for inputs/outputs folders osparc-simcore#6244
   2 of 2
   a:dask-service a:simcore-sdk +2
   
3. api-server is aware of internal S3 upload links #1745
   +0
   

Eisbock

Beta Give feedback

1. pull docker images while pulling outputs and state osparc-simcore#5626
   a:director-v2 a:dynamic-sidecar t:enhancement +3
   
2. Preparing the performance check e2e #1674
   +0
   
3. Use initialized EBS storage instead of buffer machines osparc-simcore#5864
   17 of 17
   a:autoscaling a:infra+ops +2
   

[SCA-ZMT]

@drniiken @eofli Could you please edit the description on top with more concrete use case and expectations?

[SCA-ZMT]

@drniiken could you please update the description above? thanks! :)

[mguidon]

• Costs: S3: follow up with Dustin where this is enabled
• Performance: MaG: Need to check how this performs with the public API (we should get metrics for up/download after a fix in the code)
• Speed tests (for uploads)
• Optimization of EBS: cheap machines to save costs, double throughput with RAID0 w/o efs, look into metrics of usage
• Speed: Why takes the dynamic sidecar so long to start?