Make director-v2 scalable/restartable

[sanderegg]

Currently the dv-2 has 2 main jobs:

• start/stop/schedule dynamic services
• start/stop/schedule computational services

Due to internal limitations, it is not able to be scaled up, which is a big problem for robustness of the oSparc platform.

Tasks

Beta Give feedback

1. computational scheduler: remove in process data and rely only on database before scheduling pipelines
2. dynamic scheduler: extract scheduling logic and move it away from director-v2 #5632
   1 of 9
   +0
   
3. Refactoring of in-house scheduling of computational jobs to allow scaling of simcore services and reduce load on system #6666
   1 of 1
   a:director-v2 +1
   

[mrnicegyu11]

Quick comment: In my mind it has much higher priority that the guy is restartable, scaling comes afterwards ;)

[sanderegg]

@mrnicegyu11
please define restartable because:

• restart phase: there are 2 instances running in parallel for some time (even small), which means it needs to be scalable first or unintended stuff might happen
  I think what you want is a kind of stateless service (which is also what I want), and that it can restart while osparc is running. But while we have several instances up they need to play nice with each other, then you can restart 1 instance without breaking computation pipelines or running dynamic services.