How to dynamically add jobs to a running workflow?

[tristanjahier]

Hello, I am trying to figure out how to program the following workflow:

• step 1: do some work and store the results in a DB table (lots of rows).
• step 2: read the results from the DB table, and process them but in batches of 1000 for example, because the total number of rows to process is way too high for a single job.

The problem is that when defining step 2, I cannot know at definition time what is the data to process, and how many rows there are, therefore how many jobs I need to spawn to process everything in batches.

I can handle step 2 with a job that spawns/dispatches child jobs at runtime, but I do not know how to bind them properly to the existing and already running workflow.

Is it currently possible to achieve with Venture? If so, could someone give me a hint please?

[ksassnowski]

Unfortunately, it currently isn't possible to dynamically add jobs to a running workflow. It's something I'm still working on solving.

[tristanjahier]

Understood! 🥲
Is there any way to circumvent this limitation with a dirty hack in user land?

[ksassnowski]

Not really, unfortunately.

The good news is that I think I just came up with the solution to this problem. I have a working prototype locally that let's you dynamically add jobs to a running workflow. I might be able to finish this up this weekend or next week 👍

[tristanjahier]

OK got it, thank you for the quick answers!
Don't worry, no rush. Thank you for your work man. 🙂

[travisaustin]

@ksassnowski Thanks for your awesome project.

Just curious if you made any progress on this enhancement.

[tristanjahier]

Ping! I am once again facing the situation where it would be much more convenient to do this. Does anyone knows?