-
Notifications
You must be signed in to change notification settings - Fork 300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/array node workflow parallelism #2268
Conversation
Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2268 +/- ##
===========================================
- Coverage 83.09% 53.57% -29.52%
===========================================
Files 324 180 -144
Lines 24860 18133 -6727
Branches 3761 3547 -214
===========================================
- Hits 20657 9715 -10942
- Misses 3582 7991 +4409
+ Partials 621 427 -194 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
Can you fix the unit test failure? The error:
|
Signed-off-by: Paul Dittamo <[email protected]>
@eapolinario oof on my end. Just updated - will re-request your review after the builds pass |
* default array node concurrency to -1 Signed-off-by: Paul Dittamo <[email protected]> * typo Signed-off-by: Paul Dittamo <[email protected]> * set default concurrency to None for backwards compatibility Signed-off-by: Paul Dittamo <[email protected]> * update unit test - hashed name Signed-off-by: Paul Dittamo <[email protected]> --------- Signed-off-by: Paul Dittamo <[email protected]> Signed-off-by: Jan Fiedler <[email protected]>
Tracking issue
flyteorg/flyte#3924
Why are the changes needed?
Want to add support for array node map tasks to utilize threads under the parent workflow's set parallelism. Decrease confusion on default/expected map task parallelism
What changes were proposed in this pull request?
Set default parallelism to None. If a user bumps their Flyte version to include flyteorg/flyte#5214 this will default the array node map task to utilize the parent wf's parallelism. If a user has an older flyte version this will map to 0, which will maintain the existing parallelism behavior.
NOTE this will cause cache misses on the first re-runs as the hashed name will change
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
flyteorg/flyte#5062 + flyteorg/flyte#5214 were already merged
Docs link