Override RayJob configs #1763
Override RayJob configs #1763
Conversation
Signed-off-by: Kevin Su <[email protected]>
pingsutw
requested review from
wild-endeavor,
kumare3,
eapolinario and
cosmicBboy
as code owners
Signed-off-by: Kevin Su <[email protected]>
| @@ -34,6 +34,8 @@ class RayJobConfig: | |||
| head_node_config: typing.Optional[HeadNodeConfig] = None | |||
| runtime_env: typing.Optional[dict] = None | |||
| address: typing.Optional[str] = None | |||
| # TODO: This config should be added to flyteidl. https://github.com/flyteorg/flyteidl/blob/95e11cca2dac18b727f122bbc4456ea6ab499289/protos/flyteidl/plugins/ray.proto#L8 | |||
| config_override: typing.Dict[str, str] = None | |||
There was a problem hiding this comment.
I wonder if the type needs to be Dict[str, Any] or there're other more elegant ways to handle this.
Configs for head/worker spec in RayJob are buried under a few layers, e.g.
rayClusterSpec:
headGroupSpec:
template:
spec:
resources: ...
You also have different specs for head and worker groups
There was a problem hiding this comment.
I also wonder if having this generic config is beneficial vs extending the RayJobConfig with specific options? For example, we could add a namespace option to RayJobConfig, a resources (or limits) to the HeadNodeConfig and WorkerNodeConfig, etc.
The reason why I think it would be better to go that direction is because the ray backend plugin will have to know this configurations options explicitly anyways, because it needs to override specific parts of the RayJob and RayCluster CRDs
There was a problem hiding this comment.
The problem is that get_config will be serialized to proto map<string, string>, so we can only use Dict[str, str] here.
We definitely can extend RayJobConfig, but we need to update IDL first here.
There was a problem hiding this comment.
RayJobConfig will be serialized to RayJob proto we defined in the IDL, then flytepropeller will unmarshal it.
TL;DR
WIP
Type
Are all requirements met?
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/