You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Aug 31, 2022. It is now read-only.
It would be good to have a way to evaluate the task, if all it's dependencies are OK but not run it.
Something like FloRunner.evaluateTask(task). This can be useful if the the task you want to evaluate
a) runs for a long time (data pipelines)
b) makes a permanent or hard to revert change in an external system
The text was updated successfully, but these errors were encountered:
This is interesting. One thought is that this might require Flo to gain an understanding of external dependencies. Currently Flo does not have any external dependency primitive or representation. Everything is just tasks. For example, evaluating a bigquery input dependency is done by running a lookup task. But Flo doesn't understand that it is a lookup or a dependency resolution.
To compare with Luigi, it does not have any first class external dependency primitive either, but at least it has ExternalTask.
An easy to implement Flo analog to the luigi ExternalTask could be a ExternalDependencyOperator that could be implemented by BigQueryLookupOperator, etc. This would then allow the implementation of a general mechanism for evaluating just the dependencies of a Flo dag.
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
It would be good to have a way to evaluate the task, if all it's dependencies are OK but not run it.
Something like FloRunner.evaluateTask(task). This can be useful if the the task you want to evaluate
a) runs for a long time (data pipelines)
b) makes a permanent or hard to revert change in an external system
The text was updated successfully, but these errors were encountered: