Skip to content
This repository has been archived by the owner on Aug 31, 2022. It is now read-only.

[Feature Request] Evaluation of tasks without running it. #127

Open
joychugh opened this issue Aug 2, 2018 · 1 comment
Open

[Feature Request] Evaluation of tasks without running it. #127

joychugh opened this issue Aug 2, 2018 · 1 comment

Comments

@joychugh
Copy link

joychugh commented Aug 2, 2018

It would be good to have a way to evaluate the task, if all it's dependencies are OK but not run it.
Something like FloRunner.evaluateTask(task). This can be useful if the the task you want to evaluate
a) runs for a long time (data pipelines)
b) makes a permanent or hard to revert change in an external system

@danielnorberg
Copy link
Contributor

This is interesting. One thought is that this might require Flo to gain an understanding of external dependencies. Currently Flo does not have any external dependency primitive or representation. Everything is just tasks. For example, evaluating a bigquery input dependency is done by running a lookup task. But Flo doesn't understand that it is a lookup or a dependency resolution.

To compare with Luigi, it does not have any first class external dependency primitive either, but at least it has ExternalTask.

An easy to implement Flo analog to the luigi ExternalTask could be a ExternalDependencyOperator that could be implemented by BigQueryLookupOperator, etc. This would then allow the implementation of a general mechanism for evaluating just the dependencies of a Flo dag.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants