Configuring tasks / using as CLI

[NickCrews]

Hey there! This looks like a great idea, and I think I will use it when a need arises. I have a need that I don't think fits the normal pytask usecase, and I'm looking for advice.

I am writing library A that is used for data ingestion for application B. A downloads some data and transforms it into the format that B expects. A needs to be slightly configurable, eg B needs to tell A to ingest data from before 2022 and after 2010, and where type is "foo" or "bar". At max 5 different orthogonal options.

Since the downloading and parsing of data consists of several steps, with results persisted on disk, and recalculation is expensive, I was thinking that I could use pytask for it. I was thinking of writing a simple wrapper around the python programmatic API. However, I don't see a clean way of adding these configuration options. For instance, the @produces and @depends_on would need to be different depending on the args used, so they would need to be defined dynamically???

Am I just using the wrong tool for the job? Is there a good way of doing what I'm imagining? Thank you!

[tobiasraabe]

Hi, sounds like an interesting question and I have wondered whether there are use cases where you would use pytask in the backend of another application.

First, whether it is a good idea really depends on how many of pytask's features you need and how easy it would be for you to replicate them. For example, ...

• If your underlying DAG does not really change, hardcoding it might be fine.
• If you want to do some automatic error handling when some part of the data processing fails, maybe you have more control without pytask.
• ...

Maybe there is also no harm in starting with pytask and, then, replacing it with your own code.

Secondly, is it feasible to do that? Here is one simple solution.

In the docs are some examples where tasks are dynamically created based on some configuration in a dictionary (https://pytask-dev.readthedocs.io/en/stable/how_to_guides/bp_scalable_repetitions_of_tasks.html).

What you could do is parse the user configuration with B and store it in some global variable like a dictionary at the module level. Program A contains a call to the programmatic API and collects a pre-written task module that imports this dictionary and dynamically creates the tasks like in the docs article.