Support for higher order objects in Kubernetes Integration

[kannon92]

This was brought up on a PR for spark CRD (#22253 (comment)) but I think it warrants its own discussion.

At a high level, Kubernetes users would like to submit higher order objects like Jobs and CRDs. There is also a lot of momentum on improving the Job API so it would be easier to run more complicated batch workloads on Kubernetes. This is useful because you can run many pods with a single job and reduce strain on airflow. And other projects like Kubeflow use CRDs to represent training of models.

The Kubernetes community has been working on improving the Job API and is focused on pushing that to be the standard for running batch jobs. It would be useful to consider how to support Jobs but I also think it may be worth considering a general CRD operator so we don't limit it to a single way of submission.

#17490 and #22253 (comment))

[vsoch]

I am absolutely interested in this as well (mentioned PR comment!) and would like to see a means to interact with Kubernetes - CRD and Batch objects like jobs - more readily. I can most definitely offer up developer time for this.

[potiuk]

Sure. I think it's a good idea in general. Maybe some community members would like comment here, but more likely it should be at the level of devlist thread + AIP proposal.

This seems an important enough topic to be discussed there rather than in GitHub discussions (which are mostly used for quick questions/answers) than implementing big new feature like that. Kubernetes Pod Operator is one of the most used ways of executing tasks. The devlist and Airlfow Improvement Proposals is the way that - following the Apache Software Foundation inclusivity and transparency such significant changes should be discussed ("if it did not happen on the devlist, it did not happen").

You can find information on the devlist at https://airflow.apache.org/community/

Maybe you will get some more feedback here, but my suggestion is to write such draft proposal (looking at others including some research and details on what can be done there) and start dicussion thread on the devlist.