Airflow times out on Long Running Kubernetes Jobs with logging gaps

[alittlesliceoftom]

This is listed too in other locations:

https://issues.apache.org/jira/browse/AIRFLOW-3534
https://issues.apache.org/jira/browse/AIRFLOW-4526

Only known fix is to disable log stream:
When disabling the get_logs parameter everything works because the log stream is skipped.

Suggested fix - up the retry count on the below line, or allow this to be configured.

airflow/airflow/kubernetes/pod_launcher.py

Line 198 in b23fc13

@tenacity.retry(stop=tenacity.stop_after_attempt(3), wait=tenacity.wait_exponential(), reraise=True)

[alittlesliceoftom]

Happy to raise this as a PR, I.e. changing
@tenacity.retry(stop=tenacity.stop_after_attempt(3), wait=tenacity.wait_exponential(), reraise=True)
to
@tenacity.retry(stop=tenacity.stop_after_attempt(5), wait=tenacity.wait_exponential(), reraise=True)
Which I think would solve the issue for most people and make this much more usable, but I'm not used to this code base!

[alittlesliceoftom]

(note my version exp issue was 1.1.14

[alittlesliceoftom]

Note, I tried this fix and it didn't work. The best thread I have found is this one:

#7428 (comment)

[eladkal]

Does this happens with latest Airflow version?

[potiuk]

Converted it to discussion. We have revamped K8S integration since quite heavily and the log streaming part was specifically rewritten using more stable and newer k8s library so this issue is not relevant any more most likely (we can always reopen if it is in the latest version - so please upgrade and double-check @alittlesliceoftom

[Ans-hrfc]

Hi @potiuk , still happening with Airflow 2.5.1 running in composer

[potiuk]

I am not sure why 2.5.1 when 2.5.3 (with only bugfixes) is there for many months. I suggest you describe all details, logs, circumstances etc. - first to composer team and see wha they tell. They provide paid support and have specific configuration in their K8S so maybe they will be able to find the reason.

[lorenh516]

Also happening with Airflow 2.9.3 running in composer for various circumstances/ DAGs.

[potiuk]

Well. If you can provide more details and logs and open an issue about it that will allow someone to reproduce it or at least guess what could be reason, please do that. Otherwise "various circumstances/DAGs" are hardly making it possible for anyone who would try to diagnose it.