You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Composer v2 was released this week, and it uses GKE Autopilot under the hood. With GKE Autopilot, we can no longer use node pool affinity with the KubernetesPodOperator. In anticipation of eventually migrating to Composer v2 we can utilize the GKEStartPodOperator, a child of the KubernetesPodOperator in one of two ways:
Create a long lasting GKE Cluster with node pools that is managed outside of Airflow, use the GKEStartPodOperator with node pool affinity as needed.
Programmatically create an ephemeral GKE Cluster with node pools using the GKECreateClusterOperator, use the GKEStartPodOperator to launch pods with node pool affinity, and tear down the cluster with the GKEDeleteClusterOperator
The text was updated successfully, but these errors were encountered:
Composer v2 was released this week, and it uses GKE Autopilot under the hood. With GKE Autopilot, we can no longer use node pool affinity with the
KubernetesPodOperator
. In anticipation of eventually migrating to Composer v2 we can utilize theGKEStartPodOperator
, a child of theKubernetesPodOperator
in one of two ways:GKEStartPodOperator
with node pool affinity as needed.GKECreateClusterOperator
, use theGKEStartPodOperator
to launch pods with node pool affinity, and tear down the cluster with theGKEDeleteClusterOperator
The text was updated successfully, but these errors were encountered: