diff --git a/charts/flyte-core/README.md b/charts/flyte-core/README.md index 3c75dae2b4..5ccb9f60f0 100644 --- a/charts/flyte-core/README.md +++ b/charts/flyte-core/README.md @@ -60,13 +60,15 @@ helm install gateway bitnami/contour -n flyte | cloud_events.eventsPublisher.eventTypes[0] | string | `"all"` | | | cloud_events.eventsPublisher.topicName | string | `"arn:aws:sns:us-east-2:123456:123-my-topic"` | | | cloud_events.type | string | `"aws"` | | -| cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"podAnnotations":{},"service_account_name":"flyteadmin","standaloneDeployment":false,"templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain | +| cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"podAnnotations":{},"podEnv":{},"podLabels":{},"service_account_name":"flyteadmin","standaloneDeployment":false,"templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain | | cluster_resource_manager.config | object | `{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}}` | Configmap for ClusterResource parameters | | cluster_resource_manager.config.cluster_resources | object | `{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}` | ClusterResource parameters Refer to the [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#ClusterResourceConfig) to customize. | | cluster_resource_manager.config.cluster_resources.refreshInterval | string | `"5m"` | How frequently to run the sync process | | cluster_resource_manager.config.cluster_resources.standaloneDeployment | bool | `false` | Starts the cluster resource manager in standalone mode with requisite auth credentials to call flyteadmin service endpoints | | cluster_resource_manager.enabled | bool | `true` | Enables the Cluster resource manager component | | cluster_resource_manager.podAnnotations | object | `{}` | Annotations for ClusterResource pods | +| cluster_resource_manager.podEnv | object | `{}` | Additional ClusterResource container environment variables | +| cluster_resource_manager.podLabels | object | `{}` | Labels for ClusterResource pods | | cluster_resource_manager.service_account_name | string | `"flyteadmin"` | Service account name to run with | | cluster_resource_manager.templates | list | `[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]` | Resource templates that should be applied | | cluster_resource_manager.templates[0] | object | `{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"}` | Template for namespaces resources | @@ -111,8 +113,8 @@ helm install gateway bitnami/contour -n flyte | configmap.schedulerConfig.scheduler.profilerPort | int | `10254` | | | configmap.task_logs | object | `{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":false}}}` | Section that configures how the Task logs are displayed on the UI. This has to be changed based on your actual logging provider. Refer to [structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/logs#LogConfig) to understand how to configure various logging engines | | configmap.task_logs.plugins.logs.cloudwatch-enabled | bool | `false` | One option is to enable cloudwatch logging for EKS, update the region and log group accordingly | -| configmap.task_resource_defaults | object | `{"task_resources":{"defaults":{"cpu":"100m","memory":"500Mi","storage":"500Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}` | Task default resources configuration Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#TaskResourceConfiguration). | -| configmap.task_resource_defaults.task_resources | object | `{"defaults":{"cpu":"100m","memory":"500Mi","storage":"500Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}` | Task default resources parameters | +| configmap.task_resource_defaults | object | `{"task_resources":{"defaults":{"cpu":"100m","ephemeralStorage":"500Mi","memory":"500Mi"},"limits":{"cpu":2,"ephemeralStorage":"20Mi","gpu":1,"memory":"1Gi"}}}` | Task default resources configuration Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#TaskResourceConfiguration). | +| configmap.task_resource_defaults.task_resources | object | `{"defaults":{"cpu":"100m","ephemeralStorage":"500Mi","memory":"500Mi"},"limits":{"cpu":2,"ephemeralStorage":"20Mi","gpu":1,"memory":"1Gi"}}` | Task default resources parameters | | daskoperator | object | `{"enabled":false}` | Optional: Dask Plugin using the Dask Operator | | daskoperator.enabled | bool | `false` | - enable or disable the dask operator deployment installation | | databricks | object | `{"enabled":false,"plugin_config":{"plugins":{"databricks":{"databricksInstance":"dbc-a53b7a3c-614c","entrypointFile":"dbfs:///FileStore/tables/entrypoint.py"}}}}` | Optional: Databricks Plugin allows us to run the spark job on the Databricks platform. | @@ -128,6 +130,8 @@ helm install gateway bitnami/contour -n flyte | datacatalog.image.tag | string | `"v1.10.7-b2"` | Docker image tag | | datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | +| datacatalog.podEnv | object | `{}` | Additional Datacatalog container environment variables | +| datacatalog.podLabels | object | `{}` | Labels for Datacatalog pods | | datacatalog.priorityClassName | string | `""` | Sets priorityClassName for datacatalog pod(s). | | datacatalog.replicaCount | int | `1` | Replicas count for Datacatalog deployment | | datacatalog.resources | object | `{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Datacatalog deployment | @@ -161,6 +165,7 @@ helm install gateway bitnami/contour -n flyte | flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | | flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | +| flyteadmin.podLabels | object | `{}` | Labels for Flyteadmin pods | | flyteadmin.priorityClassName | string | `""` | Sets priorityClassName for flyteadmin pod(s). | | flyteadmin.replicaCount | int | `1` | Replicas count for Flyteadmin deployment | | flyteadmin.resources | object | `{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flyteadmin deployment | @@ -192,6 +197,8 @@ helm install gateway bitnami/contour -n flyte | flyteconsole.image.tag | string | `"v1.10.2"` | | | flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment | | flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods | +| flyteconsole.podEnv | object | `{}` | Additional Flyteconsole container environment variables | +| flyteconsole.podLabels | object | `{}` | Labels for Flyteconsole pods | | flyteconsole.priorityClassName | string | `""` | Sets priorityClassName for flyte console pod(s). | | flyteconsole.replicaCount | int | `1` | Replicas count for Flyteconsole deployment | | flyteconsole.resources | object | `{"limits":{"cpu":"500m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}` | Default resources requests and limits for Flyteconsole deployment | @@ -213,6 +220,8 @@ helm install gateway bitnami/contour -n flyte | flytepropeller.manager | bool | `false` | | | flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | | flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | +| flytepropeller.podEnv | object | `{}` | Additional Flytepropeller container environment variables | +| flytepropeller.podLabels | object | `{}` | Labels for Flytepropeller pods | | flytepropeller.priorityClassName | string | `""` | Sets priorityClassName for propeller pod(s). | | flytepropeller.replicaCount | int | `1` | Replicas count for Flytepropeller deployment | | flytepropeller.resources | object | `{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"100Mi"}}` | Default resources requests and limits for Flytepropeller deployment | @@ -239,6 +248,8 @@ helm install gateway bitnami/contour -n flyte | flytescheduler.image.tag | string | `"v1.10.7-b2"` | Docker image tag | | flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | +| flytescheduler.podEnv | object | `{}` | Additional Flytescheduler container environment variables | +| flytescheduler.podLabels | object | `{}` | Labels for Flytescheduler pods | | flytescheduler.priorityClassName | string | `""` | Sets priorityClassName for flyte scheduler pod(s). | | flytescheduler.resources | object | `{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flytescheduler deployment | | flytescheduler.runPrecheck | bool | `true` | Whether to inject an init container which waits on flyteadmin | diff --git a/charts/flyte-core/templates/_helpers.tpl b/charts/flyte-core/templates/_helpers.tpl index 0221874d5c..2c3b059841 100755 --- a/charts/flyte-core/templates/_helpers.tpl +++ b/charts/flyte-core/templates/_helpers.tpl @@ -28,6 +28,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flyteadmin.podLabels" -}} +{{ include "flyteadmin.labels" . }} +{{- with .Values.flyteadmin.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "flytescheduler.name" -}} flytescheduler {{- end -}} @@ -44,6 +51,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flytescheduler.podLabels" -}} +{{ include "flytescheduler.labels" . }} +{{- with .Values.flytescheduler.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "flyteclusterresourcesync.name" -}} flyteclusterresourcesync {{- end -}} @@ -59,6 +73,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flyteclusterresourcesync.podLabels" -}} +{{ include "flyteclusterresourcesync.labels" . }} +{{- with .Values.cluster_resource_manager.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "datacatalog.name" -}} datacatalog {{- end -}} @@ -74,6 +95,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "datacatalog.podLabels" -}} +{{ include "datacatalog.labels" . }} +{{- with .Values.datacatalog.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "flyteagent.name" -}} flyteagent {{- end -}} @@ -89,6 +117,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flyteagent.podLabels" -}} +{{ include "flyteagent.labels" . }} +{{- with .Values.flyteagent.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "flytepropeller.name" -}} flytepropeller {{- end -}} @@ -104,6 +139,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flytepropeller.podLabels" -}} +{{ include "flytepropeller.labels" . }} +{{- with .Values.flytepropeller.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "flytepropeller-manager.name" -}} flytepropeller-manager {{- end -}} @@ -119,6 +161,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flytepropeller-manager.podLabels" -}} +{{ include "flytepropeller-manager.labels" . }} +{{- with .Values.flytepropeller.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + {{- define "flyte-pod-webhook.name" -}} flyte-pod-webhook {{- end -}} @@ -139,6 +188,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flyteconsole.podLabels" -}} +{{ include "flyteconsole.labels" . }} +{{- with .Values.flyteconsole.podLabels }} +{{ toYaml . }} +{{- end }} +{{- end -}} + # Optional blocks for secret mount {{- define "databaseSecret.volume" -}} diff --git a/charts/flyte-core/templates/admin/deployment.yaml b/charts/flyte-core/templates/admin/deployment.yaml index 8b6dff34af..5a081224a8 100755 --- a/charts/flyte-core/templates/admin/deployment.yaml +++ b/charts/flyte-core/templates/admin/deployment.yaml @@ -16,7 +16,7 @@ spec: {{- with .Values.flyteadmin.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} - labels: {{ include "flyteadmin.labels" . | nindent 8 }} + labels: {{ include "flyteadmin.podLabels" . | nindent 8 }} spec: securityContext: fsGroup: 65534 diff --git a/charts/flyte-core/templates/clusterresourcesync/deployment.yaml b/charts/flyte-core/templates/clusterresourcesync/deployment.yaml index bf13c08080..b4d4847c38 100644 --- a/charts/flyte-core/templates/clusterresourcesync/deployment.yaml +++ b/charts/flyte-core/templates/clusterresourcesync/deployment.yaml @@ -16,7 +16,7 @@ spec: {{- with .Values.cluster_resource_manager.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} - labels: {{ include "flyteclusterresourcesync.labels" . | nindent 8 }} + labels: {{ include "flyteclusterresourcesync.podLabels" . | nindent 8 }} spec: containers: - command: @@ -25,6 +25,12 @@ spec: - {{ .Values.flyteadmin.configPath }} - clusterresource - run + {{- if .Values.cluster_resource_manager.podEnv }} + env: + {{- with .Values.cluster_resource_manager.podEnv }} + {{- toYaml . | nindent 10 }} + {{- end }} + {{- end }} image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" name: sync-cluster-resources diff --git a/charts/flyte-core/templates/console/deployment.yaml b/charts/flyte-core/templates/console/deployment.yaml index 0c9a948f5f..09c9e741c8 100644 --- a/charts/flyte-core/templates/console/deployment.yaml +++ b/charts/flyte-core/templates/console/deployment.yaml @@ -16,7 +16,7 @@ spec: {{- with .Values.flyteconsole.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} - labels: {{ include "flyteconsole.labels" . | nindent 8 }} + labels: {{ include "flyteconsole.podLabels" . | nindent 8 }} spec: securityContext: runAsUser: 1000 @@ -33,13 +33,18 @@ spec: name: flyte-console-config ports: - containerPort: 8080 - {{- if .Values.flyteconsole.ga.enabled }} env: + {{- if .Values.flyteconsole.ga.enabled }} - name: ENABLE_GA value: "{{ .Values.flyteconsole.ga.enabled }}" - name: GA_TRACKING_ID value: "{{ .Values.flyteconsole.ga.tracking_id }}" {{- end }} + {{- if .Values.flyteconsole.podEnv -}} + {{- with .Values.flyteconsole.podEnv }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} resources: {{ toYaml .Values.flyteconsole.resources | nindent 10 }} volumeMounts: - mountPath: /srv/flyte diff --git a/charts/flyte-core/templates/datacatalog/deployment.yaml b/charts/flyte-core/templates/datacatalog/deployment.yaml index e9f2c84ec8..72c18ac61b 100644 --- a/charts/flyte-core/templates/datacatalog/deployment.yaml +++ b/charts/flyte-core/templates/datacatalog/deployment.yaml @@ -16,7 +16,7 @@ spec: {{- with .Values.datacatalog.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} - labels: {{ include "datacatalog.labels" . | nindent 8 }} + labels: {{ include "datacatalog.podLabels" . | nindent 8 }} spec: securityContext: fsGroup: 1001 @@ -47,6 +47,12 @@ spec: {{- with .Values.datacatalog.extraArgs }} {{- tpl (toYaml .) $ | nindent 8 }} {{- end }} + {{- if .Values.datacatalog.podEnv }} + env: + {{- with .Values.datacatalog.podEnv }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} image: "{{ .Values.datacatalog.image.repository }}:{{ .Values.datacatalog.image.tag }}" imagePullPolicy: "{{ .Values.datacatalog.image.pullPolicy }}" name: datacatalog diff --git a/charts/flyte-core/templates/flytescheduler/deployment.yaml b/charts/flyte-core/templates/flytescheduler/deployment.yaml index 443354efd7..fa5cfc4fd9 100755 --- a/charts/flyte-core/templates/flytescheduler/deployment.yaml +++ b/charts/flyte-core/templates/flytescheduler/deployment.yaml @@ -17,7 +17,7 @@ spec: {{- with .Values.flytescheduler.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} - labels: {{ include "flytescheduler.labels" . | nindent 8 }} + labels: {{ include "flytescheduler.podLabels" . | nindent 8 }} spec: securityContext: fsGroup: 65534 @@ -48,6 +48,12 @@ spec: - run - --config - {{ .Values.flytescheduler.configPath }} + {{- if .Values.flytescheduler.podEnv }} + env: + {{- with .Values.flytescheduler.podEnv -}} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} image: "{{ .Values.flytescheduler.image.repository }}:{{ .Values.flytescheduler.image.tag }}" imagePullPolicy: "{{ .Values.flytescheduler.image.pullPolicy }}" name: flytescheduler diff --git a/charts/flyte-core/templates/propeller/deployment.yaml b/charts/flyte-core/templates/propeller/deployment.yaml index 72f94fcc1b..0efb3b5dc9 100644 --- a/charts/flyte-core/templates/propeller/deployment.yaml +++ b/charts/flyte-core/templates/propeller/deployment.yaml @@ -26,9 +26,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} {{- if .Values.flytepropeller.manager }} - labels: {{ include "flytepropeller-manager.labels" . | nindent 8 }} + labels: {{ include "flytepropeller-manager.podLabels" . | nindent 8 }} {{- else }} - labels: {{ include "flytepropeller.labels" . | nindent 8 }} + labels: {{ include "flytepropeller.podLabels" . | nindent 8 }} {{- end }} spec: securityContext: @@ -65,6 +65,11 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace + {{- if .Values.flytepropeller.podEnv -}} + {{- with .Values.flytepropeller.podEnv -}} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} image: "{{ .Values.flytepropeller.image.repository }}:{{ .Values.flytepropeller.image.tag }}" imagePullPolicy: "{{ .Values.flytepropeller.image.pullPolicy }}" {{- if .Values.flytepropeller.manager }} diff --git a/charts/flyte-core/templates/propeller/webhook.yaml b/charts/flyte-core/templates/propeller/webhook.yaml index 2934bfcf21..75866a8c2b 100644 --- a/charts/flyte-core/templates/propeller/webhook.yaml +++ b/charts/flyte-core/templates/propeller/webhook.yaml @@ -26,9 +26,12 @@ spec: app: {{ template "flyte-pod-webhook.name" . }} app.kubernetes.io/name: {{ template "flyte-pod-webhook.name" . }} app.kubernetes.io/version: {{ .Values.flytepropeller.image.tag }} + {{- with .Values.flytepropeller.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} annotations: configChecksum: {{ include (print .Template.BasePath "/propeller/configmap.yaml") . | sha256sum | trunc 63 | quote }} - {{- with .Values.flyteadmin.podAnnotations }} + {{- with .Values.flytepropeller.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} spec: @@ -60,7 +63,7 @@ spec: fieldPath: metadata.namespace volumeMounts: - name: config-volume - mountPath: /etc/flyte/config + mountPath: /etc/flyte/config {{- end }} containers: - name: webhook @@ -81,6 +84,11 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace + {{- if .Values.flytepropeller.podEnv -}} + {{- with .Values.flytepropeller.podEnv -}} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- end }} volumeMounts: - name: config-volume mountPath: /etc/flyte/config diff --git a/charts/flyte-core/values.yaml b/charts/flyte-core/values.yaml index 896b17da91..36a9b22e5e 100755 --- a/charts/flyte-core/values.yaml +++ b/charts/flyte-core/values.yaml @@ -89,9 +89,11 @@ flyteadmin: - limitranges # -- Specifies the verbs (actions) that this ClusterRole can perform on the specified resources verbs: - - '*' + - "*" # -- Annotations for Flyteadmin pods podAnnotations: {} + # -- Labels for Flyteadmin pods + podLabels: {} # -- nodeSelector for Flyteadmin deployment nodeSelector: {} # -- tolerations for Flyteadmin deployment @@ -158,6 +160,10 @@ flytescheduler: imagePullSecrets: [] # -- Annotations for Flytescheduler pods podAnnotations: {} + # -- Additional Flytescheduler container environment variables + podEnv: {} + # -- Labels for Flytescheduler pods + podLabels: {} # -- nodeSelector for Flytescheduler deployment nodeSelector: {} # -- tolerations for Flytescheduler deployment @@ -216,6 +222,10 @@ datacatalog: imagePullSecrets: [] # -- Annotations for Datacatalog pods podAnnotations: {} + # -- Additional Datacatalog container environment variables + podEnv: {} + # -- Labels for Datacatalog pods + podLabels: {} # -- nodeSelector for Datacatalog deployment nodeSelector: {} # -- tolerations for Datacatalog deployment @@ -282,6 +292,10 @@ flytepropeller: imagePullSecrets: [] # -- Annotations for Flytepropeller pods podAnnotations: {} + # -- Additional Flytepropeller container environment variables + podEnv: {} + # -- Labels for Flytepropeller pods + podLabels: {} # -- nodeSelector for Flytepropeller deployment nodeSelector: {} # -- tolerations for Flytepropeller deployment @@ -344,6 +358,10 @@ flyteconsole: type: ClusterIP # -- Annotations for Flyteconsole pods podAnnotations: {} + # -- Additional Flyteconsole container environment variables + podEnv: {} + # -- Labels for Flyteconsole pods + podLabels: {} # -- nodeSelector for Flyteconsole deployment nodeSelector: {} # -- tolerations for Flyteconsole deployment @@ -357,7 +375,6 @@ flyteconsole: # -- Sets priorityClassName for flyte console pod(s). priorityClassName: "" - # It will enable the redoc route in ingress deployRedoc: false @@ -489,37 +506,37 @@ db: configmap: clusters: labelClusterMap: {} -# labelClusterMap: -# team1: -# - id: testcluster -# weight: 1 -# team2: -# - id: testcluster2 -# weight: 0.5 -# - id: testcluster3 -# weight: 0.5 + # labelClusterMap: + # team1: + # - id: testcluster + # weight: 1 + # team2: + # - id: testcluster2 + # weight: 0.5 + # - id: testcluster3 + # weight: 0.5 clusterConfigs: [] -# clusterConfigs: -# - name: "testcluster" -# endpoint: "testcluster_endpoint" -# auth: -# type: "file_path" -# tokenPath: "/path/to/testcluster/token" -# certPath: "/path/to/testcluster/cert" -# - name: "testcluster2" -# endpoint: "testcluster2_endpoint" -# enabled: true -# auth: -# type: "file_path" -# tokenPath: "/path/to/testcluster2/token" -# certPath: "/path/to/testcluster2/cert" -# - name: "testcluster3" -# endpoint: "testcluster3_endpoint" -# enabled: true -# auth: -# type: "file_path" -# tokenPath: "/path/to/testcluster3/token" -# certPath: "/path/to/testcluster3/cert" + # clusterConfigs: + # - name: "testcluster" + # endpoint: "testcluster_endpoint" + # auth: + # type: "file_path" + # tokenPath: "/path/to/testcluster/token" + # certPath: "/path/to/testcluster/cert" + # - name: "testcluster2" + # endpoint: "testcluster2_endpoint" + # enabled: true + # auth: + # type: "file_path" + # tokenPath: "/path/to/testcluster2/token" + # certPath: "/path/to/testcluster2/cert" + # - name: "testcluster3" + # endpoint: "testcluster3_endpoint" + # enabled: true + # auth: + # type: "file_path" + # tokenPath: "/path/to/testcluster3/token" + # certPath: "/path/to/testcluster3/cert" # -- Configuration for Flyte console UI console: @@ -571,7 +588,7 @@ configmap: eventVersion: 2 testing: host: http://flyteadmin - + # -- Authentication configuration auth: authorizedUris: @@ -621,11 +638,11 @@ configmap: defaults: cpu: 100m memory: 500Mi - storage: 500Mi + ephemeralStorage: 500Mi limits: cpu: 2 memory: 1Gi - storage: 20Mi + ephemeralStorage: 20Mi gpu: 1 # -- Admin Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/launchplan#AdminConfig) @@ -824,6 +841,10 @@ cluster_resource_manager: service_account_name: flyteadmin # -- Annotations for ClusterResource pods podAnnotations: {} + # -- Additional ClusterResource container environment variables + podEnv: {} + # -- Labels for ClusterResource pods + podLabels: {} # -- Configmap for ClusterResource parameters config: # -- ClusterResource parameters @@ -922,4 +943,4 @@ databricks: databricks: entrypointFile: dbfs:///FileStore/tables/entrypoint.py # Databricks account - databricksInstance: dbc-a53b7a3c-614c \ No newline at end of file + databricksInstance: dbc-a53b7a3c-614c diff --git a/charts/flyteagent/README.md b/charts/flyteagent/README.md index 80107d8e2b..9db05060d0 100644 --- a/charts/flyteagent/README.md +++ b/charts/flyteagent/README.md @@ -24,6 +24,8 @@ A Helm chart for Flyte agent | nameOverride | string | `""` | | | nodeSelector | object | `{}` | nodeSelector for flyteagent deployment | | podAnnotations | object | `{}` | Annotations for flyteagent pods | +| podEnv | object | `{}` | Additional flyteagent pod container environment variables | +| podLabels | object | `{}` | Labels for flyteagent pods | | ports.containerPort | int | `8000` | | | ports.name | string | `"agent-grpc"` | | | priorityClassName | string | `""` | Sets priorityClassName for datacatalog pod(s). | diff --git a/charts/flyteagent/templates/_helpers.tpl b/charts/flyteagent/templates/_helpers.tpl index fffa18304a..40411a61b9 100755 --- a/charts/flyteagent/templates/_helpers.tpl +++ b/charts/flyteagent/templates/_helpers.tpl @@ -28,6 +28,13 @@ helm.sh/chart: {{ include "flyte.chart" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} +{{- define "flyteagent.podLabels" -}} +{{ include "flyteagent.labels" . }} +{{- with .Values.podLabels }} +{{- toYaml . }} +{{- end }} +{{- end -}} + # Optional blocks for secret mount {{- define "agentSecret.volume" -}} @@ -43,4 +50,4 @@ app.kubernetes.io/managed-by: {{ .Release.Service }} {{- define "flyteagent.servicePort" -}} {{ include .Values.ports.containerPort}} -{{- end }} \ No newline at end of file +{{- end }} diff --git a/charts/flyteagent/templates/agent/deployment.yaml b/charts/flyteagent/templates/agent/deployment.yaml index a8b7a3889e..82359cec64 100644 --- a/charts/flyteagent/templates/agent/deployment.yaml +++ b/charts/flyteagent/templates/agent/deployment.yaml @@ -14,7 +14,7 @@ spec: {{- with .Values.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} - labels: {{ include "flyteagent.labels" . | nindent 8 }} + labels: {{ include "flyteagent.podLabels" . | nindent 8 }} spec: {{- if .Values.priorityClassName }} priorityClassName: {{ .Values.priorityClassName }} @@ -23,6 +23,12 @@ spec: - command: - pyflyte - serve + {{- if .Values.podEnv }} + env: + {{- with .Values.podEnv }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: "{{ .Values.image.pullPolicy }}" name: flyteagent diff --git a/charts/flyteagent/values.yaml b/charts/flyteagent/values.yaml index d1d7ac5cbf..b682600d46 100755 --- a/charts/flyteagent/values.yaml +++ b/charts/flyteagent/values.yaml @@ -56,6 +56,10 @@ serviceAccount: imagePullSecrets: [] # -- Annotations for flyteagent pods podAnnotations: {} +# -- Additional flyteagent pod container environment variables +podEnv: {} +# -- Labels for flyteagent pods +podLabels: {} # -- nodeSelector for flyteagent deployment nodeSelector: {} # -- tolerations for flyteagent deployment diff --git a/deployment/eks/flyte_aws_scheduler_helm_generated.yaml b/deployment/eks/flyte_aws_scheduler_helm_generated.yaml index 9cf02f3d6a..c4f557836a 100644 --- a/deployment/eks/flyte_aws_scheduler_helm_generated.yaml +++ b/deployment/eks/flyte_aws_scheduler_helm_generated.yaml @@ -192,10 +192,12 @@ data: task_resources: defaults: cpu: 1000m + ephemeralStorage: 500Mi memory: 1000Mi storage: 1000Mi limits: cpu: 2 + ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -847,7 +849,7 @@ spec: template: metadata: annotations: - configChecksum: "2b5c85969f2bd85bb51a084f9fd72c20c3aca94be99e53cb4c4e9f78e77ebc5" + configChecksum: "85f2694a4138443026b87878dbbc5f1e9f52aa54eb87ef4c64117d1d91e1a7f" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -1094,6 +1096,7 @@ spec: name: flyte-console-config ports: - containerPort: 8080 + env: resources: limits: cpu: 250m diff --git a/deployment/eks/flyte_helm_controlplane_generated.yaml b/deployment/eks/flyte_helm_controlplane_generated.yaml index 4d2a2d3910..47a75cae25 100644 --- a/deployment/eks/flyte_helm_controlplane_generated.yaml +++ b/deployment/eks/flyte_helm_controlplane_generated.yaml @@ -173,10 +173,12 @@ data: task_resources: defaults: cpu: 1000m + ephemeralStorage: 500Mi memory: 1000Mi storage: 1000Mi limits: cpu: 2 + ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -553,7 +555,7 @@ spec: template: metadata: annotations: - configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" + configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -800,6 +802,7 @@ spec: name: flyte-console-config ports: - containerPort: 8080 + env: resources: limits: cpu: 250m @@ -932,7 +935,7 @@ spec: template: metadata: annotations: - configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" + configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte diff --git a/deployment/eks/flyte_helm_generated.yaml b/deployment/eks/flyte_helm_generated.yaml index 9b0b74a962..910b2746b8 100644 --- a/deployment/eks/flyte_helm_generated.yaml +++ b/deployment/eks/flyte_helm_generated.yaml @@ -204,10 +204,12 @@ data: task_resources: defaults: cpu: 1000m + ephemeralStorage: 500Mi memory: 1000Mi storage: 1000Mi limits: cpu: 2 + ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -878,7 +880,7 @@ spec: template: metadata: annotations: - configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" + configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -1125,6 +1127,7 @@ spec: name: flyte-console-config ports: - containerPort: 8080 + env: resources: limits: cpu: 250m @@ -1257,7 +1260,7 @@ spec: template: metadata: annotations: - configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" + configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte diff --git a/deployment/gcp/flyte_helm_controlplane_generated.yaml b/deployment/gcp/flyte_helm_controlplane_generated.yaml index ec7d3dadf5..a31446875c 100644 --- a/deployment/gcp/flyte_helm_controlplane_generated.yaml +++ b/deployment/gcp/flyte_helm_controlplane_generated.yaml @@ -178,10 +178,12 @@ data: task_resources: defaults: cpu: 500m + ephemeralStorage: 500Mi memory: 500Mi storage: 500Mi limits: cpu: 2 + ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -568,7 +570,7 @@ spec: template: metadata: annotations: - configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" + configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -815,6 +817,7 @@ spec: name: flyte-console-config ports: - containerPort: 8080 + env: resources: limits: cpu: 250m @@ -947,7 +950,7 @@ spec: template: metadata: annotations: - configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" + configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte diff --git a/deployment/gcp/flyte_helm_generated.yaml b/deployment/gcp/flyte_helm_generated.yaml index de946d8ad4..0627ba986c 100644 --- a/deployment/gcp/flyte_helm_generated.yaml +++ b/deployment/gcp/flyte_helm_generated.yaml @@ -209,10 +209,12 @@ data: task_resources: defaults: cpu: 500m + ephemeralStorage: 500Mi memory: 500Mi storage: 500Mi limits: cpu: 2 + ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -901,7 +903,7 @@ spec: template: metadata: annotations: - configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" + configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -1148,6 +1150,7 @@ spec: name: flyte-console-config ports: - containerPort: 8080 + env: resources: limits: cpu: 250m @@ -1280,7 +1283,7 @@ spec: template: metadata: annotations: - configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" + configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte diff --git a/deployment/sandbox/flyte_helm_generated.yaml b/deployment/sandbox/flyte_helm_generated.yaml index a8da64485c..cade17e8d6 100644 --- a/deployment/sandbox/flyte_helm_generated.yaml +++ b/deployment/sandbox/flyte_helm_generated.yaml @@ -334,10 +334,12 @@ data: task_resources: defaults: cpu: 100m + ephemeralStorage: 500Mi memory: 200Mi storage: 5Mi limits: cpu: 2 + ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 20Mi @@ -6690,7 +6692,7 @@ spec: template: metadata: annotations: - configChecksum: "45f0232531c0d1494809cf83387a95b2fc802019ea095de7a24ccd4f8de86ec" + configChecksum: "82d6ffa2a2dd83eb11c491a95af43fdede659d6b5b400b6edcd88291a28c4f4" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -7040,7 +7042,7 @@ spec: template: metadata: annotations: - configChecksum: "45f0232531c0d1494809cf83387a95b2fc802019ea095de7a24ccd4f8de86ec" + configChecksum: "82d6ffa2a2dd83eb11c491a95af43fdede659d6b5b400b6edcd88291a28c4f4" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte diff --git a/docs/concepts/component_architecture/flytepropeller_architecture.rst b/docs/concepts/component_architecture/flytepropeller_architecture.rst index a04f6dbe4d..c04edbf617 100644 --- a/docs/concepts/component_architecture/flytepropeller_architecture.rst +++ b/docs/concepts/component_architecture/flytepropeller_architecture.rst @@ -19,7 +19,7 @@ A Flyte :ref:`workflow ` is represented as a Directed Acycli - ``DynamicNodes`` add nodes to the DAG. - ``WorkflowNodes`` allow embedding workflows within each other. -FlytePropeller is responsible for scheduling and tracking execution of Flyte workflows. It is implemented using a K8s controller and adheres to the established K8s design principles. In this scheme, resources are periodically evaluated and the goal is to transition from the observed state to a requested state. +FlytePropeller is responsible for scheduling and tracking execution of Flyte workflows. It is implemented using a K8s controller and adheres to the established K8s design principles. In this scheme, resources are periodically evaluated and the goal is to transition from the observed state to a requested state. In our case, workflows are the resources and they are iteratively evaluated to transition from the current state to success. During each loop, the current workflow state is established as the phase of workflow nodes and subsequent tasks, and FlytePropeller performs operations to transition this state to success. The operations may include scheduling (or rescheduling) node executions, evaluating dynamic or branch nodes, etc. These design decisions ensure that FlytePropeller can scale to manage a large number of concurrent workflows without performance degradation. @@ -33,7 +33,7 @@ Components FlyteWorkflow CRD / K8s Integration ----------------------------------- -Workflows in Flyte are maintained as Custom Resource Definitions (CRDs) in Kubernetes, which are stored in the backing etcd cluster. Each execution of a workflow definition results in the creation of a new FlyteWorkflow CR (Custom Resource) which maintains a state for the entirety of processing. CRDs provide variable definitions to describe both resource specifications (spec) and status' (status). The FlyteWorkflow CRD uses the spec subsection to detail the workflow DAG, embodying node dependencies, etc. The status subsection tracks workflow metadata including overall workflow status, node/task phases, status/phase transition timestamps, etc. +Workflows in Flyte are maintained as Custom Resource Definitions (CRDs) in Kubernetes, which are stored in the backing etcd cluster. Each execution of a workflow definition results in the creation of a new FlyteWorkflow CR (Custom Resource) which maintains a state for the entirety of processing. CRDs provide variable definitions to describe both resource specifications (spec) and status (status). The FlyteWorkflow CRD uses the spec subsection to detail the workflow DAG, embodying node dependencies, etc. The status subsection tracks workflow metadata including overall workflow status, node/task phases, status/phase transition timestamps, etc. K8s exposes a powerful controller/operator API that enables entities to track creation/updates over a specific resource type. FlytePropeller uses this API to track FlyteWorkflows, meaning every time an instance of the FlyteWorkflow CR is created/updated, the FlytePropeller instance is notified. FlyteAdmin is the common entry point, where initialization of FlyteWorkflow CRs may be triggered by user workflow definition executions, automatic relaunches, or periodically scheduled workflow definition executions. However, it is conceivable to manually create FlyteWorkflow CRs, but this will have limited visibility and usability. @@ -69,7 +69,15 @@ NodeHandlers FlytePropeller includes a robust collection of NodeHandlers to support diverse evaluation of the workflow DAG: -* **TaskHandler (Plugins)**: These are responsible for executing plugin specific tasks. This may include contacting FlyteAdmin to schedule K8s pod to perform work, calling a web API to begin/track evaluation, and much more. The plugin paradigm exposes an extensible interface for adding functionality to Flyte workflows. +* **TaskHandler (Plugins)**: These are responsible for executing tasks in the Flyte cluster. There are mainly 3 kinds of tasks for the task handler: + + 1. **Pod Task**: Create a pod in the Kubernetes cluster, execute the task, and then delete the pod. + + 2. **K8s Operator Backend Plugin**: Install a specific Kubernetes Operator (e.g., Spark, Ray, and Kubeflow) in the cluster, create pods by the Kubernetes Operator, execute the task, and then delete the pods. + + 3. **Web API Task**: Send REST/gRPC requests to a server and return the response. + Note: The Web API Task will not start a pod. + * **DynamicHandler**: Flyte workflow CRs are initialized using a DAG compiled during the registration process. The numerous benefits of this approach are beyond the scope of this document. However, there are situations where the complete DAG is unknown at compile time. For example, when executing a task on each value of an input list. Using Dynamic nodes, a new DAG subgraph may be dynamically compiled during runtime and linked to the existing FlyteWorkflow CR. * **WorkflowHandler**: This handler allows embedding workflows within another workflow definition. The API exposes this functionality using either (1) an inline execution, where the workflow function is invoked directly resulting in a single FlyteWorkflow CR with an appended sub-workflow, or (2) a launch plan, which uses a TODO to create a separate sub-FlyteWorkflow CR whose execution state is linked to the parent FlyteWorkflow CR. * **BranchHandler**: The branch handler allows the DAG to follow a specific control path based on input (or computed) values. @@ -79,3 +87,10 @@ FlyteAdmin Events ----------------- It should be noted that the WorkflowExecutor, NodeExecutor, and TaskHandlers send events to FlyteAdmin, enabling it to track workflows in near real-time. + +FlytePlugins +------------ + +Here is an overview architecture of FlytePlugins: + +.. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/concepts/architecture/flytepropeller_plugins_architecture.png diff --git a/docs/concepts/data_management.rst b/docs/concepts/data_management.rst index 33a7f499a1..0d4edbd0a8 100644 --- a/docs/concepts/data_management.rst +++ b/docs/concepts/data_management.rst @@ -168,7 +168,7 @@ Between Tasks Bringing in Your Own Datastores for Raw Data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Flytekit has a pluggable `data persistence layer `__. +Flytekit has a pluggable data persistence layer. This is driven by PROTOCOL. For example, it is theoretically possible to use S3 ``s3://`` for metadata and GCS ``gcs://`` for raw data. It is also possible to create your own protocol ``my_fs://``, to change how data is stored and accessed. But for Metadata, the data should be accessible to Flyte control plane. diff --git a/docs/deployment/plugins/k8s/index.rst b/docs/deployment/plugins/k8s/index.rst index 53b6ebd543..2199f099e8 100644 --- a/docs/deployment/plugins/k8s/index.rst +++ b/docs/deployment/plugins/k8s/index.rst @@ -267,6 +267,7 @@ Spin up a cluster `__, please ensure: * You have the correct kubeconfig and have selected the correct Kubernetes context. + * You have configured the correct flytectl settings in ``~/.flyte/config.yaml``. .. note:: @@ -277,6 +278,88 @@ Spin up a cluster helm repo add flyteorg https://flyteorg.github.io/flyte + .. tabs:: + + If you have installed Flyte using the `flyte-sandbox Helm chart `__, please ensure: + + * You have the correct kubeconfig and have selected the correct Kubernetes context. + + * You have configured the correct flytectl settings in ``~/.flyte/config.yaml``. + + * You have the correct kubeconfig and have selected the correct Kubernetes context. + * You have configured the correct flytectl settings in ``~/.flyte/config.yaml``. + + .. tabs:: + + .. group-tab:: Helm chart + + .. tabs:: + + .. group-tab:: Spark + + create the following four files and apply them using ``kubectl apply -f ``: + + 1. ``serviceaccount.yaml`` + + .. code-block:: yaml + + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + namespace: "{{ namespace }}" + annotations: + eks.amazonaws.com/role-arn: "{{ defaultIamRole }}" + + 2. ``spark_role.yaml`` + + .. code-block:: yaml + + apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: spark-role + namespace: "{{ namespace }}" + rules: + - apiGroups: + - "" + resources: + - pods + - services + - configmaps + verbs: + - "*" + + 3. ``spark_service_account.yaml`` + + .. code-block:: yaml + + apiVersion: v1 + kind: ServiceAccount + metadata: + name: spark + namespace: "{{ namespace }}" + annotations: + eks.amazonaws.com/role-arn: "{{ defaultIamRole }}" + + 4. ``spark_role_binding.yaml`` + + .. code-block:: yaml + + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: spark-role-binding + namespace: "{{ namespace }}" + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: spark-role + subjects: + - kind: ServiceAccount + name: spark + namespace: "{{ namespace }}" + Install the Kubernetes operator ------------------------------- @@ -751,7 +834,123 @@ Specify plugin configuration sidecar: sidecar container_array: k8s-array spark: spark - + + .. group-tab:: Flyte sandbox + + Create a file named ``values-override.yaml`` and add the following config to it: + + .. note:: + + Within the flyte-binary block, the value of inline.storage.signedURL.stowConfigOverride.endpoint should be set to the corresponding node Hostname/IP on the MinIO pod if you are deploying on a Kubernetes cluster. + + .. code-block:: yaml + + flyte-binary: + nameOverride: flyte-sandbox + enabled: true + configuration: + database: + host: '{{ printf "%s-postgresql" .Release.Name | trunc 63 | trimSuffix "-" }}' + password: postgres + storage: + metadataContainer: my-s3-bucket + userDataContainer: my-s3-bucket + provider: s3 + providerConfig: + s3: + disableSSL: true + v2Signing: true + endpoint: http://{{ printf "%s-minio" .Release.Name | trunc 63 | trimSuffix "-" }}.{{ .Release.Namespace }}:9000 + authType: accesskey + accessKey: minio + secretKey: miniostorage + logging: + level: 5 + plugins: + kubernetes: + enabled: true + templateUri: |- + http://localhost:30080/kubernetes-dashboard/#/log/{{.namespace }}/{{ .podName }}/pod?namespace={{ .namespace }} + inline: + task_resources: + defaults: + cpu: 500m + ephemeralStorage: 0 + gpu: 0 + memory: 1Gi + limits: + cpu: 0 + ephemeralStorage: 0 + gpu: 0 + memory: 0 + storage: + signedURL: + stowConfigOverride: + endpoint: http://localhost:30002 + plugins: + k8s: + default-env-vars: + - FLYTE_AWS_ENDPOINT: http://{{ printf "%s-minio" .Release.Name | trunc 63 | trimSuffix "-" }}.{{ .Release.Namespace }}:9000 + - FLYTE_AWS_ACCESS_KEY_ID: minio + - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage + spark: + spark-config-default: + - spark.driver.cores: "1" + - spark.hadoop.fs.s3a.aws.credentials.provider: "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" + - spark.hadoop.fs.s3a.endpoint: http://{{ printf "%s-minio" .Release.Name | trunc 63 | trimSuffix "-" }}.{{ .Release.Namespace }}:9000 + - spark.hadoop.fs.s3a.access.key: "minio" + - spark.hadoop.fs.s3a.secret.key: "miniostorage" + - spark.hadoop.fs.s3a.path.style.access: "true" + - spark.kubernetes.allocation.batch.size: "50" + - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl" + - spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" + - spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A" + - spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" + - spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A" + - spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" + - spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A" + inlineConfigMap: '{{ include "flyte-sandbox.configuration.inlineConfigMap" . }}' + clusterResourceTemplates: + inlineConfigMap: '{{ include "flyte-sandbox.clusterResourceTemplates.inlineConfigMap" . }}' + deployment: + image: + repository: flyte-binary + tag: sandbox + pullPolicy: Never + waitForDB: + image: + repository: bitnami/postgresql + tag: sandbox + pullPolicy: Never + rbac: + # This is strictly NOT RECOMMENDED in production clusters, and is only for use + # within local Flyte sandboxes. + # When using cluster resource templates to create additional namespaced roles, + # Flyte is required to have a superset of those permissions. To simplify + # experimenting with new backend plugins that require additional roles be created + # with cluster resource templates (e.g. Spark), we add the following: + extraRules: + - apiGroups: + - '*' + resources: + - '*' + verbs: + - '*' + enabled_plugins: + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + - spark + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + spark: spark + .. group-tab:: Dask .. tabs:: @@ -817,7 +1016,7 @@ Upgrade the deployment ```` with the name of your namespace (e.g., ``flyte``), and ```` with the name of your YAML file. - .. group-tab:: Flyte core + .. group-tab:: Flyte core / sandbox .. code-block:: bash @@ -830,4 +1029,4 @@ Wait for the upgrade to complete. You can check the status of the deployment pod .. code-block:: bash - kubectl get pods -n --all-namespaces + kubectl get pods -n flyte diff --git a/docs/index.md b/docs/index.md index 40a0a745f7..797b323842 100644 --- a/docs/index.md +++ b/docs/index.md @@ -135,6 +135,8 @@ Have questions or need support? The best way to reach us is through Slack: :hidden: Introduction +Quickstart guide +Getting started with workflow development Flyte Fundamentals Core Use Cases ``` diff --git a/docs/introduction.md b/docs/introduction.md index b902f32786..8e61c433a4 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -4,375 +4,35 @@ jupytext: text_representation: extension: .md format_name: myst -kernelspec: - display_name: Python 3 - language: python - name: python3 # override the toc-determined page navigation order -next-page: getting_started/flyte_fundamentals -next-page-title: Flyte Fundamentals +next-page: getting_started/quickstart_guide +next-page-title: Quickstart guide --- (getting_started_index)= -# Getting Started +# Introduction to Flyte -## Introduction to Flyte +Flyte is a workflow orchestrator that unifies machine learning, data engineering, and data analytics stacks for building robust and reliable applications. Flyte features: +* Reproducible, repeatable workflows +* Strongly typed interfaces +* Structured datasets to enable easy conversion of dataframes between types, and column-level type checking +* Easy movement of data between local and cloud storage +* Easy tracking of data lineages +* Built-in data and artifact visualization -Flyte is a workflow orchestrator that seamlessly unifies data, -machine learning, and analytics stacks for building robust and reliable -applications. +For a full list of feature, see the [Flyte features page](https://flyte.org/features). -This introduction provides a quick overview of how to get Flyte up and running -on your local machine. +## Basic Flyte components -````{dropdown} Want to try Flyte on the browser? -:title: text-muted -:animate: fade-in-slide-down +Flyte is made up of a user plane, control plane, and data plane. +* The **user plane** contains the elements you need to develop the code that will implement your application's directed acyclic graph (DAG). These elements are FlyteKit and Flytectl. Data scientists and machine learning engineers primarily work in the user plane. +* The **control plane** is part of the Flyte backend that is configured by platform engineers or others tasked with setting up computing infrastructure. It consists of FlyteConsole and FlyteAdmin, which serves as the main Flyte API to process requests from clients in the user plane. The control plane sends workflow execution requests to the data plane for execution, and stores information such as current and past running workflows, and provides that information upon request. +* The **data plane** is another part of the Flyte backend that contains FlytePropeller, the core engine of Flyte that executes workflows. FlytePropeller is designed as a [Kubernetes Controller](https://kubernetes.io/docs/concepts/architecture/controller/). The data plane sends status events back to the control plane so that information can be stored and surfaced to end users. -The introduction below is also available on a hosted sandbox environment, where -you can get started with Flyte without installing anything locally. +## Next steps -```{link-button} https://sandbox.union.ai/ ---- -classes: try-hosted-flyte btn-warning btn-block -text: Try Hosted Flyte Sandbox ---- -``` - -```{div} text-muted -*Courtesy of [Union.ai](https://www.union.ai/)* -``` - -```` - -(getting_started_installation)= - -## Installation - -```{admonition} Prerequisites -:class: important - -[Install Docker](https://docs.docker.com/get-docker/) and ensure that you -have the Docker daemon running. - -Flyte supports any [OCI-compatible](https://opencontainers.org/) container -technology (like [Podman](https://podman.io/), -[LXD](https://linuxcontainers.org/lxd/introduction/), and -[Containerd](https://containerd.io/)) when running tasks on a Flyte cluster, but -for the purpose of this guide, `flytectl` uses Docker to spin up a local -Kubernetes cluster so that you can interact with it on your machine. -``` - -First install [flytekit](https://pypi.org/project/flytekit/), Flyte's Python SDK and [Scikit-learn](https://scikit-learn.org/stable). - -```{prompt} bash $ -pip install flytekit flytekitplugins-deck-standard scikit-learn -``` - -Then install [flytectl](https://docs.flyte.org/projects/flytectl/en/latest/), -which the command-line interface for interacting with a Flyte backend. - -````{tabbed} Homebrew - -```{prompt} bash $ -brew install flyteorg/homebrew-tap/flytectl -``` - -```` - -````{tabbed} Curl - -```{prompt} bash $ -curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin -``` - -```` - -## Creating a Workflow - -The first workflow we'll create is a simple model training workflow that consists -of three steps that will: - -1. 🍷 Get the classic [wine dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#wine-recognition-dataset) - using [sklearn](https://scikit-learn.org/stable/). -2. 📊 Process the data that simplifies the 3-class prediction problem into a - binary classification problem by consolidating class labels `1` and `2` into - a single class. -3. 🤖 Train a `LogisticRegression` model to learn a binary classifier. - -First, we'll define three tasks for each of these steps. Create a file called -`example.py` and copy the following code into it. - -```{code-cell} python -:tags: [remove-output] - -import pandas as pd -from sklearn.datasets import load_wine -from sklearn.linear_model import LogisticRegression - -import flytekit.extras.sklearn -from flytekit import task, workflow - - -@task -def get_data() -> pd.DataFrame: - """Get the wine dataset.""" - return load_wine(as_frame=True).frame - -@task -def process_data(data: pd.DataFrame) -> pd.DataFrame: - """Simplify the task from a 3-class to a binary classification problem.""" - return data.assign(target=lambda x: x["target"].where(x["target"] == 0, 1)) - -@task -def train_model(data: pd.DataFrame, hyperparameters: dict) -> LogisticRegression: - """Train a model on the wine dataset.""" - features = data.drop("target", axis="columns") - target = data["target"] - return LogisticRegression(max_iter=3000, **hyperparameters).fit(features, target) -``` - -As we can see in the code snippet above, we defined three tasks as Python -functions: `get_data`, `process_data`, and `train_model`. - -In Flyte, **tasks** are the most basic unit of compute and serve as the building -blocks 🧱 for more complex applications. A task is a function that takes some -inputs and produces an output. We can use these tasks to define a simple model -training workflow: - -```{code-cell} python -@workflow -def training_workflow(hyperparameters: dict) -> LogisticRegression: - """Put all of the steps together into a single workflow.""" - data = get_data() - processed_data = process_data(data=data) - return train_model( - data=processed_data, - hyperparameters=hyperparameters, - ) -``` - -```{note} -A task can also be an isolated piece of compute that takes no inputs and -produces no output, but for the most part to do something useful a task -is typically written with inputs and outputs. -``` - -A **workflow** is also defined as a Python function, and it specifies the flow -of data between tasks and, more generally, the dependencies between tasks 🔀. - -::::{dropdown} {fa}`info-circle` The code above looks like Python, but what do `@task` and `@workflow` do exactly? -:title: text-muted -:animate: fade-in-slide-down - -Flyte `@task` and `@workflow` decorators are designed to work seamlessly with -your code-base, provided that the *decorated function is at the top-level scope -of the module*. - -This means that you can invoke tasks and workflows as regular Python methods and -even import and use them in other Python modules or scripts. - -:::{note} -A {func}`task ` is a pure Python function, while a {func}`workflow ` -is actually a [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) that -only supports a subset of Python's semantics. Learn more in the -{ref}`Flyte Fundamentals ` section. -::: - -:::: - -(intro_running_flyte_workflows)= - -## Running Flyte Workflows in Python - -You can run the workflow in ``example.py`` on a local Python by using `pyflyte`, -the CLI that ships with `flytekit`. - -```{prompt} bash $ -pyflyte run example.py training_workflow \ - --hyperparameters '{"C": 0.1}' -``` - -:::::{dropdown} {fa}`info-circle` Running into shell issues? -:title: text-muted -:animate: fade-in-slide-down - -If you're using Bash, you can ignore this 🙂 -You may need to add .local/bin to your PATH variable if it's not already set, -as that's not automatically added for non-bourne shells like fish or xzsh. - -To use pyflyte, make sure to set the /.local/bin directory in PATH - -:::{code-block} fish -set -gx PATH $PATH ~/.local/bin -::: -::::: - - - -:::::{dropdown} {fa}`info-circle` Why use `pyflyte run` rather than `python example.py`? -:title: text-muted -:animate: fade-in-slide-down - -`pyflyte run` enables you to execute a specific workflow using the syntax -`pyflyte run `. - -Keyword arguments can be supplied to ``pyflyte run`` by passing in options in -the format ``--kwarg value``, and in the case of ``snake_case_arg`` argument -names, you can pass in options in the form of ``--snake-case-arg value``. - -::::{note} -If you want to run a workflow with `python example.py`, you would have to write -a `main` module conditional at the end of the script to actually run the -workflow: - -:::{code-block} python -if __name__ == "__main__": - training_workflow(hyperparameters={"C": 0.1}) -::: - -This becomes even more verbose if you want to pass in arguments: - -:::{code-block} python -if __name__ == "__main__": - import json - from argparse import ArgumentParser - - parser = ArgumentParser() - parser.add_argument("--hyperparameters", type=json.loads) - ... # add the other options - - args = parser.parse_args() - training_workflow(hyperparameters=args.hyperparameters) -::: - -:::: - -::::: - -(getting_started_flyte_cluster)= - -## Running Workflows in a Flyte Cluster - -You can also use `pyflyte run` to execute workflows on a Flyte cluster. -To do so, first spin up a local demo cluster. `flytectl` uses Docker to create -a local Kubernetes cluster and minimal Flyte backend that you can use to run -the example above: - -```{important} -Before you start the local cluster, make sure that you allocate a minimum of -`4 CPUs` and `3 GB` of memory in your Docker daemon. If you're using the -[Docker Desktop](https://www.docker.com/products/docker-desktop/), you can -do this easily by going to: - -`Settings > Resources > Advanced` - -Then set the **CPUs** and **Memory** sliders to the appropriate levels. -``` - -```{prompt} bash $ -flytectl demo start -``` - -````{div} shadow p-3 mb-8 rounded -**Expected Output:** - -```{code-block} -👨‍💻 Flyte is ready! Flyte UI is available at http://localhost:30080/console 🚀 🚀 🎉 -❇️ Run the following command to export sandbox environment variables for accessing flytectl - export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml -🐋 Flyte sandbox ships with a Docker registry. Tag and push custom workflow images to localhost:30000 -📂 The Minio API is hosted on localhost:30002. Use http://localhost:30080/minio/login for Minio console -``` - -```{important} -Make sure to export the `FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml` environment -variable in your shell. -``` -```` - -Then, run the workflow on the Flyte cluster with `pyflyte run` using the -`--remote` flag: - -```{prompt} bash $ -pyflyte run --remote example.py training_workflow \ - --hyperparameters '{"C": 0.1}' -``` - -````{div} shadow p-3 mb-8 rounded - -**Expected Output:** A URL to the workflow execution on your demo Flyte cluster: - -```{code-block} -Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/ to see execution in the console. -``` - -Where ```` is a unique identifier for the workflow execution. - -```` - - -## Inspect the Results - -Navigate to the URL produced by `pyflyte run`. This will take you to -FlyteConsole, the web UI used to manage Flyte entities such as tasks, -workflows, and executions. - -![getting started console](https://github.com/flyteorg/static-resources/raw/main/flytesnacks/getting_started/getting_started_console.gif) - - -```{note} -There are a few features about FlyteConsole worth pointing out in the GIF above: - -- The default execution view shows the list of tasks executing in sequential order. -- The right-hand panel shows metadata about the task execution, including logs, inputs, outputs, and task metadata. -- The **Graph** view shows the execution graph of the workflow, providing visual information about the topology - of the graph and the state of each node as the workflow progresses. -- On completion, you can inspect the outputs of each task, and ultimately, the overarching workflow. -``` - -## Summary - -🎉 **Congratulations! In this introductory guide, you:** - -1. 📜 Created a Flyte script, which trains a binary classification model. -2. 🚀 Spun up a demo Flyte cluster on your local system. -3. 👟 Ran a workflow locally and on a demo Flyte cluster. - - -## What's Next? - -Follow the rest of the sections in the documentation to get a better -understanding of the key constructs that make Flyte such a powerful -orchestration tool 💪. - -```{admonition} Recommendation -:class: tip - -If you're new to Flyte we recommend that you go through the -{ref}`Flyte Fundamentals ` and -{ref}`Core Use Cases ` section before diving -into the other sections of the documentation. -``` - -```{list-table} -:header-rows: 0 -:widths: 10 30 - -* - {ref}`🔤 Flyte Fundamentals ` - - A brief tour of the Flyte's main concepts and development lifecycle -* - {ref}`🌟 Core Use Cases ` - - An overview of core uses cases for data, machine learning, and analytics - practitioners. -* - {ref}`📖 User Guide ` - - A comprehensive view of Flyte's functionality for data scientists, - ML engineers, data engineers, and data analysts. -* - {ref}`📚 Tutorials ` - - End-to-end examples of Flyte for data/feature engineering, machine learning, - bioinformatics, and more. -* - {ref}`🚀 Deployment Guide ` - - Guides for platform engineers to deploy a Flyte cluster on your own - infrastructure. -``` +* To quickly try out Flyte on your machine, follow the {ref}`Quickstart guide `. +* To create a Flyte project that can be used to package workflow code for deployment to a Flyte cluster, see {ref}`"Getting started with workflow development" `. +* To set up a Flyte cluster, see the [Deployment documentation](https://docs.flyte.org/en/latest/deployment/index.html).