Merge branch 'master' into feature/array-node-support-workflow-parall…

…elism Signed-off-by: Paul Dittamo <[email protected]>
flyteorg · Apr 4, 2024 · 78a7941 · 78a7941
2 parents 0bafb47 + 452538a
commit 78a7941
Show file tree

Hide file tree

Showing 463 changed files with 24,043 additions and 24,531 deletions.
diff --git a/.github/workflows/validate-helm-charts.yaml b/.github/workflows/validate-helm-charts.yaml
@@ -28,6 +28,9 @@ jobs:
           python-version: "3.10"
           check-latest: true
 
+      - uses: unionai/[email protected]
+        name: Setup flytectl
+
       - name: Set up chart-testing
         uses: helm/chart-testing-action@v2
 
@@ -41,15 +44,16 @@ jobs:
 
       - name: Run chart-testing (lint)
         if: steps.charts-changed.outputs.changed == 'true'
-        run: ct lint --target-branch ${{ github.event.repository.default_branch }}
-
-      - name: Create KinD cluster
-        if: steps.charts-changed.outputs.changed == 'true'
-        uses: helm/kind-action@v1
-
-      - name: Run chart-testing (install)
-        if: steps.charts-changed.outputs.changed == 'true'
-        run: ct install --target-branch ${{ github.event.repository.default_branch }}
+        run: |
+          ct lint \
+            --target-branch ${{ github.event.repository.default_branch }} \
+            --validate-maintainers=false \
+            --check-version-increment=false \
+            --chart-repos spark-operator=https://kubeflow.github.io/spark-operator \
+            --chart-repos dask=https://helm.dask.org \
+            --chart-repos bitnami=https://charts.bitnami.com/bitnami \
+            --chart-repos twuni=https://helm.twun.io \
+            --chart-repos kubernetes-dashboard=https://kubernetes.github.io/dashboard
 
   validate-manifests:
     needs:

diff --git a/CHANGELOG/CHANGELOG-v1.11.1-b0.md b/CHANGELOG/CHANGELOG-v1.11.1-b0.md
@@ -0,0 +1,2 @@
+# Flyte v1.11.1-b0
+
diff --git a/CHANGELOG/CHANGELOG-v1.11.1-b1.md b/CHANGELOG/CHANGELOG-v1.11.1-b1.md
@@ -0,0 +1,45 @@
+# Flyte v1.11.1-b1
+
+## What's Changed
+* sagemaker agent backend setup documentation by @samhita-alla in https://github.com/flyteorg/flyte/pull/5064
+* add first version of gke-starter values file by @cjidboon94 in https://github.com/flyteorg/flyte/pull/5026
+* Fix open ai secret name by @eapolinario in https://github.com/flyteorg/flyte/pull/5098
+* Allow setting a ExecutionClusterLabel when triggering a Launchplan/Workflow/Task by @RRap0so in https://github.com/flyteorg/flyte/pull/4998
+* Improve audience mismatch debugging by @ddl-rliu in https://github.com/flyteorg/flyte/pull/5078
+* docs(sandbox): Add guide for running newest flyteconsole in flyte sandbox by @MortalHappiness in https://github.com/flyteorg/flyte/pull/5100
+* Remove unnecessary step and fix numbering in code examples by @eapolinario in https://github.com/flyteorg/flyte/pull/5104
+* fix rendering of flyte-core and flyteagent charts by @pbrogan12 in https://github.com/flyteorg/flyte/pull/5048
+* Bump golang.org/x/net from 0.3.1-0.20221206200815-1e63c2f08a10 to 0.7.0 in /docker/sandbox-bundled/bootstrap by @dependabot in https://github.com/flyteorg/flyte/pull/3390
+* Update repeated value filters with ValueNotIn support by @troychiu in https://github.com/flyteorg/flyte/pull/5110
+* Update container builds from go 1.21.5 to 1.21.latest by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/5097
+* Add optional org param to ProjectGetRequest by @katrogan in https://github.com/flyteorg/flyte/pull/5118
+* Stop building read-the-docs for flyteidl by @eapolinario in https://github.com/flyteorg/flyte/pull/5120
+* Bump google.golang.org/grpc and otelgrpc by @eapolinario in https://github.com/flyteorg/flyte/pull/5121
+* Fix broken link in "Mapping Python to Flyte types" table by @neverett in https://github.com/flyteorg/flyte/pull/5122
+* Bump version of otel and grpc in flytestdlib by @eapolinario in https://github.com/flyteorg/flyte/pull/5123
+* [flyteadmin] Show diff structure when re-registration two different task with same ids by @austin362667 in https://github.com/flyteorg/flyte/pull/4924
+* Set flyteadmin grpc port correctly in config / Flyte-core flyteadmin / datacatalog expose ports by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/5013
+* Add flyte-core missing priorityClassName to webhook values by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/4987
+* Update environment_setup.md by @ssen85 in https://github.com/flyteorg/flyte/pull/4963
+* [Docs] Open external links in new tab by @MortalHappiness in https://github.com/flyteorg/flyte/pull/4966
+* Fix separateGrpcIngress flag not working in flyte-binary helm chart  by @lowc1012 in https://github.com/flyteorg/flyte/pull/4946
+* docs(contribute): Change go mod tidy to make go-tidy by @MortalHappiness in https://github.com/flyteorg/flyte/pull/5131
+* Fix execution phase by @troychiu in https://github.com/flyteorg/flyte/pull/5127
+* Add trailing slash to compile make target by @eapolinario in https://github.com/flyteorg/flyte/pull/4648
+* Change retry error from RuntimeError to FlyteRecoverableException by @dansola in https://github.com/flyteorg/flyte/pull/5128
+* Adapt ray flyteplugin to Kuberay 1.1.0 by @ByronHsu in https://github.com/flyteorg/flyte/pull/5067
+* Boilerplate simplification by @eapolinario in https://github.com/flyteorg/flyte/pull/5134
+* Upgrade cloudevents to v2.15.2 by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/5142
+* update sagemaker agent setup doc as secrets aren't required anymore by @samhita-alla in https://github.com/flyteorg/flyte/pull/5138
+* Upgrade logrus to v1.9.3 everywhere by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/5139
+* Upgrade go-restful to v3.12.0 by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/5140
+* Regenerate ray pflags by @eapolinario in https://github.com/flyteorg/flyte/pull/5149
+* Split access token into half and store to avoid "securecookie: the value is too long" error by @yubofredwang in https://github.com/flyteorg/flyte/pull/4863
+
+## New Contributors
+* @cjidboon94 made their first contribution in https://github.com/flyteorg/flyte/pull/5026
+* @ddl-rliu made their first contribution in https://github.com/flyteorg/flyte/pull/5078
+* @pbrogan12 made their first contribution in https://github.com/flyteorg/flyte/pull/5048
+* @austin362667 made their first contribution in https://github.com/flyteorg/flyte/pull/4924
+* @ssen85 made their first contribution in https://github.com/flyteorg/flyte/pull/4963
+* @dansola made their first contribution in https://github.com/flyteorg/flyte/pull/5128
diff --git a/Dockerfile b/Dockerfile
@@ -2,7 +2,7 @@ ARG FLYTECONSOLE_VERSION=latest
 FROM ghcr.io/flyteorg/flyteconsole:${FLYTECONSOLE_VERSION} AS flyteconsole
 
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-bookworm AS flytebuilder
+FROM --platform=${BUILDPLATFORM} golang:1.21-bookworm AS flytebuilder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"

diff --git a/Dockerfile.datacatalog b/Dockerfile.datacatalog
@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'FLYTEORG/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/flyteorg/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -15,14 +15,15 @@ RUN apk add git openssh-client make curl
 RUN mkdir /artifacts
 
 # Pull GRPC health probe binary for liveness and readiness checks
-RUN GRPC_HEALTH_PROBE_VERSION=v0.4.11 && \
+RUN GRPC_HEALTH_PROBE_VERSION=v0.4.25 && \
     wget -qO/artifacts/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \
     chmod +x /artifacts/grpc_health_probe && \
-    echo 'ded15e598d887ccc47bf2321371950bbf930f5e4856b9f75712ce4b2b5120480  /artifacts/grpc_health_probe' > .grpc_checksum && \
+    echo 'd14037ad94518eac8dbe57c146d6c2ca808f7f32600ee0c4057ef4b03ee0e42e  /artifacts/grpc_health_probe' > .grpc_checksum && \
     sha256sum -c .grpc_checksum
 
 WORKDIR /go/src/github.com/flyteorg/datacatalog
 
+COPY boilerplate ../boilerplate
 COPY datacatalog .
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot ../flytecopilot

diff --git a/Dockerfile.flyteadmin b/Dockerfile.flyteadmin
@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -15,14 +15,15 @@ RUN apk add git openssh-client make curl
 RUN mkdir /artifacts
 
 # Pull GRPC health probe binary for liveness and readiness checks
-RUN GRPC_HEALTH_PROBE_VERSION=v0.4.11 && \
+RUN GRPC_HEALTH_PROBE_VERSION=v0.4.25 && \
     wget -qO/artifacts/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \
     chmod +x /artifacts/grpc_health_probe && \
-    echo 'ded15e598d887ccc47bf2321371950bbf930f5e4856b9f75712ce4b2b5120480  /artifacts/grpc_health_probe' > .grpc_checksum && \
+    echo 'd14037ad94518eac8dbe57c146d6c2ca808f7f32600ee0c4057ef4b03ee0e42e  /artifacts/grpc_health_probe' > .grpc_checksum && \
     sha256sum -c .grpc_checksum
 
 WORKDIR /go/src/github.com/flyteorg/flyteadmin
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin .
 COPY flytecopilot ../flytecopilot

diff --git a/Dockerfile.flytecopilot b/Dockerfile.flytecopilot
@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -13,6 +13,7 @@ RUN apk add git openssh-client make curl
 
 WORKDIR /go/src/github.com/flyteorg/flytecopilot
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot .

diff --git a/Dockerfile.flytepropeller b/Dockerfile.flytepropeller
@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -13,6 +13,7 @@ RUN apk add git openssh-client make curl
 
 WORKDIR /go/src/github.com/flyteorg/flytepropeller
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot ../flytecopilot

diff --git a/Dockerfile.flytescheduler b/Dockerfile.flytescheduler
@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -16,6 +16,7 @@ RUN mkdir /artifacts
 
 WORKDIR /go/src/github.com/flyteorg/flyteadmin
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin .
 COPY flytecopilot ../flytecopilot

diff --git a/Makefile b/Makefile
@@ -20,7 +20,7 @@ cmd/single/dist:
 .PHONY: compile
 compile: cmd/single/dist
 	go build -tags console -v -o flyte -ldflags=$(LD_FLAGS) ./cmd/
-	mv ./flyte ${GOPATH}/bin || echo "Skipped copying 'flyte' to ${GOPATH}/bin"
+	mv ./flyte ${GOPATH}/bin/ || echo "Skipped copying 'flyte' to ${GOPATH}/bin"
 
 .PHONY: linux_compile
 linux_compile: cmd/single/dist
@@ -56,7 +56,9 @@ install-conda-lock:
 
 .PHONY: conda-lock
 conda-lock: install-conda-lock
-	conda-lock -f monodocs-environment.yaml --without-cuda --lockfile monodocs-environment.lock.yaml
+	conda-lock -f monodocs-environment.yaml --without-cuda \
+		--lockfile monodocs-environment.lock.yaml \
+		--platform=osx-64 --platform=osx-arm64 --platform=linux-64
 
 .PHONY: stats
 stats:
@@ -112,3 +114,12 @@ go-tidy:
 	make -C flyteplugins go-tidy
 	make -C flytestdlib go-tidy
 	make -C flytecopilot go-tidy
+
+.PHONY: lint-helm-charts
+lint-helm-charts:
+	# This pressuposes that you have act installed
+	act pull_request -W .github/workflows/validate-helm-charts.yaml --container-architecture linux/amd64 -e charts/event.json
+
+.PHONY: clean
+clean: ## Remove the HTML files related to the Flyteconsole.
+	rm -rf cmd/single/dist
diff --git a/...g/boilerplate/flyte/docker_build/Makefile → boilerplate/flyte/docker_build/Makefile b/...g/boilerplate/flyte/docker_build/Makefile → boilerplate/flyte/docker_build/Makefile
diff --git a/...boilerplate/flyte/docker_build/Readme.rst → boilerplate/flyte/docker_build/Readme.rst b/...boilerplate/flyte/docker_build/Readme.rst → boilerplate/flyte/docker_build/Readme.rst
diff --git a/...rplate/flyte/docker_build/docker_build.sh → ...rplate/flyte/docker_build/docker_build.sh b/...rplate/flyte/docker_build/docker_build.sh → ...rplate/flyte/docker_build/docker_build.sh
diff --git a/boilerplate/flyte/end2end/run-tests.py b/boilerplate/flyte/end2end/run-tests.py
@@ -75,9 +75,11 @@ def schedule_workflow_groups(
         workflow_group_item = list(
             filter(lambda item: item["name"] == wf_group, parsed_manifest)
         )
-        workflows = []
-        if workflow_group_item:
-            workflows = workflow_group_item[0]["examples"]
+        if not workflow_group_item:
+            continue
+        workflows = workflow_group_item[0].get("examples")
+        if not workflows:
+            continue
         executions_by_wfgroup[wf_group] = [
             execute_workflow(remote, tag, workflow[0], workflow[1], cluster_pool_name)
             for workflow in workflows

diff --git a/...ate/flyte/flyte_golang_compile/Readme.rst → ...ate/flyte/flyte_golang_compile/Readme.rst b/...ate/flyte/flyte_golang_compile/Readme.rst → ...ate/flyte/flyte_golang_compile/Readme.rst
diff --git a/...ang_compile/flyte_golang_compile.Template → ...ang_compile/flyte_golang_compile.Template b/...ang_compile/flyte_golang_compile.Template → ...ang_compile/flyte_golang_compile.Template
diff --git a/...late/flyte/flyte_golang_compile/update.sh → ...late/flyte/flyte_golang_compile/update.sh b/...late/flyte/flyte_golang_compile/update.sh → ...late/flyte/flyte_golang_compile/update.sh
diff --git a/...erplate/flyte/github_workflows/Readme.rst → ...erplate/flyte/github_workflows/Readme.rst b/...erplate/flyte/github_workflows/Readme.rst → ...erplate/flyte/github_workflows/Readme.rst
diff --git a/...thub_workflows/boilerplate_automation.yml → ...thub_workflows/boilerplate_automation.yml b/...thub_workflows/boilerplate_automation.yml → ...thub_workflows/boilerplate_automation.yml
diff --git a/...erplate/flyte/github_workflows/master.yml → ...erplate/flyte/github_workflows/master.yml b/...erplate/flyte/github_workflows/master.yml → ...erplate/flyte/github_workflows/master.yml
diff --git a/...e/flyte/github_workflows/pull_request.yml → ...e/flyte/github_workflows/pull_request.yml b/...e/flyte/github_workflows/pull_request.yml → ...e/flyte/github_workflows/pull_request.yml
diff --git a/...lerplate/flyte/github_workflows/stale.yml → boilerplate/flyte/github_workflows/stale.yml b/...lerplate/flyte/github_workflows/stale.yml → boilerplate/flyte/github_workflows/stale.yml
diff --git a/...lerplate/flyte/github_workflows/update.sh → boilerplate/flyte/github_workflows/update.sh b/...lerplate/flyte/github_workflows/update.sh → boilerplate/flyte/github_workflows/update.sh
diff --git a/...e/golang_dockerfile/Dockerfile.GoTemplate → ...e/golang_dockerfile/Dockerfile.GoTemplate b/...e/golang_dockerfile/Dockerfile.GoTemplate → ...e/golang_dockerfile/Dockerfile.GoTemplate
diff --git a/...rplate/flyte/golang_dockerfile/Readme.rst → ...rplate/flyte/golang_dockerfile/Readme.rst b/...rplate/flyte/golang_dockerfile/Readme.rst → ...rplate/flyte/golang_dockerfile/Readme.rst
diff --git a/...erplate/flyte/golang_dockerfile/update.sh → ...erplate/flyte/golang_dockerfile/update.sh b/...erplate/flyte/golang_dockerfile/update.sh → ...erplate/flyte/golang_dockerfile/update.sh