Artifact protos and related changes (#4474)

* use-personal

* tidy

* update port to 30004

Signed-off-by: Yee Hing Tong <[email protected]>

* Artifacts bundle (#7)

* Remove private repos (#4078)

Signed-off-by: Yee Hing Tong <[email protected]>

* nit from monorepo merge

Signed-off-by: Yee Hing Tong <[email protected]>

* Artifacts bring in idl (#4204)

Signed-off-by: Yee Hing Tong <[email protected]>

* Artifacts bring admin monorepo (#4203)

Main pr to bring in the boilerplate code for artifact service into the main pr.

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Artf/triggers (#4394)

This PR adds the trigger concept.

Other changes:
* Forgot to make the index on artifactkey unique.
* Fix to create artifact, it wasn't pulling back out the artifact key.
* Updated the Artifact service to handle the passing of artifacts created by the events processor to the trigger handling components. Currently, and this is potentially a design flaw, the events processor has no way of contacting the trigger component directly.

Signed-off-by: Yee Hing Tong <[email protected]>

* make trigger key index unique (#4410)

Signed-off-by: Yee Hing Tong <[email protected]>

* revert data proxy changes (#4411)

Signed-off-by: Yee Hing Tong <[email protected]>

* artf/ fix issue with trigger (#4412)

Signed-off-by: Yee Hing Tong <[email protected]>

* sandbox lite dockerfile (#4413)

Signed-off-by: Yee Hing Tong <[email protected]>

* More pgconn handling fixes (#4420)

Signed-off-by: Yee Hing Tong <[email protected]>

* Artf/switch event (#4428)

Most of the changes here are generated proto changes. Actual code changes are:
IDL
* Remove the supplemental fields in the CloudEventTaskExecution object and move them to CloudEventNodeExecution object.
* Remove some fields that the artifact service ended up not using (parent_node_execution and scheduled_at)

in the cloudevent publisher, change the code filling in of the aforementioned supplemental information to happen for node execution events instead of task execution events.
* Remove the deleted fields.

On the event handling side, move the logic to the handling of the node event instead of the task event.
Signed-off-by: Yee Hing Tong <[email protected]>

* Artf/lints (#4429)

* lint fixes
* rename sandbox_utils without underscore

Signed-off-by: Yee Hing Tong <[email protected]>

* look for other places where pgconn is cast and add handling (#4430)

Add additional handling of pgconn
* Change datacatalog logic to use flytestdlib function instead.
* Change flytestdlib constants to capitalize
* Add handling of the other type to Admin code directly

Signed-off-by: Yee Hing Tong <[email protected]>

* add artifacts branch to buf push (#4448)

Signed-off-by: squiishyy <[email protected]>

* add dummy comment (#4451)

Signed-off-by: squiishyy <[email protected]>

* add dummy comment again (#4453)

Signed-off-by: squiishyy <[email protected]>

* artf/updates to source (#4443)


Signed-off-by: Yee Hing Tong <[email protected]>

* add a feature gate (#4472)


Signed-off-by: Yee Hing Tong <[email protected]>

* fix bug in execution_manager with lpExpectedInputs, remove the standalone artifact client and move into idl clientset, update local config file, remove a deprecated function (#4473)

Signed-off-by: Yee Hing Tong <[email protected]>

* initial deletion of artifact code

Signed-off-by: Yee Hing Tong <[email protected]>

* running make manifests in docker/sandbox-bundled to update generated files

Signed-off-by: Yee Hing Tong <[email protected]>

* now compiles (#4481)

Signed-off-by: Yee Hing Tong <[email protected]>

* add a debug line

Signed-off-by: Yee Hing Tong <[email protected]>

* more debugging

Signed-off-by: Yee Hing Tong <[email protected]>

* clean up go mod and switch the order of the input extraction to check for uri first

Signed-off-by: Yee Hing Tong <[email protected]>

* change artifacts http mappings in protos

Signed-off-by: squiishyy <[email protected]>

* testing removal of go mod pin

Signed-off-by: Yee Hing Tong <[email protected]>

* update go mod some more

Signed-off-by: Yee Hing Tong <[email protected]>

* copilot go mod

Signed-off-by: Yee Hing Tong <[email protected]>

* change artifacts http mappings again (#4562)

Signed-off-by: squiishyy <[email protected]>

* panic and revert gorm log level change

Signed-off-by: Yee Hing Tong <[email protected]>

* revert local to 5

Signed-off-by: Yee Hing Tong <[email protected]>

* make helm

Signed-off-by: Yee Hing Tong <[email protected]>

* Reclaim space prior to running functional tests in single-binary

Signed-off-by: Eduardo Apolinario <[email protected]>

* Pin flytectl version to 0.8.0

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Pin flytectl version to 0.8.0"

This reverts commit cfa4745.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Use flytagent v1.10.2

Signed-off-by: Eduardo Apolinario <[email protected]>

* Debug single-binary

Signed-off-by: Eduardo Apolinario <[email protected]>

* Use an infinite loop instead of watch (term env var is not set)

Signed-off-by: Eduardo Apolinario <[email protected]>

* Get logs from single-binary pod

Signed-off-by: Eduardo Apolinario <[email protected]>

* Debug single-binary pod: use a different way of choosing the pod

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix flytectl version

Signed-off-by: Eduardo Apolinario <[email protected]>

* Use port 5432 to connect to postgres

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Use port 5432 to connect to postgres"

This reverts commit 8ded67b.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Fix flytectl version"

This reverts commit 3968d40.

Signed-off-by: Eduardo Apolinario <[email protected]>

* fix database error

Signed-off-by: Yee Hing Tong <[email protected]>

* revert .github/workflows/single-binary.yml by pulling from master

Signed-off-by: Yee Hing Tong <[email protected]>

* Review feedback

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: squiishyy <[email protected]>
Co-authored-by: Joe Eschen <[email protected]>
Co-authored-by: squiishyy <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>

.github/workflows/flyteidl-buf-publish.yml

@@ -3,6 +3,7 @@ name: Publish flyteidl Buf Package
 on:
   push:
     branches:
+      - artifacts-shell
       - artifacts
       - master
     paths:

charts/flyte-sandbox/Chart.lock

@@ -15,4 +15,4 @@ dependencies:
   repository: https://charts.bitnami.com/bitnami
   version: 12.1.9
 digest: sha256:e7155e540bbdb98f690eb12e2bd301a19d8b36833336f6991410cb44d8d9bb5e
-generated: "2023-03-31T09:25:07.80904-07:00"
+generated: "2023-10-28T10:05:34.269916+08:00"

charts/flyte-sandbox/README.md

@@ -40,7 +40,7 @@ A Helm chart for the Flyte local sandbox
 | flyte-binary.configuration.inline.task_resources.limits.gpu | int | `0` |  |
 | flyte-binary.configuration.inline.task_resources.limits.memory | int | `0` |  |
 | flyte-binary.configuration.inlineConfigMap | string | `"{{ include \"flyte-sandbox.configuration.inlineConfigMap\" . }}"` |  |
-| flyte-binary.configuration.logging.level | int | `6` |  |
+| flyte-binary.configuration.logging.level | int | `5` |  |
 | flyte-binary.configuration.logging.plugins.kubernetes.enabled | bool | `true` |  |
 | flyte-binary.configuration.logging.plugins.kubernetes.templateUri | string | `"http://localhost:30080/kubernetes-dashboard/#/log/{{.namespace }}/{{ .podName }}/pod?namespace={{ .namespace }}"` |  |
 | flyte-binary.configuration.storage.metadataContainer | string | `"my-s3-bucket"` |  |

charts/flyte-sandbox/values.yaml

@@ -29,7 +29,7 @@ flyte-binary:
           accessKey: minio
           secretKey: miniostorage
     logging:
-      level: 6
+      level: 5
       plugins:
         kubernetes:
           enabled: true

charts/flyteagent/README.md

@@ -20,7 +20,7 @@ A Helm chart for Flyte agent
 | fullnameOverride | string | `""` |  |
 | image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
 | image.repository | string | `"ghcr.io/flyteorg/flyteagent"` | Docker image for flyteagent deployment |
-| image.tag | string | `"1.10.1"` | Docker image tag |
+| image.tag | string | `"1.10.2"` | Docker image tag |
 | nameOverride | string | `""` |  |
 | nodeSelector | object | `{}` | nodeSelector for flyteagent deployment |
 | podAnnotations | object | `{}` | Annotations for flyteagent pods |

charts/flyteagent/values.yaml

@@ -23,7 +23,7 @@ image:
   # -- Docker image for flyteagent deployment
   repository: ghcr.io/flyteorg/flyteagent
   # -- Docker image tag
-  tag: 1.10.1 # FLYTEAGENT_TAG
+  tag: 1.10.2 # FLYTEAGENT_TAG
   # -- Docker image pull policy
   pullPolicy: IfNotPresent
 ports:

datacatalog/go.mod

@@ -9,17 +9,17 @@ require (
 	github.com/gofrs/uuid v4.2.0+incompatible
 	github.com/golang/glog v1.1.0
 	github.com/golang/protobuf v1.5.3
-	github.com/jackc/pgconn v1.10.1
+	github.com/jackc/pgconn v1.14.1
 	github.com/mitchellh/mapstructure v1.5.0
 	github.com/spf13/cobra v1.7.0
 	github.com/spf13/pflag v1.0.5
 	github.com/stretchr/testify v1.8.4
 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.35.0
 	go.opentelemetry.io/otel v1.19.0
 	google.golang.org/grpc v1.56.1
-	gorm.io/driver/postgres v1.2.3
-	gorm.io/driver/sqlite v1.5.0
-	gorm.io/gorm v1.25.1
+	gorm.io/driver/postgres v1.5.3
+	gorm.io/driver/sqlite v1.5.4
+	gorm.io/gorm v1.25.4
 	gorm.io/plugin/opentelemetry v0.1.4
 )
 
@@ -46,6 +46,7 @@ require (
 	github.com/flyteorg/stow v0.3.8 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
 	github.com/ghodss/yaml v1.0.0 // indirect
+	github.com/go-gormigrate/gormigrate/v2 v2.1.1 // indirect
 	github.com/go-logr/logr v1.2.4 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-openapi/jsonpointer v0.19.6 // indirect
@@ -65,10 +66,9 @@ require (
 	github.com/jackc/chunkreader/v2 v2.0.1 // indirect
 	github.com/jackc/pgio v1.0.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
-	github.com/jackc/pgproto3/v2 v2.2.0 // indirect
-	github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
-	github.com/jackc/pgtype v1.9.0 // indirect
-	github.com/jackc/pgx/v4 v4.14.0 // indirect
+	github.com/jackc/pgproto3/v2 v2.3.2 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
+	github.com/jackc/pgx/v5 v5.4.3 // indirect
 	github.com/jinzhu/inflection v1.0.0 // indirect
 	github.com/jinzhu/now v1.1.5 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
@@ -79,7 +79,7 @@ require (
 	github.com/mailru/easyjson v0.7.7 // indirect
 	github.com/mattn/go-colorable v0.1.12 // indirect
 	github.com/mattn/go-isatty v0.0.14 // indirect
-	github.com/mattn/go-sqlite3 v1.14.15 // indirect
+	github.com/mattn/go-sqlite3 v1.14.17 // indirect
 	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
@@ -145,8 +145,6 @@ replace (
 	github.com/flyteorg/flyte/flyteplugins => ../flyteplugins
 	github.com/flyteorg/flyte/flytepropeller => ../flytepropeller
 	github.com/flyteorg/flyte/flytestdlib => ../flytestdlib
-	gorm.io/driver/sqlite => gorm.io/driver/sqlite v1.1.1
-	gorm.io/gorm => gorm.io/gorm v1.22.4
 	k8s.io/api => k8s.io/api v0.28.2
 	k8s.io/apimachinery => k8s.io/apimachinery v0.28.2
 	k8s.io/client-go => k8s.io/client-go v0.28.2