Update storage config in value-gcp.yaml

[pingsutw]

Tracking issue

NA

Why are the changes needed?

The current config is incorrect. Not sure how does it work before.

flyte/flytestdlib/storage/config.go

Lines 47 to 65 in 74b0a1b

	type Config struct {
	Type Type `json:"type" pflag:",Sets the type of storage to configure [s3/minio/local/mem/stow]."`
	// Deprecated: Please use StowConfig instead
	Connection ConnectionConfig `json:"connection"`
	Stow StowConfig `json:"stow,omitempty" pflag:",Storage config for stow backend."`
	// Container here is misleading, it refers to a Bucket (AWS S3) like blobstore entity. In some terms it could be a table
	InitContainer string `json:"container" pflag:",Initial container (in s3 a bucket) to create -if it doesn't exist-.'"`
	// By default, if this is not enabled, multiple containers are not supported by the storage layer. Only the configured `container` InitContainer will be allowed to requests data from. But, if enabled then data will be loaded to written to any
	// container specified in the DataReference.
	MultiContainerEnabled bool `json:"enable-multicontainer" pflag:",If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered"`
	// Caching is recommended to improve the performance of underlying systems. It caches the metadata and resolving
	// inputs is accelerated. The size of the cache is large so understand how to configure the cache.
	// TODO provide some default config choices
	// If this section is skipped, Caching is disabled
	Cache CachingConfig `json:"cache"`
	Limits LimitsConfig `json:"limits" pflag:",Sets limits for stores."`
	DefaultHTTPClient HTTPClientConfig `json:"defaultHttpClient" pflag:",Sets the default http client config."`
	SignedURL SignedURLConfig `json:"signedUrl" pflag:",Sets config for SignedURL."`

What changes were proposed in this pull request?

Update storage config

How was this patch tested?

local single binary

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.96%. Comparing base (4a95abf) to head (74b0a1b).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5003      +/-   ##
==========================================
- Coverage   58.97%   58.96%   -0.01%     
==========================================
  Files         645      645              
  Lines       55562    55562              
==========================================
- Hits        32767    32762       -5     
- Misses      20200    20205       +5     
  Partials     2595     2595              

Flag	Coverage Δ
unittests	58.96% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pingsutw]

@katrogan / @davidmirror-ops do you know how does it work before? we don't support gsc type.

flyte/flytestdlib/storage/config.go

Lines 47 to 65 in 74b0a1b

	type Config struct {
	Type Type `json:"type" pflag:",Sets the type of storage to configure [s3/minio/local/mem/stow]."`
	// Deprecated: Please use StowConfig instead
	Connection ConnectionConfig `json:"connection"`
	Stow StowConfig `json:"stow,omitempty" pflag:",Storage config for stow backend."`
	// Container here is misleading, it refers to a Bucket (AWS S3) like blobstore entity. In some terms it could be a table
	InitContainer string `json:"container" pflag:",Initial container (in s3 a bucket) to create -if it doesn't exist-.'"`
	// By default, if this is not enabled, multiple containers are not supported by the storage layer. Only the configured `container` InitContainer will be allowed to requests data from. But, if enabled then data will be loaded to written to any
	// container specified in the DataReference.
	MultiContainerEnabled bool `json:"enable-multicontainer" pflag:",If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered"`
	// Caching is recommended to improve the performance of underlying systems. It caches the metadata and resolving
	// inputs is accelerated. The size of the cache is large so understand how to configure the cache.
	// TODO provide some default config choices
	// If this section is skipped, Caching is disabled
	Cache CachingConfig `json:"cache"`
	Limits LimitsConfig `json:"limits" pflag:",Sets limits for stores."`
	DefaultHTTPClient HTTPClientConfig `json:"defaultHttpClient" pflag:",Sets the default http client config."`
	SignedURL SignedURLConfig `json:"signedUrl" pflag:",Sets config for SignedURL."`

[katrogan]

not sure! I know for our GCP deployments we've been setting type: stow

[davidmirror-ops]

For the reference implementation, we've been using storage.type: gcs with no issues

[katrogan]

aha @pingsutw we overwrite it here:

flyte/charts/flyte-core/templates/_helpers.tpl

Line 228 in f9c1a97

type: stow

[davidmirror-ops]

Any idea about the difference between minio and local? We've been using sandbox (rewritten as minio) for on-prem deployments with flyte-core. Maybe local makes more sense.