Releases: fluid-cloudnative/fluid
v1.0.3
New Features
- Dynamic mount now supports exposing mount status updates to users.
- Supports configuring current data flow operations based on the affinity of previous data operations to optimize processing efficiency.
- Enables data plane monitoring of JindoRuntime using Prometheus.
- Allows configuration of imagePullSecret in Runtime level.
Bug Fixes.
- Fixed an issue where configuring affinity/nodeSelector/schedulerName/imagePullSecrets parameters in CronDataload caused preloading failures.
- Resolved an issue where upgrading the Fluid version caused data operation failures due to datasets with the same name.
What's Changed
- build(deps): bump github/codeql-action from 3.26.2 to 3.26.5 by @dependabot in #4296
- Chore: Fixed docker build command in Makefile by @omerap12 in #4299
- build(deps): bump github.com/prometheus/client_golang from 1.20.1 to 1.20.2 by @dependabot in #4295
- add fuse properties with default xengine by @frankleaf in #4302
- Build docker images for releasing v1.0.2 by @cheyang in #4303
- build(deps): bump github.com/onsi/gomega from 1.34.1 to 1.34.2 by @dependabot in #4300
- Prepare for v1.0.3 by @cheyang in #4304
- Docker build for fixing fuse properties of jindocache by @cheyang in #4307
- build(deps): bump github/codeql-action from 3.26.5 to 3.26.6 by @dependabot in #4305
- build(deps): bump google.golang.org/grpc from 1.65.0 to 1.66.0 by @dependabot in #4301
- build(deps): bump github.com/felixge/fgprof from 0.9.4 to 0.9.5 by @dependabot in #4306
- Sort packages, optimize Dockerfile by @omerap12 in #4309
- build(deps): bump golang.org/x/net from 0.28.0 to 0.29.0 by @dependabot in #4310
- build(deps): bump github.com/prometheus/client_golang from 1.20.2 to 1.20.3 by @dependabot in #4311
- build(deps): bump google.golang.org/grpc from 1.66.0 to 1.66.1 by @dependabot in #4314
- build(deps): bump google.golang.org/grpc from 1.66.1 to 1.66.2 by @dependabot in #4315
- enhance: NodeUnpublishVolume returns early when target path is not existed by @TrafalgarZZZ in #4318
- enhance: optimize fluid helm chart by @TrafalgarZZZ in #4319
- build(deps): bump github/codeql-action from 3.26.6 to 3.26.7 by @dependabot in #4320
- build(deps): bump com.google.protobuf:protobuf-java from 3.16.3 to 3.25.5 in /samples/hdfs by @dependabot in #4323
- build(deps): bump github.com/prometheus/client_golang from 1.20.3 to 1.20.4 by @dependabot in #4321
- build(deps): bump google.golang.org/grpc from 1.66.2 to 1.67.0 by @dependabot in #4325
- build(deps): bump github/codeql-action from 3.26.7 to 3.26.8 by @dependabot in #4326
- Use Variables for simplify edits and enhance readability. by @omerap12 in #4313
- doc: fix installation guide for how to set resources for controllers by @TrafalgarZZZ in #4330
- bugfix: fix data operation failure when upgrading cache engine implementation with same Dataset name by @TrafalgarZZZ in #4329
- build(deps): bump github/codeql-action from 3.26.8 to 3.26.9 by @dependabot in #4332
- build(deps): bump google.golang.org/grpc from 1.67.0 to 1.67.1 by @dependabot in #4333
- fix: fix slice init length by @cuishuang in #4335
- Refactor Makefile: Add .PHONY declarations by @omerap12 in #4334
- build(deps): bump golang.org/x/time from 0.6.0 to 0.7.0 by @dependabot in #4336
- support dataflow affinity using any preceding data operations by @xliuqq in #4324
- bugfix: avoid setting "storage.data-dirs.capacity" when using MEM as tieredstore's medium type by @TrafalgarZZZ in #4339
- feat: enable client-side metrics scraping for JindoCache engine by @TrafalgarZZZ in #4331
- Build docker image for supporting dataflow affinity by @cheyang in #4340
- Make dataflow affinity's annotations consistent with others by @TrafalgarZZZ in #4342
- build(deps): bump github/codeql-action from 3.26.9 to 3.26.11 by @dependabot in #4338
- Revert PR #4339 by @TrafalgarZZZ in #4343
- bugfix: add startup probe to gha-e2e minio setup by @TrafalgarZZZ in #4344
- refactor: remove
tieredstore
argument in func base.BuildRuntimeInfo by @TrafalgarZZZ in #4341 - feat: support imagePullSecrets in thinRuntime & thinRuntimeProfile by @Syspretor in #4348
- Make metrics scrape target flexible with mountModeSelector by @TrafalgarZZZ in #4350
- Fix: fix some typo and log fmt error by @Syspretor in #4347
- feat: support imagePullSecrets in jindoRuntime by @Syspretor in #4349
- feat: support mountMode configuration to skip checking mount ready by @Syspretor in #4346
- Docker build for fixing the skipping check mount ready by @cheyang in #4351
- build(deps): bump golang.org/x/net from 0.29.0 to 0.30.0 by @dependabot in #4337
- docs updated by @vaibhav93076 in #4352
- build(deps): bump github/codeql-action from 3.26.11 to 3.26.12 by @dependabot in #4353
- refactor: sync schedule info to cache nodes by @TrafalgarZZZ in #4355
- dataflow ensure the dependent operation was completed by @xliuqq in #4345
- [WIP] fix: fix the dataload with cronJob failure in jindoCacheRuntime by @Syspretor in #4359
- Fix: fix the invalid fmt of dataload cronJob helm template by @Syspretor in #4360
- Build docker images for fixing cron dataload with annotations by @cheyang in #4362
- Prepare for v1.0.4 by @cheyang in #4365
- build(deps): bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5 by @dependabot in #4361
- Revert to v1.0.3 by @cheyang in #4372
- fix some issues of helm chart by @Syspretor in #4373
- bugfix: fix NodeUnpublishVolume when checking corrupted mount points by @TrafalgarZZZ in #4374
- Docker build for fixing NodeUnpublishVolume when checking corrupted by @cheyang in #4375
New Contributors
- @cuishuang made their first contribution in #4335
- @vaibhav93076 made their first contribution in #4352
Full Changelog: v1.0.2...v1.0.3
v1.0.2
New Features
- Support setting resources requests and limits for DataProcess
- Add dataset in used info to mutated app pods
- Support hierarchical imagePullSecrets configuration in AlluxioRuntime
- Support subpath in dynamic-mount
What's Changed
- Bump golang.org/x/net from 0.26.0 to 0.27.0 by @dependabot in #4200
- Prepare 1.0.2 by @cheyang in #4207
- Delete the unused rbac roles for vineyard runtime. by @dashanji in #4208
- Update docker image to fluid 1.0.2 by @cheyang in #4210
- Bump github.com/golang/glog from 1.2.1 to 1.2.2 by @dependabot in #4204
- bugfix: support sync ak/sk secret key-values in JindoRuntime by @TrafalgarZZZ in #4212
- del create pod role of juicefs runtime by @zwwhdls in #4215
- refactor: separate image and tag in helm values by @TrafalgarZZZ in #4216
- Docker build for juicefs runtime refactor by @cheyang in #4218
- Add no cache option to Makefile by @cheyang in #4217
- update scorecard link by @fengshunli in #4220
- feat: pass extra args when building fuse sidecar mutators by @TrafalgarZZZ in #4223
- enhance: add github actions workflow e2e tests by @TrafalgarZZZ in #4224
- minorfix: set read-all permission for e2e functionality check workflow by @TrafalgarZZZ in #4226
- bugfix: fix kind e2e test images by @TrafalgarZZZ in #4232
- fix Variable Naming Convention for DEFAULT_MASTER_RPC_PORT in CamelCase by @SouthWest7 in #4234
- Change Constant Variable Naming Convention for CSI_DRIVER in camelCase format by @jajlin in #4229
- bugfix: fix version replace in gha-e2e by @TrafalgarZZZ in #4235
- Update docker images for refactoring sidecar mutator by @cheyang in #4230
- juicefs: use exec in juicefs pod to avoid ignore signal by @zwwhdls in #4236
- update juicefs default image by @zwwhdls in #4237
- Bump ossf/scorecard-action from 2.3.3 to 2.4.0 by @dependabot in #4239
- Feat:support dynamic mount of subpath format JuiceFS mount-points by @Syspretor in #4241
- Upgrade controller runtime to v0.17.5 and k8s api to v0.29.5 by @xliuqq in #4160
- add dataflow affinity doc by @xliuqq in #4243
- Bump github.com/onsi/gomega from 1.33.1 to 1.34.1 by @dependabot in #4240
- Build docker images for upgrading K8s to 1.29 by @cheyang in #4244
- enhance: validate dataset mounts before reconciling thinruntimes by @TrafalgarZZZ in #4245
- feat: support hierarchical imagePullSecrets configuration in AlluxioRuntime by @EvanCley in #4242
- update adopters by @kiwilab-cn in #4251
- Bump github/codeql-action from 2.22.11 to 3.25.15 by @dependabot in #4250
- jindo: fix master and worker ports of jindo runtime by @lx1036 in #4247
- bugfix: fix options affecting sharedOptions in other mounts by @TrafalgarZZZ in #4253
- bugfix: Support data process operation type for JindoCache engine by @TrafalgarZZZ in #4254
- Build docker images for adding validation for runtime by @cheyang in #4246
- Bump golang.org/x/time from 0.5.0 to 0.6.0 by @dependabot in #4249
- bugfix: support setting resources for dataprocess by @TrafalgarZZZ in #4258
- Build docker images for fixing resources of data process by @cheyang in #4260
- build(deps): bump golang.org/x/net from 0.27.0 to 0.28.0 by @dependabot in #4259
- Add helm version option in Makefile by @omerap12 in #4262
- Fix circleci issue by @cheyang in #4264
- add customized label affinity demo for data flow by @xliuqq in #4261
- build(deps): bump github/codeql-action from 3.25.15 to 3.26.0 by @dependabot in #4266
- Make JindoRuntime sync Ufs total size with a longer timeout by @TrafalgarZZZ in #4267
- Build docker imagews for fixing sync ufs issue by @cheyang in #4268
- optimize mount logic && fix calculating ufs total size for JindoRuntime by @TrafalgarZZZ in #4274
- Build docker images for fix calculating ufs total size by @cheyang in #4278
- build(deps): bump github/codeql-action from 3.26.0 to 3.26.2 by @dependabot in #4279
- refactor: kubeclient exec utils package by @TrafalgarZZZ in #4270
- Refactor label keys in RuntimeInfo by @omerap12 in #4271
- Build docker images for refecator exec by @cheyang in #4280
- disable metadata auto-sync for ThinRuntime by default by @TrafalgarZZZ in #4281
- enhance: add timeout for kind-e2e github actions tests by @TrafalgarZZZ in #4283
- Bind all reconcile loops with limited context.Context by @omerap12 in #4265
- build(deps): bump github.com/prometheus/client_golang from 1.19.1 to 1.20.1 by @dependabot in #4284
- feat: add mounted dataset info to mutated app pods by @TrafalgarZZZ in #4285
- doc: update api doc to v1.0.2 by @TrafalgarZZZ in #4289
- Build docker images for adding info of dataset in use by @cheyang in #4290
- clean-up: remove unused functions by @omerap12 in #4292
- use annotation for exposed affinity for dataflow by @xliuqq in #4277
- remove redundant update for performance by @lx1036 in #4248
- cleanup: Remove unnecessary comment by @omerap12 in #4293
New Contributors
- @SouthWest7 made their first contribution in #4234
- @jajlin made their first contribution in #4229
- @Syspretor made their first contribution in #4241
- @EvanCley made their first contribution in #4242
- @kiwilab-cn made their first contribution in #4251
- @lx1036 made their first contribution in #4247
Full Changelog: v1.0.1...v1.0.2
v1.0.1
New Features
- Support data flow affinity scheduling.
- Support dynamic updates for dataset mount points.
- Support data acceleration for Fluid Persistent Volume Claim in sidecar mode
What's Changed
- Prepare 1.0.1 by @cheyang in #3886
- fix jindo rbac remove secret get by @frankleaf in #3974
- support parallel tasks for cron datamigrate by @xliuqq in #3732
- Support to configure the networkMode and podMetadata of vineyard components. by @dashanji in #3981
- Bump golangci/golangci-lint-action from 4 to 5 by @dependabot in #4007
- Upgrade controller runtime to 0.14.7 and Kubernetes deps to 1.26.10 by @cheyang in #4068
- Use generic Set for volumeLock by @cheyang in #4070
- fix code scaning alerts —— Incomplete regular expression for hostnames by @xliuqq in #4052
- Upgrade k8s to 1.26.15 by @cheyang in #4077
- Update docker file golang 1.12.10 by @cheyang in #4083
- Fix security issue introduced in vineyard samples by @dashanji in #4096
- feat: allow to disable metadata sync for thinruntime by @TrafalgarZZZ in #4091
- Bump golangci/golangci-lint-action from 5 to 6 by @dependabot in #4093
- refactor: deprecate fuse global mode by @TrafalgarZZZ in #4097
- support dataflow affinity by @xliuqq in #3829
- Fix crd generation issue by @cheyang in #4099
- Bump github.com/golang/glog from 1.0.0 to 1.2.1 by @dependabot in #3798
- Bump google.golang.org/grpc from 1.56.3 to 1.64.0 by @dependabot in #4100
- Support to configure the cache size of vineyard fuse by @dashanji in #3982
- upgrade jindocache to 6.4.0 version by @frankleaf in #4126
- Bump golang.org/x/time from 0.3.0 to 0.5.0 by @dependabot in #4111
- Fix the inconsistent rpc socket port with assigned port in vineyard worker. by @dashanji in #4141
- refactor: remove deprecated data operation interface by @xliuqq in #4130
- Fix NPE of runtime controller by @cheyang in #4139
- Bump github.com/onsi/gomega from 1.27.10 to 1.33.1 by @dependabot in #4142
- Bump github.com/prometheus/client_golang from 1.16.0 to 1.19.1 by @dependabot in #4147
- fix default value by @frankleaf in #4144
- Bump up the vineyard to v0.22.2 by @dashanji in #4153
- Bump github.com/agiledragon/gomonkey/v2 from 2.10.1 to 2.11.0 by @dependabot in #4151
- addons: add dynamic mount examples by @TrafalgarZZZ in #4148
- bugfix: check corrupted mount point in utils.RemoveSymlink by @TrafalgarZZZ in #4163
- Bump github.com/spf13/cobra from 1.7.0 to 1.8.0 by @dependabot in #4158
- Bump go.uber.org/zap from 1.24.0 to 1.27.0 by @dependabot in #3727
- refactor: rename
CreateUpdatePodForSchedulingHandler
toFluidMutatingHandler
by @TrafalgarZZZ in #4168 - minorfix: pin libfuse version in dynamic mount example by @TrafalgarZZZ in #4167
- Refactor admission webhook handler registration for better error handling by @cheyang in #4170
- refactor: avoid passing namespace to mutation functions by @TrafalgarZZZ in #4171
- minorfix: carefully handle pod.namespace to avoid side effect by @TrafalgarZZZ in #4172
- Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #4164
- feat: support multiround sidecar injection by @TrafalgarZZZ in #4175
- docs: Correcting the usage steps of the CONTABUTING.md document by @Wercurial in #4176
- move dataflow affinity logic to fluidapp controller by @xliuqq in #4138
- Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 by @dependabot in #4166
- minorfix: do not skip mutation for fluid pods by @TrafalgarZZZ in #4184
- Bump github.com/felixge/fgprof from 0.9.3 to 0.9.4 by @dependabot in #4169
- bugfix: fix application controller manage pod lifecycle by @TrafalgarZZZ in #4185
- Bump github.com/go-logr/logr from 1.4.1 to 1.4.2 by @dependabot in #4174
- dataset support pvc storageSize by @wangshulei098 in #4178
- docs: dev guide for en missing GOPATH setup by @backsapce in #4188
- Bump k8s.io/klog/v2 from 2.120.1 to 2.130.1 by @dependabot in #4183
- bugfix: allow users to override juicefs attr-cache and entry-cache options by @TrafalgarZZZ in #4194
- Bump google.golang.org/grpc from 1.64.0 to 1.65.0 by @dependabot in #4191
- Update go.mod to comply with Go 1.21 toolchain syntax by @cheyang in #4196
- bugfix: fix early break when cleaning mount point in NodeUnpublishVolume by @TrafalgarZZZ in #4198
- Bump github.com/agiledragon/gomonkey/v2 from 2.11.0 to 2.12.0 by @dependabot in #4199
- bugfix: fix jindoruntime name cannot contain jindofs by @TrafalgarZZZ in #4202
- revert: "upgrade jindocache to 6.4.0 version (#4126)" by @TrafalgarZZZ in #4205
- Build docker images for reverting jindoruntime from 6.4 to 6.2 by @cheyang in #4206
Full Changelog: v1.0.0...v1.0.1
v1.0.0
Highlights
- Introduced a flexible, configurable, tiered data locality scheduling capability to optimize the affinity between application pods and data cache.
- Provided DataProcess for custom data operations, and add three trigger strategies for data operations for all the data operations – once, onEvent, Cron.
- Provided a Python SDK with dataflow capabilities for data scientists.
- Implemented A new runtime for sharing in-memory immutable data: VineyardRuntime is supported in Fluid.
- Major improvements have been made in:
- Improved security measures by defining stricter, minimum necessary cluster role permissions for Fluid components.
- Significantly optimized FUSE Recovery features to enhance their performance and reliability.
- Made significant updates to the project documentation and website.
We acknowledge all the contributors who put their effort in the version release. Credits to @zwwhdls, @xliuqq, @wangshli, @Hexilee, @weixiao-huang, @abowloflrf, @uniqueni, @TrafalgarZZZ, @dashanji, @cheyang, @fengshunli, @frankleaf, @zhang-x-z, @BillyChen1, @wushiming540, @RongGu and many others for their great contributions to Fluid v1.0.0 !
New Features
Data Operation Features
- Add cron for data migrate.
- Support cron dataload.
- Make toleration configurable in helm charts.
- Data process crd definition.
- Support dataflow.
- Data operations support resources.
- Support ssh password free pod for parallel data migrate.
Vineyard Runtime
- Add the Vineyard runtime CRD definitions.
- Add the helm chart for Vineyard runtime.
- Add the controller and RBAC yaml for Vineyard runtime.
- Implement Vineyard runtime engine and controller.
- Add a replicas field to the Vineyard Runtime and delete the svc suffix in the Vineyard helm chart.
Enhancements & Optimizations
CSI Plugin & FUSE Recovery
- Check path existence and lock in NodeUnpublishVolume.
- umountDuplicate larger than the threshold.
- Fix csi plugin concurrency issue on FuseRecovery and NodeUnpublishVolume.
- Enhancement: recover fuse according to multiple peer group options.
- Support symlink for NodePublish.
Bug Fix
- Bugfix: Support batch/v1beta1 cronjobs for compatibility before Kubernetes v1.21.
- Bugfix: fix csi plugin loop mount bug
- Bugfix: reconcile thinruntime failed when dataset is deleted
- Bugfix:fix thinruntime stuck bug when deleting it
- Bugfix:fix worker&fuse options & worker tiredstore
- Bugfix: del metadata sync && del duplicate metrics
- Bugfix: clean up orphaned thinruntime resources
- Bugfix: fix Alluxio master in HA mode start error
- Bugfix: pass AccessModes to thinruntime fuse container
- Bugfix: fix fatal error: concurrent map writes for runtime controllers
- Bugfix: fix incorrect conversion between integer types
Website & Documentation Update
Full Changelog: v0.9.0...v1.0.0
v0.9.3
What's Changed
- Prepare 0.9.3 by @cheyang in #3638
- enhance: remove jindoruntime's fsGroup (#3632) by @frankleaf in #3641
- Remove fsgroup from alluxio runtime (#3635) by @TrafalgarZZZ in #3654
- cherry-pick to 0.9: del metadata sync && del duplicate metrics (#3380) by @TrafalgarZZZ @Hexilee in #3738
- fix rbac role issue by @BillyChen1 in #3753
- Build docker images for 0.9.3 by @cheyang in #3762
Full Changelog: v0.9.2...v0.9.3
v0.9.2
What's Changed
- Prepare 0.9.2 by @cheyang in #3374
- enhancement for 0.9: remove juicefs fuse readiness probe (#3435) by @zwwhdls in #3436
- Cherry-pick to 0.9: check path existence and lock in NodeUnpublishVolume (#3284) by @TrafalgarZZZ in #3450
- fail fast with wrong kubelet rootdir (#3331) by @wangshli in #3451
- umountDuplicate larger than the threshold (#3429) by @TrafalgarZZZ in #3452
- Cherry-pick to 0.9: fix csi plugin concurrency issue on FuseRecovery and NodeUnpublishVolume (#3448) by @TrafalgarZZZ in #3453
- Fix the problem of using option to append Dataset public option (#3454) by @lgy1027 in #3459
- Build docker images for release 0.9.2 by @cheyang in #3466
Full Changelog: v0.9.1...v0.9.2
v0.8.7
What's Changed
- Update to 0.8.7 by @cheyang in #3223
- Change reinvocationPolicy to IfNeeded by @lizzzcai in #3226
- Cherry-pick bugfix for setting fuse args when using Alluxio 2.8+ by @TrafalgarZZZ in #3231
- fix default master journal type setting (#3281) by @VincentLeeMax in #3291
- Mount kubelet.conf with HostToContainer mount propagation by @cheyang in #3318
- [BUGFIX] Fix unexpected dataset phase after helm install failure (#2751) by @TrafalgarZZZ in #3350
- Use mountinfo to check mountinfo by @zwwhdls in #3351
- Docker build 0.8.7 by @cheyang in #3352
Full Changelog: v0.8.6...v0.8.7
v0.9.1
v0.9.0
v0.9.0
Breaking Changes
- Change matching pod requests mode of webhook from namespaceSelector to objectSelector
Features
- Add thinRuntime to simplify integration with third-party storage systems
- Addon component for Fluid's open source CubeFS, NFS
- Support for accessing data across namespaces
- Support for subDataset
- Native acceleration system EFCRuntime for distributed file systems NFS, GPFS
- Support dataMigrate for data migration operations (currently only supported by JuiceFSRuntime)
- Add customizable configuration for cache cleanup timeout and maximum retry times, Webhook timeout limits
- Add Dataload configuration for ImagePullSecrets, node affinity
- RBAC permission reduction
- Upgrade to golang 1.18
- Support for installing Fluid via Helm Repo
Refactoring
- Use data operation framework to construct data migrate, load, backup behaviors
Bug Fix
- JindoRuntime should support configurable env variables
- Could not set nodeAffinity in dataset
- Runtime helm release stuck in "pending-install" status
- CSI failed to recover FUSE mount point for AlluxioRuntime
- [JuiceFS] FUSE pod scheduled failed because of conflict port
- Fluid csi on rke2 k8s(1.22) use mount output empty,it caused app pod not work
Runtime Upgrade
- AlluxioRuntime is upgrade from v2.8.2 to v2.9.1
- JindoRuntime is upgraded from from 4.5.1 to 4.6.7
- JuicefsRuntime is upgraded from v1.0.0 to v1.0.4
Credits to @yangyuliufeng ,@zwwhdls, @hahchenchen,@frankleaf, @xliuqq, @abowloflrf, @odidev , @allenhaozi, @uniqueni, @yangjun289519474, @wang-mask , @fengshunli , @zhang-x-z , @wangshli @ssz1997 , @myccccccc , @baowj-678 , @BillyChen1 , @Yanghaihai1020, @lizzzcai , @weixiao-huang ,@TrafalgarZZZ and many others for their great contributions to Fluid v0.9.0!
v0.8.6
What's Changed
- Add approvers for release 0.8 by @cheyang in #2864
- Update fluid version to 0.8.6 by @cheyang in #2843
- add reinvocationPolicy by @cheyang in #3047
- Cherry-pick fix worker volumes in alluxio chart (#2330) by @wangshli in #3191
- Fix rbacs and limit CSI Plugin's node related access by @TrafalgarZZZ
Full Changelog: v0.8.5...v0.8.6