Skip to content

v0.18.0
Compare
Choose a tag to compare
@github-actions github-actions released this 06 Aug 16:53
· 1012 commits to develop since this release
v0.18.0
78069b4

Features

Bug Fixes

Documentation

Code Refactoring

Tests

Continuous Integration

Chores

Commits

  • da500b3: feat(core & webserver): bring lightweight version of namespaces (brian.mulier) #4219
  • 735ed6e: fix(core, jdbc): purge deleted executions (Loïc Mathieu) #4282
  • f39a6ac: feat(core, jdbc): add a PurgeLogs task (Loïc Mathieu) #4298
  • 2416760: feat(core,jdbc,webserver): find trigger by worker id (Loïc Mathieu) #4336
  • 0125d50: feat(core, jdbc, webserver): allow to filter logs by triggerId (Loïc Mathieu) #4337
  • a5dc8e4: feat(core, script): use a volume instead of a bind for handling files in the Docker task runner (Loïc Mathieu) #4349
  • f78a765: Revert "feat(cli): provide descriptive server start error (#4171)" (YannC)
  • 24e0063: closes #4371 (Anna Geller)
  • 5bab548: fix(ui):hide switch if trigger is disabled from source code (#4422) (YannC) #4422
  • 89af5cb: Fix/waitfor issue (#4421) (YannC) #4421
  • 68a7527: related to #4396 (Anna Geller)
  • 54aa618: feat(core,jdbc,webserver): delete flow logs (Loïc Mathieu) #4438
  • ui: regular dependency updates (#4469) #4469 (Miloš Paunović)
  • 3462815: feat(core, webserver): delete execution related logs, metrics and storage (Loïc Mathieu) #4427
  • uploadfiles fix (#4484) #4484 (YannC)
  • 69a1e8a: Revert "chore(deps): bump kafkaVersion from 3.7.1 to 3.8.0" (#4515) (YannC) #4515
  • c73d832: fix #4371 (Anna Geller)
  • 3aead5b: fix(core, script): Docker and Proces should always be a new instance (Loïc Mathieu)

Breaking Changes

Docker image tags

Most of our users leverage a Kestra version with all plugins included. We've changed our Docker image tags to ensure that the default Docker image includes all plugins. To use a lightweight version of Kestra without plugins, you will now have to explicitly use the suffix *-no-plugins.

Before upgrading to Kestra 0.18.0, make sure to adjust the image in your deployment as follows:

  • kestra/kestra:latest to always use the latest stable version incl. all plugins
  • kestra/kestra:latest-no-plugins to always use the latest stable version of Kestra without plugins (e.g. only your custom plugins).

If your deployment strategy is to pin the version, make sure to change the image to:

  • kestra/kestra:v0.18.0 if you want to have all plugins included in the image
  • kestra/kestra:v0.18.0-no-plugins if you prefer to use only your custom plugins.

If you are using the develop images that include the most recently developed (but not yet released) features and bug fixes, make sure to change the image to:

  • kestra/kestra:develop if you want to have all plugins included in the image
  • kestra/kestra:develop-no-plugins if you prefer to use only your custom plugins.

TL;DR of what you need to do on your end:

  • If you use the develop-full or latest-full image with all plugins, cut the -full suffix from your Docker image tag, and you're good to go!
  • If you use the develop or latest image without plugins, add -no-plugins suffix to the image tag.

We believe that this change will simplify your deployments. If you have any questions or need help with this change, please reach out via Slack.

The kestra-ee binary has been renamed to kestra

If you are using the kestra-ee CLI, note that it has been renamed to kestra. This change is intended to avoid confusion between the open-source and Enterprise Edition binaries.

The runner property is deprecated in favor of the taskRunner

Starting with Kestra 0.18.0, the taskRunner property has precedence over the runner property. In Kestra 0.18.0, task runners are out of Beta, and runner is now deprecated. You can still use the deprecated runner property when explicitly specified, but remember that Docker-type taskRunner is now the default way of running script tasks in Kestra. We recommend switching any existing pluginDefaults using runner to taskRuner.

In the 3-month Beta period, we gave everyone in the community the opportunity to try all task runner plugins. With the General Availability, some plugins are exclusively available on the Cloud and Enterprise editions of Kestra. As a result, their plugin paths have changed as follows:

  • io.kestra.plugin.ee.kubernetes.runner.Kubernetes
  • io.kestra.plugin.ee.azure.runner.Batch
  • io.kestra.plugin.ee.aws.runner.Batch
  • io.kestra.plugin.ee.gcp.runner.Batch
  • io.kestra.plugin.ee.gcp.runner.CloudRun

If you want to try them out, please reach out.

Changes in JSON function and filter (non-breaking)

We've made several improvements to JSON and ION handling. To avoid confusion between the json filter and json() function, we've renamed:

  1. the json filter to toJson — it converts an object or data structure into a JSON string e.g. {{ [1, 2, 3] | toJson }}
  2. the json() function to fromJson — it converts a JSON string into a data structure or object so that you can access further attributes using a dot notation e.g. {{ fromJson(kv('YOUR_JSON_KEY')).some_attribute }}

This renaming has been done with an alias to the old name:

  • The json filter has been renamed to toJson — using the old filter will emit a warning in the server logs
  • The json() function has been renamed to fromJson — using the old function will emit a warning in the server logs

New mechanism of the Purge tasks (non-breaking)

We've changed the mechanism of the Purge tasks to make it more performant and reliable:

  • io.kestra.plugin.core.storage.Purge has been renamed to io.kestra.plugin.core.execution.PurgeExecutions to reflect that it only purges data related to executions (e.g. not including trigger logs, to purge those you should use the PurgeLogs task) — we've added an alias so that using the old task type will still work but it will emit a warning. We recommend using the new task type.
  • io.kestra.plugin.core.storage.PurgeExecution has been renamed to io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles to reflect that it can purge all execution-related data incl. inputs to an Execution and Execution outputs — we've added an alias so that using the old task type will still work but it will emit a warning. Also here, we recommend adjusting your flow code to match the new task type.

Starting with the 0.18.0 release, the recommended way to clean internal storage files, executions and logs is using a combination of io.kestra.plugin.core.execution.PurgeExecutions but without logs and the newly added io.kestra.plugin.core.log.PurgeLogs task that removes all logs (both Execution logs and Trigger logs) in a performant batch operation. Combining those two together will give you the same functionality as the old io.kestra.plugin.core.storage.Purge task but in a more performant and reliable way (roughly 10x faster as compared to the old task).

id: purge
namespace: company.team
description: |
  This flow will remove all executions and logs older than one month.
  We recommend running this flow daily to avoid running out of disk space.

tasks:
  - id: purge_executions
    type: io.kestra.plugin.core.execution.PurgeExecutions
    endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
    purgeLog: false
    
  - id: purge_logs
    type: io.kestra.plugin.core.log.PurgeLogs
    endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "@daily"

KV Store permission must be added manually for now (non-breaking)

Starting with the 0.18.0 release, we've introduced a new KVSTORE permission that allows users to access the new KV Store functionality. This permission needs to be added manually to all Roles that need access to the KV Store.

If you have any Users and Groups that want to leverage the new KV Store functionality, make sure to update their RBAC to include the new KVSTORE permission.

The latest kestra terraform provider now uses plugin_defaults instead of task_defaults

The 0.18.0 version of the Terraform Provider now uses the property plugin_defaults instead of task_defaults in the kestra_namespace resource. Simply replace task_defaults with plugin_defaults in your Terraform configuration before you upgrade your Kestra Terraform provider.

Assets 3
Loading