Distribute defaults across multiple files (#438)

Closes #55
bakdata · Feb 27, 2024 · b5fdbc8 · b5fdbc8
1 parent 31a1142
commit b5fdbc8
Show file tree

Hide file tree

Showing 63 changed files with 502 additions and 296 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,7 +3,5 @@
 .DS_Store
 .envrc
 .vscode
-pipelines/
-defaults/
 site/
 scratch*
diff --git a/action.yaml b/action.yaml
@@ -11,9 +11,6 @@ inputs:
   working-directory:
     description: "The root directory containing the config.yaml, pipelines folder and defaults"
     default: "."
-  defaults:
-    description: "Defaults folder path"
-    required: false
   config:
     description: "Directory containing the config*.yaml file(s)"
     required: false
@@ -86,4 +83,4 @@ runs:
     - name: ${{ inputs.command }} ${{ inputs.pipeline }} pipeline
       shell: bash
       working-directory: ${{inputs.working-directory}}
-      run: kpops ${{ inputs.command }} ${{ inputs.pipeline }} ${{ (inputs.defaults != '' && format('--defaults {0}', inputs.defaults)) || '' }} ${{ (inputs.config != '' && format('--config {0}', inputs.config)) || '' }} ${{ (inputs.environment != '' && format('--environment {0}', inputs.environment)) || '' }} ${{ (inputs.filter-type != '' && format('--filter-type {0}', inputs.filter-type)) || '' }} ${{ (inputs.parallel == 'true' && '--parallel') || '' }}
+      run: kpops ${{ inputs.command }} ${{ inputs.pipeline }} ${{ (inputs.config != '' && format('--config {0}', inputs.config)) || '' }} ${{ (inputs.environment != '' && format('--environment {0}', inputs.environment)) || '' }} ${{ (inputs.filter-type != '' && format('--filter-type {0}', inputs.filter-type)) || '' }} ${{ (inputs.parallel == 'true' && '--parallel') || '' }}
diff --git a/docs/docs/resources/pipeline-config/config.yaml b/docs/docs/resources/pipeline-config/config.yaml
@@ -1,8 +1,5 @@
 # CONFIGURATION
 #
-# The path to the folder containing the defaults.yaml file and the environment
-# defaults files.
-defaults_path: .
 # Custom Python module defining project-specific KPOps components
 components_module: null
 # Base directory to the pipelines (default is current working directory)

diff --git a/docs/docs/resources/variables/cli_env_vars.env b/docs/docs/resources/variables/cli_env_vars.env
@@ -7,8 +7,6 @@
 #
 # Path to the dir containing config.yaml files
 KPOPS_CONFIG_PATH=.
-# Path to defaults folder
-KPOPS_DEFAULT_PATH # No default value, not required
 # Path to dotenv file. Multiple files can be provided. The files will
 # be loaded in order, with each file overriding the previous one.
 KPOPS_DOTENV_PATH # No default value, not required

diff --git a/docs/docs/resources/variables/cli_env_vars.md b/docs/docs/resources/variables/cli_env_vars.md
@@ -3,7 +3,6 @@ These variables are a lower priority alternative to the commands' flags. If a va
 |        Name        |Default Value|Required|                                                                                Description                                                                                 |
 |--------------------|-------------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 |KPOPS_CONFIG_PATH   |.            |False   |Path to the dir containing config.yaml files                                                                                                                                |
-|KPOPS_DEFAULT_PATH  |             |False   |Path to defaults folder                                                                                                                                                     |
 |KPOPS_DOTENV_PATH   |             |False   |Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one.                                         |
 |KPOPS_ENVIRONMENT   |             |False   |The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development).|
 |KPOPS_PIPELINE_PATH |             |True    |Path to YAML with pipeline definition                                                                                                                                       |

diff --git a/docs/docs/resources/variables/config_env_vars.env b/docs/docs/resources/variables/config_env_vars.env
@@ -4,11 +4,6 @@
 # alternative to the settings in `config.yaml`. Variables marked as
 # required can instead be set in the global config.
 #
-# defaults_path
-# The path to the folder containing the defaults.yaml file and the
-# environment defaults files. Paths can either be absolute or relative
-# to `config.yaml`
-KPOPS_DEFAULTS_PATH=.
 # components_module
 # Custom Python module defining project-specific KPOps components
 KPOPS_COMPONENTS_MODULE # No default value, not required

diff --git a/docs/docs/resources/variables/config_env_vars.md b/docs/docs/resources/variables/config_env_vars.md
@@ -1,22 +1,21 @@
 These variables are a lower priority alternative to the settings in `config.yaml`. Variables marked as required can instead be set in the global config.
 
-|                       Name                       |             Default Value              |Required|                                                                     Description                                                                      |               Setting name                |
-|--------------------------------------------------|----------------------------------------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------|
-|KPOPS_DEFAULTS_PATH                               |.                                       |False   |The path to the folder containing the defaults.yaml file and the environment defaults files. Paths can either be absolute or relative to `config.yaml`|defaults_path                              |
-|KPOPS_COMPONENTS_MODULE                           |                                        |False   |Custom Python module defining project-specific KPOps components                                                                                       |components_module                          |
-|KPOPS_PIPELINE_BASE_DIR                           |.                                       |False   |Base directory to the pipelines (default is current working directory)                                                                                |pipeline_base_dir                          |
-|KPOPS_KAFKA_BROKERS                               |                                        |True    |The comma separated Kafka brokers address.                                                                                                            |kafka_brokers                              |
-|KPOPS_DEFAULTS_FILENAME_PREFIX                    |defaults                                |False   |The name of the defaults file and the prefix of the defaults environment file.                                                                        |defaults_filename_prefix                   |
-|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME|${pipeline.name}-${component.name}      |False   |Configures the value for the variable ${output_topic_name}                                                                                            |topic_name_config.default_output_topic_name|
-|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME |${pipeline.name}-${component.name}-error|False   |Configures the value for the variable ${error_topic_name}                                                                                             |topic_name_config.default_error_topic_name |
-|KPOPS_SCHEMA_REGISTRY__ENABLED                    |False                                   |False   |Whether the Schema Registry handler should be initialized.                                                                                            |schema_registry.enabled                    |
-|KPOPS_SCHEMA_REGISTRY__URL                        |http://localhost:8081/                  |False   |Address of the Schema Registry.                                                                                                                       |schema_registry.url                        |
-|KPOPS_KAFKA_REST__URL                             |http://localhost:8082/                  |False   |Address of the Kafka REST Proxy.                                                                                                                      |kafka_rest.url                             |
-|KPOPS_KAFKA_CONNECT__URL                          |http://localhost:8083/                  |False   |Address of Kafka Connect.                                                                                                                             |kafka_connect.url                          |
-|KPOPS_TIMEOUT                                     |300                                     |False   |The timeout in seconds that specifies when actions like deletion or deploy timeout.                                                                   |timeout                                    |
-|KPOPS_CREATE_NAMESPACE                            |False                                   |False   |Flag for `helm upgrade --install`. Create the release namespace if not present.                                                                       |create_namespace                           |
-|KPOPS_HELM_CONFIG__CONTEXT                        |                                        |False   |Name of kubeconfig context (`--kube-context`)                                                                                                         |helm_config.context                        |
-|KPOPS_HELM_CONFIG__DEBUG                          |False                                   |False   |Run Helm in Debug mode                                                                                                                                |helm_config.debug                          |
-|KPOPS_HELM_CONFIG__API_VERSION                    |                                        |False   |Kubernetes API version used for `Capabilities.APIVersions`                                                                                            |helm_config.api_version                    |
-|KPOPS_HELM_DIFF_CONFIG__IGNORE                    |                                        |True    |Set of keys that should not be checked.                                                                                                               |helm_diff_config.ignore                    |
-|KPOPS_RETAIN_CLEAN_JOBS                           |False                                   |False   |Whether to retain clean up jobs in the cluster or uninstall the, after completion.                                                                    |retain_clean_jobs                          |
+|                       Name                       |             Default Value              |Required|                                    Description                                    |               Setting name                |
+|--------------------------------------------------|----------------------------------------|--------|-----------------------------------------------------------------------------------|-------------------------------------------|
+|KPOPS_COMPONENTS_MODULE                           |                                        |False   |Custom Python module defining project-specific KPOps components                    |components_module                          |
+|KPOPS_PIPELINE_BASE_DIR                           |.                                       |False   |Base directory to the pipelines (default is current working directory)             |pipeline_base_dir                          |
+|KPOPS_KAFKA_BROKERS                               |                                        |True    |The comma separated Kafka brokers address.                                         |kafka_brokers                              |
+|KPOPS_DEFAULTS_FILENAME_PREFIX                    |defaults                                |False   |The name of the defaults file and the prefix of the defaults environment file.     |defaults_filename_prefix                   |
+|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME|${pipeline.name}-${component.name}      |False   |Configures the value for the variable ${output_topic_name}                         |topic_name_config.default_output_topic_name|
+|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME |${pipeline.name}-${component.name}-error|False   |Configures the value for the variable ${error_topic_name}                          |topic_name_config.default_error_topic_name |
+|KPOPS_SCHEMA_REGISTRY__ENABLED                    |False                                   |False   |Whether the Schema Registry handler should be initialized.                         |schema_registry.enabled                    |
+|KPOPS_SCHEMA_REGISTRY__URL                        |http://localhost:8081/                  |False   |Address of the Schema Registry.                                                    |schema_registry.url                        |
+|KPOPS_KAFKA_REST__URL                             |http://localhost:8082/                  |False   |Address of the Kafka REST Proxy.                                                   |kafka_rest.url                             |
+|KPOPS_KAFKA_CONNECT__URL                          |http://localhost:8083/                  |False   |Address of Kafka Connect.                                                          |kafka_connect.url                          |
+|KPOPS_TIMEOUT                                     |300                                     |False   |The timeout in seconds that specifies when actions like deletion or deploy timeout.|timeout                                    |
+|KPOPS_CREATE_NAMESPACE                            |False                                   |False   |Flag for `helm upgrade --install`. Create the release namespace if not present.    |create_namespace                           |
+|KPOPS_HELM_CONFIG__CONTEXT                        |                                        |False   |Name of kubeconfig context (`--kube-context`)                                      |helm_config.context                        |
+|KPOPS_HELM_CONFIG__DEBUG                          |False                                   |False   |Run Helm in Debug mode                                                             |helm_config.debug                          |
+|KPOPS_HELM_CONFIG__API_VERSION                    |                                        |False   |Kubernetes API version used for `Capabilities.APIVersions`                         |helm_config.api_version                    |
+|KPOPS_HELM_DIFF_CONFIG__IGNORE                    |                                        |True    |Set of keys that should not be checked.                                            |helm_diff_config.ignore                    |
+|KPOPS_RETAIN_CLEAN_JOBS                           |False                                   |False   |Whether to retain clean up jobs in the cluster or uninstall the, after completion. |retain_clean_jobs                          |
diff --git a/docs/docs/schema/config.json b/docs/docs/schema/config.json
@@ -163,17 +163,6 @@
             "title": "Defaults Filename Prefix",
             "type": "string"
         },
-        "defaults_path": {
-            "default": ".",
-            "description": "The path to the folder containing the defaults.yaml file and the environment defaults files. Paths can either be absolute or relative to `config.yaml`",
-            "examples": [
-                "defaults",
-                "."
-            ],
-            "format": "path",
-            "title": "Defaults Path",
-            "type": "string"
-        },
         "helm_config": {
             "allOf": [
                 {

diff --git a/docs/docs/user/core-concepts/defaults.md b/docs/docs/user/core-concepts/defaults.md
@@ -14,15 +14,40 @@ It is possible, although not recommended, to add settings that are specific to a
 
 ### Configuration
 
-It is possible to set specific `defaults` for each `environment` by adding files called `defaults_{environment}.yaml` to the defaults folder at `defaults_path`. The defaults are loaded based on the currently set environment.
+KPOps allows using multiple default values. The `defaults.yaml` (or `defaults_<env>.yaml`) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single `pipeline.yaml` file.
+KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the `pipeline_base_dir` is defined.
+
+The deepest `defaults.yaml` file in the folder hierarchy (i.e., the closest one to the `pipeline.yaml`) overwrites the higher-level defaults' values.
 
 It is important to note that `defaults_{environment}.yaml` overrides only the settings that are explicitly set to be different from the ones in the base `defaults` file.
 
 <!-- dprint-ignore-start -->
 
+??? Example "defaults merge priority"
+    Imagine the following folder structure, where the `pipeline_base_dir` is configured to `pipelines`:
+
+    ```
+    └─ pipelines
+       └── distributed-defaults
+           ├── defaults.yaml
+           ├── defaults_dev.yaml
+           └── pipeline-deep
+               ├── defaults.yaml
+               └── pipeline.yaml
+    ```
+
+    KPOps picks up the defaults in the following order (high to low priority):
+
+    - `./pipelines/distributed-defaults/pipeline-deep/defaults.yaml`
+    - `./pipelines/distributed-defaults/defaults_dev.yaml`
+    - `./pipelines/distributed-defaults/defaults.yaml`
+
+<!-- dprint-ignore-end -->
+
+<!-- dprint-ignore-start -->
+
 !!! tip
     `defaults` is the default value of `defaults_filename_prefix`.
-    Together with `defaults_path` and `environment` it can be changed in [`config.yaml`](../config/#__codelineno-0-16)
 
 <!-- dprint-ignore-end -->
 

diff --git a/docs/docs/user/migration-guide/v3-v4.md b/docs/docs/user/migration-guide/v3-v4.md
@@ -0,0 +1,41 @@
+# Migrate from V3 to V4
+
+## [Distribute defaults across multiple files](https://github.com/bakdata/kpops/pull/438/)
+
+<!-- dprint-ignore-start -->
+
+!!! warning inline end
+    **The `--defaults` flag is removed**
+
+<!-- dprint-ignore-end -->
+
+It is possible now to use multiple default values. The `defaults.yaml` (or `defaults_<env>.yaml`) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single `pipeline.yaml` file.
+KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the `pipeline_base_dir` is defined.
+
+For example, imagine the following folder structure:
+
+```
+└─ pipelines
+   └── distributed-defaults
+       ├── defaults.yaml
+       ├── defaults_dev.yaml
+       └── pipeline-deep
+           ├── defaults.yaml
+           └── pipeline.yaml
+```
+
+The `pipeline_base_dir` is configured to `pipelines`. Now if we generate this pipeline with the following command:
+
+```bash
+kpops generate \
+      --environment dev
+      ./pipelines/distributed-defaults/pipeline-deep/pipeline.yaml
+```
+
+The defaults would be picked in the following order (high to low priority):
+
+- `./pipelines/distributed-defaults/pipeline-deep/defaults.yaml`
+- `./pipelines/distributed-defaults/defaults_dev.yaml`
+- `./pipelines/distributed-defaults/defaults.yaml`
+
+The deepest `defaults.yaml` file in the folder hierarchy (i.e., the closest one to the `pipeline.yaml`) overwrites the higher-level defaults' values.