Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manifest toSection with Strimzi KafkaTopic #545

Merged
merged 55 commits into from
Dec 10, 2024
Merged

Manifest toSection with Strimzi KafkaTopic #545

Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
c218e02
Introduce KPOps operation and manifest resources for deployment
raminqaf Nov 28, 2024
4a29cbd
Introduce KPOps operation and manifest resources for deployment
raminqaf Nov 28, 2024
7ccd04f
Propose refactoring for #541
disrupted Nov 28, 2024
834cd14
Replace `manifest` method with `manifest_deploy`
disrupted Nov 28, 2024
6b3a57a
Merge branch 'main' of github.com:bakdata/kpops into feat/add-operations
raminqaf Dec 2, 2024
be817c3
Merge branch 'feat/kubernetes-manifests' of github.com:bakdata/kpops …
raminqaf Dec 2, 2024
3ce5c95
Introduce KPOps operation and manifest resources for deployment
raminqaf Dec 2, 2024
56f72a1
Introduce KPOps operation and manifest resources for deployment
raminqaf Dec 2, 2024
48de4e2
Merge branch 'main' of github.com:bakdata/kpops into feat/add-operations
raminqaf Dec 2, 2024
af9b23b
Update files
raminqaf Dec 2, 2024
4d1cc1e
Update files
raminqaf Dec 2, 2024
c717002
Update files
raminqaf Dec 2, 2024
975b10e
fix init command
raminqaf Dec 2, 2024
7b47919
clean ups
raminqaf Dec 2, 2024
9fcd55f
clean ups
raminqaf Dec 2, 2024
aaed48d
clean ups
raminqaf Dec 2, 2024
ece6e9f
clean ups
raminqaf Dec 2, 2024
3cc771e
Update files
raminqaf Dec 2, 2024
a7e3c00
Update files
raminqaf Dec 2, 2024
0ec82e2
Update files
raminqaf Dec 2, 2024
af25f92
address reviews
raminqaf Dec 3, 2024
9e1938e
add tests
raminqaf Dec 3, 2024
9a4dc35
Update files
raminqaf Dec 4, 2024
64cc384
refactor argo sync wave
raminqaf Dec 4, 2024
15fe717
Manifest toSection with Strimzi Kafka topic
raminqaf Dec 4, 2024
0c62a53
Address reviews
raminqaf Dec 4, 2024
ab5b74e
Address reviews
raminqaf Dec 4, 2024
8838bfa
Define pydantic model to representig Kubernetes manifest
raminqaf Dec 4, 2024
f1ed671
Define pydantic model to representig Kubernetes manifest
raminqaf Dec 4, 2024
a24de23
Update files
raminqaf Dec 4, 2024
32ea4dc
Update files
raminqaf Dec 4, 2024
2a4838a
Merge remote-tracking branch 'origin/v9' into feat/add-strimzi-topic
raminqaf Dec 4, 2024
79e1af4
Update files
raminqaf Dec 4, 2024
b2b2983
Update files
raminqaf Dec 4, 2024
d894e57
Update files
raminqaf Dec 4, 2024
e67908e
Update files
raminqaf Dec 4, 2024
0a2e7c8
Update files
raminqaf Dec 4, 2024
4ebd633
Update files
raminqaf Dec 5, 2024
ea3237b
Update files
raminqaf Dec 5, 2024
1070fde
clean ups
raminqaf Dec 5, 2024
50e81df
Update files
raminqaf Dec 5, 2024
4b187bb
clean ups
raminqaf Dec 6, 2024
c25a883
add cluster name label
raminqaf Dec 6, 2024
8ea444c
Update files
raminqaf Dec 6, 2024
16d88c7
Update files
raminqaf Dec 9, 2024
3f71255
Update files
raminqaf Dec 9, 2024
2f1b510
Update files
raminqaf Dec 9, 2024
baae8e3
fix tests
raminqaf Dec 9, 2024
ac780e4
Update files
raminqaf Dec 9, 2024
253157f
Update files
raminqaf Dec 9, 2024
b32b6a4
Update files
raminqaf Dec 9, 2024
62a6b7c
Merge branch 'v9' of github.com:bakdata/kpops into feat/add-strimzi-t…
raminqaf Dec 10, 2024
794d1a6
change strimzi label
raminqaf Dec 10, 2024
c41af8d
add property
raminqaf Dec 10, 2024
7f1adb7
add tests
raminqaf Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"
pipeline_base_dir: tests/pipeline
kafka_topic_resource_label: "my-cluster"
7 changes: 7 additions & 0 deletions docs/docs/resources/variables/config_env_vars.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,10 @@ KPOPS_RETAIN_CLEAN_JOBS=False
# operation_mode
# The operation mode of KPOps (managed, manifest, argo).
KPOPS_OPERATION_MODE=managed
# kafka_topic_resource_label
# The label to identify the KafkaTopic resources managed by the Topic
# Operator. This does not have to be the name of the Kafka cluster. It
# can be the label assigned to the KafkaTopic resource. If you deploy
# more than one Topic Operator, the labels must be unique for each.
# That is, the operators cannot manage the same resources.
KPOPS_KAFKA_TOPIC_RESOURCE_LABEL # No default value, not required
41 changes: 21 additions & 20 deletions docs/docs/resources/variables/config_env_vars.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,23 @@
These variables take precedence over the settings in `config.yaml`. Variables marked as required can instead be set in the global config.

| Name | Default Value |Required| Description | Setting name |
|--------------------------------------------------|----------------------------------------|--------|----------------------------------------------------------------------------------|-------------------------------------------|
|KPOPS_PIPELINE_BASE_DIR |. |False |Base directory to the pipelines (default is current working directory) |pipeline_base_dir |
|KPOPS_KAFKA_BROKERS | |True |The comma separated Kafka brokers address. |kafka_brokers |
|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME|${pipeline.name}-${component.name} |False |Configures the value for the variable ${output_topic_name} |topic_name_config.default_output_topic_name|
|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME |${pipeline.name}-${component.name}-error|False |Configures the value for the variable ${error_topic_name} |topic_name_config.default_error_topic_name |
|KPOPS_SCHEMA_REGISTRY__ENABLED |False |False |Whether the Schema Registry handler should be initialized. |schema_registry.enabled |
|KPOPS_SCHEMA_REGISTRY__URL |http://localhost:8081/ |False |Address of the Schema Registry. |schema_registry.url |
|KPOPS_SCHEMA_REGISTRY__TIMEOUT |30 |False |Operation timeout in seconds. |schema_registry.timeout |
|KPOPS_KAFKA_REST__URL |http://localhost:8082/ |False |Address of the Kafka REST Proxy. |kafka_rest.url |
|KPOPS_KAFKA_REST__TIMEOUT |30 |False |Operation timeout in seconds. |kafka_rest.timeout |
|KPOPS_KAFKA_CONNECT__URL |http://localhost:8083/ |False |Address of Kafka Connect. |kafka_connect.url |
|KPOPS_KAFKA_CONNECT__TIMEOUT |30 |False |Operation timeout in seconds. |kafka_connect.timeout |
|KPOPS_CREATE_NAMESPACE |False |False |Flag for `helm upgrade --install`. Create the release namespace if not present. |create_namespace |
|KPOPS_HELM_CONFIG__CONTEXT | |False |Name of kubeconfig context (`--kube-context`) |helm_config.context |
|KPOPS_HELM_CONFIG__DEBUG |False |False |Run Helm in Debug mode |helm_config.debug |
|KPOPS_HELM_CONFIG__API_VERSION | |False |Kubernetes API version used for `Capabilities.APIVersions` |helm_config.api_version |
|KPOPS_HELM_DIFF_CONFIG__IGNORE | |True |Set of keys that should not be checked. |helm_diff_config.ignore |
|KPOPS_RETAIN_CLEAN_JOBS |False |False |Whether to retain clean up jobs in the cluster or uninstall the, after completion.|retain_clean_jobs |
|KPOPS_OPERATION_MODE |managed |False |The operation mode of KPOps (managed, manifest, argo). |operation_mode |
| Name | Default Value |Required| Description | Setting name |
|--------------------------------------------------|----------------------------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------|
|KPOPS_PIPELINE_BASE_DIR |. |False |Base directory to the pipelines (default is current working directory) |pipeline_base_dir |
|KPOPS_KAFKA_BROKERS | |True |The comma separated Kafka brokers address. |kafka_brokers |
|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME|${pipeline.name}-${component.name} |False |Configures the value for the variable ${output_topic_name} |topic_name_config.default_output_topic_name|
|KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME |${pipeline.name}-${component.name}-error|False |Configures the value for the variable ${error_topic_name} |topic_name_config.default_error_topic_name |
|KPOPS_SCHEMA_REGISTRY__ENABLED |False |False |Whether the Schema Registry handler should be initialized. |schema_registry.enabled |
|KPOPS_SCHEMA_REGISTRY__URL |http://localhost:8081/ |False |Address of the Schema Registry. |schema_registry.url |
|KPOPS_SCHEMA_REGISTRY__TIMEOUT |30 |False |Operation timeout in seconds. |schema_registry.timeout |
|KPOPS_KAFKA_REST__URL |http://localhost:8082/ |False |Address of the Kafka REST Proxy. |kafka_rest.url |
|KPOPS_KAFKA_REST__TIMEOUT |30 |False |Operation timeout in seconds. |kafka_rest.timeout |
|KPOPS_KAFKA_CONNECT__URL |http://localhost:8083/ |False |Address of Kafka Connect. |kafka_connect.url |
|KPOPS_KAFKA_CONNECT__TIMEOUT |30 |False |Operation timeout in seconds. |kafka_connect.timeout |
|KPOPS_CREATE_NAMESPACE |False |False |Flag for `helm upgrade --install`. Create the release namespace if not present. |create_namespace |
|KPOPS_HELM_CONFIG__CONTEXT | |False |Name of kubeconfig context (`--kube-context`) |helm_config.context |
|KPOPS_HELM_CONFIG__DEBUG |False |False |Run Helm in Debug mode |helm_config.debug |
|KPOPS_HELM_CONFIG__API_VERSION | |False |Kubernetes API version used for `Capabilities.APIVersions` |helm_config.api_version |
|KPOPS_HELM_DIFF_CONFIG__IGNORE | |True |Set of keys that should not be checked. |helm_diff_config.ignore |
|KPOPS_RETAIN_CLEAN_JOBS |False |False |Whether to retain clean up jobs in the cluster or uninstall the, after completion. |retain_clean_jobs |
|KPOPS_OPERATION_MODE |managed |False |The operation mode of KPOps (managed, manifest, argo). |operation_mode |
|KPOPS_KAFKA_TOPIC_RESOURCE_LABEL | |False |The label to identify the KafkaTopic resources managed by the Topic Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaTopic resource. If you deploy more than one Topic Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.|kafka_topic_resource_label |
13 changes: 13 additions & 0 deletions docs/docs/schema/config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,19 @@
},
"description": "Configuration for Kafka REST Proxy."
},
"kafka_topic_resource_label": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The label to identify the KafkaTopic resources managed by the Topic Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaTopic resource. If you deploy more than one Topic Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.",
"title": "Kafka Topic Resource Label"
},
"operation_mode": {
"allOf": [
{
Expand Down
13 changes: 13 additions & 0 deletions kpops/components/streams_bootstrap/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@

import pydantic
from pydantic import Field
from typing_extensions import override

from kpops.component_handlers.helm_wrapper.model import HelmRepoConfig
from kpops.components.base_components import KafkaApp
from kpops.components.base_components.helm_app import HelmApp
from kpops.components.streams_bootstrap.model import StreamsBootstrapValues
from kpops.manifests.kubernetes import KubernetesManifest
from kpops.manifests.strimzi.kafka_topic import StrimziKafkaTopic
from kpops.utils.docstring import describe_attr

if TYPE_CHECKING:
Expand Down Expand Up @@ -81,3 +84,13 @@ def warning_for_latest_image_tag(self) -> Self:
f"The image tag for component '{self.name}' is set or defaulted to 'latest'. Please, consider providing a stable image tag."
)
return self

@override
def manifest_deploy(self) -> tuple[KubernetesManifest, ...]:
resource = super().manifest_deploy()
if self.to:
resource = resource + tuple(
StrimziKafkaTopic.from_topic(topic) for topic in self.to.kafka_topics
)

return resource
4 changes: 4 additions & 0 deletions kpops/config/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ class KpopsConfig(BaseSettings):
default=OperationMode.MANAGED,
description="The operation mode of KPOps (managed, manifest, argo).",
)
kafka_topic_resource_label: str | None = Field(
raminqaf marked this conversation as resolved.
Show resolved Hide resolved
default=None,
description="The label to identify the KafkaTopic resources managed by the Topic Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaTopic resource. If you deploy more than one Topic Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.",
)

model_config = SettingsConfigDict(
env_prefix=ENV_PREFIX,
Expand Down
Empty file added kpops/manifests/strimzi/__init__.py
View file
Open in desktop
Empty file.
Loading