Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update resource operator documentation #246

Merged
merged 2 commits into from
Sep 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 32 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,41 @@ Authored by [StreamNative](https://streamnative.io), this Pulsar Resources Oper

Currently, the Pulsar Resources Operator provides full lifecycle management for the following Pulsar resources, including creation, update, and deletion.

- Tenants
- Namespaces
- Topics
- Permissions
- Packages
- Functions
- Sinks
- Sources
- [Tenants](docs/pulsar_tenant.md)
- [Namespaces](docs/pulsar_namespace.md)
- [Topics](docs/pulsar_topic.md)
- [Permissions](docs/pulsar_permission.md)
- [Packages](docs/pulsar_package.md)
- [Functions](docs/pulsar_function.md)
- [Sinks](docs/pulsar_sink.md)
- [Sources](docs/pulsar_source.md)
- [Geo-Replication](docs/pulsar_geo_replication.md)

## Lifecycle Management

The Pulsar Resources Operator provides a flexible approach to managing the lifecycle of Pulsar resources through the `PulsarResourceLifeCyclePolicy`. This policy determines how Pulsar resources are handled when their corresponding Kubernetes custom resources are deleted. For more details on lifecycle management, please refer to the [PulsarResourceLifeCyclePolicy documentation](docs/pulsar_resource_lifecycle.md).

There are two available options for the lifecycle policy:

1. `CleanUpAfterDeletion`: When set, the Pulsar resource (such as a tenant, namespace, or topic) will be deleted from the Pulsar cluster when its corresponding Kubernetes custom resource is deleted. This is the default policy.

2. `KeepAfterDeletion`: When set, the Pulsar resource will remain in the Pulsar cluster even after its corresponding Kubernetes custom resource is deleted.

You can specify the lifecycle policy in the custom resource definition:

```yaml
apiVersion: pulsar.streamnative.io/v1beta1
kind: PulsarTenant
metadata:
name: my-tenant
spec:
pulsarResourceLifeCyclePolicy: KeepAfterDeletion
```

# Installation

The Pulsar Resources Operator is an independent controller, it doesn’t need to be installed with the pulsar operator. You can install it when you need the feature. And it is built with the [Operator SDK](https://github.com/operator-framework/operator-sdk), which is part of the [Operator framework](https://github.com/operator-framework/).


You can install the Pulsar Resources Operator using the officially supported `pulsar-resources-operator` Helm [chart](https://github.com/streamnative/charts/tree/master/charts/pulsar-resources-operator). It provides Custom Resource Definitions (CRDs) and Controllers to manage the Pulsar resources.

## Prerequisites
Expand Down Expand Up @@ -98,6 +118,7 @@ Before creating Pulsar resources, you must create a resource called `PulsarConne
In this tutorial, a Kubernetes namespace called `test` is used for examples, which is the namespace that the pulsar cluster installed.

- [PulsarConnection](docs/pulsar_connection.md)
- [PulsarResourceLifeCyclePolicy](docs/pulsar_resource_lifecycle.md)
- [PulsarTenant](docs/pulsar_tenant.md)
- [PulsarNamespace](docs/pulsar_namespace.md)
- [PulsarTopic](docs/pulsar_topic.md)
Expand All @@ -106,8 +127,9 @@ In this tutorial, a Kubernetes namespace called `test` is used for examples, whi
- [PulsarFunction](docs/pulsar_function.md)
- [PulsarSink](docs/pulsar_sink.md)
- [PulsarSource](docs/pulsar_source.md)
- [PulsarGeoReplication](docs/pulsar_geo_replication.md)

## Contributing
# Contributing

Contributions are warmly welcomed and greatly appreciated!
The project follows the typical GitHub pull request model.
Expand Down
61 changes: 49 additions & 12 deletions api/v1alpha1/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,34 +38,71 @@ type ValueOrSecretRef struct {
SecretRef *SecretKeyRef `json:"secretRef,omitempty"`
}

// PulsarAuthentication use the token or OAuth2 for pulsar authentication
// PulsarAuthentication defines the authentication configuration for Pulsar resources.
// It supports two authentication methods: Token-based and OAuth2-based.
// Only one authentication method should be specified at a time.
type PulsarAuthentication struct {
// Token specifies the configuration for token-based authentication.
// This can be either a direct token value or a reference to a secret containing the token.
// If using a secret, the token should be stored under the specified key in the secret.
// +optional
Token *ValueOrSecretRef `json:"token,omitempty"`

// OAuth2 specifies the configuration for OAuth2-based authentication.
// This includes all necessary parameters for setting up OAuth2 authentication with Pulsar.
// For detailed information on the OAuth2 fields, refer to the PulsarAuthenticationOAuth2 struct.
// +optional
OAuth2 *PulsarAuthenticationOAuth2 `json:"oauth2,omitempty"`
}

// PulsarResourceLifeCyclePolicy indicates whether it will keep or delete the resource
// in pulsar cluster after resource is deleted by controller
// KeepAfterDeletion or CleanUpAfterDeletion
// PulsarResourceLifeCyclePolicy defines the behavior for managing Pulsar resources
// when the corresponding custom resource (CR) is deleted from the Kubernetes cluster.
// This policy allows users to control whether Pulsar resources should be retained or
// removed from the Pulsar cluster after the CR is deleted.
type PulsarResourceLifeCyclePolicy string

const (
// KeepAfterDeletion keeps the resource in pulsar cluster when cr is deleted
// KeepAfterDeletion instructs the operator to retain the Pulsar resource
// in the Pulsar cluster even after the corresponding CR is deleted from Kubernetes.
// Use this option when:
// - You want to preserve data or configurations in Pulsar for backup or future use.
// - You're performing temporary maintenance on Kubernetes resources without affecting Pulsar.
// - You need to migrate or recreate Kubernetes resources without losing Pulsar data.
KeepAfterDeletion PulsarResourceLifeCyclePolicy = "KeepAfterDeletion"
// CleanUpAfterDeletion deletes the resource in pulsar cluster when cr is deleted

// CleanUpAfterDeletion instructs the operator to remove the Pulsar resource
// from the Pulsar cluster when the corresponding CR is deleted from Kubernetes.
// Use this option when:
// - You want to ensure complete removal of resources to free up Pulsar cluster capacity.
// - You're decommissioning services or environments and need to clean up all associated resources.
// - You want to maintain strict synchronization between Kubernetes and Pulsar resources.
CleanUpAfterDeletion PulsarResourceLifeCyclePolicy = "CleanUpAfterDeletion"
)

// PulsarAuthenticationOAuth2 indicates the parameters which are need by pulsar OAuth2
// PulsarAuthenticationOAuth2 represents the configuration for Pulsar OAuth2 authentication.
// This struct aligns with Pulsar's OAuth2 authentication mechanism as described in
// https://pulsar.apache.org/docs/3.3.x/security-oauth2/ and
// https://docs.streamnative.io/docs/access-cloud-clusters-oauth
type PulsarAuthenticationOAuth2 struct {
IssuerEndpoint string `json:"issuerEndpoint"`
ClientID string `json:"clientID"`
Audience string `json:"audience"`
Key *ValueOrSecretRef `json:"key"`
Scope string `json:"scope,omitempty"`
// IssuerEndpoint is the URL of the OAuth2 authorization server.
// This is typically the base URL of your identity provider's OAuth2 service.
IssuerEndpoint string `json:"issuerEndpoint"`

// ClientID is the OAuth2 client identifier issued to the client during the registration process.
ClientID string `json:"clientID"`

// Audience is the intended recipient of the token. In Pulsar's context, this is usually
// the URL of your Pulsar cluster or a specific identifier for your Pulsar service.
Audience string `json:"audience"`

// Key is either the client secret or the path to a JSON credentials file.
// For confidential clients, this would be the client secret.
// For public clients using JWT authentication, this would be the path to the JSON credentials file.
Key *ValueOrSecretRef `json:"key"`

// Scope is an optional field to request specific permissions from the OAuth2 server.
// If not specified, the default scope defined by the OAuth2 server will be used.
Scope string `json:"scope,omitempty"`
}

// IsPulsarResourceReady returns true if resource satisfies with these condition
Expand Down
47 changes: 36 additions & 11 deletions api/v1alpha1/pulsarconnection_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,59 +22,81 @@ import (
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.

// PulsarConnectionSpec defines the desired state of PulsarConnection
// It specifies the configuration for connecting to a Pulsar cluster.
//
// For plaintext (non-TLS) Pulsar clusters:
// - Set AdminServiceURL to "http://<admin-service-host>:<port>"
// - Set BrokerServiceURL to "pulsar://<broker-service-host>:<port>"
//
// For TLS-enabled Pulsar clusters:
// - Set AdminServiceSecureURL to "https://<admin-service-host>:<port>"
// - Set BrokerServiceSecureURL to "pulsar+ssl://<broker-service-host>:<port>"
// - Optionally set BrokerClientTrustCertsFilePath if using custom CA certificates
type PulsarConnectionSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file

// AdminServiceURL is the admin service url of the pulsar cluster
// AdminServiceURL is the HTTP(S) URL for the Pulsar cluster's admin service.
// This URL is used for administrative operations.
// +optional
// +kubebuilder:validation:Pattern="^https?://.+$"
AdminServiceURL string `json:"adminServiceURL"`

// Authentication defines authentication configurations
// Authentication defines the authentication configuration for connecting to the Pulsar cluster.
// It supports both token-based and OAuth2-based authentication methods.
// +optional
Authentication *PulsarAuthentication `json:"authentication,omitempty"`

// BrokerServiceURL is the broker service url of the pulsar cluster
// BrokerServiceURL is the non-TLS URL for connecting to Pulsar brokers.
// Use this for non-secure connections to the Pulsar cluster.
// +optional
// +kubebuilder:validation:Pattern="^pulsar?://.+$"
BrokerServiceURL string `json:"brokerServiceURL,omitempty"`

// BrokerServiceSecureURL is the broker service url for secure connection.
// BrokerServiceSecureURL is the TLS-enabled URL for secure connections to Pulsar brokers.
// Use this for encrypted communications with the Pulsar cluster.
// +optional
// +kubebuilder:validation:Pattern="^pulsar\\+ssl://.+$"
BrokerServiceSecureURL string `json:"brokerServiceSecureURL,omitempty"`

// AdminServiceSecureURL is the admin service url for secure connection.
// AdminServiceSecureURL is the HTTPS URL for secure connections to the Pulsar admin service.
// Use this for encrypted administrative operations.
// +optional
// +kubebuilder:validation:Pattern="^https://.+$"
AdminServiceSecureURL string `json:"adminServiceSecureURL,omitempty"`

// BrokerClientTrustCertsFilePath Path for the trusted TLS certificate file for outgoing connection to a server (broker)
// BrokerClientTrustCertsFilePath is the file path to the trusted TLS certificate
// for outgoing connections to Pulsar brokers. This is used for TLS verification.
// +optional
BrokerClientTrustCertsFilePath string `json:"brokerClientTrustCertsFilePath,omitempty"`

// ClusterName indicates the local cluster name of the pulsar cluster. It should
// set when enabling the Geo Replication
// ClusterName specifies the name of the local Pulsar cluster.
// When setting up Geo-Replication between Pulsar instances, this should be enabled to identify the cluster.
// +optional
ClusterName string `json:"clusterName,omitempty"`
}

// PulsarConnectionStatus defines the observed state of PulsarConnection
// PulsarConnectionStatus defines the observed state of PulsarConnection.
// It provides information about the current status of the Pulsar connection.
type PulsarConnectionStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make" to regenerate code after modifying this file

// ObservedGeneration is the most recent generation observed for this resource.
// It corresponds to the metadata generation, which is updated on mutation by the API Server.
// This field is used to track whether the controller has processed the latest changes.
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`

// SecretKeyHash is the hash of the secret ref
// SecretKeyHash is the hash of the secret reference used for authentication.
// This is used to detect changes in the secret without exposing sensitive information.
// The controller should update this hash when the secret changes.
// +optional
SecretKeyHash string `json:"secretKeyHash,omitempty"`

// Represents the observations of a connection's current state.
// Conditions represent the latest available observations of the connection's current state.
// It follows the Kubernetes conventions for condition types and status.
// The "Ready" condition type is typically used to indicate the overall status.
// +patchMergeKey=type
// +patchStrategy=merge
// +listType=map
Expand All @@ -95,6 +117,8 @@ type PulsarConnectionStatus struct {
//+kubebuilder:printcolumn:name="READY",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`

// PulsarConnection is the Schema for the pulsarconnections API
// It represents a connection to a Pulsar cluster and includes both the desired state (Spec)
// and the observed state (Status) of the connection.
type PulsarConnection struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Expand All @@ -106,6 +130,7 @@ type PulsarConnection struct {
//+kubebuilder:object:root=true

// PulsarConnectionList contains a list of PulsarConnection
// This type is used by the Kubernetes API to return multiple PulsarConnection objects.
type PulsarConnectionList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Expand Down
Loading
Loading