Deploy Connectors in a RBAC cluster (confluentinc#1537)

* [ANSIENG-2938] | Update Minimum Ansible version to 2.14 * [ANSIENG-2939] | kafka_connector plugin to support username, password, certs and keys * [ANSIENG-2939] | Add Role Bindings for Connectors * [ANSIENG-2939] | Add connect producer and consumer jaas configs * [ANSIENG-2939] | Deploy Connectors for Multiple clusters with RBAC * [ANSIENG-2939] | Deploy connectors on non-RBAC cluster * [ANSIENG-2939] | Create Connector on a RBAC cluster * [ANSIENG-2939] | [ANSIENG-2939] | Create Role Bindings for Connector User White list topics * [ANSIENG-2939] | Update docs for Connector deployment * [ANSIENG-2939] | Update hosts_example.yml for Connectors deployment
mansisinha · Jan 22, 2024 · 739fb41 · 739fb41
1 parent 5586dbb
commit 739fb41
Show file tree

Hide file tree

Showing 24 changed files with 719 additions and 85 deletions.
diff --git a/docs/MOLECULE_SCENARIOS.md b/docs/MOLECULE_SCENARIOS.md
@@ -18,11 +18,11 @@ Validates that Confluent CLI is installed.
 
 ***
 
-### molecule/archive-plain-debian9
+### molecule/archive-plain-debian10
 
-#### Scenario archive-plain-debian9 test's the following:
+#### Scenario archive-plain-debian10 test's the following:
 
-Archive installation of Confluent Platform on Debian 9.
+Archive installation of Confluent Platform on Debian 10.
 
 SASL Protocol Plain.
 
@@ -32,23 +32,21 @@ Kafka Connect Confluent Hub Plugins logic (Installs jcustenborder/kafka-connect-
 
 Custom log dirs for all components.
 
-#### Scenario archive-plain-debian9 verify test's the following:
+#### Scenario archive-plain-debian10 verify test's the following:
 
-Validates that SASL SSL protocol is set across all components
+Validates that SASL SSL protocol is set across all components.
 
 Validates that custom log4j configuration is in place.
 
 Validates that Java 17 is in Use
 
-Validates that Confluent CLI is installed.
-
 ***
 
-### molecule/archive-plain-debian10
+### molecule/archive-plain-debian9
 
-#### Scenario archive-plain-debian10 test's the following:
+#### Scenario archive-plain-debian9 test's the following:
 
-Archive installation of Confluent Platform on Debian 10.
+Archive installation of Confluent Platform on Debian 9.
 
 SASL Protocol Plain.
 
@@ -58,14 +56,16 @@ Kafka Connect Confluent Hub Plugins logic (Installs jcustenborder/kafka-connect-
 
 Custom log dirs for all components.
 
-#### Scenario archive-plain-debian10 verify test's the following:
+#### Scenario archive-plain-debian9 verify test's the following:
 
-Validates that SASL SSL protocol is set across all components.
+Validates that SASL SSL protocol is set across all components
 
 Validates that custom log4j configuration is in place.
 
 Validates that Java 17 is in Use
 
+Validates that Confluent CLI is installed.
+
 ***
 
 ### molecule/archive-plain-rhel-fips
@@ -140,6 +140,8 @@ Kafka Connect Confluent Hub Plugins logic (Installs jcustenborder/kafka-connect-
 
 Custom log dirs for all components.
 
+Deploy Connector on Connect Cluster.
+
 #### Scenario archive-plain-ubuntu2004 verify test's the following:
 
 Validates that protocol is set to sasl plain.
@@ -148,13 +150,15 @@ Validates that protocol is set to SASL SSL.
 
 Validates log4j config.
 
+Validates that Connector is Running.
+
 ***
 
 ### molecule/archive-scram-rhel
 
 #### Scenario archive-scram-rhel test's the following:
 
-Archive Installation of Confluent Platform on RHEL8.
+Archive Installation of Confluent Platform on Oracle Linux 8.
 
 SASL SCRAM protocol.
 
@@ -172,6 +176,32 @@ Validates that TLS is configured properly.
 
 ***
 
+### molecule/archive-zookeeper-tls-rhel-fips
+
+#### Scenario archive-zookeeper-tls-rhel-fips test's the following:
+
+Installs Confluent Platform on Rocky linux 8
+
+Enables SASL SCRAM Auth on Zookeeper.
+
+TLS enabled.
+
+Customer zookeeper root.
+
+Jolokia has TLS disabled.
+
+FIPS enabled
+
+#### Scenario archive-zookeeper-tls-rhel-fips verify test's the following:
+
+Validates that Zookeeper is using TLS.
+
+Validates that other components are using SCRAM for auth.
+
+Validates that FIPS is in use in OpenSSL.
+
+***
+
 ### molecule/broker-scale-up
 
 #### Scenario broker-scale-up test's the following:
@@ -376,19 +406,23 @@ Installation of Confluent Platform on Oracle Linux 9.
 
 Kerberos enabled with custom client config path
 
+Creates a Connector in Connect cluster
+
 #### Scenario kerberos-rhel verify test's the following:
 
 Validates that Kerberos is enabled across all components.
 
 Validates that SASL SSL Plaintext is enabled across all components.
 
+Validates that Connector is running
+
 ***
 
 ### molecule/ksql-scale-up
 
 #### Scenario ksql-scale-up test's the following:
 
-Installation of Confluent Platform on centos7.
+Installation of Confluent Platform on RHEL9.
 
 MTLS enabled.
 
@@ -606,7 +640,7 @@ Validates that Control Center Can connect to each KSQL cluster.
 
 #### Scenario plain-customcerts-rhel-fips test's the following:
 
-Installation of Confluent Platform on RHEL8.
+Installation of Confluent Platform on Oracle Linux 8.
 
 TLS enabled.
 
@@ -1026,6 +1060,8 @@ RBAC Additional System Admin.
 
 Use Java 11 package
 
+Creates a Connector in connect cluster
+
 #### Scenario rbac-mds-mtls-existing-keystore-truststore-ubuntu verify test's the following:
 
 Validates that keystores are present on all components.
@@ -1076,7 +1112,7 @@ Validates that FIPS is in use on both clusters.
 
 #### Scenario rbac-mds-scram-custom-rhel test's the following:
 
-Installs two Confluent Platform Clusters on RHEL8.
+Installs two Confluent Platform Clusters on Rocky Linux 9.
 
 RBAC enabled.
 
@@ -1146,6 +1182,8 @@ RBAC Additional System Admin.
 
 Provided SSL Principal Mapping rule
 
+Creates two unique Connectors in Connect cluster.
+
 #### Scenario rbac-mtls-rhel-fips verify test's the following:
 
 Validates TLS version across all components.
@@ -1168,7 +1206,7 @@ Validates that FIPS is in use in OpenSSL.
 
 #### Scenario rbac-mtls-rhel8 test's the following:
 
-Installs Confluent Platform Cluster on RHEL8.
+Installs Confluent Platform Cluster on Oracle Linux 8.
 
 RBAC enabled.
 
@@ -1206,6 +1244,8 @@ Control Center disabled, metrics reporters enabled.
 
 LdapAuthenticateCallbackHandler for AuthN
 
+Creates two unique Connectors in Connect cluster
+
 #### Scenario rbac-plain-provided-debian9 verify test's the following:
 
 Validates Metrics reporter without C3.
@@ -1242,6 +1282,8 @@ SSO authentication using OIDC in Control center using Azure IdP
 
 FIPS enabled
 
+Installs Two unique Kafka Connect Clusters with unique connectors.
+
 #### Scenario rbac-scram-custom-rhel-fips verify test's the following:
 
 Validates keystore is present across all components.
@@ -1258,6 +1300,8 @@ Validates OIDC authenticate api for SSO in Control Center
 
 Validates that FIPS is in use in OpenSSL.
 
+Validates that both the Connectors are Running
+
 ***
 
 ### molecule/scram-rhel
@@ -1368,29 +1412,3 @@ Validates that Secrets protection is applied to the correct properties.
 
 ***
 
-### molecule/archive-zookeeper-tls-rhel-fips
-
-#### Scenario archive-zookeeper-tls-rhel-fips test's the following:
-
-Installs Confluent Platform on RHEL8
-
-Enables SASL SCRAM Auth on Zookeeper.
-
-TLS enabled.
-
-Customer zookeeper root.
-
-Jolokia has TLS disabled.
-
-FIPS enabled
-
-#### Scenario archive-zookeeper-tls-rhel-fips verify test's the following:
-
-Validates that Zookeeper is using TLS.
-
-Validates that other components are using SCRAM for auth.
-
-Validates that FIPS is in use in OpenSSL.
-
-***
-
diff --git a/docs/VARIABLES.md b/docs/VARIABLES.md
@@ -2020,6 +2020,14 @@ Default:  ""
 
 ***
 
+### kafka_connect_connector_white_list
+
+Set this variable with a comma separated list of Topics for Kafka Connect Connector to produce/consume from.  This is a mandatory variable when creating Connector in RBAC cluster.
+
+Default:  ""
+
+***
+
 ### kafka_connect_skip_restarts
 
 Boolean used for disabling of systemd service restarts when rootless install is executed

diff --git a/docs/hosts_example.yml b/docs/hosts_example.yml
@@ -184,6 +184,10 @@ all:
     #       tasks.max: "1"
     #       file: "path/to/file.txt"
     #       topics: "test_topic"
+    #
+    ## To manage the connector on an RBAC cluster, set the following variable with the list of Topics for Kafka Connect Connector to produce/consume.
+    ## The variable should contain the list of all the topics of the Connectors in a Connect cluster. eg.
+    # kafka_connect_connector_white_list: "test_topic1,test_topic2"
 
     #### Configuring logredactor ####
     ## To configure logredactor for all components, set the following variables ##

diff --git a/molecule/archive-plain-ubuntu2004/molecule.yml b/molecule/archive-plain-ubuntu2004/molecule.yml
@@ -4,6 +4,7 @@
 ### SSL Enabled.
 ### Kafka Connect Confluent Hub Plugins logic (Installs jcustenborder/kafka-connect-spooldir:2.0.43).
 ### Custom log dirs for all components.
+### Deploy Connector on Connect Cluster.
 
 platforms:
   - name: ${KRAFT_CONTROLLER:-zookeeper}1
@@ -129,3 +130,12 @@ provisioner:
         kafka_connect_log_dir: /connect/logs
         ksql_log_dir: /ksql/logs/
         control_center_log_dir: /c3/logs
+
+        kafka_connect_connectors:
+          - name: sample-connector-1
+            config:
+              connector.class: "org.apache.kafka.connect.tools.VerifiableSourceConnector"
+              tasks.max: "1"
+              file: "/etc/kafka/connect-distributed.properties"
+              topic: "test_topic"
+              throughput: "1000"
diff --git a/molecule/archive-plain-ubuntu2004/verify.yml b/molecule/archive-plain-ubuntu2004/verify.yml
@@ -2,6 +2,7 @@
 ### Validates that protocol is set to sasl plain.
 ### Validates that protocol is set to SASL SSL.
 ### Validates log4j config.
+### Validates that Connector is Running.
 
 - name: Verify - kafka_controller
   hosts: kafka_controller
@@ -72,6 +73,31 @@
         property: security.protocol
         expected_value: SASL_SSL
 
+    - name: Get Connectors on connect cluster1
+      uri:
+        url: "https://kafka-connect1:8083/connectors"
+        status_code: 200
+        validate_certs: false
+
+      register: connectors
+
+    - name: Assert Connector Created
+      assert:
+        that:
+          - connectors.json[0] == "sample-connector-1"
+        fail_msg: "Connector not created"
+        quiet: true
+
+    - name: Wait for Connector tasks to return Running
+      uri:
+        url: https://kafka-connect1:8083/connectors/sample-connector-1/status
+        status_code: 200
+        validate_certs: false
+      register: connector_status_response
+      until: connector_status_response.json.tasks[0].state == 'RUNNING'
+      retries: 10
+      delay: 5
+
 - name: Verify - ksql
   hosts: ksql
   gather_facts: false

diff --git a/molecule/connect-scale-up/molecule.yml b/molecule/connect-scale-up/molecule.yml
@@ -163,9 +163,21 @@ provisioner:
       # connect clusters
       ssl:
         kafka_connect_group_id: connect-ssl
+        # Create Connectors with ssl
+        kafka_connect_ssl_enabled: true
+        kafka_connect_ssl_mutual_auth_enabled: true
+        kafka_connect_connectors:
+          - name: sample-connector-3
+            config:
+              connector.class: "org.apache.kafka.connect.tools.VerifiableSinkConnector"
+              tasks.max: "5"
+              file: "/etc/kafka/connect-distributed.properties"
+              topics: "test_topic"
+              key.converter: "org.apache.kafka.connect.json.JsonConverter"
+              value.converter: "org.apache.kafka.connect.json.JsonConverter"
       cluster1:
         kafka_connect_group_id: connect-cluster1
-        # Create Connectors not working w ssl on molecule
+        # Create Connectors without ssl
         kafka_connect_ssl_enabled: false
         kafka_connect_ssl_mutual_auth_enabled: false
         kafka_connect_connectors:
@@ -179,7 +191,7 @@ provisioner:
               value.converter: "org.apache.kafka.connect.json.JsonConverter"
       cluster2:
         kafka_connect_group_id: connect-cluster2
-        # Create Connectors not working w ssl
+        # Create Connectors without ssl
         kafka_connect_ssl_enabled: false
         kafka_connect_ssl_mutual_auth_enabled: false
         kafka_connect_connectors: