Kroxy system tests (kroxylicious#628)

* added strimzi system test files to try to run one simple test * STs baseline * first test run producing and consuming messages * Removing useless stuff * Moved quay org env variable to Environment class * Added skipSTs to avoid running them on github actions * fix some sonar issues * fix javadoc for kroxy class * renamed module to kroxylicious-systemtests and refactor most of the kubectl commands * removed commented lines * Added cert manager url as a constant * moved method to test utils * Added cert manager namespace as constant * Use kubernetes java package instead of kubectl command for producer and consumers * Added test-clients for consumer and producer * delete files * Refactor some classes * Added license headers * Launch Kroxy as a resource, removing all kustomize commands * Added first test cases: restart brokers and kroxy with more than one replica * Fix sonarCloud bugs and rename some packages * improved kafka utils * fix some code smells * added description and dependency management * added enforcer rule for system tests and rename kroxy class by kroxylicious * added license header * rename isSuccess method in ExecResult * Removed restart zookeeper method in kafkaSteps class * removed CustomResourceStatus enum as it is always Ready * Removed serializable from ExecResult and removed VALUES from Environment * removed intellij javadocs file and added to gitignore * redo javadoc for environment and contants interfaces and set to private those necessary * Created installation url env variable for Strimzi * created env variable for Kroxy URL as well to be able to push the image to any repository * cached the load of the yaml from url for strimzi and cert manager and set to debug the level of logging for exec * added some fixes from comments * improved the way to check the kafka broker is restarted * Fix security issue * Added kroxy extension with ephemeral namespace * Improved kubeClient usage * removed testNamespace from KubeClusterResource * remove bug from sonar * removed stored resources management and delete resources from ResourceManager * Split test cases for kroxy on kubernetes and kroxy on bare metal * reformat * fix kroxy app * increasing the timeout for checking load balancer * Log the exception when encountered * fix javadoc build * defend against NPE's * Adding Sam changes * improved app * Track the k8s namespace in the junit extension context. * Added toLowerCase to resolveParameter * remove sonar coverage from system tests and replaced some waitFor by await * improved await failures and test coverage exclusions for sonar * Address sonar security warnings * fix exceptions for awaitality * address security issues, fix shaded packages from testcontainers * Address security issues * renamed certManager field * renamed strimzi field * Added more logger info for test suites * reduce code smells * use built in enforcer rule rather than a custom one to ban the use of system tests in other modules * comments fixes * updated DEV_GUIDE and pom files * fix pom file * Removed getBootstrap from Kroxylicious app since it is not used * DEV_GUIDE update * Disabled kroxylicious app system tests to focus currently on kubernetes * Latest comment fixes * removed unused variable * adding new fixes in comments * fix pom file * removed commented lines * remove code smell * adding private constructors --------- Signed-off-by: Lukas Kral <[email protected]> Signed-off-by: Francisco Vila <[email protected]> Co-authored-by: Lukas Kral <[email protected]> Co-authored-by: Sam Barker <[email protected]>
franvila · Nov 23, 2023 · c029d82 · c029d82
1 parent 86acead
commit c029d82
Show file tree

Hide file tree

Showing 58 changed files with 4,950 additions and 1 deletion.
diff --git a/.gitignore b/.gitignore
@@ -74,6 +74,9 @@ out/
 # mpeltonen/sbt-idea plugin
 .idea_modules/
 
+ # Javadoc plugin
+.idea/**/intellij-javadocs-*.xml
+
 # JIRA plugin
 atlassian-ide-plugin.xml
 

diff --git a/DEV_GUIDE.md b/DEV_GUIDE.md
@@ -68,6 +68,7 @@ The running of the tests can be controlled with the following Maven properties:
 |--------------------|-------------------------------------------------------------------------------------------|
 | `-DskipUTs=true`   | skip unit tests                                                                           |
 | `-DskipITs=true`   | skip integration tests                                                                    |
+| `-DskipSTs=true`   | skip system tests                                                                         |
 | `-DskipTests=true` | skip all tests                                                                            |
 | `-Pdebug`          | enables logging so you can see what the Kafka clients, Proxy and in VM brokers are up to. |
 
@@ -308,6 +309,50 @@ You'll see an API response.  If the service_timeout change is effective, the soc
 will continue for 3 minutes.  If `socat` terminates after about 10 seconds, the workaround
 has been applied ineffectively.
 
+## Running system tests locally
+### Prerequisites
+* minikube
+* User must have access to a container registry such as [quay.io](https://quay.io) or [docker.io](https://docker.io).
+     Create a public accessible repository within the registry named `kroxylicious`.
+
+### Environment variables
+* `KROXYLICIOUS_IMAGE_REPO`: url to the image of kroxylicious to be used. Default value: `quay.io/kroxylicious/kroxylicious-developer`
+* `KROXYLICIOUS_VERSION`: version of kroxylicious to be used. Default value: `0.4.0-SNAPSHOT`
+* `KAFKA_VERSION`: kafka version to be used. Default value: `3.6.0`
+* `STRIMZI_URL`: url where to download strimzi. Default value: `khttps://strimzi.io/install/latest?namespace=kafka`
+
+### Launch system tests
+First of all, the code must be compiled and the distribution artifacts created:
+
+```shell
+mvn clean install -Dquick -Pdist
+```
+
+If the tests are going to be run against local changes, 
+upload your package to the `kroxylicious` repository in the container registry:
+
+```shell
+PUSH_IMAGE=true REGISTRY_DESTINATION=<container_registry>/<myorg>/kroxylicious ./scripts/deploy-image.sh 
+```
+
+Start minikube:
+```shell
+minikube start
+```
+
+Then, you can run them from system test or root folder:
+
+* Run the system tests from [kroxylicious-systemtests](kroxylicious-systemtests) folder:
+
+```shell
+KROXYLICIOUS_IMAGE_REPO=<container_registry>/<myorg>/kroxylicious mvn clean integration-test
+```
+
+* Run them from root folder of kroxylicious project:
+
+```shell
+KROXYLICIOUS_IMAGE_REPO=<container_registry>/<myorg>/kroxylicious mvn clean verify -DskiptITs=true -DskiptUTs=true -DskipSTs=false
+```
 
 ## Rendering documentation
 

diff --git a/kroxylicious-systemtests/pom.xml b/kroxylicious-systemtests/pom.xml
@@ -0,0 +1,115 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Copyright Kroxylicious Authors.
+
+    Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>io.kroxylicious</groupId>
+        <artifactId>kroxylicious-parent</artifactId>
+        <version>0.4.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>kroxylicious-systemtests</artifactId>
+    <description>
+        The intention of this module is to test Kroxylicious against a kafka instance deployed in a kubernetes cluster/minikube
+        using Strimzi, to simulate a real scenario for end-to-end testing treating Kroxylicious as a black box.
+    </description>
+    <dependencies>
+        <dependency>
+            <groupId>io.fabric8</groupId>
+            <artifactId>kubernetes-client-api</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.fabric8</groupId>
+            <artifactId>kubernetes-client</artifactId>
+            <scope>runtime</scope>
+        </dependency>
+        <dependency>
+            <groupId>io.fabric8</groupId>
+            <artifactId>kubernetes-httpclient-jdk</artifactId>
+            <scope>runtime</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.junit.jupiter</groupId>
+            <artifactId>junit-jupiter-engine</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>io.strimzi</groupId>
+            <artifactId>api</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.junit.jupiter</groupId>
+            <artifactId>junit-jupiter-api</artifactId>
+            <scope>compile</scope>
+        </dependency>
+        <!-- third party dependencies - runtime and compile -->
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-api</artifactId>
+        </dependency>
+
+        <!-- third party dependencies - test -->
+        <dependency>
+            <groupId>org.apache.logging.log4j</groupId>
+            <artifactId>log4j-slf4j2-impl</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.maven.enforcer</groupId>
+            <artifactId>enforcer-api</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.maven</groupId>
+            <artifactId>maven-core</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.hamcrest</groupId>
+            <artifactId>hamcrest</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.junit.jupiter</groupId>
+            <artifactId>junit-jupiter-params</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.awaitility</groupId>
+            <artifactId>awaitility</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>commons-io</groupId>
+            <artifactId>commons-io</artifactId>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-surefire-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>integration-test</id>
+                        <goals>
+                            <goal>test</goal>
+                        </goals>
+                        <phase>integration-test</phase>
+                        <configuration>
+                            <skipTests>${skipSTs}</skipTests>
+                            <includes>
+                                <include>**/ST*.java</include>
+                                <include>**/*ST.java</include>
+                            </includes>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+</project>
diff --git a/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/Constants.java b/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/Constants.java
@@ -0,0 +1,142 @@
+/*
+ * Copyright Kroxylicious Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+package io.kroxylicious.systemtests;
+
+import java.time.Duration;
+
+import static io.kroxylicious.systemtests.Environment.KAFKA_VERSION_DEFAULT;
+
+/**
+ * The interface Constants.
+ */
+public interface Constants {
+
+    /**
+     * The deployment name for kroxylicous
+     */
+    String KROXY_DEPLOYMENT_NAME = "kroxylicious-proxy";
+    /**
+     * The service name for kroxylicious. Used for the bootstrap url
+     */
+    String KROXY_SERVICE_NAME = "kroxylicious-service";
+    /**
+     * The constant KROXY_CONFIG_NAME.
+     */
+    String KROXY_CONFIG_NAME = "kroxylicious-config";
+    /**
+     * Strimzi cluster operator deployment name
+     */
+    String STRIMZI_DEPLOYMENT_NAME = "strimzi-cluster-operator";
+    /**
+     * The default namespace used for kubernetes deployment
+     */
+    String KROXY_DEFAULT_NAMESPACE = "kafka";
+
+    /**
+     * The cert-manager namespace for kubernetes deployment
+     */
+    String CERT_MANAGER_NAMESPACE = "cert-manager";
+
+    /**
+     * API versions of Strimzi CustomResources
+     */
+    String KAFKA_API_VERSION_V1BETA2 = "kafka.strimzi.io/v1beta2";
+
+    /**
+     * Kind of Strimzi CustomResources
+     */
+    String KAFKA_KIND = "Kafka";
+    /**
+     * Kind of kafka topics
+     */
+    String KAFKA_TOPIC_KIND = "KafkaTopic";
+    /**
+     * Kind of kafka users
+     */
+    String KAFKA_USER_KIND = "KafkaUser";
+    /**
+     * Kind of kafka node pools
+     */
+    String KAFKA_NODE_POOL_KIND = "KafkaNodePool";
+    /**
+     * Kind of pods
+     */
+    String POD_KIND = "Pod";
+
+    /**
+     * Kind of config maps
+     */
+    String CONFIG_MAP_KIND = "ConfigMap";
+
+    /**
+     * Kind of services
+     */
+    String SERVICE_KIND = "Service";
+
+    /**
+     * Load balancer type name.
+     */
+    String LOAD_BALANCER_TYPE = "LoadBalancer";
+
+    /**
+     * Listener names for Kafka cluster
+     */
+    String PLAIN_LISTENER_NAME = "plain";
+    /**
+     * Listener name for tls
+     */
+    String TLS_LISTENER_NAME = "tls";
+
+    /**
+     * Strimzi related labels and annotations
+     */
+    String STRIMZI_DOMAIN = "strimzi.io/";
+    /**
+     * Strimzi cluster label
+     */
+    String STRIMZI_CLUSTER_LABEL = STRIMZI_DOMAIN + "cluster";
+
+    /**
+     * Polls and timeouts constants
+     */
+    long POLL_INTERVAL_FOR_RESOURCE_READINESS_MILLIS = Duration.ofSeconds(5).toMillis();
+    /**
+     * Poll interval for resource deletion in milliseconds
+     */
+    long POLL_INTERVAL_FOR_RESOURCE_DELETION_MILLIS = Duration.ofSeconds(1).toMillis();
+
+    /**
+     * Global timeout in milliseconds
+     */
+    long GLOBAL_TIMEOUT_MILLIS = Duration.ofMinutes(5).toMillis();
+    /**
+     * Global Poll interval in milliseconds
+     */
+    long GLOBAL_POLL_INTERVAL_MILLIS = Duration.ofSeconds(1).toMillis();
+
+    /**
+     * Kubernetes related constants
+     */
+    String DEPLOYMENT = "Deployment";
+    /**
+     * Strimzi kafka image url in quay
+     */
+    String STRIMZI_KAFKA_IMAGE = "quay.io/strimzi/kafka:latest-kafka-" + KAFKA_VERSION_DEFAULT;
+
+    /**
+     * The cert manager url to install it on kubernetes
+     */
+    String CERT_MANAGER_URL = "https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml";
+    /**
+     * kafka consumer client label to identify the consumer test client
+     */
+    String KAFKA_CONSUMER_CLIENT_LABEL = "kafka-consumer-client";
+    /**
+     * kafka producer client label to identify the producer test client
+     */
+    String KAFKA_PRODUCER_CLIENT_LABEL = "kafka-producer-client";
+}
diff --git a/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/Environment.java b/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/Environment.java
@@ -0,0 +1,71 @@
+/*
+ * Copyright Kroxylicious Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+package io.kroxylicious.systemtests;
+
+import java.util.function.Function;
+
+/**
+ * The type Environment.
+ */
+public class Environment {
+
+    private Environment() {
+    }
+
+    /**
+     * Env. variables names
+     */
+    private static final String KAFKA_VERSION_ENV = "KAFKA_VERSION";
+    private static final String KROXY_VERSION_ENV = "KROXYLICIOUS_VERSION";
+    private static final String KROXY_IMAGE_REPO_ENV = "KROXYLICIOUS_IMAGE_REPO";
+    private static final String STRIMZI_URL_ENV = "STRIMZI_URL";
+
+    /**
+     * The kafka version default value
+     */
+    public static final String KAFKA_VERSION_DEFAULT = "3.6.0";
+
+    /**
+     * The kroxy version default value
+     */
+    public static final String KROXY_VERSION_DEFAULT = "0.4.0-SNAPSHOT";
+    /**
+     * The url where kroxylicious image lives to be downloaded.
+     */
+    public static final String KROXY_IMAGE_REPO_DEFAULT = "quay.io/kroxylicious/kroxylicious-developer";
+
+    /**
+     * The strimzi installation url for kubernetes.
+     */
+    public static final String STRIMZI_URL_DEFAULT = "https://strimzi.io/install/latest?namespace=" + Constants.KROXY_DEFAULT_NAMESPACE;
+
+    /**
+     * KAFKA_VERSION env variable assignment
+     */
+    public static final String KAFKA_VERSION = getOrDefault(KAFKA_VERSION_ENV, KAFKA_VERSION_DEFAULT);
+
+    /**
+     * KROXY_VERSION env variable assignment
+     */
+    public static final String KROXY_VERSION = getOrDefault(KROXY_VERSION_ENV, KROXY_VERSION_DEFAULT);
+    /**
+     * STRIMZI_URL env variable assignment
+     */
+    public static final String STRIMZI_URL = getOrDefault(STRIMZI_URL_ENV, STRIMZI_URL_DEFAULT);
+    /**
+     * KROXY_IMAGE_REPO env variable assignment
+     */
+    public static final String KROXY_IMAGE_REPO = getOrDefault(KROXY_IMAGE_REPO_ENV, KROXY_IMAGE_REPO_DEFAULT);
+
+    private static String getOrDefault(String varName, String defaultValue) {
+        return getOrDefault(varName, String::toString, defaultValue);
+    }
+
+    private static <T> T getOrDefault(String varName, Function<String, T> converter, T defaultValue) {
+        return System.getenv(varName) != null ? converter.apply(System.getenv(varName)) : defaultValue;
+    }
+}