Skip to content

Latest commit

 

History

History
156 lines (115 loc) · 5.92 KB

README.md

File metadata and controls

156 lines (115 loc) · 5.92 KB
Raw

Kafka Connect Playground

This repository provides a Docker Compose setup to create a Kafka Connect playground for testing connectors and different scenarios. The setup includes Zookeeper, Kafka Broker, Schema Registry, Kafka Connect, MySQL, phpMyAdmin, and Kafkacat.

Prerequisites

  • Docker
  • Docker Compose

Services

Zookeeper

Zookeeper is used to manage and coordinate Kafka brokers.

  • Image: confluentinc/cp-zookeeper:7.0.1
  • Ports: 2181

Kafka Broker

The Kafka broker handles the storage and retrieval of messages.

  • Image: confluentinc/cp-kafka:7.0.1
  • Ports: 9092
  • Depends on: Zookeeper

Schema Registry

Schema Registry provides a serving layer for your metadata.

  • Image: confluentinc/cp-schema-registry:7.0.1
  • Ports: 8081
  • Depends on: Kafka Broker

Kafka Connect

Kafka Connect is used to stream data between Kafka and other systems.

  • Image: confluentinc/cp-kafka-connect-base:7.0.1
  • Ports: 8083
  • Depends on: Kafka Broker, Schema Registry

MySQL

MySQL is used as a sample database for testing JDBC connectors.

  • Image: mysql:9.0.0
  • Ports: 3306
  • Environment Variables:
    • MYSQL_ROOT_PASSWORD=Admin123
    • MYSQL_USER=mysqluser
    • MYSQL_PASSWORD=mysqlpw
  • Volumes:
    • ${PWD}/mysql/00_setup_db.sql:/docker-entrypoint-initdb.d/00_setup_db.sql
    • ${PWD}/mysql/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh

phpMyAdmin

phpMyAdmin is a web interface for managing MySQL databases.

  • Image: phpmyadmin:5.2.1-apache
  • Ports: 8080
  • Depends on: MySQL

Kafkacat

Kafkacat is a command-line utility for interacting with Kafka.

  • Image: edenhill/kcat:1.7.1
  • Depends on: Kafka Broker, Schema Registry

Setup

  1. Clone the repository:

    git clone <repository-url>
cd <repository-directory>

  2. Start Services:

    docker compose up or
docker compose up -d

  3. Access phpMyAdmin:

  4. Access Kafka Connect:

Kafka Connectors

The following connectors are pre-installed

  • HTTP Source Connector: confluentinc/kafka-connect-http-source:0.2.0
  • JDBC Connector: confluentinc/kafka-connect-jdbc:10.7.6

Customizing Connectors

To add more connectors, modify the command section in the kafka-connect service in the docker-compose.yml file.

Example: Creating a JDBC Sink Connector

To create a JDBC Sink Connector, use the following command:

curl -i -X PUT -H "Accept:application/json" \
    -H "Content-Type:application/json" http://localhost:8083/connectors/sink-jdbc-mysql-00/config \
    -d '{
          "connector.class"     : "io.confluent.connect.jdbc.JdbcSinkConnector",
          "connection.url"      : "jdbc:mysql://mysql:3306/freetestapi",
          "connection.user"     : "mysqluser",
          "connection.password" : "mysqlpw",
          "topics"              : "freetestapi-books",
          "tasks.max"           : "4",
          "auto.create"         : "true",
          "key.converter"       : "org.apache.kafka.connect.json.JsonConverter",
          "key.converter.schemas.enable": "false",
          "value.converter"     : "org.apache.kafka.connect.json.JsonConverter",
          "value.converter.schemas.enable": "false"
        }'

Cheat sheet for Kafka connect rest APIs

Command Description
curl /connectors Get list of connectors
curl -X POST -H "Content-Type: application/json" /connectors -d 'json' Create a connector
curl /connectors/[name] Get information about a single connector
curl /connectors/[name]/config Get config of a single connector
curl -X PUT -H "Content-Type: application/json" /connectors/[name]/config -d 'json' Create or update a connector config
curl /connectors/[name]/status Get the status of a connector
curl -X POST /connectors/[name]/restart Restart a connector
curl -X PUT /connectors/[name]/pause Pause a connector
curl -X PUT /connectors/[name]/resume Resume a connector
curl -X DELETE /connectors/[name] Delete a connector
curl /connectors/[name]/tasks Get list of tasks for a connector
curl /connectors/[name]/tasks/[number]/status Get the status of a single task
curl -X POST /connectors/[name]/tasks/[number]/restart Restart a single task
curl /connector-plugins/ Get list of plugins

Using kcat examples

kcat (formerly kafkacat) is a command-line utility that you can use to test and debug Apache Kafka® deployments. You can use kcat to produce, consume, and list topic and partition information for Kafka.

  • list topics

     docker exec kafkacat kcat -b broker:29092 -L | grep topic

  • Consuming messages from a topic

    docker exec kafkacat kcat -b broker:29092 -t freetestapi-books -C -J | jq '.'

Notes

The Kafka broker is configured to allow large message sizes (up to 200MB). The MySQL service uses an initialization script to set up the database.