Kafka docker builds

The no-surprises Kafka docker image. Public build at solsson/kafka, with 100k+ pulls to date.

Design goals:

• Transparent build: self-contained in Dockerfile && directly from source.
• Recommend use of image SHAs for security and stability (#11).
• Same basic platform choices as the more thoroughly validated Confluent Platform distribution (#5, #9).
• Supports the other tools bundled with Kafka distributions - Zookeeper, topic admin, Connect & Streams (#7).
• Encourage conventions to clearly show all non-default config, to help Kafka beginners.
• Avoid recommending --net=host as such practices are impractical in multi-node environments.
• Support Kubernetes; transparent and tweakable cluster setups like Yolean/kubernetes-kafka.

How to use

The default entrypoint docker run solsson/kafka will list "bin" scripts and sample config files. Make a guess like docker run --entrypoint ./bin/kafka-server-start.sh solsson/kafka or docker run --entrypoint ./bin/kafka-topics.sh solsson/kafka to see tool-specific help.

You most likely need to mount your own config files, or for ./bin/kafka-server-start.sh use overrides like:

  --override zookeeper.connect=zookeeper:2181
  --override log.dirs=/var/lib/kafka/data/topics
  --override log.retention.hours=-1
  --override broker.id=0
  --override advertised.listener=PLAINTEXT://kafka-0:9092

Beware of log4j.properties' location if you mount config. Kafka's bin scripts will guess path unless you set a KAFKA_LOG4J_OPTS env.

We avoid environment variable rules for config override, used in wurstmeister and cp-kafka, because bin scripts are quite heavy on env use anyway so you'd get a toxic mix. Also for cluster setups to share config across instances we tend to need bash tricks. See for example this gotcha and the overrides needed for external access.

Upgrade from pre 0.11 images

Earlier images used ./bin/kafka-server-start.sh as entrypoint and had the zookeeper.connect=zookeeper:2181 (instead of localhost:2181) built in. At upgrade use the command recommended above to restore that functionality.

Build and test locally

To build your own kafka image simply run docker build ./kafka.

When we develop locally --- stream processing images, monitoring, compliance with kubernetes-kafka etc --- we use a build-contract.

Build and test using: docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/:/source solsson/build-contract test. However... while timing issues remain you need some manual intervention:

compose='docker-compose -f build-contracts/docker-compose.yml'
$compose up -d zookeeper kafka-0
$compose logs zookeeper kafka-0
# can we create topics using the image's provided script?
$compose up test-topic-create-1
# can a producer send messages using snappy (has issues before with a class missing in the image)
$compose up test-snappy-compression
$compose up test-consume-all
# demo the log/file aggregation image
docker-compose -f build-contracts/docker-compose.files-aggregation.yml up
# demo the JMX->kafka image
docker-compose -f build-contracts/docker-compose.monitoring.yml up

Why is the repo named dockerfiles?

This repo used to contain misc dockerfiles, but they've moved to separate repositories for dockerization projects. We've kept the repository name to avoid breaking the automated build of solsson/kafka in Docker Hub.

For legacy Dockerfiles from this repo (if you navigated to here from a Docker Hub solsson image), see https://github.com/solsson/dockerfiles/tree/misc-dockerfiles.

Native builds

Very experimental.

NOPUSH=true IMAGE_NAME=solsson/kafka:nativeagent ./hooks/build
docker-compose -f native/docker-compose.yml down
docker run --rm --entrypoint chown -v $(pwd)/native/configs:/configs busybox -R $(id -u) /configs
NOPUSH=true IMAGE_NAME=solsson/kafka:native ./hooks/build

To test the native images reuse the usecases script:

for build in kafka-topics kafka-configs kafka-consumer-groups zookeeper-server-start; do
  docker tag solsson/kafka:native-$build solsson/kafka:nativeagent-$build
done
./native/native-usecases.sh
docker-compose -f native/docker-compose.yml down
git restore --source=HEAD --staged --worktree -- native/configs/
# The cli image should simply combine the supported commands
docker run --rm --entrypoint sh solsson/kafka:native-cli -c 'ls ./bin/'