Kafka with KRaft

[iblutrifork]

Add a docker compose for local development, which starts a Kafka instance running KRaft, instead of Zookeeper.

This PR uses the Kafka docker image compiled from https://github.com/trifork/cheetah-infrastructure-utils-kafka/pull/77.

This new docker compose runs a Kafka in KRaft mode in both controller and broker mode combined.
Controller mode is when Kafka acts as Zookeeper (runs the consensus algorithm).
Broker mode is when Kafka acts like a message queue (the way it always was).

Changes from the way we ran Kafka using Zookeeper:

• Added a new listener for the KRaft controller: CONTROLLER://kafka:9095
• The listener did not work with lowercase letters so I used capital letters
• Removed the specific zookeeper options from server.properties
• Added KRaft section to the server.properties
• The cluster ID is something random generated following https://sleeplessbeastie.eu/2021/10/22/how-to-generate-kafka-cluster-id/
• The docker file uses an image built from https://github.com/trifork/cheetah-infrastructure-utils-kafka/pull/77
• The execute command starts by formatting a directory for the KRaft metadata bin/kafka-storage.sh format ...
• Renamed server.properties to kraft.properties

[github-actions]

🦙 MegaLinter status: ⚠️ WARNING

Descriptor	Linter	Files	Fixed	Errors	Elapsed time
✅ BASH	shellcheck	5		0	0.24s
⚠️ BASH	shfmt	5		1	0.03s
✅ COPYPASTE	jscpd	yes		no	1.08s
✅ DOCKERFILE	hadolint	1		0	0.09s
✅ JSON	jsonlint	4		0	0.33s
⚠️ JSON	prettier	4		1	2.03s
✅ JSON	v8r	4		0	2.71s
✅ REPOSITORY	git_diff	yes		no	0.01s
✅ REPOSITORY	grype	yes		no	15.17s
✅ REPOSITORY	trivy	yes		no	7.79s
✅ REPOSITORY	trivy-sbom	yes		no	5.93s
✅ REPOSITORY	trufflehog	yes		no	4.3s
⚠️ YAML	prettier	19		1	1.94s
✅ YAML	v8r	19		0	6.02s
⚠️ YAML	yamllint	19		1	1.0s

See detailed report in MegaLinter reports

MegaLinter is graciously provided by

[cthtrifork]

Great work. Did you verify it together with redpanda as well?

Also with OAuth2 please:
https://github.com/trifork/cheetah-development-infrastructure/blob/main/docker-compose/kafka.yaml#L106C10-L106C131

[iblutrifork]

Great work. Did you verify it together with redpanda as well?

Also with OAuth2 please: https://github.com/trifork/cheetah-development-infrastructure/blob/main/docker-compose/kafka.yaml#L106C10-L106C131

Tested with both by creating a topic and writing a message to it through the redpanda UI