Kafka Docker Image

This project is meant to create an optimised docker image to run kafka containers as ‘statefulset’ into kubernetes/openshift.

Obviously, the docker image can be used locally for testing or development purposes.

Build and publish a kafka docker image

To get a docker image ready with default values type:

$ make clean-all docker-build docker-test docker-push

To get your own image:

$ export KAFKA_HOME="/opt/kafka"
$ export SCALA_VERSION="2.13"
$ export KAFKA_VERSION="2.5.0"
$ docker build --build-arg SCALA_VERSION=$SCALA_VERSION --build-arg KAFKA_VERSION=$KAFKA_VERSION --build-arg KAFKA_HOME=$KAFKA_HOME \
-t your-org/kafka:${SCALA_VERSION}-${KAFKA_VERSION} .

NOTE: build-args are optional arguments if you want different values from default ones in the Dockerfile

The built docker image will contain a kafka distribution (${SCALA_VERSION}-${KAFKA_VERSION}) under the directory $KAFKA_HOME.

The provided scripts are:

• kafka_download.sh : This script is used to download the suitable release.
• kafka_env.sh : It purpose is load the default environments variables.
• kafka_setup.sh : Configure kafka and zookeeper dynamically , based on utils-docker project
• kafka_server.sh : A central script to manage kafka and optional zookeeper processes.
• kafka_server_status.sh : Checks kafka server status.

Public docker images are available HERE

Getting started with a single docker container locally

The example bellow shows you how to run an all-in-one docker kafka container (with zookeeper as internal sidecar):

$ docker run -it -p 9092:9092 -p 2181:2181 \
  -e "SETUP_DEBUG=true" \
  -e "SERVER_advertised_listeners=PLAINTEXT://localhost:9092" \
  -e "SERVER_listener_security_protocol_map=PLAINTEXT:PLAINTEXT" \
  -e "SERVER_listeners=PLAINTEXT://0.0.0.0:9092" \
  -h kafka engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION}
Writing environment variables to file :
PREFIX           : SERVER_
DEST_FILE        : /opt/kafka/config/server.properties
EXCLUSIONS       :
CREATE_FILE      : true
OVERRIDE         : true
FROM_SEPARATOR   : _
TO_SEPARATOR     : .
LOWER            : true
.......................................
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : SERVER_log_dirs --> log.dirs=/opt/kafka/logs
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : SERVER_zookeeper_connect --> zookeeper.connect=localhost:2181
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : SERVER_broker_id --> broker.id=-1
.......................................
Writing environment variables to file :
PREFIX           : ZK_
DEST_FILE        : /opt/kafka/config/zookeeper.properties
EXCLUSIONS       :
CREATE_FILE      : true
OVERRIDE         : true
FROM_SEPARATOR   : _
TO_SEPARATOR     : .
LOWER            : false
.......................................
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : ZK_dataDir --> dataDir=/opt/kafka/zookeeper/data
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : ZK_clientPort --> clientPort=2181
[DEBUG] [2017-01-31_20:17:26] -  [  ADD   ] : ZK_dataLogDir --> dataLogDir=/opt/kafka/zookeeper/data-log
...
[2017-01-31 20:17:28,150] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-01-31 20:17:28,308] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x159f62cc8c00000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
...
[2017-01-31 20:17:29,646] INFO Kafka version : 2.3.0 (org.apache.kafka.common.utils.AppInfoParser)
[2017-01-31 20:17:29,646] INFO Kafka commitId : f10ef2720b03b247 (org.apache.kafka.common.utils.AppInfoParser)
[2017-01-31 20:17:29,647] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

NOTE: We’ve passed a SETUP_DEBUG environment variable (SETUP_DEBUG=true) to view the setup process details.

Setting up

Users can provide parameters to config files just adding environment variables with specific name patterns.

This table collects the patterns of variable names which will are written in each file:

So we can configure our kafka server via environment variables directly:

$ docker run -it -d -e "LOG4J_log4j_rootLogger=DEBUG, stdout" -e "SERVER_log_retention_hours=24"\
engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION}

Also you may use --env-file option to load these variables from a file.

And, of course, you could provide your own property files directly by option -v with the suitable properties files.

The override option of kafka server is preserved and anybody can use it on this way:

$ docker run -it \
  -e "SETUP_DEBUG=true" \
  -h kafka engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION} \
 /bin/bash -c "kafka_server.sh start --override advertised.host.name=kafka"
 [2017-02-04 19:06:10,504] INFO KafkaConfig values:
    advertised.host.name = kafka
...
[2017-02-04 19:06:11,693] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

Run local zookeeper

By default a zookeeper process is started too, as we said previously.
This behaviour is managed by the env variable KAFKA_ZK_LOCAL (defaults to “true”).

External zookeeper

If you want to deploy a kafka server without a local zookeeper then you should provide these env values:

• KAFKA_ZK_LOCAL=false
• SERVER_zookeeper_connect=\[,\]

For instance:

$ docker run -it \
  -e "KAFKA_ZK_LOCAL=false" \
  -e "SERVER_zookeeper_connect=zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181" \
  engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION}

Kubernetes

In k8s directory there are some examples and utilities for Kubernetes

Openshift

In openshift directory there are some resources for Openshift.

Extra Dockers

Another great kafka docker images can be found at:

• https://hub.docker.com/r/spotify/kafka/
• https://hub.docker.com/r/wurstmeister/kafka

Author

Enrique Garcia engapa@gmail.com