Kafka streaming with Spark and Flink example

Example project on how to use Apache Kafka and streaming consumers, namely:

Producer sending random number words to Kafka
Consumer using Kafka to output received messages
Streaming Consumer using Apache Spark to count words occurrences
Streaming Consumer using Apache Flink to count words occurrences

Requirements

Docker
Docker Compose
Java 8
Maven

Build

Build Java project
```
 mvn clean package
```

Build Docker image

 docker build -t kafka-spark-flink-example .

Run

Start docker containers
```
 docker-compose up -d
```

Check

Check producer logs

 docker logs kafka-spark-flink-example_kafka-producer_1 -f

Output should be similar to:

 [main] INFO  org.davidcampos.kafka.producer.KafkaProducerExample - Sent (74b23319-084c-4309-80a7-c0d6f107a092, eight) to topic example @ 1525127107909.

Check consumer with Spark logs

 docker logs kafka-spark-flink-example_kafka-consumer-spark_1 -f

Output should be similar to:

 (two,3)
 (one,3)
 (nine,5)
 (six,8)
 (three,2)
 (five,2)
 (four,9)
 (seven,3)
 (eight,6)
 (ten,6)

Check consumer with Flink logs

 docker logs kafka-spark-flink-example_kafka-consumer-flink_1 -f

Output should be similar to:

 1> (ten,85)
 4> (nine,104)
 1> (ten,86)
 4> (five,91)
 4> (one,94)
 4> (six,90)
 1> (three,89)
 4> (six,91)
 4> (five,92)

Kafka Web UI

Kafka Manager Web UI available at http://localhost:9000.

Spark Web UI

Spark Web UI available at http://localhost:4040.

Stop

Stop docker containers
```
 docker-compose down
```