项目作者: blachniet

项目描述 :
Set up 2 local, single-broker Kafka clusters and one MirrorMaker 2.0 instance to replicate topics.
高级语言:
项目地址: git://github.com/blachniet/demo-kafka-mirrormaker-2.git
创建时间: 2020-08-11T01:06:45Z
项目社区:https://github.com/blachniet/demo-kafka-mirrormaker-2

开源协议:

下载


Kafka MirrorMaker 2 Demo

Kafka v2.4.0 introduced MirrorMaker 2.0 (MM2); a new multi-cluster, cross-datacenter replication engine. MirrorMaker 2.0 is well documented in the KIP-382 proposal and the MirrorMaker 2.0 README.md, but I learn-by-doing.

This demo sets up 2 local, single-broker Kafka clusters and one MirrorMaker 2.0 instance to replicate topics from cluster A to cluster B. Once you have this environment set up, it’s quick and easy to play with different configuration options.

Clusters Topic Kafka Port ZooKeeper Port
A t1 9091 2181
B t2 9092 2182

Diagram

Set up

  1. Clone this repository and cd into the directory.

  2. Download a Kafka release and extract it to a ./kafka/.

  3. Clean up, if you’ve run this before.

    1. rm -rf /tmp/zk1
    2. rm -rf /tmp/zk2
    3. rm -rf /tmp/kafka-logs-1
    4. rm -rf /tmp/kafka-logs-2
  4. Run each of the following commands in a separate shell to start your two single-node Kafka clusters (each with their own single-node ZooKeeper instance).

    1. kafka/bin/zookeeper-server-start.sh zk1.properties
    2. kafka/bin/kafka-server-start.sh server1.properties
    3. kafka/bin/zookeeper-server-start.sh zk2.properties
    4. kafka/bin/kafka-server-start.sh server2.properties
  5. Create topics.

    1. kafka/bin/kafka-topics.sh --create --topic t1 --bootstrap-server localhost:9091 --partitions 1 --replication-factor 1
    2. kafka/bin/kafka-topics.sh --create --topic t2 --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
  6. Start MirrorMaker.

    1. kafka/bin/connect-mirror-maker.sh mm2.properties

Common tasks

  1. # List topics
  2. kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9091
  3. kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
  4. # Produce
  5. kafka/bin/kafka-console-producer.sh --topic t1 --bootstrap-server localhost:9091
  6. kafka/bin/kafka-console-producer.sh --topic t2 --bootstrap-server localhost:9092
  7. # Consume
  8. kafka/bin/kafka-console-consumer.sh --topic t1 --bootstrap-server localhost:9091
  9. kafka/bin/kafka-console-consumer.sh --topic t2 --bootstrap-server localhost:9092
  10. # Consumer replicated topic
  11. kafka/bin/kafka-console-consumer.sh --topic A.t1 --bootstrap-server localhost:9092