Kafka>> demo-kafka-mirrormaker-2>> 返回
项目作者： blachniet

项目描述 ：
Set up 2 local, single-broker Kafka clusters and one MirrorMaker 2.0 instance to replicate topics.
高级语言：
项目地址: git://github.com/blachniet/demo-kafka-mirrormaker-2.git
创建时间： 2020-08-11T01:06:45Z
项目社区：https://github.com/blachniet/demo-kafka-mirrormaker-2
开源协议：
下载


Kafka MirrorMaker 2 Demo

Kafka v2.4.0 introduced MirrorMaker 2.0 (MM2); a new multi-cluster, cross-datacenter replication engine. MirrorMaker 2.0 is well documented in the KIP-382 proposal and the MirrorMaker 2.0 README.md, but I learn-by-doing.

This demo sets up 2 local, single-broker Kafka clusters and one MirrorMaker 2.0 instance to replicate topics from cluster A to cluster B. Once you have this environment set up, it’s quick and easy to play with different configuration options.

Clusters Topic Kafka Port ZooKeeper Port
A t1 9091 2181
B t2 9092 2182

Diagram

Set up

  1. Clone this repository and cd into the directory.

  2. Download a Kafka release and extract it to a ./kafka/.

  3. Clean up, if you’ve run this before.

    1.  rm -rf /tmp/zk1
    2.  rm -rf /tmp/zk2
    3.  rm -rf /tmp/kafka-logs-1
    4.  rm -rf /tmp/kafka-logs-2

  4. Run each of the following commands in a separate shell to start your two single-node Kafka clusters (each with their own single-node ZooKeeper instance).

    1.  kafka/bin/zookeeper-server-start.sh zk1.properties
    2.  kafka/bin/kafka-server-start.sh server1.properties
    3.  kafka/bin/zookeeper-server-start.sh zk2.properties
    4.  kafka/bin/kafka-server-start.sh server2.properties

  5. Create topics.

    1.  kafka/bin/kafka-topics.sh --create --topic t1 --bootstrap-server localhost:9091 --partitions 1 --replication-factor 1
    2.  kafka/bin/kafka-topics.sh --create --topic t2 --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

  6. Start MirrorMaker.

    1.  kafka/bin/connect-mirror-maker.sh mm2.properties

Common tasks

  1. # List topics
  2. kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9091
  3. kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
  5. # Produce
  6. kafka/bin/kafka-console-producer.sh --topic t1 --bootstrap-server localhost:9091
  7. kafka/bin/kafka-console-producer.sh --topic t2 --bootstrap-server localhost:9092
  9. # Consume
  10. kafka/bin/kafka-console-consumer.sh --topic t1 --bootstrap-server localhost:9091
  11. kafka/bin/kafka-console-consumer.sh --topic t2 --bootstrap-server localhost:9092
  13. # Consumer replicated topic
  14. kafka/bin/kafka-console-consumer.sh --topic A.t1 --bootstrap-server localhost:9092