项目作者: srigumm

项目描述 :
Using "Debezium" Kafka CDC connector plugin to source data from MongoDB Cluster into KAFKA topics.
高级语言: Shell
项目地址: git://github.com/srigumm/Mongo-To-Kafka-CDC.git
创建时间: 2019-05-22T14:07:47Z
项目社区:https://github.com/srigumm/Mongo-To-Kafka-CDC

开源协议:

下载


KAFKA CDC Source Connector - using “Debezium CDC Connector “ to import mongodb data into kafka topics

Debezium CDC Connector for MongoDB:

A Kafka CDC connector to source data from MongoDB Cluster, available as part of confluent connector plugins
https://www.confluent.io/connector/debezium-mongodb-cdc-connector/

**For more details about the plugin owners,visit https://debezium.io/docs/connectors/mongodb/

How to run this connector in local docker container?

1: Start your kafka server

2: Standup Mongodb clusters(replicaset) and update the below setting in scripts/cdcsetup.sh

  1. - "mongodb.hosts": "rs0/mongo1:27017",
  2. - "mongodb.name": "rawmaterial",
  3. - "database.whitelist": "inventory" (**your database here **)
  4. Note: you can use our docker-compose file to spinup a new mango cluster along with connectors.
  5. Path: src/kafka-mongo-import-export/scripts/mongoclustersetup.sh
  6. Note: cdcsetup.sh has a curl POST call that issues a "create new connector" instruction to the existing connector process(running on 8083)

3: Build docker images&containers in local docker repository

  1. - cd <<root>> i.e cd eCupcakesFactoryOnContainers
  2. - Run => "docker-compose up"
  3. Note: our docker-compose file would standup a new mongodb replicaset, starts the kafka connector process(on 8083) and runs a source connector for importing mongodata.

4: Verify whether the connector works or not.

  1. Create some collections in the configured mongodb(i.e inventory as per the step2) cluster(on primary node):
  2. use inventory
  3. db.ingredients.insert({"Id":132,"Flavour":"Cookies","Quantity":11})
  4. verify kafka if a new topic topic got created:
  5. When this connector runs successfully, it creates new topics in kafa with the below name format:
  6. <<logicalname>>.<<dbname>>.<<collectioname>>
  7. ex: as per the configuration in step2, new topic name that gets created is - rawmaterial.inventory.<<collectioname>>

5. Cleanup => Kill your containers when you are done.

  1. docker-compose down
  2. docker-compose rm