项目作者: wlu-mstr

项目描述 :
A kafka connect sink specifically from (source) debezium-postgres to (sink) HBase
高级语言: Java
项目地址: git://github.com/wlu-mstr/debezium-postgres-hbase-sink.git
创建时间: 2017-05-12T07:53:31Z
项目社区:https://github.com/wlu-mstr/debezium-postgres-hbase-sink

开源协议:

下载


debezium-postgres-hbase-sink

A kafka connect sink specifically from (source) debezium-postgres to (sink) HBase

Debezium-postgres

PostgreSQL’s logical decoding feature was first introduced in version 9.4 and is a mechanism which allows the extraction of the changes which were commited to the transaction log and the processing of these changes in a user-friendly manner via the help of an output plugin.

Debezium is a great project for Postgres CDC atop of the logical decoding feature(https://github.com/debezium/debezium).

debezium/decoderbufs

Any changes commit to PG tables are captured and the change events are sent to kafka through kafka connect source task. The events are Protobuf formated (https://github.com/debezium/postgres-decoderbufs).

We want to poll the change events out from kafka,’decode’ them and put to HBase.

Goal

So, rather than a generic hbase-sink for kafka connect (such as https://github.com/mravi/kafka-connect-hbase), this project is very specific:

  • it is a hbase sink for kafka connect,
  • BUT it consumes only Debezium-Postgres CDC generated by decoderbufs (github.com/debezium/postgres-decoderbufs)