Kafka>> ruby-kafka>> 返回
项目作者： rodolfobandeira

项目描述 ：
Simple test of queue (producer/consumer) using Ruby and Apache Kafka + Zookeeper
高级语言： Ruby
项目地址: git://github.com/rodolfobandeira/ruby-kafka.git
创建时间： 2017-08-03T02:17:26Z
项目社区：https://github.com/rodolfobandeira/ruby-kafka
开源协议：
下载


ruby-kafka

Simple test of queue (producer/consumar) using Ruby and Apache Kafka + Zookeeper

1) Install Zookeeper: sudo apt-get install zookeeper

2) Downloading Apache Kafka:

wget http://apache.parentingamerica.com/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz

3) Installing Kafka:

sudo mkdir /usr/local/kafka

sudo tar -xvf kafka_2.11-0.11.0.0.tgz -C /usr/local/kafka/

4) Special config to make it work on a single node:

sudo vim /usr/local/kafka/kafka_2.11-0.11.0.0/config/server.properties

  1. port = 9092
  2. advertised.host.name = localhost

5) Starting Apache Kafka in background. Notice that you DON’T need sudo.

nohup /usr/local/kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh /usr/local/kafka/kafka_2.11-0.11.0.0/config/server.properties &

6) Let’s create a iptables simple firewall allowing only localhost to connect to our Zookeeper and Kafka. Remember that the idea of Zookeeper and Kafta is having a distributed queue system where you can spread your queue on multiple machines. (Huge stuff brow); So, since the idea here is just study using one single machine on digital ocean, here is how to block h4ck3rs from internet connecting to your toys.

vim iptables.sh

  1. #!/bin/bash
  3. echo "Closing Kafka and Zookeeper ports from external world. Allowing just locally."
  5. iptables -A INPUT -p tcp --dport 2181 -s 127.0.0.0/8 -j ACCEPT
  6. iptables -A INPUT -p tcp --dport 44337 -s 127.0.0.0/8 -j ACCEPT
  7. iptables -A INPUT -p tcp --dport 34232 -s 127.0.0.0/8 -j ACCEPT
  8. iptables -A INPUT -p tcp --dport 9092 -s 127.0.0.0/8 -j ACCEPT
  9. iptables -A INPUT -p tcp --dport 2181 -j DROP
  10. iptables -A INPUT -p tcp --dport 44337 -j DROP
  11. iptables -A INPUT -p tcp --dport 34232 -j DROP
  12. iptables -A INPUT -p tcp --dport 9092 -j DROP

Cleaning iptables before starting our rules. Careful if you have previous rules!! You can check the rules with sudo iptables -L

sudo iptables -F

sudo ./iptables.sh

Now testing using ruby:

gem install ruby-kafka

git clone git@github.com:rodolfobandeira/ruby-kafka.git

cd ruby-kafka

ruby producer.rb (Repete 3 times)

ruby consumer.rb (Yeahh)

  1. Hello, World! 617f4877-e34d-4bab-9993-4f95da626549
  2. 1
  4. Hello, World! f14e791a-092f-4100-b35d-c51d716e5e57
  5. 2
  7. Hello, World! 87c302aa-df8e-41fa-88b5-657148f029d1
  8. 3

Reference:

https://gist.github.com/monkut/07cd1618102cbae8d587811654c92902

https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/

https://stackoverflow.com/questions/35788697/leader-not-available-kafka-in-console-producer

https://github.com/zendesk/ruby-kafka#consuming-messages-from-kafka

https://github.com/rodolfobandeira/ruby-kafka