kafka quick start
Kafka是由LinkedIn开发的一个分布式的消息系统，使用Scala编写
kafka是基于zookeeper的，在启动kafka之前要先启动zookeeper，kafka默认自带zookeeper，不需要自带的可以自行安装。
1.下载安装
官网下载，解压到 /usr/local下
shifulong kafka_2.9.1-0.8.2.2$ pwd /usr/local/kafka_2.9.1-0.8.2.2 shifulong kafka_2.9.1-0.8.2.2$ ll total 32 drwxr-xr-x@ 8 shifulong staff 272 10 12 10:37 ./ drwxrwxr-x 23 root admin 782 10 12 10:17 ../ -rw-r--r--@ 1 shifulong staff 11358 9 3 09:34 LICENSE -rw-r--r--@ 1 shifulong staff 162 9 3 09:34 NOTICE drwxr-xr-x@ 21 shifulong staff 714 9 3 09:44 bin/ drwxr-xr-x@ 12 shifulong staff 408 10 15 14:16 config/ drwxr-xr-x@ 18 shifulong staff 612 9 3 09:44 libs/ drwxr-xr-x 125 shifulong staff 4250 10 15 14:03 logs/
2.启动zookeeper
shifulong kafka_2.9.1-0.8.2.2$ bin/zookeeper-server-start.sh config/zookeeper.properties
3.启动kafka
由于到单机模拟4台kafka，需要启动4个server
config/server.properties 拷贝四分，做如下修改
broker.id全局唯一，从0开始
config/server.properties: //默认配置
broker.id=0
port=9093
log.dir=/tmp/kafka-logs-1
config/server-1.properties:
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
config/server-3.properties:
broker.id=3
port=9095
log.dir=/tmp/kafka-logs-3启动4个broker
shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server.properties shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server-1.properties shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server-2.properties shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server-3.properties
4.创建topic
- 1.假设实验环境中Kafka集群只有一个broker，log.dirs=/tmp/kafka-logs
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic test_single_broker
执行完创建命令后，在/tmp/kafka-logs 目录下 会自动创建四个partition，以topic名称+编号命名(从0开始)
drwxr-xr-x 4 shifulong wheel 136 10 15 14:36 test_single_broker-0/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:36 test_single_broker-1/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:36 test_single_broker-2/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:36 test_single_broker-3/
- 2.假设实验环境中Kafka集群只有4个broker，
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 5 --topic test_multi_broker
/tmp/kafka-logs 目录下
drwxr-xr-x 4 shifulong wheel 136 10 15 14:43 test_multi_broker-2/
/tmp/kafka-logs-1 目录下
drwxr-xr-x 4 shifulong wheel 136 10 15 14:43 test_multi_broker-3/
/tmp/kafka-logs-2 目录下
drwxr-xr-x 4 shifulong wheel 136 10 15 14:43 test_multi_broker-0/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:43 test_multi_broker-4/
/tmp/kafka-logs-3 目录下
drwxr-xr-x 4 shifulong wheel 136 10 15 14:43 test_multi_broker-1/
上面的测试证明了如下问题
在集群环境中通一个topic的多个partition的均匀的在每个broker上创建，
- 3.多broker 并且 replication-factor > 1 的情况
与上面的例子不同的是，--replication-factor 2的参数是2(创建1个副本)
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 5 --topic test_multi_replication_topic
/tmp/kafka-logs
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-2/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-3/
/tmp/kafka-logs-1
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-0/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-3/
/tmp/kafka-logs-2
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-0/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-1/
/tmp/kafka-logs-3
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-1/
drwxr-xr-x 4 shifulong wheel 136 10 15 14:54 test_multi_repli_topic-2/
可以看到每个partition创建了2此在不同的broker上
并且创建是有一定规律的
有一篇较为详细的博客介绍分配的规则：
Kafka集群partition replication自动分配分析 http://blog.csdn.net/lizhitao/article/details/41778193
5.发消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test_multi_replication_topic
6.消费消息
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test_multi_replication_topic --from-beginning