kafka的三种部署模式
- - CSDN博客云计算推荐文章*kafka 0.8.1.1的安装部署. 相关部署视频地址： http://edu.51cto.com/course/course_id-2374.html. kafka的部署模式为3种模式. 2)单机多broker模式 (伪集群). 3)多机多broker模式 (真正的集群模式). 1.在hadoopdn2机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面.
kafka的三种部署模式
标签：
kafka
模式
| 发表时间：2014-11-18 19:40 | 作者：0flying0
出处：http://blog.csdn.net
/*************
*kafka 0.8.1.1的安装部署
*blog:www.r66r.net
*qq:26571864
kafka的部署模式为3种模式
1)单broker模式
2)单机多broker模式 (伪集群)
3)多机多broker模式 (真正的集群模式)
第一种模式安装
1.在hadoopdn2机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面
2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=0 默认不用修改
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000 默认不用修改
zookeeper.connect=hadoopdn2:2181
3.启动kafka的broker
> bin/kafka-server-start.sh config/server.properties
正常启动如下：
[2014-11-18 10:36:32,196] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.version=2.6.32-220.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.dir=/opt/hadoop/kafka/kafka_2.9.2-0.8.1.1 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,197] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@c2f8b5a (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,231] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,238] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,262] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x349c07dcd7a0002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,266] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-11-18 10:36:32,415] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2014-11-18 10:36:32,422] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-11-18 10:36:32,502] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-11-18 10:36:32,503] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2014-11-18 10:36:32,634] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-11-18 10:36:32,716] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-11-18 10:36:32,887] INFO Registered broker 0 at path /brokers/ids/0 with address JobTracker:9092. (kafka.utils.ZkUtils$)
[2014-11-18 10:36:32,941] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2014-11-18 10:36:33,034] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
4.创建topics
> bin/kafka-topics.sh --create --zookeeper hadoopdn2:2181 --replication-factor 1 --partitions 1 --topic test
查看队列列表
> bin/kafka-topics.sh --list --zookeeper hadoopdn2:2181
查看队列明细
> bin/kafka-topics.sh --describe --zookeeper hadoopdn2:2181 --topic test
Topic[队列]:test PartitionCount[分区数量]:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
第一行是所有Partition的总结。后面的行是每个partition一行。
查看帮助文档
> bin/kafka-topics.sh --help 查看与topics相关的指令
第二种模式部署：
1.为第二个broker创建server的配置文件
> cp server.properties server1.properties
2.修改server1.properties
broker.id=1
port=9093
log.dirs=/opt/hadoop/kafka/kafka-logs-server1
zookeeper.connect=hadoopdn2:2181
3.启动kafka的broker
> nohup bin/kafka-server-start.sh config/server1.properties &
4.通过zookeeper的客户端可以查看当前的broker
[zk: hadoopdn2:2181(CONNECTED) 7] ls /
[zookeeper, admin, consumers, config, controller, brokers, controller_epoch]
[zk: hadoopdn2:2181(CONNECTED) 8] ls /brokers
[topics, ids]
[zk: hadoopdn2:2181(CONNECTED) 9] ls /brokers/ids
[1, 0]
5.查看队列情况
$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
6.修改test队列的参数
$ bin/kafka-topics.sh --zookeeper hadoopdn2:2181 --partitions 3 --topic test --alter
$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0[在broker0上面] Isr: 0
Topic: test Partition: 1 Leader: 1 Replicas: 1[在broker1上面] Isr: 1
Topic: test Partition: 2 Leader: 0 Replicas: 0[在broker0上面] Isr: 0
第三种部署方式：
1.在hadoopdn3机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面
2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=2 必须修改保证每个broker的ID唯一
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000 默认不用修改
zookeeper.connect=hadoopdn2:2181
3.通过zookeeper的客户端查看
[zk: hadoopdn2:2181(CONNECTED) 10] ls /brokers/ids
[2, 1, 0]
broker的id为2的已经注册到zookeeper上面了
到此为止，kafka的部署模式已经完整。
*kafka 0.8.1.1的安装部署
*blog:www.r66r.net
*qq:26571864
**************/
相关部署视频地址： http://edu.51cto.com/course/course_id-2374.html
kafka的部署模式为3种模式
1)单broker模式
2)单机多broker模式 (伪集群)
3)多机多broker模式 (真正的集群模式)
第一种模式安装
1.在hadoopdn2机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面
2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=0 默认不用修改
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000 默认不用修改
zookeeper.connect=hadoopdn2:2181
3.启动kafka的broker
> bin/kafka-server-start.sh config/server.properties
正常启动如下：
[2014-11-18 10:36:32,196] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.version=2.6.32-220.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.dir=/opt/hadoop/kafka/kafka_2.9.2-0.8.1.1 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,197] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@c2f8b5a (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,231] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,238] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,262] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x349c07dcd7a0002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,266] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-11-18 10:36:32,415] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2014-11-18 10:36:32,422] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-11-18 10:36:32,502] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-11-18 10:36:32,503] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2014-11-18 10:36:32,634] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-11-18 10:36:32,716] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-11-18 10:36:32,887] INFO Registered broker 0 at path /brokers/ids/0 with address JobTracker:9092. (kafka.utils.ZkUtils$)
[2014-11-18 10:36:32,941] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2014-11-18 10:36:33,034] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
4.创建topics
> bin/kafka-topics.sh --create --zookeeper hadoopdn2:2181 --replication-factor 1 --partitions 1 --topic test
查看队列列表
> bin/kafka-topics.sh --list --zookeeper hadoopdn2:2181
查看队列明细
> bin/kafka-topics.sh --describe --zookeeper hadoopdn2:2181 --topic test
Topic[队列]:test PartitionCount[分区数量]:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
第一行是所有Partition的总结。后面的行是每个partition一行。
查看帮助文档
> bin/kafka-topics.sh --help 查看与topics相关的指令
第二种模式部署：
1.为第二个broker创建server的配置文件
> cp server.properties server1.properties
2.修改server1.properties
broker.id=1
port=9093
log.dirs=/opt/hadoop/kafka/kafka-logs-server1
zookeeper.connect=hadoopdn2:2181
3.启动kafka的broker
> nohup bin/kafka-server-start.sh config/server1.properties &
4.通过zookeeper的客户端可以查看当前的broker
[zk: hadoopdn2:2181(CONNECTED) 7] ls /
[zookeeper, admin, consumers, config, controller, brokers, controller_epoch]
[zk: hadoopdn2:2181(CONNECTED) 8] ls /brokers
[topics, ids]
[zk: hadoopdn2:2181(CONNECTED) 9] ls /brokers/ids
[1, 0]
5.查看队列情况
$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
6.修改test队列的参数
$ bin/kafka-topics.sh --zookeeper hadoopdn2:2181 --partitions 3 --topic test --alter
$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0[在broker0上面] Isr: 0
Topic: test Partition: 1 Leader: 1 Replicas: 1[在broker1上面] Isr: 1
Topic: test Partition: 2 Leader: 0 Replicas: 0[在broker0上面] Isr: 0
第三种部署方式：
1.在hadoopdn3机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面
2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=2 必须修改保证每个broker的ID唯一
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000 默认不用修改
zookeeper.connect=hadoopdn2:2181
3.通过zookeeper的客户端查看
[zk: hadoopdn2:2181(CONNECTED) 10] ls /brokers/ids
[2, 1, 0]
broker的id为2的已经注册到zookeeper上面了
到此为止，kafka的部署模式已经完整。
作者：0flying0 发表于2014-11-18 11:40:17 原文链接
阅读：101 评论：0 查看评论
相关 [kafka 模式] 推荐：
使用 Kafka、Debezium 和 Kubernetes 实现应用现代化的模式
- - InfoQ - 促进软件开发领域知识与创新的传播本文最初发表于 RedHat 的开发者站点，经原作者 Bilgin Ibryam 许可，由 InfoQ 中文站翻译分享. “我们建造计算机的方式与建造城市的方式是一样的，那就是随着时间的推移，依然毫无计划，并且要建造在废墟之上. Ellen Ullman 在 1998 年写下了这样一句话，但它今天依然适用于我们构建现代应用程序的方式，那就是，随着时间的推移，我们要在遗留的软件上构建应用，而且仅仅有短期的计划.
kafka监控之kafka-run-class.sh
- - 开源软件 - ITeye博客kafka自带了很多工具类，在源码kafka.tools里可以看到：. 这些类该如何使用呢，kafka的设计者早就为我们考虑到了，在${KAFKA_HOME}/bin下，有很多的脚本，其中有一个kafka-run-class.sh，通过这个脚本，可以调用其中的tools的部分功能，如调用kafka.tools里的ConsumerOffsetChecker.scala,.
闲扯kafka mq
- - 开源软件 - ITeye博客本文主要讲解关于kafka mq的设计思想及个人理解. 关于kafka的详细信息，大家可以参考官网的文献 http://kafka.apache.org/documentation.html这是一篇相当不错的文章，值得仔细研读. 第一个问题：消息队列（Message Queue）是干嘛用的. 首先，要对消息队列有一个基本的理解.
Kafka优化
- - ITeye博客配置优化都是修改server.properties文件中参数值. 1.网络和io操作线程配置优化. # broker处理消息的最大线程数. # broker处理磁盘IO的线程数. 一般num.network.threads主要处理网络io，读写缓冲区数据，基本没有io等待，配置线程数量为cpu核数加1.
Kafka Connect简介
- - 鸟窝Kafka 0.9+增加了一个新的特性 Kafka Connect,可以更方便的创建和管理数据流管道. 它为Kafka和其它系统创建规模可扩展的、可信赖的流数据提供了一个简单的模型，通过 connectors可以将大数据从其它系统导入到Kafka中，也可以从Kafka中导出到其它系统. Kafka Connect可以将完整的数据库注入到Kafka的Topic中，或者将服务器的系统监控指标注入到Kafka，然后像正常的Kafka流处理机制一样进行数据流处理.
kafka consumer group offset
- - 开源软件 - ITeye博客 kafka0.9及以前版本kafka offset 保存在zookeeper， 因频繁读写zookeeper性能不高；从0.10开始，主题分区offset存储于kafka独立主题中. 管理监控kafka主题及分区offset至关重要，原网上很开源流行工具KafkaOffsetMonitor、kafka-manager，旧版offset保存于zookeeper，kafka consumer无相应API，从kafka0.10.1.1以后提供相应API读取主题分区offset（也可以调用KafkaClient API，kafka管理API由scala语言编写）.
Kafka设计解析（二）：Kafka High Availability （上）
- -Kafka在0.8以前的版本中，并不提供High Availablity机制，一旦一个或多个Broker宕机，则宕机期间其上所有Partition都无法继续提供服务. 若该Broker永远不能再恢复，亦或磁盘故障，则其上数据将丢失. 而Kafka的设计目标之一即是提供数据持久化，同时对于分布式系统来说，尤其当集群规模上升到一定程度后，一台或者多台机器宕机的可能性大大提高，对Failover要求非常高.
GitHub - andreas-schroeder/kafka-health-check: Health Check for Kafka Brokers.
- -At AutoScout24, to keep the OS up to date of our clusters running on AWS, we
perform regular in-place rolling updates. As we run immutable servers, we
terminate each broker and replace them with fresh EC2 instances (keeping the
previous broker ids).
Kafka编程实例
- - CSDN博客云计算推荐文章 Producer是一个应用程序，它创建消息并发送它们到Kafka broker中. 这些producer在本质上是不同. 比如，前端应用程序，后端服务，代理服务，适配器对于潜在的系统，Hadoop对于的Producer. 这些不同的Producer能够使用不同的语言实现，比如java、C和Python.