Kafka+ZooKeeper安装

  • A+
所属分类:Linux

系统环境:Centos 7

软件版本:zooKeeper 3.6.0

 

IP 安装服务
172.17.1.26 ZooKeeper
172.17.1.27 ZooKeeper
172.17.1.28 ZooKeeper,Kafka
172.17.1.29 ZooKeeper,Kafka
172.17.1.30 ZooKeeper,Kafka

一.JDK

yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel

二.ZooKeeper

1.下载zooKeeper

wget https://downloads.apache.org/zookeeper/zookeeper-3.6.0/apache-zookeeper-3.6.0-bin.tar.gz

2.解压并创建目录

tar zxvf apache-zookeeper-3.6.0-bin.tar.gz && mv apache-zookeeper-3.6.0-bin /data/zookeeper && cd /data/zookeeper/ && mkdir -p data logs

3.修改配置文件

cp conf/zoo_sample.cfg conf/zoo.cfg;
sed -i 's#dataDir=/tmp/zookeeper#dataDir=/data/zookeeper/data#' conf/zoo.cfg;
sed -i '13a\dataLogDir=/data/zookeeper/logs' conf/zoo.cfg;
sed -i '$a\server.1=172.17.1.26:2888:3888' conf/zoo.cfg;
sed -i '$a\server.2=172.17.1.27:2889:3889' conf/zoo.cfg;
sed -i '$a\server.3=172.17.1.28:2890:3890' conf/zoo.cfg;
sed -i '$a\server.4=172.17.1.29:2891:3891' conf/zoo.cfg;
sed -i '$a\server.5=172.17.1.30:2892:3892' conf/zoo.cfg;

4.创建启动脚本

tee -a  zookeeper.sh <<-'EOF'
#!/bin/bash
#
#chkconfig: 2345 20 90
#description: start and stop zookeeper
#
CMD=bin/zkServer.sh
start()
{
 $CMD start
 echo "zookeeper is running background..."
}

status()
{
 $CMD status
 echo "zookeeper is running..."
}
stop()
{
 $CMD stop
 echo "zookeeper is stopped"
}
restart()
{
 $CMD restart
 echo "zookeeper is restart"
}
case "$1" in
 start)
 start
 ;;
 stop)
 stop
 ;;
 status)
 status
 ;;
 restart)
 restart
 ;;
 *)
 echo $"Usage: $0 {start|stop|status|restart}"
esac
EOF

添加执行权限

chmod +x zookeeper.sh

5.传输到其他节点

scp -P 65322 -r /data/zookeeper/ user@172.17.1.27:/data

在所有节点依次增加myid文件

echo 1 >> /data/zookeeper/data/myid;
echo 2 >> /data/zookeeper/data/myid;
echo 3 >> /data/zookeeper/data/myid;
echo 4 >> /data/zookeeper/data/myid;
echo 5 >> /data/zookeeper/data/myid;

6.添加环境变量(所有节点都执行)

tee -a /etc/profile <<-'EOF'
ZOOKEEPER=/data/zookeeper
PATH=$PATH:$ZOOKEEPER/bin
EOF

配置生效

source /etc/profile;

7.添加防火墙规则

firewall-cmd --zone=public --add-port=2181/tcp --permanent;
firewall-cmd --zone=public --add-port=2888-2892/tcp --permanent;
firewall-cmd --zone=public --add-port=3888-3892/tcp --permanent;
firewall-cmd --reload;

8.启动服务

/data/zookeeper/zookeeper.sh start;
/data/zookeeper/zookeeper.sh stop;

9.测试

bin/zkCli.sh -server ip:port

三.Kafka

1.下载

wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.4.1/kafka_2.13-2.4.1.tgz;
tar zxvf kafka_2.13-2.4.1.tgz;
mv kafka_2.13-2.4.1 /data/kafka;

2.修改配置文件

vim /data/kafka/config/server.properties
broker.id=1           # 这里是kafka节点名称,不可以重复,在集群内部要唯一
listeners=PLAINTEXT://:9092    #broker的默认端口端口为9092
log.dirs=/data/kafka/kafka-logs
zookeeper.connect=172.17.1.26:2181,172.17.1.27:2181,172.17.1.28:2181,172.17.1.29:2181,172.17.1.30:2181          #zookeeper连接地址,集群以逗号分隔

3.创建目录

mkdir -p /data/kafka/{kafka-logs,logs}

4.启动停止脚本

tee -a /data/kafka/start.sh <<-'EOF'
#/bin/bash
nohup /data/kafka/bin/kafka-server-start.sh /data/kafka/config/server.properties &> /data/kafka/logs/kafka.log &
EOF
tee -a /data/kafka/stop.sh <<-'EOF'
/data/kafka/bin/kafka-server-stop.sh 
EOF
chmod +x /data/kafka/start.sh;
chmod +x /data/kafka/stop.sh;

5.传到其他节点,修改broker.id

scp -P 65322 -r /data/kafka user@172.17.1.28:/data

6.添加防火墙规则

firewall-cmd --zone=public --add-port=9092/tcp --permanent;firewall-cmd --reload;

7.测试创建两个分区,两个副本

/data/kafka/bin/kafka-topics.sh --create --zookeeper 172.17.1.26:2181,172.17.1.27:2181,172.17.1.28:2181,172.17.1.29:2181,172.17.1.30:2181 --replication-factor 2 --partitions 2 --topic kafka

8.查看副本状态

/data/kafka/bin/kafka-topics.sh --describe --zookeeper 172.17.1.26:2181,172.17.1.27:2181,172.17.1.28:2181,172.17.1.29:2181,172.17.1.30:2181 --topic kafka

Kafka+ZooKeeper安装

avatar

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: