Kafka环境搭建及使用教程

admin

kafka里自带了一个zookeeper，可以不需要再去下载zookeeper。在启动kafka前，需要先启动zookeeper。

（1）首先去修改config目录下的zookeeper.properties
! ?: s  k: r1 e3 l% B（2）修改config目录下的server.properties，修改log.dirs和zookeeper.connect。log.dirs是日志存放文件夹，zookeeper.connect是zookeeper连接地址（端口和clientPort保持一致）。
) C8 V! a  B- R0 w) _) J$ D0 o( @$ h( Y
8 q  {% }* D5 T# j9 B' V4 z至此，kafka单机版环境已搭建完毕。（如果对里面的参数没修改的话，可不修改，直接用默认的）
6 p9 c# V! Q$ X0 c0 [1 E
3 h" n* C: L- U% P( c: f

kafka使用教程
（1）启动kafka自带的zookeeper(加了-daemon，可以让这个进程挂在后台进行，否则会发现是个阻塞进程，只能再开一个终端进行其他操作)
3 l9 X, u& S0 L, a( E% T2 M3 g& G5 W" ~bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
% l/ y" v1 W8 J3 g
（2）启动kafka
/ y# l1 T8 a8 b  Pbin/kafka-server-start.sh -daemon config/server.properties
0 D# ^4 e9 _+ h9 v
（3）创建一个topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

9 v& }. s5 n) Z: K
8 @8 `7 t2 {0 a  W* k% `% h0 l（4）查看topic列表
6 M8 c5 b- v. ~! r9 C$ jbin/kafka-topics.sh -list -zookeeper localhost:2181
* r0 }( e) g. s3 d2 [
（5）创建生产者进程  （9092是默认的broker-list端口）
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
" g8 z, X7 |, \( S
（6）创建消费者进程
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

4 }2 L- P  p# }% R+ S' Q（7）外网访问 vi config/server.properties
6 l$ s0 ]6 ]/ e2 E, w0 I5 }- ]1 Padvertised.listeners=PLAINTEXT://192.168.8.65:9092
7 N' u, ?, n  b0 B

admin

cat config/zookeeper.properties
+ X& O& k* s* X7 P! o: j; i* J# Licensed to the Apache Software Foundation (ASF) under one or more
, X) @. Z& r/ M- d0 G' E3 y# contributor license agreements.  See the NOTICE file distributed with
6 ?) e% f8 t+ q# E, a1 @# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# W/ ~" Q! v! M7 x8 V1 E2 F: p; D# the License.  You may obtain a copy of the License at
3 a  i% T) @- k  Q# Q#
1 f( w+ F0 X; _#    http://www.apache.org/licenses/LICENSE-2.0
#
/ b* Y7 R' F  }, R: _# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
9 \: l! r: A/ @& c( l7 q+ k$ M. q# limitations under the License.
# the directory where the snapshot is stored.
) \+ r& C# d3 Z8 v3 VdataDir=/mnt/kafka_2.13-3.3.1/data/zookeeper
# the port at which the clients will connect
, g% \7 {5 [) p) \' F1 _" jclientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
$ X7 S4 w: W4 g3 s. Z5 w. P# Disable the adminserver by default to avoid port conflicts.
2 w* Q! ]/ q9 G6 ~# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080


  q- q# o% F" q3 f! enohup bin/zookeeper-server-start.sh config/zookeeper.properties &

admin

[root@ceph2 kafka_2.13-3.3.1]# egrep -v "^#|^$" config/server.properties
broker.id=0
listeners=PLAINTEXT://192.168.0.233:9092
num.network.threads=3
2 ?) x2 Q# }" H3 n' ?" Fnum.io.threads=8
socket.send.buffer.bytes=102400
0 W5 N1 J+ x' m7 v& Ysocket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
' @! ~7 Z& S8 o- S, r; [8 slog.dirs=/var/log/kafka-logs
9 }& g, P4 s8 u5 u7 @5 z) d5 Znum.partitions=1
num.recovery.threads.per.data.dir=1
5 l5 Q" u6 C$ I  ?offsets.topic.replication.factor=1
1 V0 l# L3 d* {transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
, j5 d- V- w1 ]* D" P  Xlog.retention.hours=168
: L# W8 t% U3 o" a3 ?- A7 l) ilog.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
5 A6 k5 I" e5 U% L3 |7 Szookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

8 ~3 t: k4 ^4 e! v: b/ o3 q5 A
- [+ l3 Q1 y4 x9 r' F4 D0 {8 ^  o2 q[root@ceph2 kafka_2.13-3.3.1]# nohup bin/kafka-server-start.sh config/server.properties &


. D- b. u% t. b7 j2 I
1 d. H( G2 H6 I  S! S: unetstat  -ntlp |grep java
tcp6       0      0 192.168.0.233:9092      :::*                    LISTEN      21222/java         
4 N- R0 j8 [% f0 l  X# k) ntcp6       0      0 :::2181                 :::*                    LISTEN      20780/java         
: p# i) J9 `3 {$ \& Atcp6       0      0 :::41467                :::*                    LISTEN      21222/java         
- C* W5 W4 J! m1 r, G$ r8 _9 }tcp6       0      0 :::40571                :::*                    LISTEN      20780/java