将设为首页浏览此站
开启辅助访问 天气与日历 收藏本站联系我们切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 35|回复: 2
收起左侧

Kafka环境搭建及使用教程

[复制链接]
发表于 2023-1-3 10:58:57 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
kafka里自带了一个zookeeper,可以不需要再去下载zookeeper。在启动kafka前,需要先启动zookeeper。4 z# U7 T( ?1 w9 Q$ ~  M& q. K/ X

$ @, y  \% t' W4 J/ \5 T& P% \(1)首先去修改config目录下的zookeeper.properties
  b/ d4 W) b7 s* }8 n(2)修改config目录下的server.properties,修改log.dirs和zookeeper.connect。log.dirs是日志存放文件夹,zookeeper.connect是zookeeper连接地址(端口和clientPort保持一致)。
. |6 S) r9 s. z
- A) F5 x. D" u4 C至此,kafka单机版环境已搭建完毕。(如果对里面的参数没修改的话,可不修改,直接用默认的). K  V1 ?- {1 p
! J& s6 h  h; c$ g/ [1 i

9 _$ E% ]$ \8 M9 C% n0 n+ k" m2 }9 _; A  V5 Z1 W; j! |6 m/ |6 d
kafka使用教程
/ K! z+ q3 c! y: q, P. D, M(1)启动kafka自带的zookeeper(加了-daemon,可以让这个进程挂在后台进行,否则会发现是个阻塞进程,只能再开一个终端进行其他操作)# a" U  n' {5 a" R
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties+ k8 P/ d1 O5 Z
2 L8 h& ?, ^) h6 B
(2)启动kafka
6 P3 ~4 S5 E& p, vbin/kafka-server-start.sh -daemon config/server.properties; P/ v0 X* S7 X6 k
* z9 M1 k; E; o7 n
(3)创建一个topic( v9 B5 n/ x7 x5 ?3 ^5 q9 r
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
, }% q8 m9 y) k- n
' G) [5 U! u( L
+ P; x  A4 T+ e& P- |+ M(4)查看topic列表  {  A8 k* g; A2 L1 D
bin/kafka-topics.sh -list -zookeeper localhost:2181/ o' L& ?# v! {% A' W7 u0 ?

1 q) @" M( p( g2 h7 s; p+ |(5)创建生产者进程  (9092是默认的broker-list端口)
$ h2 J8 Q) }& bbin/kafka-console-producer.sh --broker-list localhost:9092 --topic test+ d7 J5 a( y0 |
3 x( Q+ g  Z& l- J9 ~
(6)创建消费者进程/ v1 @$ Q% c, ~$ @
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  _7 k4 ^. }3 b- w* M' E' j
6 R- x: H" C3 G+ m, t+ H(7)外网访问 vi config/server.properties+ K8 p& J0 }& B
advertised.listeners=PLAINTEXT://192.168.8.65:9092
0 E+ l$ r/ K4 V
 楼主| 发表于 2023-1-3 11:28:57 | 显示全部楼层
cat config/zookeeper.properties - y# f/ N4 j& n
# Licensed to the Apache Software Foundation (ASF) under one or more
, X) J* U4 z! i) [$ _* Q# contributor license agreements.  See the NOTICE file distributed with) G0 `( D% }3 T5 a. I  j' J
# this work for additional information regarding copyright ownership.5 j2 }! a8 w% T% Z1 U0 L
# The ASF licenses this file to You under the Apache License, Version 2.0
& Y2 A2 F1 t; Y9 Y# (the "License"); you may not use this file except in compliance with
* c8 o7 _0 W! U5 g! X' E, X2 ~# the License.  You may obtain a copy of the License at" s# ^: U, M1 [9 ?! d$ |0 d0 g
#
0 G( J! T4 [" G( Z5 ]) n1 H! F#    http://www.apache.org/licenses/LICENSE-2.06 Z. C+ p' C7 a- _* X$ a! n9 a
# 0 T. J3 z; ^( `# c/ n  B
# Unless required by applicable law or agreed to in writing, software
  g+ Y' N) o! L! j: Z2 d3 v5 e* [( U# distributed under the License is distributed on an "AS IS" BASIS,
( f1 e& k. e) v5 `# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.4 Z4 k% l8 v2 f, {/ ^+ V1 K
# See the License for the specific language governing permissions and8 T% F1 _, f* D( N8 N7 X& p$ R
# limitations under the License.
" c! G" h* N0 P# ?- Y" Z# the directory where the snapshot is stored.
* G& S6 m  U+ z3 LdataDir=/mnt/kafka_2.13-3.3.1/data/zookeeper
6 }% S2 q0 z$ f0 r9 T# the port at which the clients will connect. n6 |( h! z) t; H
clientPort=2181
9 e) J1 Z; K: j0 ~$ ]. z* V# disable the per-ip limit on the number of connections since this is a non-production config
3 o7 y* Z. K1 _4 a( r0 G0 ?maxClientCnxns=0
( s% G6 j8 l$ I4 I  D6 m% }  Z# Disable the adminserver by default to avoid port conflicts.
; e- l' X# w! E# Set the port to something non-conflicting if choosing to enable this
# C# g( G& w# }5 R- O% E+ {/ @admin.enableServer=false
9 K' }4 D6 d7 z$ F# admin.serverPort=8080
3 s9 j0 `, M6 U4 F, `; [7 a
1 U  M5 i' `5 Z; O1 g$ w8 o& c- O$ N# \; m
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &/ s, Y" B3 M' J; a7 x: s. J
 楼主| 发表于 2023-1-3 11:40:15 | 显示全部楼层
[root@ceph2 kafka_2.13-3.3.1]# egrep -v "^#|^$" config/server.properties
6 R1 r" V  F" [9 ~. ~1 [broker.id=0
% j$ \% v9 W8 K* nlisteners=PLAINTEXT://192.168.0.233:9092) f  Z8 O$ y% m- ?
num.network.threads=3
7 C$ T3 Y5 N5 |# X; ], Qnum.io.threads=8
) a5 J* I, W0 ]0 M" w% N, X) {# N& |socket.send.buffer.bytes=102400
, N. v+ A2 d$ y# R: rsocket.receive.buffer.bytes=102400, {! f3 I% a  {; u$ E, U
socket.request.max.bytes=104857600
& J; h0 k9 }, j* {3 R$ z' I# Tlog.dirs=/var/log/kafka-logs+ l0 I: k$ m/ F1 y4 Y5 W3 o
num.partitions=1
2 U* T7 ]' E& P! j7 Z! a  V+ Hnum.recovery.threads.per.data.dir=1
" o& W7 t) x6 koffsets.topic.replication.factor=1
0 F, C7 Q5 _2 N! xtransaction.state.log.replication.factor=1& ~2 ?, o$ F3 g+ H) k) s
transaction.state.log.min.isr=11 c# ^0 \2 {& F! _
log.retention.hours=1680 X- J( {$ [( _. i: J: I; M1 `
log.retention.check.interval.ms=300000& G% o5 K0 D, k/ X% E7 Q
zookeeper.connect=localhost:2181
7 W1 {  @; A) X' e2 L) Y& a3 Czookeeper.connection.timeout.ms=18000* D$ W0 m3 E: W2 m% i
group.initial.rebalance.delay.ms=0) K# r3 L, k2 w$ q" f; }  D

# |. L$ a+ X6 \2 A" ]0 B) h; k( n1 x3 K& y" N8 w
[root@ceph2 kafka_2.13-3.3.1]# nohup bin/kafka-server-start.sh config/server.properties &
5 B7 c) W5 x0 t0 a4 N
! q1 h, Z4 P9 ]# l
% _8 S* p% @  @; {  c% y
7 O) F+ ?: f: y' j" i/ mnetstat  -ntlp |grep java
: }& J! L& J: wtcp6       0      0 192.168.0.233:9092      :::*                    LISTEN      21222/java         
' h! T3 f% l8 Itcp6       0      0 :::2181                 :::*                    LISTEN      20780/java         
" ^2 L, l3 f' B1 z! gtcp6       0      0 :::41467                :::*                    LISTEN      21222/java          9 V2 m6 E9 O+ R; \" o0 r
tcp6       0      0 :::40571                :::*                    LISTEN      20780/java          0 M8 x- D3 K! ?  {) s$ {: v  ~3 t

$ p1 @) n; F- C( d
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现 点击这里给我发消息

GMT+8, 2023-1-31 23:55 , Processed in 0.043085 second(s), 22 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2022 Comsenz Inc.

快速回复 返回顶部 返回列表