找回密码
 注册
查看: 383|回复: 2

Kafka环境搭建及使用教程

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2023-1-3 10:58:57 | 显示全部楼层 |阅读模式
kafka里自带了一个zookeeper,可以不需要再去下载zookeeper。在启动kafka前,需要先启动zookeeper。7 u' A# v/ ^, k

6 {) g, W, R% M% `(1)首先去修改config目录下的zookeeper.properties
/ ?6 B$ r: y0 w. [! o(2)修改config目录下的server.properties,修改log.dirs和zookeeper.connect。log.dirs是日志存放文件夹,zookeeper.connect是zookeeper连接地址(端口和clientPort保持一致)。
' u; \1 a9 ^3 J" H- |. j2 Y7 W9 v$ Y7 G: g0 t. J9 e
至此,kafka单机版环境已搭建完毕。(如果对里面的参数没修改的话,可不修改,直接用默认的). r- D* O; F! a6 H3 M) y2 d7 [

6 n$ c7 z' g6 ?; h8 S ) k$ u: O- D5 E7 u0 A1 A
* J* z7 D* w! s6 w( W
kafka使用教程4 p5 _, H" h& U4 n. |. c4 Q4 h1 ^- A
(1)启动kafka自带的zookeeper(加了-daemon,可以让这个进程挂在后台进行,否则会发现是个阻塞进程,只能再开一个终端进行其他操作)
4 _# u! R$ I7 z  S7 y: B) Z  k% }bin/zookeeper-server-start.sh -daemon config/zookeeper.properties& d  w# R. j/ ?5 j2 q5 G

5 S( @" k8 y$ ?: K(2)启动kafka' n/ j  ?' c" u& `
bin/kafka-server-start.sh -daemon config/server.properties+ E2 t' @: W: G8 s

4 ]1 d. ~) f0 u4 a5 d2 B(3)创建一个topic
! g: b9 u6 C  Y, U, nbin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
5 ^9 i, U& u" D! \- Z' E
6 w+ W: }+ m/ R
, j. I# x8 w$ A! C. c(4)查看topic列表
! D8 Z2 H- u. x) g4 \6 c( X, ]' Rbin/kafka-topics.sh -list -zookeeper localhost:2181
( n6 [& w+ l3 Y) B% ~5 S5 p3 g6 M/ g6 f5 N4 n
(5)创建生产者进程  (9092是默认的broker-list端口)) w+ o+ I6 L3 F; v) X* D& }, T
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
  N# V  K) y* n# o8 y: l2 a9 M  \& Y* |9 r
(6)创建消费者进程
9 k( P4 B1 B8 h( W) kbin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning5 H* {2 S$ S$ U  ?$ h/ `* b3 Q
$ }% R1 {3 a8 h# a0 X
(7)外网访问 vi config/server.properties% ?7 {- R. L  u6 P' w& W; W  z
advertised.listeners=PLAINTEXT://192.168.8.65:9092" P' s2 s/ n! g' L

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2023-1-3 11:28:57 | 显示全部楼层
cat config/zookeeper.properties
  y) I% G9 k: o0 A7 X# Licensed to the Apache Software Foundation (ASF) under one or more
7 j1 N/ g8 S( N5 n9 x# contributor license agreements.  See the NOTICE file distributed with
  r0 q# g2 k' I( j6 P( T( w* s/ N# this work for additional information regarding copyright ownership.  @8 b, k. ^# i
# The ASF licenses this file to You under the Apache License, Version 2.0
0 f# F* k* X! _4 x4 c# (the "License"); you may not use this file except in compliance with7 N* T. q7 w6 _1 G: R4 g; x# z# O
# the License.  You may obtain a copy of the License at. Y3 T3 a. `: z) E! Q
#
1 l9 M5 d! _+ b! R  {, P% r! k#    http://www.apache.org/licenses/LICENSE-2.0
8 \" q% [, N* D" N  x% j6 P#
" ?* S" j# O- K- N7 N% A3 c! t# Unless required by applicable law or agreed to in writing, software
. J/ ^3 X' a, [6 O# distributed under the License is distributed on an "AS IS" BASIS,# R5 o0 c2 k- x4 V; b" W
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* C1 H) m2 w- i1 A+ y+ [1 A# B7 j# See the License for the specific language governing permissions and
/ p: v" g6 Q$ x- Q# limitations under the License.6 g8 [3 W4 h& P5 f8 \  @2 S9 R% V
# the directory where the snapshot is stored.# d0 z; c3 \9 q4 C/ S7 f
dataDir=/mnt/kafka_2.13-3.3.1/data/zookeeper
3 `, N# x3 u6 X1 `5 ]9 J" P) t# the port at which the clients will connect
5 b$ u: A& c+ X2 f/ LclientPort=2181
$ ^) q: g5 N  T- w# n' ^( X# disable the per-ip limit on the number of connections since this is a non-production config8 ?* z6 V6 {3 l% e, X4 Q8 Q
maxClientCnxns=0# l% n- Z1 ~- L7 h4 }
# Disable the adminserver by default to avoid port conflicts.0 O' {& P1 n( k1 T* ~
# Set the port to something non-conflicting if choosing to enable this
. S  y3 N% \0 `% V% X, s  e5 ]$ w& Hadmin.enableServer=false
3 d8 v$ o: a/ Y8 K4 T3 h" B" O# ]# admin.serverPort=8080
% d; M) m/ X6 W2 `& W# B, O$ h% ^% w! b; f/ f2 c9 k

% z" ^. t! p+ T2 q* S& @nohup bin/zookeeper-server-start.sh config/zookeeper.properties &( n9 Z+ M7 I- ], V' G

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2023-1-3 11:40:15 | 显示全部楼层
[root@ceph2 kafka_2.13-3.3.1]# egrep -v "^#|^$" config/server.properties
5 M( ]) e, g3 h5 E% t& U! Zbroker.id=04 f! q( e1 a7 @8 \
listeners=PLAINTEXT://192.168.0.233:9092+ {$ _) {% ^" @
num.network.threads=3
! ~* s- A9 J6 q0 |$ W& jnum.io.threads=8- Q$ O% o. H5 P% {, u3 H8 P1 q. n. B
socket.send.buffer.bytes=102400* F% q9 z. Y$ j& h  V
socket.receive.buffer.bytes=102400% K( a) z, ^: S5 \' ?; A
socket.request.max.bytes=104857600
7 S& f8 a. o7 l# o5 D6 E: plog.dirs=/var/log/kafka-logs
9 a9 p* B/ s' `  q, N% ]0 j# Tnum.partitions=1# g6 Q1 x' E) \& p! c8 _
num.recovery.threads.per.data.dir=1; a, M  h. }( y
offsets.topic.replication.factor=1
% D9 W+ f2 {8 H% t2 ^( P4 k( gtransaction.state.log.replication.factor=1
# A/ w7 q0 Q- I% Etransaction.state.log.min.isr=1
2 {9 b4 @1 v9 T7 R5 ^log.retention.hours=168
9 E$ L/ o& `' D$ c: a; Glog.retention.check.interval.ms=300000
, c4 N4 K- j: w- l# ]' T, czookeeper.connect=localhost:21817 A4 D' c4 H+ J# ~* X
zookeeper.connection.timeout.ms=18000
( ?' [1 f3 u6 [9 n8 ?' n, igroup.initial.rebalance.delay.ms=0( a# l! E7 x5 L3 B$ T3 ^- j/ p0 H
8 c. ~% b5 o9 N" r; {0 g
8 }. R5 c8 \2 T
[root@ceph2 kafka_2.13-3.3.1]# nohup bin/kafka-server-start.sh config/server.properties &: s& r( G9 V7 H( Q
% x% a( r0 |7 b3 q$ Z# ~
2 J% Z0 P8 M2 l+ v
$ F; }. A9 e% h' u3 c6 V' d
netstat  -ntlp |grep java) B: Z, Y. _+ X2 z
tcp6       0      0 192.168.0.233:9092      :::*                    LISTEN      21222/java         
3 P0 L& a4 L. L6 v4 U' `( k5 A$ Ltcp6       0      0 :::2181                 :::*                    LISTEN      20780/java         
% O( D# {5 z- z0 P9 y' F4 @/ R5 stcp6       0      0 :::41467                :::*                    LISTEN      21222/java         
8 F  R# t9 T3 T6 b$ m! wtcp6       0      0 :::40571                :::*                    LISTEN      20780/java          0 W) H6 h  ]9 y6 e  A7 D
5 W! B, N  \8 ?. J6 O
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 02:06 , Processed in 0.024937 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表