|
|
介绍及特点$ M2 y1 q, T( q# ^' V V
Pacemaker：工作在资源分配层，提供资源管理器的功能
& t6 S9 \" m1 D& ` Corosync：提供集群的信息层功能，传递心跳信息和集群事务信息
, \; W3 G$ O. b( \% l, a Pacemaker + Corosync 就可以实现高可用集群架构
; {2 G- Q' S# {' }1 W* W
$ o6 |) [8 D+ `$ [. v 集群搭建
3 s6 M2 D# O( g) f以下三个节点都需要执行：$ H8 h9 u$ l7 p9 d
& o' d% a( N* ^: w- N/ H0 r b- s
# yum install pcs -y
5 Q2 E) @6 F0 _* }* d& o- }# systemctl start pcsd ; systemctl enable pcsd( K- C" L" }" G ^4 ]0 B
# echo 'hacluster' | passwd --stdin hacluster
9 s, s2 ] C; E2 v# yum install haproxy rsyslog -y
( }) |" e6 w1 n1 B9 n) K3 ~8 z# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf # 启动服务的时候，允许忽视VIP的存在( V/ b# r3 H: V; ?3 ~% U z
# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf # 开启内核转发功能
6 `# ~% C/ m$ P# sysctl -p
0 F7 \* H% l% M; X0 k/ u: M2 v, u: A* J5 c2 J
在任意节点创建用于haproxy监控Mariadb的用户; }" s# k3 e' ~7 e1 F3 J- Q
MariaDB [(none)]> CREATE USER 'haproxy'@'%' ;) M! m5 M7 C6 R* m( T
配置haproxy用于负载均衡器3 r1 m$ N# M( l; P
; \, k; l' F4 n" ?[root@controller1 ~]# egrep -v "^#|^$" /etc/haproxy/haproxy.cfg% M# r- K" v3 v! I1 K: G9 l
log 127.0.0.1 local2
" T# P7 _+ T1 v chroot /var/lib/haproxy. T K, u5 R: f2 b
pidfile /var/run/haproxy.pid8 F; [8 k2 ?* r @* I/ C. ~
maxconn 40007 x- m& _6 f: @) N* e
user haproxy
0 f7 z2 J; X4 f$ `; c2 N' W group haproxy$ w# l# }* E; g: ], Y
daemon
- e8 i+ \ Z+ H) u9 \# @0 m' J # turn on stats unix socket) C' e; F: f" i: l6 ^3 }
stats socket /var/lib/haproxy/stats* p; p$ L# \6 |9 k" `( T7 u x4 l7 w
defaults
6 c; v0 T2 u( q7 z9 W; ? O mode http7 }9 X5 }5 P! B$ y
log global5 O4 Y" z% ]6 S y! T* t8 \
option httplog/ `1 ?! N( @0 R( j
option dontlognull; d9 G& S+ r {& j5 k8 g
option http-server-close
: Y/ n. f% b+ Z% i option forwardfor except 127.0.0.0/8: w$ q, M- N; T: ]
option redispatch
# Z& s6 N, ?7 h& n7 s' @4 ` v# H0 e retries 34 A' O% e9 O$ V) _- z$ I, ^
timeout http-request 10s2 j' b* S9 O) q) e3 X9 L+ U- G
timeout queue 1m2 d: W+ t+ ]$ V0 q" p5 l% h2 n
timeout connect 10s
( t1 A* h; K. g, S timeout client 1m
0 f4 ~3 Z% k, w- J timeout server 1m. `. @: t9 x0 z6 w
timeout http-keep-alive 10s
# z2 X2 `$ y( R3 I6 D timeout check 10s c4 c) n" s; u7 ?0 ~0 Q) V
maxconn 4000
$ w. S7 T9 P- F+ ` d2 w5 Clisten galera_cluster
1 ^4 q& }$ _+ Y, ]. u4 `( S mode tcp . @# D! j4 B' a5 `% r
bind 192.168.0.10:3306& V$ |2 }% p$ ]0 n3 A0 P2 L& u
balance source- h) j. T6 u, O# @' J& D
option mysql-check user haproxy
9 ?& [- ^7 r+ O4 ~( c3 }: f# l server controller1 192.168.0.11:3306 check inter 2000 rise 3 fall 3 backup
# J. U/ _- Y( d2 }3 z server controller2 192.168.0.12:3306 check inter 2000 rise 3 fall 3
9 Z1 |7 p+ {- ]; |' I9 t. z server controller3 192.168.0.13:3306 check inter 2000 rise 3 fall 3 backup
0 p8 ~7 k% e: h
# p0 V" l" t( w3 f: r/ D. }! elisten memcache_cluster/ H7 Z( K* \( s( I
mode tcp% J* V9 U+ M- y6 [
bind 192.168.0.10:11211
6 j7 r# n4 ?) U9 y a' A( d) d balance source
, I- z- P3 }- r: k6 a, z$ i option tcplog
. A: z% J3 c. q- J. A server controller1 192.168.0.11:11211 check inter 2000 rise 3 fall 3
; c1 ?0 _3 r! o- J! p9 Q' \1 E server controller2 192.168.0.12:11211 check inter 2000 rise 3 fall 3
0 P4 W7 N( O7 h# I& T& I: d server controller3 192.168.0.13:11211 check inter 2000 rise 3 fall 3+ k, j, @) A% l. V0 l+ H! _; A
q, |8 D3 M. N% ^' C1 l
7 L9 }* {. Z. X+ Q- m6 t& v1 F注意：' r! @1 @6 u# \8 p+ f4 M% w# v
（1）确保haproxy配置无误，建议首先修改ip和端口启动测试是否成功。
6 z( k, j7 g9 ? （2）Mariadb-Galera和rabbitmq默认监听到 0.0.0.0 修改调整监听到本地 192.168.0.x
9 u; A! l- P) T, a$ Y （3）将haproxy正确的配置拷贝到其他节点，无需手动启动haproxy服务
( H9 v: c3 y. i* W/ ]为haproxy配置日志（所有controller节点执行）：1 P* r3 a5 g4 C0 w' w% l
, `& J2 A' j/ A5 L
# vim /etc/rsyslog.conf
7 |* H8 t3 L- ~6 ~4 d…8 D1 s) x4 H7 U$ M7 H' W! r
$ModLoad imudp
5 R1 z$ U8 B6 Q7 W" @" V+ T. A$UDPServerRun 5145 m! n( K; L' s6 E! W' V) I# O& C
…
/ e+ y5 ^% L5 K0 y9 A. _+ Blocal2.* /var/log/haproxy/haproxy.log" K' O; S. x- r
…
+ d7 b. H7 R& J t: m4 A4 G9 R& E( K- D6 v5 @% A$ d0 r1 d! x
# mkdir -pv /var/log/haproxy/9 }) S; ~, K8 e" h
mkdir: created directory ‘/var/log/haproxy/’
- B$ s: P& Y) ~; z: T% r @' o; V: c( b5 r: K
# systemctl restart rsyslog
6 F2 S0 Y( w7 j$ H
+ k( P$ f- D, L% [' y! b- J启动haproxy进行验证操作：2 `1 ]; P. o6 l5 u' ~
% s# K# R( H* P& m; x: l- x# systemctl start haproxy
! ^0 s1 u0 Z; o! y& ~% i0 L[root@controller1 ~]# netstat -ntplu | grep ha+ g5 a- y( C- s( V) P* @0 K
tcp 0 0 192.168.0.10:3306 0.0.0.0:* LISTEN 15467/haproxy 5 E) v1 I9 q/ g. k' ^/ j6 N
tcp 0 0 192.168.0.10:11211 0.0.0.0:* LISTEN 15467/haproxy
& a' O9 Y, c- H, ]; a0 o9 [/ }udp 0 0 0.0.0.0:43268 0.0.0.0:* 15466/haproxy
2 S9 ~0 N6 \2 n2 B3 E8 A
, R* _& x1 |: B: _验证成功，关闭haproxy, `' \, \0 D8 Q$ h$ F U
# systemctl stop haproxy
, k& ~4 q7 v1 D1 j' _9 X8 J
7 ]* \( G$ c, j1 _2 f5 }4 d 2 G- o& a; Z2 ]5 L2 E6 \2 s T) e
在controller1节点上执行：' A" z2 Y9 \1 G/ P v4 ^) }
[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p hacluster --force
& x+ c( _: C+ X7 J, D- R" k8 o" Ocontroller3: Authorized
. Y- U6 t: B" B: e `3 u! ~+ v g. Bcontroller2: Authorized
, L. q$ @; C* b: _( Zcontroller1: Authorized8 i: ?5 B; W7 U1 F P& }1 k3 `
创建集群：
2 ^$ @3 h; L2 x' R8 @! i4 r/ K
( S. [1 [( _* e1 k4 i0 }2 v[root@controller1 ~]# pcs cluster setup --name openstack-cluster controller1 controller2 controller3 --force$ a6 H: Z: ] _' a+ I( s/ w8 f, S& ?
Destroying cluster on nodes: controller1, controller2, controller3...2 w/ F" ?$ J7 z
controller3: Stopping Cluster (pacemaker)...* W) H# J' }5 t! B) v& H
controller2: Stopping Cluster (pacemaker)...
8 N* b( T2 @$ o7 g9 e- vcontroller1: Stopping Cluster (pacemaker)...
, A# O# z: t# |- i, xcontroller3: Successfully destroyed cluster- x7 ~$ }: D$ f7 F* B$ y; Z
controller1: Successfully destroyed cluster
) f+ E$ h P' Z) T# U; I. Kcontroller2: Successfully destroyed cluster
" b: c3 G* f+ h* e( v: P0 L$ L2 t5 D7 N" A3 i& U3 W
Sending 'pacemaker_remote authkey' to 'controller1', 'controller2', 'controller3'
. I. T1 F- n+ E0 |- q: c" D- kcontroller3: successful distribution of the file 'pacemaker_remote authkey'
( v: Q; F, q2 j% B5 y, Icontroller1: successful distribution of the file 'pacemaker_remote authkey'
& [$ q- c8 K8 |0 Q- Kcontroller2: successful distribution of the file 'pacemaker_remote authkey'' d" `9 p7 [: n$ U) r5 H
Sending cluster config files to the nodes...0 t+ P: Q/ F. ]! D+ H7 i8 C' e6 l
controller1: Succeeded
1 p0 J( z. D! y& v3 }! icontroller2: Succeeded- a' i" v$ q, \+ r
controller3: Succeeded5 R& R% z/ P( t6 b3 Z F
, Y% a5 a; e: ~7 d+ ^" s2 W4 t2 Q D
Synchronizing pcsd certificates on nodes controller1, controller2, controller3...
1 K+ R6 c) v5 h- Mcontroller3: Success
5 o# T2 O N4 }! O* r/ F# |controller2: Success. Z. ^# Q; A/ z
controller1: Success
! O% Z, S5 A2 x& S/ X& s. O1 ORestarting pcsd on the nodes in order to reload the certificates...6 V) E) L7 l- P
controller3: Success% F: {( h1 D7 N% Z# k
controller2: Success# _: ` \* v7 {: Y9 ~7 x8 ]
controller1: Success# p& K S# v& r3 z- K& K
- f# p5 ~2 ?7 B# {$ K) d" |
启动集群的所有节点：& l! k: w4 y- h7 v; V/ n+ Q2 k! z
- ^7 I% k4 N: B! a
[root@controller1 ~]# pcs cluster start --all
. [$ U7 c( a- N, U8 Z4 s+ r8 `controller2: Starting Cluster...+ v; B5 T# f- A0 G9 E5 G' K
controller1: Starting Cluster...
" C2 w/ X- z: S3 O, [5 Z. `* ccontroller3: Starting Cluster...4 M& w- z1 y1 Y1 P) s
[root@controller1 ~]# pcs cluster enable --all
; m3 c# J0 p* F: a, l( n$ ccontroller1: Cluster Enabled h' {' ]5 B' T! ~
controller2: Cluster Enabled
: k& F% G. z- h* E0 g0 Ucontroller3: Cluster Enabled( j2 a/ z& c% J" d- V+ i# e0 W1 o" L7 s
# t0 D' q: O5 \; V# d查看集群信息：
# w9 T' j* s4 Q7 W* x
1 W( c3 d, l& ?, U: P[root@controller1 ~]# pcs status% `+ P8 Y9 _2 B! D
Cluster name: openstack-cluster7 Q6 ]) ?9 x( l; ]; s8 E3 X* Y
WARNING: no stonith devices and stonith-enabled is not false/ U, U+ P. u0 U) E6 A7 Q4 S" l
Stack: corosync
! r- _9 \; ~" M2 ^+ `6 HCurrent DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum' d5 G% ^* Z1 S$ t
Last updated: Thu Nov 30 19:30:43 2017
! ~4 \) Y; O9 ^6 D, |& c+ \Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3
( u9 V$ P6 R/ s B+ [- r
7 z* I, D, ~9 z$ \! H) c m3 nodes configured. {* w9 t0 b0 }" e
0 resources configured! {& s4 H& A& R/ ~7 b& T
9 ]2 \, @0 J/ }# s9 s
Online: [ controller1 controller2 controller3 ]
: c9 B& f! f/ Y
0 S1 c. k3 l8 H3 U/ D6 ]- N: S; fNo resources+ y5 d4 ^2 O1 Q2 Q$ F' i/ n
l1 ^5 ~ c u8 t( |0 S" x6 ~3 Q* S8 j* z: w# o) M% V$ ? l
Daemon Status:8 z) }" Z- F; @: C
corosync: active/enabled7 o3 o L0 C) }+ D8 y* W
pacemaker: active/enabled
3 N- l( {# A6 Q0 e pcsd: active/enabled
, N* [# l! \4 ]9 b: [6 ^. u' `8 j[root@controller1 ~]# pcs cluster status2 s' j, [+ q" N; I+ I; H
Cluster Status:
2 p! J" Y( V# h Z Stack: corosync
& q* I) c0 \0 j4 m# \2 y Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
: o5 t/ N3 M7 N) T& d8 W Last updated: Thu Nov 30 19:30:52 2017/ x0 h1 _) |# h
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3% }; c% M. `; q) W
3 nodes configured4 u0 d3 t5 \# b! m" B3 e
0 resources configured1 Y3 {9 X. y# w% P- s: o$ w0 ~2 c
. u" j* C0 }& G# q' Z7 n. q& \
PCSD Status:
" E5 Y5 o) Q( d7 \8 o controller2: Online$ r0 g- R7 j5 n3 t: y, E( O
controller3: Online
. g& S+ q- q$ i5 S9 V. D; H1 F, R4 D2 f controller1: Online- z; P7 e/ D. Y: ~
8 v6 @, i2 y" [/ b2 @2 m- Q三个节点都在线& o' E% ?- r! J
默认的表决规则建议集群中的节点个数为奇数且不低于3。当集群只有2个节点，其中1个节点崩坏，由于不符合默认的表决规则， 集群资源不发生转移，集群整体仍不可用。no-quorum-policy="ignore"可以解决此双节点的问题，但不要用于生产环境。换句话说，生 产环境还是至少要3节点。& Z* A# [$ b% D6 |" T
pe-warn-series-max、pe-input-series-max、pe-error-series-max代表日志深度。2 s% T3 p9 r9 k5 q( R! g7 k
cluster-recheck-interval是节点重新检查的频率。4 Q) E& v3 E1 p
[root@controller1 ~]# pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000 cluster-recheck-interval=5min
4 T# M6 }. h: c5 y2 T% r禁用stonith：- q" }( J3 M$ A/ w
stonith是一种能够接受指令断电的物理设备，环境无此设备，如果不关闭该选项，执行pcs命令总是含其报错信息。
4 S9 y3 z; @ I$ J s, y( o1 s6 I[root@controller1 ~]# pcs property set stonith-enabled=false
( j5 n' P/ r( U" r二个节点时，忽略节点quorum功能：
9 l1 K h; E+ o2 Q( [ R P" d[root@controller1 ~]# pcs property set no-quorum-policy=ignore. p+ G9 E9 X% g& b+ B- R! n
验证集群配置信息3 J {; J4 A: a5 d5 P- ]; L
[root@controller1 ~]# crm_verify -L -V
8 {1 n- U S/ w4 D为集群配置虚拟 ip
9 h- u- B# n/ ~! b[root@controller1 ~]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 \4 f! k4 o3 g! s2 r b+ S
ip="192.168.0.10" cidr_netmask=32 nic=eno16777736 op monitor interval=30s& h5 M' {+ g/ v0 V1 U
到此，Pacemaker+corosync 是为 haproxy服务的，添加haproxy资源到pacemaker集群
4 q' C( ]$ K& t! J. a: K X; v[root@controller1 ~]# pcs resource create lb-haproxy systemd:haproxy --clone/ u& K! R! A o* q& X$ y% w0 n! |
说明：创建克隆资源，克隆的资源会在全部节点启动。这里haproxy会在三个节点自动启动。1 ]* u' H: [& h8 v6 C% n2 M6 L2 _
查看Pacemaker资源情况
" F! X* O( A1 U( n[root@controller1 ~]# pcs resource
% P0 m) c* f+ a; e% ` ClusterIP (ocf::heartbeat:IPaddr2): Started controller1 # 心跳的资源绑定在第三个节点的# t3 N5 ^$ l' R1 R9 k
Clone Set: lb-haproxy-clone [lb-haproxy] # haproxy克隆资源
) \" m8 [' K' Y. r o; A I h4 |" O Started: [ controller1 controller2 controller3 ]
' |5 J8 _! J+ e% n; v. b注意：这里一定要进行资源绑定，否则每个节点都会启动haproxy，造成访问混乱3 h' D' `) O9 ^# i( ?- g4 I% P: K. e
将这两个资源绑定到同一个节点上
# {1 L9 }& ^4 M: r4 ]. h8 h[root@controller1 ~]# pcs constraint colocation add lb-haproxy-clone ClusterIP INFINITY
& t1 |. Z' J5 W+ o! ]% [8 k绑定成功% d7 j% D5 u8 G7 y
[root@controller1 ~]# pcs resource
g. s/ o5 \ \+ N ]; Y! ` ClusterIP (ocf::heartbeat:IPaddr2): Started controller3
4 b, c h' f- |3 C5 {9 K Clone Set: lb-haproxy-clone [lb-haproxy]3 H+ u; o0 T3 h2 |% ]- O5 a
Started: [ controller1]1 o% t7 v1 N1 g% `# `. l# K7 S
Stopped: [ controller2 controller3 ]3 P J5 `. y) ~, A/ L
配置资源的启动顺序，先启动vip，然后haproxy再启动，因为haproxy是监听到vip8 |: I# ?# x. L0 c) p
[root@controller1 ~]# pcs constraint order ClusterIP then lb-haproxy-clone4 h9 x, E4 g( l4 U" e$ l
手动指定资源到某个默认节点，因为两个资源绑定关系，移动一个资源，另一个资源自动转移。# B% A1 w% n% U$ g0 I' e
" o: A7 v+ E$ A, p7 O4 t
[root@controller1 ~]# pcs constraint location ClusterIP prefers controller1
1 w4 N' x% b0 K& w _1 q* ^[root@controller1 ~]# pcs resource
* K& S" ^% Y1 c! r/ W$ L ClusterIP (ocf::heartbeat:IPaddr2): Started controller1 C3 f+ m. `) G4 a0 T! ?
Clone Set: lb-haproxy-clone [lb-haproxy]
7 @7 S) c+ z( G# x1 t Started: [ controller1 ]
5 ?& m: Y! n5 n# p; U Stopped: [ controller2 controller3 ]
$ `$ c: N. Q# i( r% Y6 p" m( Y[root@controller1 ~]# pcs resource defaults resource-stickiness=100 # 设置资源粘性，防止自动切回造成集群不稳定$ R" F! t" _8 K- A8 L
现在vip已经绑定到controller1节点
4 t) a( v F2 I% R[root@controller1 ~]# ip a | grep global! }, B3 T8 {: i" B0 R$ c% T
inet 192.168.0.11/24 brd 192.168.0.255 scope global eno167777369 f( e* u0 R3 z; m7 J) O$ i
inet 192.168.0.10/32 brd 192.168.0.255 scope global eno16777736: R+ V! z2 l4 v! L* s
inet 192.168.118.11/24 brd 192.168.118.255 scope global eno33554992
$ m/ I, [1 I, [5 R4 G) J4 l$ ?5 E
$ |# {# C. f1 w尝试通过vip连接数据库
, @+ E# c* f. N/ F4 _Controller1:# a" h1 K* d! {* [( x, r7 L
0 y9 M) V* l' z: g; W+ O0 z
[root@controller1 haproxy]# mysql -ugalera -pgalera -h 192.168.0.10- T2 a3 w9 |$ M" Y, S$ `# T' J; s
3 Z* Z3 w) M d& t$ }
' C- i" r7 g8 H8 A9 |
Controller2:
2 t& @! u: N) G& r9 D* K4 g1 s6 O
) ~7 t+ v: d4 p. Z8 Q& W. r" \0 t
高可用配置成功。
6 _0 _2 ?' ]6 M/ V4 g& d# z $ e, Z" `/ A3 ], [4 b
测试高可用是否正常6 C1 M/ w# ^+ O9 S
在controller1节点上直接执行 poweroff -f
# [7 N H% ?$ h/ o0 g. S' r[root@controller1 ~]# poweroff -f
1 J* y2 o- G, Z8 v, rvip很快就转移到controller2节点上
' \2 T) `7 q- B2 P
H+ a2 F! M) ?9 b再次尝试访问数据库5 }* N7 k9 s4 Q6 D; w/ T% r
; \$ w+ S- j. ~/ v ' Y7 p0 N2 Q* |
无任何问题，测试成功。
' \$ }/ b5 N) }' n. o查看集群信息：
. E" d% @( p+ Z1 F J# ^. @
1 l$ N) u$ f9 f* U7 A' I3 a" `' G: Q[root@controller2 ~]# pcs status
* [6 T5 R7 \# k$ YCluster name: openstack-cluster
. F! p& E% ^. u2 d1 N$ n- UStack: corosync
/ j/ w3 x3 Y3 d$ A7 f3 ICurrent DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum6 Z& D$ D, H" Q4 f/ Y
Last updated: Thu Nov 30 23:57:28 20174 f% v/ v y' s; u2 |+ s, N
Last change: Thu Nov 30 23:54:11 2017 by root via crm_attribute on controller1
, x r) E& w9 `1 D. T; I V% h$ j4 L7 Q L1 x
3 nodes configured
$ R+ D. n; E8 W- Z' [9 s2 ~! w4 resources configured% X# Q& i1 B5 H2 t" J
: L- X# r" h5 E. X0 {Online: [ controller2 controller3 ]
+ s& [+ n4 Q6 _( lOFFLINE: [ controller1 ] # controller1 已经下线
1 ~* ]5 E* a6 f, C- g9 E4 Z# x' [, r; r! m* m% f. {0 o8 H
Full list of resources:* N: p6 P# |0 I( A
8 z$ I( T# m& o8 B B* ~7 H) N# K ClusterIP (ocf::heartbeat:IPaddr2): Started controller2
2 x* |- J& j5 t Clone Set: lb-haproxy-clone [lb-haproxy]& r9 R+ V/ }7 A
Started: [ controller2 ]
& E' N0 z- c" D1 `( q Stopped: [ controller1 controller3 ]8 O. k O r# s$ C2 g+ O( j2 j
/ R) `( l% C0 P
Daemon Status:3 A( q6 h0 R; J# H0 ?- G. _
corosync: active/enabled! a/ P: t' n9 K( b, e% @
pacemaker: active/enabled
5 H, ]/ j2 Q0 E2 ^, @6 T/ u pcsd: active/enabled |
|
发表于 2018-10-19 16:05:14