|
|
介绍及特点
& c; F- I/ M% Q Pacemaker：工作在资源分配层，提供资源管理器的功能; c* r! \! s, V& u
Corosync：提供集群的信息层功能，传递心跳信息和集群事务信息
6 P$ P7 }5 ^2 F: k* v# R- l5 ? Pacemaker + Corosync 就可以实现高可用集群架构
2 |0 A+ X! e# ?3 E9 x7 t0 L , W+ V X; h) M5 {& m& a; X
集群搭建
3 w9 c3 `. Y% _% I( j以下三个节点都需要执行：/ w1 d+ i, t6 t( z
) i& z* \+ g& d: p
# yum install pcs -y; d7 U- P; `8 P* g
# systemctl start pcsd ; systemctl enable pcsd2 |( H6 E" y- L7 I% j; w$ ?
# echo 'hacluster' | passwd --stdin hacluster
% r; t4 ~: l' N0 p0 U1 V# yum install haproxy rsyslog -y' s8 E/ W L: C8 C1 \( F" v8 o
# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf # 启动服务的时候，允许忽视VIP的存在
; r }1 C; t, w# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf # 开启内核转发功能
* g) b, b$ _% D4 `# sysctl -p8 |8 H# K# s) ]
; |$ f' I+ G- |( @; D9 O2 m* x在任意节点创建用于haproxy监控Mariadb的用户4 a2 B$ i' l4 Y* s* a, t8 @( S
MariaDB [(none)]> CREATE USER 'haproxy'@'%' ;- v; ~4 u# ^5 s
配置haproxy用于负载均衡器, |5 K: v* ^+ J/ w6 M7 q% \
$ }3 e5 o0 S* T% `; ^3 [) A; D8 I
[root@controller1 ~]# egrep -v "^#|^$" /etc/haproxy/haproxy.cfg
- D% v9 ^8 D, U/ ] s% n& S log 127.0.0.1 local2
+ C, L1 A; V, t* h W" L chroot /var/lib/haproxy$ |4 ^. S2 ^. ]; v
pidfile /var/run/haproxy.pid' f& z! ^3 P4 z) c* p& v
maxconn 40006 }5 E- ]5 e1 u; G
user haproxy: e0 E8 o8 `: Q
group haproxy! K# W, q1 `8 Z2 ~% b, X, ]8 m" V( T
daemon I. f5 r9 n {, Q/ {0 O7 r5 M
# turn on stats unix socket$ \1 k! z/ x6 I" u1 Z4 B5 d
stats socket /var/lib/haproxy/stats
$ p9 z6 s- \- V: |8 Vdefaults/ I: J" d6 K) j8 T" \% M
mode http$ O2 N/ ^6 u. |6 N; q9 z
log global! x, n$ F) @ R! i
option httplog# p0 v- }. u$ ~) e
option dontlognull g* S( m6 T) t
option http-server-close
' m O! L0 V8 E* t2 ?& S option forwardfor except 127.0.0.0/8' A+ e% c+ _2 U# k. P1 X* S
option redispatch
. E3 U9 Y# r+ z retries 3
4 t1 [3 z5 Y8 N% Y( ^+ R% a timeout http-request 10s
; ]. P1 R1 _9 w timeout queue 1m! a( @8 ]9 U( w
timeout connect 10s
6 S# r4 U7 Q- J( C& a7 @' O: Y timeout client 1m! C' v1 M6 d v3 W$ g& [5 a
timeout server 1m
# l* c8 T" E2 _9 @( G timeout http-keep-alive 10s5 w2 x- x& j! G8 V0 B
timeout check 10s& O1 X2 M! K- w! f3 L7 U6 E3 i4 d6 x/ a
maxconn 40003 g$ H- z! Q0 L/ t
listen galera_cluster
% y; q% ?" z" O! |2 c mode tcp
& Z- C c4 e0 X1 q( o bind 192.168.0.10:3306* @$ M7 u+ c: B+ B
balance source
8 `( |2 K B8 ~+ q6 O- S8 P option mysql-check user haproxy
. T/ ~: Q# \: x' S+ b) _6 ~ server controller1 192.168.0.11:3306 check inter 2000 rise 3 fall 3 backup8 `" F/ @/ m) c! J5 s/ m! N
server controller2 192.168.0.12:3306 check inter 2000 rise 3 fall 3 5 c- |# s9 p# g6 x" d8 y$ p
server controller3 192.168.0.13:3306 check inter 2000 rise 3 fall 3 backup+ V8 F# f/ M* ^8 ~" ?
+ z6 T$ ^# w; T2 J4 `' F: J2 vlisten memcache_cluster
$ Q; X" a3 j; _) x9 B$ X- } mode tcp: e, a) {1 U% K. A1 `0 p1 X
bind 192.168.0.10:11211
0 b% ] _( G7 y4 q3 o8 J3 w1 b# ^ balance source7 h6 [2 E: r( w7 n1 `4 Y( f
option tcplog& y, W; {- ~0 t- F2 m& B
server controller1 192.168.0.11:11211 check inter 2000 rise 3 fall 3 4 c; z0 D0 X8 Z, h2 k2 A' A6 R
server controller2 192.168.0.12:11211 check inter 2000 rise 3 fall 3
" [6 g P5 @& d! S' a" W# _ server controller3 192.168.0.13:11211 check inter 2000 rise 3 fall 3
* n+ h7 z& e" Y+ g5 _4 A: M4 z# b( A4 @
1 {8 n8 a1 Y* q: [* A) \
2 F( h# a4 }: r9 u% S# L, Z* z注意：; L; V |! d, ]
（1）确保haproxy配置无误，建议首先修改ip和端口启动测试是否成功。
2 ~5 O# d: P( Z" N+ Z （2）Mariadb-Galera和rabbitmq默认监听到 0.0.0.0 修改调整监听到本地 192.168.0.x
k7 G. w# z+ D) l0 d （3）将haproxy正确的配置拷贝到其他节点，无需手动启动haproxy服务
& f5 u4 q& D: f+ r! {& n" y0 Z0 Z# U, J为haproxy配置日志（所有controller节点执行）：, O) Z* g5 Z& L+ B
. i& o. s; u& X# Z0 _7 s. s# vim /etc/rsyslog.conf) G: R7 }! R. X M
…
( r0 Y' L+ a( ~' v0 k' s4 a2 A$ModLoad imudp/ T5 J" I# ]$ H
$UDPServerRun 5143 ]: Y1 T6 ~7 E" |, s5 R/ W( Y
…- G+ f( u. F; Y- q. I$ [
local2.* /var/log/haproxy/haproxy.log
- `% d& O# I/ y…! z: ?/ F- e$ o0 a/ b
7 `( S( ^* r. O) Y/ R& y1 ?0 b
# mkdir -pv /var/log/haproxy/% F4 I. T: y; e
mkdir: created directory ‘/var/log/haproxy/’6 y% V) t! p# @, b
% I# a7 `- G0 J" x! x$ c2 z# systemctl restart rsyslog% Y+ V1 h$ d1 q# [
2 R# s& s9 m2 @2 O) ^) |
启动haproxy进行验证操作：
% G" c3 h3 A% F
( S2 o3 @+ y1 }" Q O. c# systemctl start haproxy
$ d7 C) |. D; b) ?9 P+ ]: |[root@controller1 ~]# netstat -ntplu | grep ha
; l3 J& a/ p& J {5 I* Rtcp 0 0 192.168.0.10:3306 0.0.0.0:* LISTEN 15467/haproxy
; {. D/ K7 s0 o' H7 ]tcp 0 0 192.168.0.10:11211 0.0.0.0:* LISTEN 15467/haproxy
; U( [8 \! c+ d, w- zudp 0 0 0.0.0.0:43268 0.0.0.0:* 15466/haproxy
! w3 U8 P( ~+ y y/ j/ i! q h
验证成功，关闭haproxy( ~" c9 t# f9 o# }: I" s$ |+ ~3 B
# systemctl stop haproxy
7 w- ?7 K: r/ X. M+ ?
: G; y% b U5 [; m* k% ?/ v" ^ , U6 H( T% Y" ^
在controller1节点上执行：: |: D2 d/ x* j. h v7 y
[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p hacluster --force
6 `, C( x" i' o3 Tcontroller3: Authorized
* {/ T: M- o: X6 ~* r pcontroller2: Authorized
* v8 g3 v- P H" S' k, [( L$ vcontroller1: Authorized
/ x2 X! T! Q" Y" {创建集群：* @" ]1 v3 H! n
7 l1 f& [ |: J# k3 t8 }& U$ n
[root@controller1 ~]# pcs cluster setup --name openstack-cluster controller1 controller2 controller3 --force
, W, R- p$ m0 ~, pDestroying cluster on nodes: controller1, controller2, controller3...9 D( K, w& G J5 E" B
controller3: Stopping Cluster (pacemaker)...0 f4 v9 e, R" Z7 w [7 Z
controller2: Stopping Cluster (pacemaker)...7 ?4 e" v' _% g: @! _
controller1: Stopping Cluster (pacemaker)...
/ K* `3 f4 b5 u1 o/ ocontroller3: Successfully destroyed cluster7 V# k* p' X ~7 a) n4 y
controller1: Successfully destroyed cluster/ j4 `" q+ f2 q( ~
controller2: Successfully destroyed cluster* u9 P% R/ ?9 K4 f. _
( H% |+ _# o* C6 v% d. o, c' q
Sending 'pacemaker_remote authkey' to 'controller1', 'controller2', 'controller3'
% B3 c: S5 O1 _* Y3 fcontroller3: successful distribution of the file 'pacemaker_remote authkey'
; @" S# M1 ]: Hcontroller1: successful distribution of the file 'pacemaker_remote authkey'
7 @2 `, Z8 ~& N* a. x9 ucontroller2: successful distribution of the file 'pacemaker_remote authkey'
3 L# L* c% [* ?/ {) f0 Z- ]: xSending cluster config files to the nodes...) A+ n& `. R- p1 a: i
controller1: Succeeded1 G3 O3 x8 g+ ^8 H! ~+ b
controller2: Succeeded% L$ ~/ V5 U# @% a- P) G6 g5 E. A( G; p
controller3: Succeeded, [% f/ q, c- ?* n _
0 m1 e7 C1 O; C+ |' E7 m8 xSynchronizing pcsd certificates on nodes controller1, controller2, controller3...6 {4 O% w8 M# Z! a4 n
controller3: Success
9 z* y8 h. E3 G- ?9 tcontroller2: Success5 P& Q, j* x" V9 Q
controller1: Success
0 g c# D2 U5 _# S" P2 u9 \5 H6 ERestarting pcsd on the nodes in order to reload the certificates...0 r- E% z8 H& }, {. S
controller3: Success
4 w5 L) W) ~% @3 Wcontroller2: Success% A, {2 R3 B& O$ z3 Z) v
controller1: Success" k) _# T5 j# n& v
* t# c4 Q9 `$ X, g! }% z! S. y. X
启动集群的所有节点：3 G9 \1 H0 c8 V$ }7 h% c+ X+ ?
+ s# y# u" ~3 C: ] w: n/ M[root@controller1 ~]# pcs cluster start --all
) {" H) n/ d0 G8 C1 s- S( Dcontroller2: Starting Cluster...% X& Z* X* Y. O" T8 C" Z% X& \& s
controller1: Starting Cluster...4 F$ C" u6 y% [$ G- A7 x S
controller3: Starting Cluster...
/ F8 z# ^/ e6 w* O* ~- z, B4 ^[root@controller1 ~]# pcs cluster enable --all" d0 x8 l. B! J$ F5 C! X; \
controller1: Cluster Enabled6 p3 b1 X& Q* ^" a7 c& D9 s
controller2: Cluster Enabled& t6 T2 V3 Q6 O! y1 f
controller3: Cluster Enabled" P/ P, e7 L$ f
, X* P( V) H) H% e! K, E8 B
查看集群信息：0 H7 j( @3 [* K% R
! S \9 O8 X" L3 y. a: e
[root@controller1 ~]# pcs status
' E+ i4 ?- N. t/ {7 m: G# ^# |Cluster name: openstack-cluster
% y* ?' v3 {/ g- ]/ {, J7 KWARNING: no stonith devices and stonith-enabled is not false
' |" | ~% w9 uStack: corosync
, k- o5 I5 n2 d) ECurrent DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum o: l% M9 V; i) q0 l0 O
Last updated: Thu Nov 30 19:30:43 2017
[& ~$ T( V5 `0 OLast change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller36 p6 y7 S9 g1 ]: A- L; v2 k/ P0 Y
% b8 B! x+ k4 p# ]- C$ t- _0 n3 nodes configured6 k* Y& M& f; M3 T1 m
0 resources configured7 Z! a- H3 e" u; G" k
" ?9 @4 B% c* h6 y1 i7 c. v$ x
Online: [ controller1 controller2 controller3 ]
; ] b; r0 y- u
9 c! d2 J$ l( n5 c, I) FNo resources- C( B1 t- L/ w, Y0 d" a6 V
8 z$ `7 d: T) Z* i
0 g) _. y2 G. q9 }+ q
Daemon Status:8 J# W+ f0 \% V) ?
corosync: active/enabled+ E s% d* ]% W) i
pacemaker: active/enabled
& R- T0 j) X. L* ? pcsd: active/enabled/ t3 R9 q C1 s! v3 e7 {2 q/ W
[root@controller1 ~]# pcs cluster status6 u9 A9 L, T' ~
Cluster Status:
8 p5 m: A; G7 d) `& \ Stack: corosync7 o4 v* K4 A% j% j5 m
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum4 l7 E; L" m9 ]( T: g5 p
Last updated: Thu Nov 30 19:30:52 20177 l* J! I( d; y: M# {4 H c; d- p
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller38 `1 s0 j, O( z
3 nodes configured* Z7 g& k y1 W! V2 Q$ v1 ]
0 resources configured7 P3 m7 L8 r6 I% n. I
' A6 d" j( \8 ~6 ]0 T! o2 p2 XPCSD Status:1 r% ]7 Z5 v; o/ \4 V
controller2: Online
, ?' q! E/ W3 e& F2 i' u controller3: Online
5 n, b/ r0 h# g6 ~3 N controller1: Online
" A5 O) L* `% N
: k: O( m" T. Z3 o' c0 s7 y% H三个节点都在线
/ b8 R% B( G9 S8 S3 Q0 V默认的表决规则建议集群中的节点个数为奇数且不低于3。当集群只有2个节点，其中1个节点崩坏，由于不符合默认的表决规则， 集群资源不发生转移，集群整体仍不可用。no-quorum-policy="ignore"可以解决此双节点的问题，但不要用于生产环境。换句话说，生 产环境还是至少要3节点。- ^8 k+ L: ]6 g$ P5 W
pe-warn-series-max、pe-input-series-max、pe-error-series-max代表日志深度。
2 N8 o. K! X+ G2 M$ o2 t/ o& ?cluster-recheck-interval是节点重新检查的频率。
/ F- G5 Z( q; h7 M[root@controller1 ~]# pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000 cluster-recheck-interval=5min
3 r7 B' _7 |6 E, c% j, t( X) N禁用stonith：
. w0 n% s" I+ b+ M: ]stonith是一种能够接受指令断电的物理设备，环境无此设备，如果不关闭该选项，执行pcs命令总是含其报错信息。, L4 O# {- P( ]1 N. {8 |
[root@controller1 ~]# pcs property set stonith-enabled=false
7 V8 }7 |4 [: D3 V二个节点时，忽略节点quorum功能：1 D$ ^' Y- C$ _ a$ B! f
[root@controller1 ~]# pcs property set no-quorum-policy=ignore5 a7 P- W- u* F' F% Y" a+ D. Q: B; z, |
验证集群配置信息
3 o( h+ Q. R' j" A2 b& V[root@controller1 ~]# crm_verify -L -V
: C+ Y) g/ h0 c* q1 F& m" \: f+ c为集群配置虚拟 ip
# A5 k# z: P6 h" R4 g[root@controller1 ~]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 \1 W& J/ @3 {5 W. W
ip="192.168.0.10" cidr_netmask=32 nic=eno16777736 op monitor interval=30s
3 Q/ C- t( u, y到此，Pacemaker+corosync 是为 haproxy服务的，添加haproxy资源到pacemaker集群
* e9 j" R# L# r[root@controller1 ~]# pcs resource create lb-haproxy systemd:haproxy --clone
3 b" w2 X# X5 {' c: @) q说明：创建克隆资源，克隆的资源会在全部节点启动。这里haproxy会在三个节点自动启动。# t ]! u$ B2 V* q) n' t6 q1 Z
查看Pacemaker资源情况
- u7 S1 v7 B3 S: l) B! R[root@controller1 ~]# pcs resource
7 ^3 B& s4 O# o& ?4 e4 U$ V& e ClusterIP (ocf::heartbeat:IPaddr2): Started controller1 # 心跳的资源绑定在第三个节点的0 O* z9 f, z' `
Clone Set: lb-haproxy-clone [lb-haproxy] # haproxy克隆资源
# M q o! A, ^* g Started: [ controller1 controller2 controller3 ]0 Y. ?: Y9 V& F4 P
注意：这里一定要进行资源绑定，否则每个节点都会启动haproxy，造成访问混乱4 P; e/ u3 a, X9 V: K5 @
将这两个资源绑定到同一个节点上
5 C. z8 Q& D5 e+ D; w* [6 A5 o[root@controller1 ~]# pcs constraint colocation add lb-haproxy-clone ClusterIP INFINITY& e ^* v' J5 t g! K# n
绑定成功% ]% A3 B" c8 n8 V {
[root@controller1 ~]# pcs resource
, W$ Z- H+ ^4 O ClusterIP (ocf::heartbeat:IPaddr2): Started controller3
9 @7 `$ t2 c0 L4 I1 I Clone Set: lb-haproxy-clone [lb-haproxy]
4 U2 L! G6 o0 L$ }& J1 P2 Z/ r Started: [ controller1]
& X" @: b- b* d" |8 a! d+ L Stopped: [ controller2 controller3 ]
% {) n+ Z, P, B5 [1 h" q4 ^3 Q配置资源的启动顺序，先启动vip，然后haproxy再启动，因为haproxy是监听到vip
# [0 Q U w0 H, T/ m[root@controller1 ~]# pcs constraint order ClusterIP then lb-haproxy-clone
( f9 X& R+ v1 v% T$ K+ ^. q5 V8 d# I手动指定资源到某个默认节点，因为两个资源绑定关系，移动一个资源，另一个资源自动转移。
, y( e5 q) B' H, \! A3 F( v; s
. z" h4 i3 h$ Q% h& Q2 |[root@controller1 ~]# pcs constraint location ClusterIP prefers controller1
# [8 H6 j5 {" v7 L[root@controller1 ~]# pcs resource
% q( B1 O- A: z) t" C0 J ClusterIP (ocf::heartbeat:IPaddr2): Started controller16 }8 V, S0 A( E* i% E. n
Clone Set: lb-haproxy-clone [lb-haproxy]
3 Q& d$ ^( j- ~, ^6 E) {* T Started: [ controller1 ]
% @; ?6 ?. F& G+ e3 j Stopped: [ controller2 controller3 ]
* r( y% a/ B/ c7 A[root@controller1 ~]# pcs resource defaults resource-stickiness=100 # 设置资源粘性，防止自动切回造成集群不稳定: P0 a# Z+ [: k( w1 z9 W
现在vip已经绑定到controller1节点
: z2 A( H4 S. T/ s2 G' t1 n: S; X- m[root@controller1 ~]# ip a | grep global2 D8 I" H; S' ]# Q
inet 192.168.0.11/24 brd 192.168.0.255 scope global eno16777736
+ v8 \' Y3 i: j% R1 K& P inet 192.168.0.10/32 brd 192.168.0.255 scope global eno16777736' b9 k- B2 d% ^% w: k
inet 192.168.118.11/24 brd 192.168.118.255 scope global eno33554992
3 L3 |9 E, x, C6 |) b1 E% o/ ?+ m. J$ @* s6 S. |( i+ Y9 w
尝试通过vip连接数据库
: f, Z6 S) ^7 E$ ^1 k( u# aController1:
5 j! e1 B& S: N( O5 l
$ F X8 q1 l8 i3 | g[root@controller1 haproxy]# mysql -ugalera -pgalera -h 192.168.0.10
9 @4 X) ^ {; Q- q& B' t6 C! [8 _0 X: {
; x l% e& ]8 ^( k2 ?
Controller2:) |* c( }! F8 X5 X o/ g3 z& f E
# k& \- ?% M! L& P; W0 n0 c+ e
4 g3 {" P; n3 I. h, w7 u高可用配置成功。
' s \; d( {# }4 x0 |. p8 [% x& r
% s7 u0 L0 K- j2 E# i! q+ I2 ]0 z6 Y测试高可用是否正常7 x& s. \' v( A: R
在controller1节点上直接执行 poweroff -f
2 ~& r: Y/ e0 V3 f. Q[root@controller1 ~]# poweroff -f
" S' t, \) O4 e& _vip很快就转移到controller2节点上
7 Q: \1 E" e) P" Z! [; V8 E4 D" R2 {6 y0 ~: p% m
再次尝试访问数据库
- g& H( O' Q" }; y2 i; Z3 {. s7 g) J5 y+ N6 ]5 s
* u/ l3 e8 t4 S7 E8 @" m) c
无任何问题，测试成功。1 O1 f. G5 ^( `2 T% _9 P# e/ Q& _
查看集群信息：: Q5 b' t' i# M5 u" Y
V& M% N! a5 r: ~+ [& X/ q/ E[root@controller2 ~]# pcs status ( h# d) r0 s& l3 R
Cluster name: openstack-cluster
, a7 @4 U( F' i; n5 h; j. ~Stack: corosync3 s! d5 ~1 I, h6 c' J3 P
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
- k5 }. \# A, n( YLast updated: Thu Nov 30 23:57:28 20179 P2 W$ e; \2 m" r0 f! O. V
Last change: Thu Nov 30 23:54:11 2017 by root via crm_attribute on controller1
" D& }; R' D, u8 z% R) Z- w: R# w
* [6 W/ i% ]$ F9 X' S3 nodes configured
( L# k8 `9 ~" b) `8 x: Q8 y4 resources configured
2 F% Y1 [% y9 G5 |8 L
* @1 _2 U+ z }, k) {" I _7 jOnline: [ controller2 controller3 ]
& T6 L5 i* o' ~. q4 t! j' ROFFLINE: [ controller1 ] # controller1 已经下线
$ C$ L1 ]: u) a: Q6 }4 q0 k, C7 Y
* k4 R# q6 N4 i* e9 HFull list of resources:3 S8 |! a& e r8 G) n
8 D* ~5 P' ?+ u2 d5 {/ w
ClusterIP (ocf::heartbeat:IPaddr2): Started controller2
+ b( J5 `# ^# r, ^/ y7 d( G5 F Clone Set: lb-haproxy-clone [lb-haproxy]
$ Q% I+ e9 `$ s* I7 X% b0 H$ W Started: [ controller2 ]
9 `' J4 k j7 r* p# i Stopped: [ controller1 controller3 ] p8 c( l1 N4 Y. {0 V. ~
7 r5 O) w3 n5 w' Q+ z$ L
Daemon Status:
, L% L# g# o& L* X2 C corosync: active/enabled
+ @9 l2 y. N& C! P" |0 w% n pacemaker: active/enabled7 W- e, I0 ]1 g$ m- L8 j
pcsd: active/enabled |
|
发表于 2018-10-19 16:05:14