Openstack-Mitaka 高可用之 Pacemaker+corosync+pcs 高可用集群

admin

介绍及特点
! Q! ^$ H/ y' H! _1 h    Pacemaker：工作在资源分配层，提供资源管理器的功能
    Corosync：提供集群的信息层功能，传递心跳信息和集群事务信息
/ A1 e6 G  \  o" _" j8 z; Z: k1 G    Pacemaker + Corosync 就可以实现高可用集群架构
# l+ p7 e! k3 u) y& _
, f3 r' C, N3 B# ~- I 集群搭建
( k+ E7 m& t( X$ C6 W4 _以下三个节点都需要执行：

# yum install pcs -y
# systemctl start  pcsd ; systemctl enable pcsd
8 h; z8 f1 X$ f$ ^# echo 'hacluster' | passwd --stdin hacluster
7 M4 m" X6 g1 O, S# yum install haproxy  rsyslog -y
9 t$ D- a0 R* [# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf        # 启动服务的时候，允许忽视VIP的存在
" j1 i) ^" g& e$ A3 u# k# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf        # 开启内核转发功能
8 m2 L# M7 T' ?, \6 H- J# sysctl -p
7 B' E% N0 O, q5 E- w/ Z
在任意节点创建用于haproxy监控Mariadb的用户
MariaDB [(none)]> CREATE USER 'haproxy'@'%' ;
配置haproxy用于负载均衡器

[root@controller1 ~]# egrep -v "^#|^$" /etc/haproxy/haproxy.cfg
) f% `9 K9 D! p    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
0 p4 @# G! X6 R. e, q    pidfile     /var/run/haproxy.pid
    maxconn     4000
7 I6 Q. V. O9 [, }4 ^7 o    user        haproxy
/ q4 X; S; H" R    group       haproxy
1 k$ D+ T! r5 W$ W  I    daemon
    # turn on stats unix socket
# A0 p; U2 l& [( [/ P    stats socket /var/lib/haproxy/stats
+ ]4 I% ^* T5 I5 Q5 ?& x! Qdefaults
    mode                    http
    log                     global
) Z% K6 s: B- x, ]% w- N# `, e    option                  httplog
0 v: L: b$ ^4 X7 f7 n    option                  dontlognull
# g3 _! `8 ?, [    option http-server-close
    option forwardfor       except 127.0.0.0/8
% v" q$ D* u' Q$ J0 O    option                  redispatch
    retries                 3
; |8 [+ s) b" T, _3 a    timeout http-request    10s
    timeout queue           1m
( U! x+ {: M: k: y! E' M2 v    timeout connect         10s
    timeout client          1m
" c4 o+ T( F4 M, J1 U, w/ V    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
, T9 _" n+ V/ b+ I& C$ C' E9 W    maxconn                 4000
listen galera_cluster
    mode tcp            
/ W2 \' \! O- @, B: ?9 y3 D    bind 192.168.0.10:3306
    balance source
    option mysql-check user haproxy
    server controller1 192.168.0.11:3306 check inter 2000 rise 3 fall 3 backup
    server controller2 192.168.0.12:3306 check inter 2000 rise 3 fall 3
    server controller3 192.168.0.13:3306 check inter 2000 rise 3 fall 3 backup

/ J# T& S/ e2 A" D# mlisten memcache_cluster
    mode tcp
    bind 192.168.0.10:11211
    balance source
    option tcplog
    server controller1 192.168.0.11:11211 check inter 2000 rise 3 fall 3
    server controller2 192.168.0.12:11211 check inter 2000 rise 3 fall 3
3 p. |) Z! @  K    server controller3 192.168.0.13:11211 check inter 2000 rise 3 fall 3

5 |: `! Z! V' c# T
注意：
    （1）确保haproxy配置无误，建议首先修改ip和端口启动测试是否成功。
    （2）Mariadb-Galera和rabbitmq默认监听到 0.0.0.0 修改调整监听到本地 192.168.0.x
    （3）将haproxy正确的配置拷贝到其他节点，无需手动启动haproxy服务
为haproxy配置日志（所有controller节点执行）：

& Z; a) j" y% u* u# vim /etc/rsyslog.conf
3 y  c7 O$ [! [- j…
9 O  r* T+ F) U$ModLoad imudp
- ]" p& d  b; I% v" `( j9 W4 `, Q$UDPServerRun 514
9 ^3 L3 \: k& ^…
4 ?5 @" u9 F* Y7 x' blocal2.*                                                /var/log/haproxy/haproxy.log
* A9 N0 n: I( H! w…
9 A3 U5 G, I+ \7 B7 i4 K6 ]$ w
4 a6 x1 l3 e. D- D+ |6 i! r# mkdir -pv /var/log/haproxy/
mkdir: created directory ‘/var/log/haproxy/’
0 w* t( }3 j6 A5 s/ c# Y
* x7 ~& R; K5 D% h  G9 i% d# systemctl restart rsyslog
2 _) c( q! F3 m$ N
; e: F$ b+ L; J+ [& F+ k' `启动haproxy进行验证操作：

# systemctl start haproxy
1 f, m+ l% T" {; w; S0 ]$ I[root@controller1 ~]# netstat -ntplu | grep ha
tcp        0      0 192.168.0.10:3306       0.0.0.0:*               LISTEN      15467/haproxy      
tcp        0      0 192.168.0.10:11211      0.0.0.0:*               LISTEN      15467/haproxy      
3 }4 a! R+ P; B9 ]. n7 Tudp        0      0 0.0.0.0:43268           0.0.0.0:*                           15466/haproxy

验证成功，关闭haproxy
# systemctl stop haproxy
* L4 I! `) i( c6 I! @7 i7 e

/ x) d' U/ J- V2 E在controller1节点上执行：
) W* R0 u  n8 ~1 }, x% h- u[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p hacluster --force
controller3: Authorized
controller2: Authorized
% u& ^) y* w/ B5 p+ w; D3 j; \controller1: Authorized
7 ]/ \1 w1 Z+ \# s6 |% B创建集群：
$ V) W4 n+ F) j) Z
[root@controller1 ~]# pcs cluster setup --name openstack-cluster controller1 controller2 controller3  --force
4 F, u2 C: p! Y+ tDestroying cluster on nodes: controller1, controller2, controller3...
controller3: Stopping Cluster (pacemaker)...
controller2: Stopping Cluster (pacemaker)...
controller1: Stopping Cluster (pacemaker)...
9 ~9 W6 g9 X# Acontroller3: Successfully destroyed cluster
& [9 a% I! N6 C& v- h' Zcontroller1: Successfully destroyed cluster
controller2: Successfully destroyed cluster
9 P$ V" m8 b( Q- n* K: |
7 Y& B1 ]1 ~. f( f  hSending 'pacemaker_remote authkey' to 'controller1', 'controller2', 'controller3'
+ `) t" g- I; B( g/ `controller3: successful distribution of the file 'pacemaker_remote authkey'
, h! f' Q; i! Y9 a3 @$ kcontroller1: successful distribution of the file 'pacemaker_remote authkey'
controller2: successful distribution of the file 'pacemaker_remote authkey'
6 o4 v' k+ s7 t1 t$ N2 fSending cluster config files to the nodes...
0 J( d2 f+ s  Y9 p: q( j2 bcontroller1: Succeeded
controller2: Succeeded
controller3: Succeeded
, ]9 S5 {2 j9 M2 q' \8 _# b0 g$ H
Synchronizing pcsd certificates on nodes controller1, controller2, controller3...
; G% b- T$ x% X# w6 `3 m: Dcontroller3: Success
* v" f- Z$ Q0 ]. t! dcontroller2: Success
controller1: Success
0 Q, M( X5 g& Y8 Z$ f+ mRestarting pcsd on the nodes in order to reload the certificates...
controller3: Success
( ]4 e6 q9 ~& n6 x. y+ ]0 Gcontroller2: Success
: n+ K8 M7 D$ icontroller1: Success

, w- b, Z4 s+ o8 A. }2 K启动集群的所有节点：

1 ^6 e) U2 f3 q( p[root@controller1 ~]# pcs cluster start --all
8 X* ^8 U/ U9 @. S4 T+ econtroller2: Starting Cluster...
, q9 Z/ P* M' C3 e6 Y2 H& ycontroller1: Starting Cluster...
, U7 {4 D% f; A$ rcontroller3: Starting Cluster...
[root@controller1 ~]# pcs cluster enable --all
, P- r' {1 ^% Q3 q! W+ d5 ~controller1: Cluster Enabled
controller2: Cluster Enabled
controller3: Cluster Enabled
: u- E% A. Y1 g  l2 d5 }% t, `) x
查看集群信息：

[root@controller1 ~]# pcs status
% v" K4 _3 S' {- t5 A, x, O" u4 PCluster name: openstack-cluster
" P9 ~( a: Q  tWARNING: no stonith devices and stonith-enabled is not false
# [* b7 ], ~1 l+ YStack: corosync
6 L5 K' ^& _6 X7 O( K) lCurrent DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:43 2017
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3

3 nodes configured
% N, U5 |# _- U5 h# z9 i' x0 U- S0 resources configured

Online: [ controller1 controller2 controller3 ]

No resources
- H: R" O# C1 s8 y

, o, @; L% Q. r) T/ eDaemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
# j- S- u4 N: r% F- m$ }  pcsd: active/enabled
( g+ o8 k3 J) _' C2 |* [3 i[root@controller1 ~]# pcs cluster status
) B3 u( ~5 u! _8 FCluster Status:
Stack: corosync
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:52 2017
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3
3 nodes configured
, o$ K! T/ o. L+ n 0 resources configured
* ?4 I0 D! }# V& r
PCSD Status:
, Q+ g. Y: t8 E2 f& K7 w  controller2: Online
7 ]1 @, [) l, X  controller3: Online
7 K) {) W* I" }7 b) ]: A2 M( l6 H' X  controller1: Online
2 a* j3 `. g* b) u) [+ Q0 q
2 {6 k) o: F8 }' X" p" A三个节点都在线
- b/ J) I& U* N4 ]1 x1 f7 w9 G' P默认的表决规则建议集群中的节点个数为奇数且不低于3。当集群只有2个节点，其中1个节点崩坏，由于不符合默认的表决规则， 集群资源不发生转移，集群整体仍不可用。no-quorum-policy="ignore"可以解决此双节点的问题，但不要用于生产环境。换句话说，生 产环境还是至少要3节点。
( A0 n. k; k$ M2 X2 S0 q" j" Ppe-warn-series-max、pe-input-series-max、pe-error-series-max代表日志深度。
7 ]1 Q9 j" L: O1 r9 Y6 bcluster-recheck-interval是节点重新检查的频率。
4 j* O. x+ E8 T[root@controller1 ~]#  pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000 cluster-recheck-interval=5min
: Q' f9 P* c5 ]' a, A禁用stonith：
stonith是一种能够接受指令断电的物理设备，环境无此设备，如果不关闭该选项，执行pcs命令总是含其报错信息。
[root@controller1 ~]# pcs property set stonith-enabled=false
二个节点时，忽略节点quorum功能：
[root@controller1 ~]# pcs property set no-quorum-policy=ignore
验证集群配置信息
[root@controller1 ~]# crm_verify -L -V
为集群配置虚拟 ip
[root@controller1 ~]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 \
% l' |1 @) q' w; }# }; J2 [ ip="192.168.0.10" cidr_netmask=32 nic=eno16777736 op monitor interval=30s
- C/ F. C4 \* C/ e! o到此，Pacemaker+corosync 是为 haproxy服务的，添加haproxy资源到pacemaker集群
' T5 C( G* m" v! D; |7 z[root@controller1 ~]# pcs resource create lb-haproxy systemd:haproxy --clone
说明：创建克隆资源，克隆的资源会在全部节点启动。这里haproxy会在三个节点自动启动。
查看Pacemaker资源情况
& ]# |" k. F: v[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1        # 心跳的资源绑定在第三个节点的
Clone Set: lb-haproxy-clone [lb-haproxy]        # haproxy克隆资源
& Q! e# H2 l2 v3 G6 I; `2 P     Started: [ controller1 controller2 controller3 ]
注意：这里一定要进行资源绑定，否则每个节点都会启动haproxy，造成访问混乱
% L* a. P: C8 o7 Q8 L9 Z, e将这两个资源绑定到同一个节点上
[root@controller1 ~]# pcs constraint colocation add lb-haproxy-clone ClusterIP INFINITY
' j, F+ }: F6 h1 B绑定成功
[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller3
Clone Set: lb-haproxy-clone [lb-haproxy]
     Started: [ controller1]
1 i4 ~0 M9 s: g& d* U6 V9 B     Stopped: [ controller2 controller3 ]
配置资源的启动顺序，先启动vip，然后haproxy再启动，因为haproxy是监听到vip
0 p% h# g, [% v0 e9 ^[root@controller1 ~]# pcs constraint order ClusterIP then lb-haproxy-clone
; b" d* m& m# f. |. j, O8 |+ c, V手动指定资源到某个默认节点，因为两个资源绑定关系，移动一个资源，另一个资源自动转移。
$ ?9 j9 H7 v5 u5 X/ ~, d
[root@controller1 ~]# pcs constraint location ClusterIP prefers controller1
[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1
Clone Set: lb-haproxy-clone [lb-haproxy]
  @( E6 T2 C; [/ d. W- O9 C     Started: [ controller1 ]
     Stopped: [ controller2 controller3 ]
$ ]' H) k8 k8 S# d# C; |[root@controller1 ~]# pcs resource defaults resource-stickiness=100        # 设置资源粘性，防止自动切回造成集群不稳定
现在vip已经绑定到controller1节点
( G3 f" [2 ?$ G7 `  E9 b, B[root@controller1 ~]# ip a | grep global
! L; S) p/ S+ m" l8 d' g  T    inet 192.168.0.11/24 brd 192.168.0.255 scope global eno16777736
# |# v9 h5 q( j6 ?    inet 192.168.0.10/32 brd 192.168.0.255 scope global eno16777736
    inet 192.168.118.11/24 brd 192.168.118.255 scope global eno33554992

尝试通过vip连接数据库
9 A3 H* ?1 C; n0 e3 J# iController1:
# \2 |& n+ T  i6 v" d
[root@controller1 haproxy]# mysql -ugalera -pgalera -h 192.168.0.10
1 E) O% y% `) }* @4 y# P+ p: A$ U
+ |- o% j2 b! X( B5 s* p) @
9 e3 C& I1 A, @6 u1 Y Controller2:

　
% y: A) A% p2 P( d. [: g高可用配置成功。
* P7 n5 E$ Y7 v7 ~4 a
测试高可用是否正常
在controller1节点上直接执行 poweroff -f
6 k! V+ U+ R0 r5 \9 X9 p% E[root@controller1 ~]# poweroff -f
) d% r9 E5 k5 z; V3 d: Svip很快就转移到controller2节点上

再次尝试访问数据库
% ?& P: i( J6 F3 {' k, Y

( w4 `4 y! w/ }4 x/ C 无任何问题，测试成功。
查看集群信息：
/ `5 T* O# M9 g- c
3 q9 Q1 @/ v) W9 \[root@controller2 ~]# pcs status
Cluster name: openstack-cluster
8 V* ~6 h" p* R, kStack: corosync
$ P$ Y1 H% Q( @Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
2 P: j& B; N: n/ n8 tLast updated: Thu Nov 30 23:57:28 2017
Last change: Thu Nov 30 23:54:11 2017 by root via crm_attribute on controller1
5 W8 E; X) x  S5 X
3 nodes configured
4 resources configured

Online: [ controller2 controller3 ]
OFFLINE: [ controller1 ]            # controller1 已经下线

Full list of resources:
  @. M& b4 i$ O
, k$ C9 T- [# z1 o/ P: _ ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller2
* r6 x' N4 \9 M) v Clone Set: lb-haproxy-clone [lb-haproxy]
8 K6 a  Z; H3 b" e2 y1 {     Started: [ controller2 ]
: o+ s) m5 Y! L( ~     Stopped: [ controller1 controller3 ]

  T0 n+ C, U) k) i0 fDaemon Status:
1 k$ W& m. A5 Q6 f9 @  corosync: active/enabled
  pacemaker: active/enabled
6 x3 Z( y" J: x- \  ?/ \4 ~8 |( P  pcsd: active/enabled