How to make MaxScale High Available with Corosync/Pacemaker

admin

MaxScale, an open-source database-centric router for MySQL and MariaDB makes High Availability possible by hiding the complexity of backends and masking failures. MaxScale itself however is a single application running in a Linux box between the client application and the databases - so how do we make MaxScale High Available? This blog post shows how to quickly setup a Pacemaker/Corosync environment and configure MaxScale as a managed cluster resource.
, ^) l0 S+ }4 ^! U$ z! R+ b% QAnyone following the instructions detailed here, modifying configuration files and issuing system and software checks could create a complete setup with three Linux Centos 6.5 servers and unicast heartbeat mode.
In a few steps MaxScale will be ready for basic HA operations and one simple failure test, the running process manually killed, is showed as an example.
8 x3 o* g# n2 [# u7 kWe make the following assumptions here:
The solution is a quick setup example that may not be suited for all production environments.
Pacemaker/Corosync and crmsh command line tools usage is known at a basic level
9 L0 _8 j/ ]0 n1 E. L" Y1 U% f. l" EA Virtual IP is set providing the access to the MaxScale process
MaxScale is already configured and working with a MariaDB/MySQL replication setup or MariaDB Galera Cluster
/ t" ~+ l6 p% ~MaxScale process is started/stopped and monitored via /etc/init.d/maxscale LSB compatible script that is available in RPM package from version 1.0. The script might be found in the GitHub repository, for Ubuntu as well.
) S7 ^* _" t# Y6 aStep 1 - Clustering Software installation
On each cluster node do the following operations:
7 Z- N9 [& u% l" o4 b  fLet’s start enabling  a new repo
5 @9 R  V# N) {" j7 ?# vi /etc/yum.repos.d/ha-clustering.repo
! a( Q# I% t9 ?7 kand add the following lines to the file
[haclustering]
name=HA Clustering
* `: @0 y0 Q/ Y5 k" ~. D
baseurl=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/
, g' D2 p; D- V0 C# e, r
enabled=1

5 z7 h0 K7 d5 k4 r4 a2 B) sgpgcheck=0
+ C2 s) U: P$ {0 H. {# K$ LNow install the software.
# j1 d, U: g1 r! P+ c# yum install pacemaker corosync crmsh
. t) @. ?. g" }& L. k  z) b# g/ _Please note the packages versions used in this setup are:
pacemaker-1.1.10-14.el6_5.3.x86_64
8 {4 e4 E$ g1 q7 Ucorosync-1.4.5-2.4.x86_64
crmsh-2.0+git46-1.1.x86_64
Step 2 - Configuring  the system
1 m" k7 F( [$ B! F2 yLet’s begin assigning the hostname to each node:
2 W* z) U* K1 e2 Y0 I% }The node names are: node1,node,node3
# hostname node1
& w2 B# {( {$ v7 ^: p...
# hostname nodeN
and write entries in /etc/hosts:
" `5 w( w# q  }4 }For each node add the server names and current-node, that is as an alias for the current server.
# vi /etc/hosts
4 U; J% o. ~0 |6 S10.74.14.39     node1
+ q/ F& ^. {5 o+ Y4 l) ?, c10.228.103.72   node2
0 c4 x  t! f" B10.35.15.26     node3 current-node     ...
# vi /etc/hosts
10.74.14.39     node1 current-node
10.228.103.72   node2
% P/ P( x3 h4 z: z& b/ W10.35.15.26     node3
! ^: w& j. K7 w9 ]Prepare authkey for optional cryptographic use:
On one of the nodes, say node2 run the corosync-keygen utility and follow the instructions.
6 ~7 j% p! @' ]; X[root@node2 ~]# corosync-keygen
2 f1 z) ~, ~) X$ z5 v2 j; d
+ Q* n6 ^) _* O% M( |Corosync Cluster Engine Authentication key generator.
. ^0 B/ ^; q/ ?8 C6 \. b( ]       Gathering 1024 bits for key from /dev/random.
       Press keys on your keyboard to generate entropy.
After completion the key will be found in /etc/corosync/authkey.
( b2 ?1 I+ l3 D( }Now let’s create the corosync configuration file:
4 ?1 D/ d$ u5 W# V0 Y[root@node2 ~]# vi /etc/corosync/corosync.conf
Add the following content to the file:
& A, Y/ Q! p$ T2 y8 i- v, M" X; m# Please read the corosync.conf.5 manual page
! a3 y! W/ T& Xcompatibility: whitetank
2 p6 u8 [' c# R& ^6 g7 f% X, U9 z
, q0 V. u/ O5 m. N# Q2 m, Z2 M- v& o& Ttotem {
        version: 2
        secauth: off
        interface {
( Q$ M' h% K0 j3 y                member {
                        memberaddr: node1
                }
  `3 {; H9 x0 U* m! g9 U. d                member {
$ J4 D4 h8 z' D! v0 f/ {                        memberaddr: node2
                }
                member {
                        memberaddr: node3
                }
             ringnumber: 0
/ @0 N# u5 h/ }) C0 w' l0 P/ K5 l/ u                 bindnetaddr: current-node
6 d7 F( Z8 W5 [6 W: V6 }1 u7 k: O* _                 mcastport: 5405
                 ttl: 1
1 Z6 o  [* G  X: U$ M        }
        transport: udpu
}
- t* d2 ?  M: D: v* B7 g( q! d+ i
# B. a8 T/ m+ S  z* s2 i3 Clogging {
        fileline: off
        to_logfile: yes
$ [2 V+ F- Q) \        to_syslog: yes
2 f. O- N  `2 m. v- d! _        logfile: /var/log/cluster/corosync.log
8 ^/ B0 `" Y* J, |        debug: off
" g$ o7 O9 A: L8 {1 z) t! C+ r3 W        timestamp: on
3 Z9 f# w& g" k4 H. o; c        logger_subsys {
) K; p* \; O2 b                subsys: AMF
9 L& A3 K# v0 N                debug: off
6 L$ z# d" a+ F4 h2 A1 X
7 x! p5 H! E& ~+ a' ^+ [        }
& k4 p$ w' I/ h. E}
- w  [9 {) N" {% }% f
# this will start Pacemaker processes
, J$ v7 B+ Q+ g1 Uservice {
; Z6 X& t! h& s& }! Q- Wver: 0
& }( ?  M; V% u4 {" a9 h3 J( Ename: pacemaker
}
A few notes here:
Unicast UDP is used
+ Y) ~! R( p) a+ X) \bindnetaddr for Corosync process is “current-node”, that has the right value on each node due to the alias added in /etc/hosts above
Pacemaker processes are started by the Sorosync daemon, so there is no need to launch it via /etc/init.d/pacemaker start
We can now copy configuration files and auth key on each of the other nodes:
8 l: w9 j$ D) ?- p, d9 S[root@node2 ~]# scp /etc/corosync/*  root@node1:/etc/corosync/
3 n. _3 D$ `5 \4 g...
[root@node2 ~]# scp /etc/corosync/*  root@nodeN:/etc/corosync/
Step 3 - Start the Cluster
9 v: j7 q8 I6 p+ s, ^, |The Cluster can be started now but let’s do additional checks before proceeding. Corosync needs UDP port 5405 to be opened so we need to configure any firewall or iptables accordingly.
0 e5 [! v* |1 ?7 U9 Y2 q+ |For a quick start just disable iptables on each nodes:
[root@node2 ~]# service iptables stop
…
3 ~+ ?* b" P: U$ I[root@nodeN ~]# service iptables stop
( R# ?7 R& F5 [Let’s start Corosync on each node:
1 K0 ~+ Z  _9 k[root@node2 ~] #/etc/init.d/corosync start
…
[root@nodeN ~] #/etc/init.d/corosync start
1 B4 P( j6 a1 z4 v: s" |and check if the corosync daemon is successfully bound to port 5405:
[root@node2 ~] #netstat -na | grep 5405
& e# ]$ [1 p; I! p. Y
udp        0      0 10.228.103.72:5405        0.0.0.0:*
Check also if other nodes are reachable with nc utility and option UDP (-u):
3 m# h$ n' g  d/ U* m" r" e: n[root@node2 ~] #echo "check ..."  | nc -u node1 5405
* J' M+ A3 Z7 F, p0 @6 D[root@node2 ~] #echo "check ..."  | nc -u node3 5405
4 Y- m: L  g; ]* f' K% C# \…
[root@node1 ~] #echo "check ..."  | nc -u node2 5405
2 R  v( Z3 F9 n; U[root@node1 ~] #echo "check ..."  | nc -u node3 5405
" T( j6 R7 O0 G- K4 uIf the following message is displayed:
9 {9 D* b5 C0 e& p9 d" D# [nc: Write error: Connection refused
2 Y( C* Q, q) F9 F  |there is an issue with communication between the nodes, this is most likely to be an issue with the firewall configuration on your nodes.
Please check and resolve issues with your firewall configuration.
We can check the cluster status, from any node, with this command:
[root@node3 ~]# crm status
+ z, e9 W/ f6 _: VAfter a while the output will look like:
; [$ f( _+ m, z9 e/ {[root@node3 ~]# crm status
Last updated: Mon Jun 30 12:47:53 2014
0 S0 l: L( o) s+ E7 ULast change: Mon Jun 30 12:47:39 2014 via crmd on node2
3 d. Z  y5 f; \5 AStack: classic openais (with plugin)
Current DC: node2 - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
3 Nodes configured, 3 expected votes
0 Resources configured


Online: [ node1 node2 node3 ]
The Cluster has been started successfully, that’s the first achievement so far!
0 H  k0 r3 H9 v; n, L# n& APlease note, in the basic setup we will disable the following properties:
stonith
quorum policy
' I7 f+ y( v6 x$ _3 g[root@node3 ~]# crm configure property 'stonith-enabled'='false'
0 q0 z; m1 c8 D& O" q[root@node3 ~]# crm configure property 'no-quorum-policy'='ignore'
7 [$ ^9 z6 h, J- b* C, kAfter these commands the configuration is automatically updated on every node and we want to check it from another node, say node1
. U  x7 i9 A9 X( E- E[root@node1 ~]# crm configure show
& R2 j+ w. N' h7 G0 C5 |" C* c
1 f- ?4 a8 y0 _) l+ [6 U8 _node node1
# s  J% O$ o/ w1 ]7 Anode node2
node node3
property cib-bootstrap-options: \
        dc-version=1.1.10-14.el6_5.3-368c726 \
        cluster-infrastructure="classic openais (with plugin)" \
        expected-quorum-votes=3 \
' Z* [3 K5 I! p: g& k0 ^        stonith-enabled=false \
        no-quorum-policy=ignore \
- O6 q, y" m7 K# b& @- S1 v& e( f$ U        placement-strategy=balanced \
8 L/ E  K$ ?4 O' R        default-resource-stickiness=infinity
4 q: g: r5 v$ d" R# `6 W  oWell done, the Corosync / Pacemaker cluster is now ready to manage resources, in the next steps we’ll add MaxScale.
Step 4 - Check MaxScale init script
: a$ A8 o- T) s. B% D5 x5 u% }The new MaxScale /etc/init.d./maxscale script allows to start/stop/restart and monitor MaxScale process running in the system.
  s" V) l4 j- r" oThe script found in the RPM package is already working with the following path: /usr/local/skysql/maxscale
It might be necessary to modify some variables such as MAXSCALE_HOME to match the installation directory you choose when you installed MaxScale or MAXSCALE_PIDFILE or LD_LIBRARY_PATH
We assume here MaxScale is configured with a MariaDB/MySQL replication setup or MariaDB Galera Cluster and those servers might be located in the three Linux boxes we are using or anywhere else.
Following commands should be issued on each node, assuring the application could run and managed:
[root@node1 ~]# /etc/init.d/maxscale
; T8 m2 }7 d. k6 hUsage: /etc/init.d/maxscale {start|stop|status|restart|condrestart|reload}
5 ]+ m4 d# q' h' c" U& jStart
[root@node1 ~]# /etc/init.d/maxscale start
Starting MaxScale: maxscale (pid 25892) is running...      [  OK  ]
Start again
[root@node1 ~]# /etc/init.d/maxscale start
Starting MaxScale:  found maxscale (pid  25892) is running.[  OK  ]
9 B4 L$ Y" @, ?' `/ Q1 z) ^Stop
[root@node1 ~]# /etc/init.d/maxscale stop
Stopping MaxScale:                                         [  OK  ]
Stop again
/ F8 D) H5 A9 w5 k) G! ~$ t[root@node1 ~]# /etc/init.d/maxscale stop
Stopping MaxScale:                                         [FAILED]
Status (MaxScale not running)
[root@node1 ~]# /etc/init.d/maxscale status
) M- m& f) Y% XMaxScale is stopped                                        [FAILED]
$ ]6 w# |0 j8 G# O. w" k2 g; r# PStatus (MaxScale is running)
$ T4 k3 [+ A  b[root@node1 ~]# /etc/init.d/maxscale status
Checking MaxScale status: MaxScale (pid  25953) is running.[  OK  ]
As MaxScale script is LSB compatible, returns the proper exit code for each action, it’s now possible to configure the application as a resource in Pacemaker, next step will show how to do it.
/ ^# u  v9 ^) [* S1 g6 [) Z, \Step 5 - Configure MaxScale as a cluster resource
% P- q2 W* B. h* E# ]  j& zWe are assuming here MaxScale could run on each node with the same configuration file.
. _4 F, o) ?0 p8 P6 J* s" u[root@node2 ~]# crm configure primitive MaxScale lsb:maxscale \
op monitor interval=”10s” timeout=”15s” \
op start interval=”0” timeout=”15s” \
& E8 I: V1 w9 S  q7 jop stop interval=”0” timeout=”30s”
. A6 `3 g) m- t! k1 TThe command above has configured MaxScale as a LSB resource, note “lsb:maxscale”
- K8 j3 W3 f5 O& a( P# dIn Pacemaker there are two different ways for managing applications:
Resource Agents (VIP, MySQL, Filesystem etc)
6 L' R" @, D) m4 J6 p# qLSB scripts for applications that don’t require the complexity of a resource agent and custom applications, in general.
MaxScale itself manages the backend servers we had configured in etc/MaxScale.cnf service sections such as:
) L9 ^' a1 B  F4 k[RW Split Router]
type=service
router=readwritesplit
* x# j" [+ F3 K; k2 hservers=server1,server2,server3,server4,server5,server6,server7
% W; a  _) r3 _3 u( e1 K! i# }0 T6 puser=maxuser
passwd=maxpwd
So we only want Pacemaker to manage the MaxScale process and the LSB approach is well suitable here.
, E9 U$ Q; U  ?If everything is fine we should see the resource running:
) \6 z7 n' x& Q1 m0 T2 e( W[root@node2 ~]# crm status
Last updated: Mon Jun 30 13:15:34 2014
Last change: Mon Jun 30 13:15:28 2014 via cibadmin on node2
8 e+ X! }0 e" S. F1 `' TStack: classic openais (with plugin)
( {- Z# k! t  `+ _" c, N! B% `Current DC: node2 - partition with quorum
+ A. w' |5 C8 q- |6 _Version: 1.1.10-14.el6_5.3-368c726
  ?7 f7 _7 t" ]; x3 Nodes configured, 3 expected votes
* {7 d% Y% B" r4 x6 Y$ g& g0 |6 b/ e1 Resources configured
6 t" U* y2 n& D) r  K. D% q
Online: [ node1 node2 node3 ]
, B1 h( d  _* g5 K+ r( E
/ v& ]$ a% ^6 T3 H; u4 fMaxScale        (lsb:maxscale):        Started node1
Well done, another achievement here!
, a8 _% K) |+ Z% E+ P1 xWe now have MaxScale running via Pacemaker and we don’t need anymore to have it started via /etc/init.d at boot time! Pacemaker will do all the job but it needs to be started at boot: with CentOS 6.5 setup we need at least:
# chkconfig maxscale off
# chkconfig corosync on
) y1 [7 {7 c  \! Y4 ~! u  gStep 6 - Does the HA software work? Let’s see a resource restarted after a failure:
, L+ Z! w$ t/ D- a+ |MaxScale application is now managed by the HA clustering software but what does it mean?
Will the application be restarted in case of any failure? It should be!
/ @2 S0 L4 A' k3 O2 iWe try now to kill the MaxScale process and see what will happen ...
As we now MaxScale PID could be easily found in $MAXSCALE_HOME/log/maxscale.pid
& {, D& l' x- L" h1 X( V: BIn this example the PID is 26114, and we kill the process with brute force:
- @2 ~, C' w; z% |  C# `: N5 |[root@node2 ~]# kill -9 26114
% ^% t9 j! n6 u) ^  v2 ^
# B' |8 M4 ^$ J* F; z# }; r; h[root@node2 ~]# crm status
Last updated: Mon Jun 30 13:16:11 2014
. G/ a: z- a% X9 \7 WLast change: Mon Jun 30 13:15:28 2014 via cibadmin on node2
' T, A5 g$ F/ F& n4 \7 y3 qStack: classic openais (with plugin)
5 W! F/ h4 S' t% r) LCurrent DC: node2 - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
4 [' F/ Q' D1 l! z, n3 Nodes configured, 3 expected votes
1 Resources configured
" Z( i1 C! T& Z* S% N  t
* `4 R0 M+ K7 G+ ?/ b9 b. H9 qOnline: [ node1 node2 node3 ]
3 |6 s9 D( ~# H0 a9 Y. I1 Y; F
Failed actions:
    MaxScale_monitor_15000 on node1 'not running' (7): call=19, status=complete, last-rc-change='Mon Jun 30 13:16:14 2014', queued=0ms, exec=0ms
% ~8 F0 a4 c9 |/ I4 |* HNote the MaxScale_monitor failed action above and ... after a few seconds it will be started again:
[root@node2 ~]# crm status
Last updated: Mon Jun 30 13:16:22 2014
Last change: Mon Jun 30 13:15:28 2014 via cibadmin on node1
/ u8 n+ f7 Z' k1 W$ l! VStack: classic openais (with plugin)
: B& D5 X" [( L4 d/ w/ MCurrent DC: node2 - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
3 Nodes configured, 3 expected votes
1 Resources configured
7 o. L8 W+ z* }5 v; t1 S! t

Online: [ node1 node2 node3 ]
- X0 n# I: p. u
1 ~+ p: s, v9 L) E- f4 \' _ MaxScale        (lsb:maxscale):        Started node1
5 a' i2 W( ^, h- L3 T8 l8 Y! jThe Clustering HA software will keep MaxScale running in one of the three Linux boxes we have but … which node? and how could we connect to MaxScale from our client application if we don’t know where it runs?
# mysql -h $MAXSCALE_IP -P 4006 -utest -p test
" b9 Q. A0 c+ p+ j0 Y% ^3 w9 pWhat is the $MAXSCALE_IP then? Let’s Follow the last step ...
Step 7 - Add a Virtual IP (VIP) to the cluster
The solution for $MAXSCALE_IP is that MaxScale process should be contacted using one known IP, that may move across nodes with MaxScale as well.
7 f# |% q( ^2 h' EThe setup is very easy: assuming an addition IP address is available and that it can be added to one of the nodes, this i the new configuration to add:
' u3 L0 q$ O1 S9 V) N[root@node2 ~]# crm configure primitive maxscale_vip ocf:heartbeat:IPaddr2 params ip=192.168.122.125 op monitor interval=10s
There is of course another action to do: MaxScale process and the VIP must be run in the same node, so it’s mandatory to add to the configuration the group ‘maxscale_service’.
[root@node2 ~]# crm configure group maxscale_service maxscale_vip MaxScale
Here is the final configuration:
[root@node3 ~]# crm configure show
node node1
node node2
node node3
: e. j- T" D& U" x* Tprimitive MaxScale lsb:maxscale \
        op monitor interval=15s timeout=10s \
        op start interval=0 timeout=15s \
. T4 h  @8 f& Z3 V( K0 l        op stop interval=0 timeout=30s
primitive maxscale_vip IPaddr2 \
        params ip=192.168.122.125 \
        op monitor interval=10s
group maxscale_service maxscale_vip MaxScale \
        meta target-role=Started
property cib-bootstrap-options: \
* I+ p8 M& t5 p0 w; X  x        dc-version=1.1.10-14.el6_5.3-368c726 \
/ N, k( u- A, V% Y/ m$ v8 [        cluster-infrastructure="classic openais (with plugin)" \
        expected-quorum-votes=3 \
        stonith-enabled=false \
8 m: A- @! k. B' q3 E! d/ m        no-quorum-policy=ignore \
        placement-strategy=balanced \
        last-lrm-refresh=1404125486
% c5 d# v  d/ c2 l1 F7 @2 MCheck the resource status:
. b2 K7 n+ z7 Y5 y) y8 I[root@node1 ~]# crm status
Last updated: Mon Jun 30 13:51:29 2014
. L* b( j. m3 O# W$ xLast change: Mon Jun 30 13:51:27 2014 via crmd on node1
Stack: classic openais (with plugin)
( Y4 b  N' W3 E: T/ j/ N3 LCurrent DC: node2 - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
8 e( Q3 h% Y$ v7 A% B. z! z6 l3 Nodes configured, 3 expected votes
2 Resources configured
; x/ C/ a+ }# V- r: k1 y9 [6 E
Online: [ node1 node2 node3 ]
1 J( k% r7 i' M6 }8 F: m; R
* c7 {  A( h. X6 o, \" S0 H Resource Group: maxscale_service
3 o5 G0 ~. y0 R, s     maxscale_vip        (ocf::heartbeat:IPaddr2):        Started node2
     MaxScale        (lsb:maxscale):        Started node2
) g" S6 F2 e5 p; H7 L8 s/ }With both resources on node2, now MaxScale service will be reachable via the configured VIP address 192.168.122.125:
) S& b8 C+ ?5 Y1 Q+ y1 W/ Y# mysql -h 192.168.122.125 -P 4006 -utest -p test
Please note our three Linux boxes setup require now four IP addresses: one for each node plus the moving IP address assigned to MaxScale
Summary
5 N8 b; g# P& H+ z( dThe goal of this post was to present a quick HA solution for a running MaxScale setup, using a widely adopted open-source clustering solution.
Even though the main content could be seen as a basic Corosync/Pacemaker setup guide, I encourage you to look for other failure scenarios and all the cluster administrative commands such as moving resources, adding constraints that could be found through the links below.
The reader might fin the LSB script tutorials interesting too, just enabling another application to the HA side,