|
|
楼主 |
发表于 2021-10-18 11:12:53
|
显示全部楼层
[root@controller1 _data]# kolla-ansible -i multinode mariadb_recovery, u, c4 u: ]" |7 d
Attempting to restart mariadb cluster : ansible-playbook -i multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy -e common_run=true /usr/share/kolla-ansible/ansible/mariadb_recovery.yml 8 O) S# z. e$ Q6 c8 a
/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
2 O+ ?) ~ y% P! N' Q' J from cryptography.exceptions import InvalidSignature
+ R" {, v8 D& t" J; b[WARNING]: Unable to parse /var/lib/docker/volumes/mariadb/_data/multinode as an inventory source
- i% k, X! O% h[WARNING]: No inventory was parsed, only implicit localhost is available
4 S$ i* e# Q$ L6 _$ V- C, K G$ ]$ n[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
9 G0 H7 K- k2 }+ u[WARNING]: Could not match supplied host pattern, ignoring: mariadb
9 j9 z# R; F! @4 M4 R* `5 y2 _1 {$ X, G ]% v
PLAY [Detect openstack_release variable] **************************************************************************************************************************2 W6 c' W& B, k
skipping: no hosts matched& C/ y* r) M, ?2 B& U! b
PLAY [Recover mariadb] ********************************************************************************************************************************************
1 I; f8 E' N. N& ^2 d) ?. L B/ j/ S) mskipping: no hosts matched
# m: L' {$ s* APLAY RECAP ********************************************************************************************************************************************************
& g L% e, d1 f/ T. W[root@controller1 _data]# cd
# r. E+ X* w/ z9 U. x[root@controller1 ~]# kolla-ansible -i multinode mariadb_recovery
- M" P# ^& K$ O. H5 uAttempting to restart mariadb cluster : ansible-playbook -i multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy -e common_run=true /usr/share/kolla-ansible/ansible/mariadb_recovery.yml - B+ e* k- i1 ]) q. a4 u9 B% R- \
/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.: @: Z+ ? Z9 E0 \# \: n. ~7 u" X
from cryptography.exceptions import InvalidSignature6 M- R/ S4 @0 m: I2 V" F
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
8 L, @* k1 q2 Z1 Z5 C8 e[WARNING]: Found variable using reserved name: action
5 Z( d9 Z" h4 ?7 [! N
! M: w. B0 D& h6 Q- HPLAY [Detect openstack_release variable] **************************************************************************************************************************$ B) ^2 y- F+ e
TASK [Get current kolla-ansible version number] *******************************************************************************************************************
* e5 w# ]2 m R1 vskipping: [controller1]
8 O# o! `% v$ {- A4 j, bskipping: [controller2]8 O: n3 F5 ~% b% X& k% M1 c
skipping: [controller3]# h) z. }- h% @4 K
TASK [Set openstack_release variable] *****************************************************************************************************************************; K) Y6 c$ n0 _, D0 F
skipping: [controller1]
1 X. u8 x; |: h0 V& I& b& d5 c9 Zskipping: [controller2]+ o: k- d* L% Z/ ~9 X
skipping: [controller3]
* a2 z8 S- [5 u; Z7 EPLAY [Recover mariadb] ********************************************************************************************************************************************
3 i3 u" m& A" E% FTASK [Gathering Facts] ********************************************************************************************************************************************& U, C2 d2 K7 A
ok: [controller1]
) E6 Q. p/ h9 j; W( U4 G. mok: [controller3]( Z( B+ C, L ` g7 b; c0 W
ok: [controller2]
1 l) G! r4 W1 r7 z1 aTASK [common : include] *******************************************************************************************************************************************
/ E# N7 o w6 W4 e7 ?9 ^skipping: [controller1]
; \- v% B' y+ D0 J& [skipping: [controller2]
7 O* t; _3 v6 {# Pskipping: [controller3]6 U% V( y% x5 d" k) I; E
TASK [common : Registering common role has run] *******************************************************************************************************************- A$ A7 i* z' f1 P4 k6 P
skipping: [controller1]! [! v9 \2 p* Z( `. z% u
skipping: [controller2]
0 o q' D, S& `- Pskipping: [controller3]! j- P9 K. b3 q% n. k" Q0 d6 r
TASK [mariadb : include] ******************************************************************************************************************************************( v/ F1 ?/ F3 w' a2 P
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/deploy.yml for controller1, controller2, controller3
1 ]- [ h w$ mTASK [mariadb : include] ******************************************************************************************************************************************
' j: b* g9 Q5 j _* |* K- c+ Sincluded: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/config.yml for controller1, controller2, controller3- u1 n( y# t- q2 T( J/ Y
TASK [mariadb : Ensuring config directories exist] ****************************************************************************************************************& x2 S: p& d% h, x8 M) _* n
ok: [controller2] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
+ h3 ~7 P0 @5 [- zok: [controller3] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
* T; a% [. R3 L U5 r" T1 e- Z/ K. nok: [controller1] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
( K+ Z1 ?9 o* C! J( X1 [( PTASK [mariadb : Copying over config.json files for services] ******************************************************************************************************" k6 e1 c7 @3 D. n
ok: [controller2]
, r8 Y* U. A* n. X8 @& nok: [controller3]
! I* k. l% I& s& m$ w( Yok: [controller1]
- s5 k" r2 C0 cTASK [mariadb : Copying over galera.cnf] **************************************************************************************************************************
) f& {4 V Q; o[WARNING]: _execute_module no longer honors the tmp parameter. Action plugins should set self._connection._shell.tmpdir to share the tmpdir
. _& e. P9 c7 F/ }7 d[WARNING]: _execute_module no longer honors the tmp parameter. Action plugins should set self._connection._shell.tmpdir to share the tmpdir
* P9 ?5 ^! ^; g9 t! a/ V" r[WARNING]: _execute_module no longer honors the tmp parameter. Action plugins should set self._connection._shell.tmpdir to share the tmpdir
/ H$ V, h) t. D4 H/ @4 n5 \ok: [controller3]& L9 Y5 ?; j" R
ok: [controller2]
' p# ], N6 Z* ?5 o' vok: [controller1]. Q$ _) x/ P7 `& f" S1 [7 k
TASK [mariadb : Copying over wsrep-notify.sh] *********************************************************************************************************************& n# m& y3 G% e3 b" o* m0 k( B
ok: [controller2] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'}): y9 U$ {1 r, F) ]* s( ]
ok: [controller1] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
z) }7 a8 @3 o( M) B1 Y# Pok: [controller3] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})4 y6 M; I) j9 _; a4 Z7 N4 C
TASK [mariadb : Check mariadb containers] *************************************************************************************************************************
5 x( V/ B6 B, @" w* T: {) k: schanged: [controller3] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})! H, ]4 {3 @% J* `+ z& }2 l
changed: [controller1] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})! F/ p: y [3 h5 A6 A; N9 C/ B
changed: [controller2] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
/ K+ ?; f _8 v* \8 g8 S( jTASK [mariadb : include] ******************************************************************************************************************************************8 S. a6 B# j! W0 `* q
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/bootstrap.yml for controller1, controller2, controller3
! l/ L/ r* J9 ?TASK [mariadb : set_fact] *****************************************************************************************************************************************
0 c4 u9 e' j7 _6 xok: [controller1]2 e, Y1 F, j3 J& P; y
ok: [controller2]
* K V7 ^9 K3 ^6 c+ U! b" lok: [controller3]
, N' J) n M6 N8 G; Y, l1 qTASK [mariadb : include] ******************************************************************************************************************************************+ G4 h/ S( ^: f4 {+ S
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/lookup_cluster.yml for controller1, controller2, controller3
+ \/ b0 j5 S; D" h3 N9 @TASK [mariadb : Cleaning up temp file on localhost] ***************************************************************************************************************
) R, n6 ~7 I$ I: |3 ]% r$ j/ aok: [controller1 -> localhost]
$ X- N [0 o3 a! R3 H* W) H3 STASK [mariadb : Creating temp file on localhost] ******************************************************************************************************************( E. ^& s6 z- B r, H1 I! n
ok: [controller1 -> localhost]) ~% N1 a2 E2 p3 s! W, o% `) G" h
TASK [mariadb : Creating mariadb volume] **************************************************************************************************************************
6 Q E) Y+ n& V" E Qok: [controller1]2 y* U4 H. X- E
ok: [controller2]
6 n# F, w; R9 X: vok: [controller3]
; N# P' q; D/ O# Z; M0 O! yTASK [mariadb : Writing hostname of host with existing cluster files to temp file] ********************************************************************************4 M* y# {, ]$ }* w
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version / z9 V4 q1 I8 X* h
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.8 M8 d+ J) A/ N& i8 q
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version
4 ]: f4 V& [6 W# d+ q3 U2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
; g4 E& y' O3 b1 ]: G[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version ' T8 v; J, W" A" l5 s3 D
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.5 D% u+ r) Z- V* R& T! B
ok: [controller1 -> localhost]
9 F8 `# {1 `5 j( Y6 @- x# Xok: [controller2 -> localhost]
# Q8 a; L" c* k1 c9 X' `6 `3 Lok: [controller3 -> localhost]
$ D) J. K# u T. I, d0 c, bTASK [mariadb : Registering host from temp file] ******************************************************************************************************************; M9 ]6 Q/ R8 r1 O! X4 a) P. m q
ok: [controller1]8 i9 b* o3 X4 o) {) j) `5 m
ok: [controller2]
: [) u9 b+ K) C' x# ^& m, Tok: [controller3]
9 H1 `1 G6 H$ U4 P2 Q7 MTASK [mariadb : Cleaning up temp file on localhost] ***************************************************************************************************************
. L2 V7 n9 Z4 }$ |- Y8 }0 L# Aok: [controller1 -> localhost]" U- u- l- W4 m' ^ i! `
TASK [mariadb : include] ******************************************************************************************************************************************
O8 A( \) h; h8 K% v" Xskipping: [controller1]
$ @0 ]3 A o8 q7 Y. Wskipping: [controller2]
! y9 |# V% u) d5 y5 }( rskipping: [controller3]
" C" R6 V- b3 Q. p, QTASK [mariadb : include] ******************************************************************************************************************************************2 F" F0 F8 [( x& q, K& ?
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/recover_cluster.yml for controller1, controller2, controller3* k! z" U5 n% n: ]/ @
TASK [mariadb : fail] *********************************************************************************************************************************************1 J4 I; L4 F2 V m5 d9 Y& {
skipping: [controller1]
7 H* i1 J: L3 i0 dskipping: [controller2]
; d3 y! u; ^, J' U2 Tskipping: [controller3]2 {9 K+ d1 v" D8 q- b- @! a2 Z
TASK [mariadb : Cleaning up temp file on mariadb hosts] ***********************************************************************************************************6 c4 f5 w. w, N$ }& w# |6 i6 Y
ok: [controller2]; L+ I( m! i& W: s
ok: [controller1]
: Z% ?) x' Y, `% x. d/ [" \$ [. v+ ^ok: [controller3]+ v( g; Q7 L, p5 Z$ p8 R0 N+ H
TASK [mariadb : Cleaning up temp file on localhost] ***************************************************************************************************************
. x- P# q Y6 K- l/ Nok: [controller1 -> localhost]& K! y$ R$ Y' h8 v, H0 I/ N
TASK [mariadb : Stop MariaDB containers] **************************************************************************************************************************/ I% |/ X( X- h* k
ok: [controller2]
; M, ^/ N; ?0 zok: [controller1]2 f) c4 Y7 r& I* q6 l1 t0 G( g
ok: [controller3], H* t% ~0 B8 c8 d3 z
TASK [mariadb : Run MariaDB wsrep recovery] ***********************************************************************************************************************
1 c* i z9 r( Y9 i# uchanged: [controller1]
/ T. Q7 d$ q) s( h& ~5 Schanged: [controller2]
& ~; b" H2 @8 E; {- ichanged: [controller3], [2 B* X- F- N: W5 A: p
TASK [mariadb : Copying MariaDB log file to /tmp] *****************************************************************************************************************
* O3 ?# j& Z3 W! X& xchanged: [controller3]2 l u4 m" ], t! P7 _4 B
changed: [controller2]
3 P( S* R6 `' Q+ Bchanged: [controller1]
. [+ p# w: }" g3 ETASK [mariadb : Get MariaDB wsrep recovery seqno] *****************************************************************************************************************1 k: W* s/ G( c% F
changed: [controller2]4 N& t" T. s9 S! ~) A
changed: [controller1]
- Q( c! L, D3 i f0 r/ xchanged: [controller3]% e, c8 y* ^: R) u
TASK [mariadb : Removing MariaDB log file from /tmp] **************************************************************************************************************) h6 a3 H2 h( z5 F7 G& U7 \6 h
ok: [controller2]& j" b4 L5 F! W+ C
ok: [controller1]
* J5 i! P7 C! M4 T9 d6 `2 Xok: [controller3]
* v$ n( w- _0 V% C$ F0 NTASK [mariadb : Registering MariaDB seqno variable] ***************************************************************************************************************
! ] x( @$ C/ O: }ok: [controller1]% l2 I; W/ k, B6 u7 H
ok: [controller2]
, M: q4 d: [6 p; m* C9 q% x" Xok: [controller3]
E8 j) v2 d, Y, _TASK [mariadb : Comparing seqno value on all mariadb hosts] *******************************************************************************************************
3 [: `4 D! a% D1 L2 A9 m6 C" T* ~ok: [controller2] => (item=controller1)
1 j( {2 z+ N3 U5 x b5 c- [2 e$ Qok: [controller1] => (item=controller1)
0 i7 m' y- E* g# u+ U# C& j( Iok: [controller3] => (item=controller1)2 v; W5 \( p" R5 ~2 A# B
ok: [controller2] => (item=controller2)
+ h* |, j& T/ Y. L. F% wok: [controller3] => (item=controller2). \6 j8 e1 W o* B$ }2 v/ E
ok: [controller1] => (item=controller2)
4 h, g* e* i# Y8 m; o; Y# Y: Wok: [controller2] => (item=controller3)
3 \" c9 e. d7 [/ O4 z. ]! jok: [controller3] => (item=controller3)8 d$ ~) ?, a4 {) ]7 ?! h' l
ok: [controller1] => (item=controller3)
# C4 [0 j$ { p6 |9 a# @* WTASK [mariadb : Writing hostname of host with the largest seqno to temp file] *************************************************************************************
$ ~; D8 c. i; S3 ]7 a- Wskipping: [controller3]
3 p+ J$ A- p% y1 G9 iok: [controller1 -> localhost]5 h; M' k1 `) v
ok: [controller2 -> localhost]
! n, F8 M: Q% KTASK [mariadb : Registering mariadb_recover_inventory_name from temp file] ****************************************************************************************
+ e9 |( G9 Q6 y. J* J& B" c% J1 cok: [controller1]8 Q5 S4 J* O' x, H$ ~
ok: [controller2]8 v1 q* `# B- p i
ok: [controller3]
# g+ x) g' R0 \; e: | zTASK [mariadb : set_fact] *****************************************************************************************************************************************! N! `8 H, E; Z+ c# K
changed: [controller1]2 g' I! I* u$ m; h
changed: [controller2]
0 l: F+ D; x# vchanged: [controller3]2 n3 f2 D8 B9 f8 u
TASK [mariadb : Copying grastate.dat file from MariaDB container in bootstrap host] *******************************************************************************
; H$ p4 ~# i, l; @/ g+ C9 _4 W" h; ]' P0 Jskipping: [controller1]/ l$ n. ^( W1 i2 ] M) X
skipping: [controller3]
0 G& c% X) t m$ S5 \3 [8 W4 pok: [controller2]- K% p8 O# L/ p
TASK [mariadb : Set grastate.dat file from MariaDB container in bootstrap host] ***********************************************************************************4 @1 ^* v8 k+ x$ @' B8 _4 O
skipping: [controller1]
6 g. ^3 {0 z7 V1 Fskipping: [controller3]. E$ j# n! ~7 x, M# \, B0 l* S) ^
changed: [controller2]
+ P' e( K9 X( b! z1 rTASK [mariadb : Copying grastate.dat file to mariadb container] ***************************************************************************************************
# [9 o3 `' q$ |2 V* \skipping: [controller1]% o" C. L7 N# n; s) l5 a4 G
skipping: [controller3]
q: e2 `& o0 w$ w0 i2 O1 S$ cok: [controller2]
) m+ \+ {5 [2 x) C, l0 U$ ETASK [mariadb : Starting first MariaDB container] *****************************************************************************************************************
" f& e: S5 y5 | Oskipping: [controller1], P) U" p1 B- A: h8 h/ ]" P
skipping: [controller3]& i) z( ?. N; O' G& o
changed: [controller2]+ R5 {, T. W8 C1 P. n( ~
TASK [mariadb : Wait for first MariaDB container] *****************************************************************************************************************7 v% U& V$ z2 @5 p1 s9 Z, y6 b. K
skipping: [controller1]
5 @: J# N% M7 E: d9 w9 [9 Hskipping: [controller3]
" a5 q$ }* D1 {" N, s; ^6 v' Y[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
8 o* G2 b& x. r) a3 B; f2 U2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.' J5 h- ^ p" b5 _9 p6 M% k
FAILED - RETRYING: Wait for first MariaDB container (10 retries left).+ R1 w& {0 Q7 F) o! d* F0 u; c4 w
ok: [controller2]" l7 k9 T& D |. `" K
TASK [mariadb : Set first MariaDB container as primary] ***********************************************************************************************************
/ U3 _1 z4 k; ]7 D% u* d6 ?2 o9 Sskipping: [controller1]8 w; T- t+ {- R& P
skipping: [controller3]" ~; @( j" g" z: R; U1 R
changed: [controller2]
0 f9 i! h7 G# z" QTASK [mariadb : Wait for MariaDB to become operational] ***********************************************************************************************************0 @8 J% ]# F# a$ h0 O
skipping: [controller1]# P- K4 n. K u% F
skipping: [controller3]
; d c d) l( e8 fok: [controller2]
e& K+ ~- X x. KTASK [mariadb : Restart slave MariaDB container] ******************************************************************************************************************1 R5 m% Q0 T# b6 g$ _/ S
skipping: [controller2]
. F+ v/ J& k. z m: ]( T( pchanged: [controller3]- z$ m! M! U7 f5 \) ?
changed: [controller1]
& S, b5 a4 a3 q3 N' O6 }TASK [mariadb : Wait for slave MariaDB] ***************************************************************************************************************************
5 j3 G& d4 h& ]4 Z1 Tskipping: [controller2]( m/ R" a% _( B7 B
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
9 L B& [8 b% I2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
' }3 v1 K+ Y- d3 l1 \FAILED - RETRYING: Wait for slave MariaDB (10 retries left).
: C% X7 o$ {1 r7 G( I3 a5 O9 ^2 s[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
+ }1 n1 L+ \) }5 c6 ]2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
( s- H4 ~- P- p1 w0 Q0 HFAILED - RETRYING: Wait for slave MariaDB (10 retries left).
4 X0 D# T1 h$ ~4 Tok: [controller3]9 C& }& o5 t8 }3 g! F4 p
ok: [controller1]
* j% \6 e% m5 s1 q[WARNING]: flush_handlers task does not support when conditional
/ Z2 D m5 \- z" G' N" \8 d! M9 l( b. y" v1 z( U
RUNNING HANDLER [mariadb : restart slave mariadb] *****************************************************************************************************************0 P" j8 O2 I. N& a& D7 n* m# c. T% ^
skipping: [controller3]
+ ], l/ h# x$ M* _0 Vskipping: [controller1]
! z4 t S e3 a. \" X6 kskipping: [controller2]
+ v9 d. J9 N4 D0 W5 H: DRUNNING HANDLER [mariadb : restart master mariadb] ****************************************************************************************************************
# B* k* b% q# h4 _- b3 yskipping: [controller3]
# [7 Z$ p/ ~* g8 T6 eskipping: [controller1]4 K# T- S: F: u$ C9 A
skipping: [controller2]
. w8 S9 W8 a, j+ O" HTASK [mariadb : include] ******************************************************************************************************************************************
! B; |( `9 \1 [+ ~9 W' [, Uincluded: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/register.yml for controller1, controller2, controller3
6 M b3 A" m- ? K/ b- yTASK [mariadb : Creating haproxy mysql user] **********************************************************************************************************************, C: v8 \" U" _( [1 d1 C
ok: [controller1]. Q! P7 y1 s" u* h! ^% h' n o
TASK [mariadb : Cleaning up facts] ********************************************************************************************************************************3 c+ n3 ^) w" F1 G2 C0 p
ok: [controller1]
! m# q7 P$ j9 q& ?. wok: [controller2] a, K1 q0 Y5 _: X
ok: [controller3]2 }% X& W! u4 v$ q, L2 `" ^+ _
TASK [mariadb : include] ******************************************************************************************************************************************8 `, `! u1 b, y6 f
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/check.yml for controller1, controller2, controller3% o G% U; L+ C4 D. n
TASK [mariadb : Waiting for MariaDB service to be ready through VIP] **********************************************************************************************$ [7 h- i% F8 k# }1 n7 c* K
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version ' R/ D/ V1 s w( C! n
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. X2 T8 q; G% e7 j! f/ F
ok: [controller1]
' K/ f, i$ a% j* ?9 B/ H[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version , Y, p: T* e% c$ w1 L3 |
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.9 x5 _% s0 P) \$ }% c) y2 @9 _
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version : `% v F; b, t! A# z
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.( c, |) r4 u$ A- W
ok: [controller2]4 o* f* l0 g. Q9 u7 T" O
ok: [controller3]; a: z' o S1 U( x, q* C3 H" y
PLAY RECAP ********************************************************************************************************************************************************- d: a$ ~# _' ^0 I0 u h
controller1 : ok=37 changed=6 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 2 \$ Y% z! D$ c/ N7 x# T& L
controller2 : ok=37 changed=8 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 / B* E: } f8 Y' P
controller3 : ok=31 changed=6 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 , `& K7 N3 A; H& n" L1 D: _8 w& U' ?
t, {1 A& ^) K( h6 [: D0 @) `
恢复正常。
8 f% x3 N$ N: z% r1 t8 q* ^9 V. R6 z; ` V3 I- b
3 i! I. V# v6 {$ B$ ~' i3 V
有时候通过一些特殊手段可以帮助我们解决一些特别的问题，与其长时间尝试老方法，不如来点新的方式试试或许不一样的效果。5 c! k0 p8 U- g" H
% k5 p0 d9 R/ z8 n8 @/ _9 @' W/ {7 u5 N
我们首先是因为数据库恢复出问题，无法恢复数据，， 检查数据的库的值，
' |9 L$ i# [8 ~2 u- I2 r2 `" c _cat /var/lib/docker/volumes/mariadb/_data/grastate.dat 中seqno值，若该值在所有节点中存在唯一得最大值/ H( e# K9 W6 \/ J
然后我们保留这个节点上数据，
* n; Y1 Z! ~* ]$ h+ y: ?删除其他节点上的数据文件
$ w: C& X8 ^. v) @! p& d/ e6 }9 \然后通过scp或者rsync同步保留节点的数据文件 （根据个人喜好同步文件，本人建议使用rsync）( y' G ?8 G% T
然后再执行kolla-ansible数据库mariadb_recovery恢复命令：6 h2 O! B& x- T6 V2 @( M8 P
6 W( y& c( u5 M5 Z( u) }; o. E: w5 e
- ^; r* y/ ] M
4 n9 t$ }) \1 J* _+ w9 x" ]6 {2 f/ N- {4 D
差不多就可以完成数据库集群恢复了。% {2 m. u: {0 g. Y
|
|
发表于 2021-10-18 11:03:46