找回密码
 注册
查看: 616|回复: 1

记一次模拟测试cephfs reset恢复状态 1 filesystem is degraded

[复制链接]

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
发表于 2022-8-5 14:26:41 | 显示全部楼层 |阅读模式
[root@ceph-3 test]# rados ls -p metedata |grep 2.00000000 |xargs -i rados -p metedata rm {}
3 G- K2 J* W$ h# `9 _[root@ceph-3 test]# rados ls -p metedata |grep 2.00000000
! M( x' y3 k5 @$ R[root@ceph-3 test]# systemctl restart ceph-mds@
' P8 l3 k% j$ ], e3 Z& B% t1 v: m- yceph-mds@                ceph-mds@ceph-3.service  $ Z# w. d, r; \3 k% Z2 ~; g
[root@ceph-3 test]# systemctl restart ceph-mds@
6 ^2 f" v7 D0 n9 ?9 O, pceph-mds@                ceph-mds@ceph-3.service  
+ u3 \7 i4 N) A. [* M[root@ceph-3 test]# systemctl restart ceph-mds@
" _7 A0 F( r5 j* Jceph-mds@                ceph-mds@ceph-3.service  5 V; Y! E) x' s9 _
[root@ceph-3 test]# systemctl restart ceph-mds@ceph-3.service
6 t; v# J8 z+ n5 c+ c: Q  L[root@ceph-3 test]# ceph fs status
3 a0 {; b# D/ j2 D$ b                                         #Show the status of a CephFS filesystem
0 D# \6 R' C4 g% q3 \) s[root@ceph-3 test]# ceph fs status 7 M" ]5 y+ v2 E8 x
                                         #Show the status of a CephFS filesystem
: A: c  J9 l3 D. _4 I( W[root@ceph-3 test]# ceph fs status
/ C5 X3 T1 e' L2 [8 V; }cephfs - 1 clients
; p. E3 _" r8 j9 Q* F3 \======0 g! q* T0 u9 s3 m# r, z4 Z
+------+--------+--------+---------------+-------+-------+# ?. h+ k$ t6 X7 H7 J2 s6 ]
| Rank | State  |  MDS   |    Activity   |  dns  |  inos |: d! j# ~% c/ U- z/ O4 `& ~
+------+--------+--------+---------------+-------+-------+1 O% H: P- }1 |8 K0 r
|  0   | active | ceph-1 | Reqs:    0 /s |   11  |   13  |) \# r8 q. b0 F" r6 K
+------+--------+--------+---------------+-------+-------+9 C* r7 s, u: d4 a9 t5 j
+------------+----------+-------+-------+. _. o5 L/ q8 O. h# z; a
|    Pool    |   type   |  used | avail |
, f% u8 B& B1 E0 G2 A/ j0 }) a+------------+----------+-------+-------+
! _' z( I& l6 y|  metedata  | metadata | 5911  |  277G |
8 W. `  E6 k' s0 S% p| cephfsdata |   data   | 44.5k |  277G |" U" ]3 J4 B! ?
+------------+----------+-------+-------+8 A1 ^- c& [2 {9 F% i; h* E

% s; E7 {& R2 I+-------------++ x9 S0 y2 Z' T4 K
| Standby MDS |% O6 u: {$ J- \- }
+-------------+
; c8 t5 h5 p! ?6 A|    ceph-2   |
  V# ]* E8 r) E; ?0 N9 w|    ceph-3   |; o$ n, d# s% y$ X4 M9 e3 F, F
+-------------+
& @6 H1 r7 h; \' S4 qMDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
! M# c) V; x* W' k' f  l7 W[root@ceph-3 test]# ceph fs status 4 M9 ?  }/ R0 p
cephfs - 1 clients- N0 x9 V4 V7 y& k# R5 Z* m+ G
======
1 _3 E* X2 G. z. P0 E8 {6 O+------+--------+--------+----------+-------+-------+( K+ q; p3 W2 ^, i, C) u
| Rank | State  |  MDS   | Activity |  dns  |  inos |- ?8 o& P+ U0 _5 s. K% k, f
+------+--------+--------+----------+-------+-------+0 l  c% A3 L  j3 j' N9 H
|  0   | replay | ceph-3 |          |    0  |    0  |
! D( |. S2 C0 N9 C( G- [, P0 h+------+--------+--------+----------+-------+-------+8 |( l$ D! H- V/ U' v. H
+------------+----------+-------+-------+: [/ L7 y; |' u. P" Y' L
|    Pool    |   type   |  used | avail |
( f, W9 T0 h5 J+ M* I6 b1 d+------------+----------+-------+-------+
1 L0 r' Y5 m) C1 q! K  D|  metedata  | metadata | 5911  |  277G |
( `3 O4 ^! @2 D' o& M5 V) x| cephfsdata |   data   | 44.5k |  277G |
% ?4 ?/ T# x& m  p+------------+----------+-------+-------+1 l; F' x1 u* G2 ]) f$ t" O& D6 b

$ `: M  k! K8 Y/ H1 _+-------------+$ I* v! Z: D3 |# n) `. R- f+ u
| Standby MDS |1 [  x- N8 p% ]; o( X" i8 }
+-------------+
- [1 M! k$ G4 [# N8 Z$ s/ D|    ceph-2   |, N1 o. `6 m' W" h7 _
+-------------+* u0 N/ ^9 }% Y2 e' Y% P
MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)! E9 x' V" y$ f
[root@ceph-3 test]# ceph fs status 8 X2 W, d* u/ _* [2 `
cephfs - 1 clients5 h% ?5 Z7 g3 M# O! m# H' F5 G1 O
======- S4 P& `! L& Q+ j# c, t7 }6 m
+------+-----------+--------+----------+-------+-------+( M' }& s# k; z2 `# u
| Rank |   State   |  MDS   | Activity |  dns  |  inos |
6 Q+ h8 o: W# d" g: @9 V+------+-----------+--------+----------+-------+-------+
; {# q1 M# a3 j4 C3 C( R. Y|  0   | reconnect | ceph-3 |          |    0  |    0  |' g& o  L. s4 m* O8 j
+------+-----------+--------+----------+-------+-------+
( v* _+ W  ]1 k; f2 b+------------+----------+-------+-------+
4 P! X, k: k& r+ Q1 ?|    Pool    |   type   |  used | avail |: i1 R/ r. ?. ]- {2 O4 k. A# b+ E
+------------+----------+-------+-------+$ d- [' M/ i" s* ?0 ]) \4 b
|  metedata  | metadata | 5911  |  277G |7 z! @  d9 y8 _( e7 V
| cephfsdata |   data   | 44.5k |  277G |5 U% ?% x! _9 t- ?
+------------+----------+-------+-------+# v2 r5 R2 X" D# S

- n% m3 H- e6 U, l+-------------+' f- t2 E+ i3 z" L! e2 |3 q
| Standby MDS |) a3 H# n/ K5 W( n$ L, ^4 Q/ k. b
+-------------+
0 Z* t, S: W' f8 L/ U* O2 w+-------------+6 _; [# @3 r. ?6 ?' K9 ^& Z8 N
MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
" `8 e$ m4 q7 q( D" M3 Z[root@ceph-3 test]# ceph fs status
3 M, V: w0 t/ x/ r! L/ F  t7 Ecephfs - 1 clients
4 V, x0 H( g4 V======4 `( S/ \' A5 @7 }3 @( H
+------+-----------+--------+----------+-------+-------+
: h& }' E% ~$ c( P, V3 b" @. y| Rank |   State   |  MDS   | Activity |  dns  |  inos |
' [8 }) L3 \  D, r3 l+------+-----------+--------+----------+-------+-------+" }" B- i9 O+ J' {
|  0   | reconnect | ceph-3 |          |    0  |    0  |/ i4 b1 ]$ B+ e( ~
+------+-----------+--------+----------+-------+-------+0 `* O+ p' y; `
+------------+----------+-------+-------+% `$ n0 S0 O7 G' U  F
|    Pool    |   type   |  used | avail |% Y. N" X0 j' f! b
+------------+----------+-------+-------+4 R: f& w  v0 J# U% V: j- |
|  metedata  | metadata | 5911  |  277G |
5 I; o% R+ Z5 ^. `& e$ G# ?/ s4 k| cephfsdata |   data   | 44.5k |  277G |
7 S7 M* n% B% C+------------+----------+-------+-------+: F( j, U, K. h! v) n* L
* A: E4 @4 V5 V- j' q9 E; N3 Z
+-------------+( r: l3 B- m9 X4 [/ l' r9 L: f8 v
| Standby MDS |
) x5 w9 _% ^- q: |, t5 }+-------------+
- V& _- |2 L5 u; x( ^+-------------+4 m; G7 a  A* C1 N: s  K
MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)0 ~! }& m# B$ G! e
[root@ceph-3 test]# ceph fs status
% B( u8 ~, z$ \2 M* Wcephfs - 1 clients, ^0 v4 v" ^" S2 v/ j
======" [- P, c1 ~. G7 Q4 l
+------+--------+--------+----------+-------+-------+% Q6 z/ P: S* u% m' |
| Rank | State  |  MDS   | Activity |  dns  |  inos |0 b1 `* ]5 ~; H
+------+--------+--------+----------+-------+-------+) \$ j7 O$ y) Q! w
|  0   | rejoin | ceph-3 |          |    1  |    3  |% m1 P9 M3 m- c
+------+--------+--------+----------+-------+-------+3 |5 T: u6 P& X9 G! R3 i
+------------+----------+-------+-------+
1 z) V8 U0 s1 R, C0 E|    Pool    |   type   |  used | avail |' q* ?. i7 ?% u- ?
+------------+----------+-------+-------+2 C# s5 l" V. K. L( D2 F/ i
|  metedata  | metadata | 5911  |  277G |9 H) X  g" z$ ]0 ]* ^
| cephfsdata |   data   | 44.5k |  277G |6 @! n& I: H' r  ?/ t' x
+------------+----------+-------+-------+/ t3 S9 P  t: u) Y) X% w
# e& n! R9 I  i0 m3 _% p: S
+-------------+
  Y4 j8 Q/ a1 J& ]& Z# j| Standby MDS |  J6 l& O2 \! [7 C3 G
+-------------+$ [1 Y4 F. j$ o' r
|    ceph-1   |2 ^  C) o& y! s8 k4 m
+-------------+2 w' P6 Y$ n% E
MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
+ R- I' @, ~. W; ^6 O& {7 w[root@ceph-3 test]# ceph fs status
4 ]" r* c* |1 Q; E. icephfs - 1 clients& d) b/ }3 E) C3 z) G
======
2 m5 c/ n1 {2 ^, p6 x0 F+------+--------+--------+----------+-------+-------+
0 c- }6 R, A- S: b9 V, V  k; m| Rank | State  |  MDS   | Activity |  dns  |  inos |
  W: Z. a0 D$ h6 U* Q: S7 k0 ?+------+--------+--------+----------+-------+-------+
1 C, x) \2 e/ [5 `8 d( l$ N|  0   | rejoin | ceph-3 |          |    1  |    3  |( T  N+ v4 B) J' b5 G, r9 O
+------+--------+--------+----------+-------+-------+: J' e/ P+ f0 u. k
+------------+----------+-------+-------+
2 c% B) Q7 N/ x, d9 u) u% ?|    Pool    |   type   |  used | avail |! Z$ H  _- ^2 r, S* W
+------------+----------+-------+-------+5 X( @6 p3 l2 m% K
|  metedata  | metadata | 5911  |  277G |% T: p; o3 q4 F$ O. b
| cephfsdata |   data   | 44.5k |  277G |
6 Z1 M# ^: L; s* A' M' o3 @+------------+----------+-------+-------+" ?3 D0 k/ k4 y0 Q' ~

( H" c4 y2 P4 `" `9 ]+-------------+
6 k  ]2 U6 G% ^/ F| Standby MDS |2 p) K" X* P& h" m* C# b
+-------------+4 @" C5 W! \4 S
|    ceph-1   |
' {' l# G: u  l1 R/ i6 [8 O+-------------+
: E. p$ J8 `  I- f) w$ R9 cMDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
$ |  x) t8 m- P[root@ceph-3 test]# ceph fs status
/ |' D! V+ o/ V2 r# `- b/ x4 xcephfs - 1 clients
, N4 l/ P9 x5 Q/ [  W======8 m/ B" n6 }8 h. h* \$ I0 N
+------+--------+--------+----------+-------+-------+" R6 \" a0 m( J1 Z5 e& z0 Q
| Rank | State  |  MDS   | Activity |  dns  |  inos |
) F( M6 m/ w6 i+------+--------+--------+----------+-------+-------+
8 T4 p9 ]' Y' O|  0   | rejoin | ceph-3 |          |    1  |    3  |
; Z% W, p9 k) y3 [' n7 w) m+------+--------+--------+----------+-------+-------+
/ o8 }3 H7 M# J6 C' i: Y4 D+------------+----------+-------+-------+
0 m5 ?' U9 v+ x! x% H|    Pool    |   type   |  used | avail |
2 R! `8 j3 b! e! _# j( z) N+------------+----------+-------+-------+
+ V- u+ D. |6 g/ G$ {! f) }|  metedata  | metadata | 5911  |  277G |& V. B9 c& q" u7 Q* i
| cephfsdata |   data   | 44.5k |  277G |
, r; C* a/ s# Z7 f; i' C. z+------------+----------+-------+-------+( ^+ h- s2 z: s% V. C& h" V1 P

# O. s9 ~! F0 h. d) Q* q: l+-------------+5 e, A; Y* h% F* r* L. u) N
| Standby MDS |
! Y* X7 S* {$ w+ L% T% D3 G+-------------+: H. ~1 q) Z6 N+ x* Y; g
|    ceph-1   |
* ^$ k- m; f/ Q: g5 @+-------------+  `, I+ B+ j* G5 s* }3 J
MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)/ h6 J5 N* h6 V% x0 V
[root@ceph-3 test]# ceph fs status
$ @9 n; [% b- B' i! E7 V( Ecephfs - 1 clients( |" L$ y* D  r# U1 A: o
======% E: c( w1 H( K$ E
+------+--------+--------+---------------+-------+-------+
) C( t3 L' ]9 s8 ^| Rank | State  |  MDS   |    Activity   |  dns  |  inos |
: Z8 W! v5 d3 n6 T+------+--------+--------+---------------+-------+-------+
( D0 b4 C& u$ x1 Q|  0   | active | ceph-3 | Reqs:    0 /s |    1  |    3  |: T$ q# A- A6 P) D/ U9 K6 c
+------+--------+--------+---------------+-------+-------+
4 C* u/ h8 ]2 |. |1 a+------------+----------+-------+-------+- n# P  A8 D. D# }
|    Pool    |   type   |  used | avail |
! J; [8 e  [0 X  W# p; |% \  @+------------+----------+-------+-------+- P" k8 c% q* R
|  metedata  | metadata | 5911  |  277G |
' i5 R7 {2 {& @1 W5 U| cephfsdata |   data   | 44.5k |  277G |; U9 `9 i0 |) N- q+ @& s  ^
+------------+----------+-------+-------+$ n, R! c% o$ _% B

& C( G. l, g/ X/ U8 }; s7 J6 a+-------------++ i3 o) N9 z- `& M! p, }- f5 z
| Standby MDS |
( g. O6 L( g  K) R. H. _+-------------+
$ z/ s% j$ B, I. H1 p  y|    ceph-2   |4 P1 A1 {& i. m. d! M3 j
|    ceph-1   |  Q8 F, ~- t( d4 D; u( [, F5 S
+-------------+7 p6 M+ @% U# i# h- b
MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)+ [7 |/ V! i9 f. w' i+ F
[root@ceph-3 test]# rados ls -p metedata |grep 2.00000000
! e/ N2 C; O7 k9 H- }: c4 Z[root@ceph-3 test]# rados ls -p metedata |grep 2.00000000
: W. y# t4 X4 ^& }1 Y[root@ceph-3 test]# ls+ m4 g& L: [; e2 P
10000000000.00000000  recoverfile.png6 v# ~9 z" F+ }0 Q
[root@ceph-3 test]# + x8 ^4 C* u. D
[root@ceph-3 test]# # u+ W8 X& _3 |, x
[root@ceph-3 test]# cd
( G$ z! i/ }2 C( `! h[root@ceph-3 ~]# rados ls -p metedata
% S. e& A$ E) i6 G601.000000001 P' }: g, f6 J1 [
600.00000000
( c8 Y( x7 y1 r; ~( c" `1 H) P% \603.000000000 j& n: Y* a  H. g( X, E
1.00000000.inode
. N+ n3 {+ z3 t$ x1 f200.00000000
' ^: h) Y0 ^5 e7 F. }& U9 o200.00000001
6 z+ k& {" d0 H' G606.000000009 k1 L; z9 t8 s5 t
607.00000000
4 l$ [+ J  m8 r) R) P608.00000000: g& _7 o3 ?3 e1 T
604.00000000
: Q, a+ p5 c- f1 ^0 X500.00000000% I9 W. f0 R$ b( V# P& y
mds_snaptable
+ R5 j- h$ o" K* z) S605.00000000- l" _( S6 S/ C6 {& ?+ |* o
mds0_inotable: a2 h7 Q4 F, K* f( p8 f. j
100.000000002 ^$ m4 y; G$ L( X  e, V  r
mds0_sessionmap
4 P+ b, Z) f! Q7 _1 j  F609.00000000
8 i& p3 G: v+ x% c) t4 L- B400.00000000
( w% O) L$ o: w( T100.00000000.inode
: F5 h" D  ?. u1 M& w; F) i8 N! c1.00000000$ G0 M6 b( o; u3 n
[root@ceph-3 ~]# rados ls -p metedata |grep 602
: B: c6 V+ }9 ~+ A4 y[root@ceph-3 ~]# cd /mnt/^C' {' }3 b) ?1 M& O. o, E3 ]
[root@ceph-3 ~]# ceph -s( F( l' p, q6 ?2 B# i5 J) q4 W$ O
  cluster:) S  m; c: q/ r7 _& [& J8 r
    id:     57bf4711-2218-46af-99d6-9c68ae230ce1
' ~: z# b9 V4 K1 H$ h' N    health: HEALTH_ERR7 U: @0 p- i8 Y4 o" b5 {
            1 filesystem is degraded# F3 @- n$ d3 R4 U
            1 mds daemon damaged
/ H) j  Z, J& I; I
* R' }: c0 i( f( Q1 q; _  services:
; C6 ~1 C( }/ O1 Y8 k3 s    mon: 3 daemons, quorum ceph-2,ceph-1,ceph-3
4 ?$ p$ d/ A' i( k) v8 T    mgr: ceph-2(active), standbys: ceph-1, ceph-3
$ [' O( Q9 g# ?2 ^! [4 I. v# A& x    mds: cephfs-0/1/1 up , 3 up:standby, 1 damaged, j$ A4 n# S' T" g: d
    osd: 6 osds: 6 up, 6 in
% O* }* A7 u2 C: b$ @% t4 |+ t1 h( p9 X  |$ U# ^" y- t3 H7 \) s! i
  data:) M" N/ b% A& H# L" x' \2 {2 m
    pools:   2 pools, 256 pgs" N  g( a& K6 U% R; K8 @9 q$ K" z
    objects: 21 objects, 50.4KiB
8 g/ M5 F6 c5 Q" a7 e    usage:   738MiB used, 584GiB / 585GiB avail
' w5 [5 x; P8 H6 [3 @3 O# Q    pgs:     256 active+clean9 E4 c2 O2 j( I! a% ^$ o

2 }& k4 Z2 N8 I# g3 z3 W[root@ceph-3 ~]# ls
% ~: {6 `6 \" x; k2 l( Jrecovery_cephfs.py
; t6 v2 e/ G6 k[root@ceph-3 ~]# ls
/ A. b* \3 L# d4 B. Wrecovery_cephfs.py+ M& C5 R* S) N8 x5 Z" |0 ~6 q
[root@ceph-3 ~]#
' }1 v8 {% }, h& W[root@ceph-3 ~]# ls7 {% b1 T' n/ t) }- |6 f
recovery_cephfs.py
2 A: O3 v4 X' N. {; d[root@ceph-3 ~]# df -TH+ ^# _7 C# [; D

4 V% K3 h; q/ I. o% \; Y+ F
. h% a7 m3 }8 a7 r% l( \  R7 s: M* }7 |& u$ d
^C& e7 {( ]7 @6 V" u
[root@ceph-3 ~]# ls3 f, ^: L* B/ f6 g5 T
recovery_cephfs.py
; \6 @" }7 @" H; g) F& m& ^2 H7 P[root@ceph-3 ~]# ls
8 `" p9 X, {) o5 ]3 T# Srecovery_cephfs.py
3 [! @% n4 U% l& g# V( U[root@ceph-3 ~]# ls* ]4 @1 M1 i1 r$ y! {
recovery_cephfs.py
$ c8 l: Y' F1 @[root@ceph-3 ~]# ceph -s+ E2 _* }3 R% x: W( j
  cluster:9 t4 |/ U# K" a# b5 j
    id:     57bf4711-2218-46af-99d6-9c68ae230ce17 i3 P: S% F( L/ `3 x5 R& n
    health: HEALTH_ERR  B9 c1 q' ?! w  B# J4 ^  I7 S
            1 filesystem is degraded; k4 j5 T, s: D/ d, O  T5 D
            1 mds daemon damaged
% H- z2 J$ }/ j& r& C/ M+ X" k& t! m+ p- A9 W  C, {9 B
  services:8 R0 Z5 k' ]% N" v) ?7 [  a
    mon: 3 daemons, quorum ceph-2,ceph-1,ceph-3. s" E# t$ Z/ Z  x8 K* K
    mgr: ceph-2(active), standbys: ceph-1, ceph-3+ T" m- i& j) a9 @$ R
    mds: cephfs-0/1/1 up , 3 up:standby, 1 damaged8 j8 D2 `; T3 T0 T8 e, t
    osd: 6 osds: 6 up, 6 in2 v) y1 O+ }- o, ], S% m! I" H: Y

. w1 T' D; Q3 J8 B# R  data:
5 P5 s; q* J6 l" ]" e) \+ r    pools:   2 pools, 256 pgs
; @% e+ A: v: ~/ s. k; _    objects: 21 objects, 50.4KiB
  \; x7 ~0 h  o6 X. D: B    usage:   738MiB used, 584GiB / 585GiB avail
" A( D9 F9 |! W% p. O$ l    pgs:     256 active+clean
& J9 u; J; u3 P+ Z+ w7 Y! A' K" ]3 o5 n0 Q: e7 ^
[root@ceph-3 ~]# ceph fs status s/ L) @3 l4 ~! M$ a7 X# `/ p* |, {( O! {( A

! P; Z! n  e+ }* K9 L. x" m, N+-------------+
8 ?' I! m8 N. j  b! v) {, Y; l| Standby MDS |* J7 a% F' p( Q4 t5 x- {
+-------------+* S! [1 z# H/ @+ b! q
|    ceph-2   |
, j7 W, x) P/ A|    ceph-1   |" e, l' q& B8 ?& C9 }
|    ceph-3   |% l9 N. k+ x& S5 {. A7 \
+-------------+
: e' x5 ^) p5 E' k( j" y) Q4 |; _MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)" X. M, g" n5 {# s
[root@ceph-3 ~]# ceph fs status
( z  z# A+ K3 hcephfs - 0 clients
2 N- Y4 ~3 E0 B0 ~; U6 T======
7 K0 u9 I3 Q& Z, f: k/ F+------+--------+-----+----------+-----+------+
: `* @$ C/ e9 y6 \  e/ ?+ _| Rank | State  | MDS | Activity | dns | inos |
, P3 Q$ N: ?  l( C6 i, |/ k+------+--------+-----+----------+-----+------+
7 b. \* {/ \3 r9 x. U! n4 c|  0   | failed |     |          |     |      |! Y$ |* g$ J- ^2 v
+------+--------+-----+----------+-----+------+
/ X# L" j6 b( G) [* t+------------+----------+-------+-------+0 g+ X; u" G! d' _
|    Pool    |   type   |  used | avail |! r6 n) H, y$ H& n
+------------+----------+-------+-------+
: @. B8 m! G! _" w|  metedata  | metadata | 6023  |  277G |
8 m5 b4 Z' A' y+ k3 ~0 ?| cephfsdata |   data   | 44.5k |  277G |7 b6 F$ d5 W& e/ {5 @2 Y
+------------+----------+-------+-------+
7 R( ~# s; `0 N2 b  Y
' }! f; s) d( ^+-------------+
- Q7 b* v1 {$ t" J8 X; b| Standby MDS |" \% c5 n7 ]+ r% _
+-------------+
3 g9 t! @5 L" ?" K5 k% Z( h|    ceph-2   |; E- ^" a& _% p* F2 n
|    ceph-1   |+ |7 a0 H" q) P% i" g  G( T5 R  q
|    ceph-3   |
# [6 d1 G" ]1 }, G+-------------+
1 g0 S% p6 `7 b; `9 Q2 Q2 YMDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
: f( M' N+ U7 a( h9 I: p: p[root@ceph-3 ~]# ceph fs status
1 F: q: D6 J: [cephfs - 0 clients+ E% k+ r" @+ I0 _* }$ X# M' O/ x
======
- A1 E( l; L6 A4 c+------+--------+-----+----------+-----+------+
5 u- j# R; C& V2 E! q| Rank | State  | MDS | Activity | dns | inos |3 N3 D/ ^. y) r) p) C1 X9 _. L6 G
+------+--------+-----+----------+-----+------+
# {$ F! V; l6 i( w1 I|  0   | failed |     |          |     |      |
) c3 m, ^/ f# O7 u7 N& \6 M4 f3 L+------+--------+-----+----------+-----+------+# Z3 X9 Z" @( E. K( K4 o
+------------+----------+-------+-------+
+ ?, _- x: K5 E* F. i. a5 I- G|    Pool    |   type   |  used | avail |! p' R2 f* z) K& n% T9 D7 c
+------------+----------+-------+-------+
4 {- l2 R" A* u2 ^) \* S- D5 U3 [/ W|  metedata  | metadata | 6023  |  277G |; k& |5 |3 r; o9 t+ d8 o+ ]
| cephfsdata |   data   | 44.5k |  277G |1 m5 C( F! |2 T# q% c
+------------+----------+-------+-------+
7 t, o( k& [# t6 S0 M/ w8 X' Z, p5 G' z7 ]3 P: ?& j9 t7 o
+-------------+! ?' y7 h* O8 a7 `7 ~' X- l, o
| Standby MDS |
9 a7 `6 d, q8 p% E" ~, b7 O4 Y+-------------+. o" j: h- h; h+ T2 Q
|    ceph-2   |6 k1 R" Z/ Q2 S8 h# ]
|    ceph-1   |
  }) _0 k7 O6 O( o+ w0 y|    ceph-3   |9 S  `( ]$ B( X$ f5 ^8 c6 _
+-------------+
& }/ N5 p, v* U; D" `) q8 MMDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
) T3 I: ]/ b9 Q: `# s" H' J[root@ceph-3 ~]# df -TH! @1 {' d& i; ?* i  _7 l+ q, N# A! F" ^
^C/ C8 d* b/ m: l5 Q! e. b
[root@ceph-3 ~]# ls; j# f# P; f8 {) d7 J' {6 v
recovery_cephfs.py- B* c" }& T  }5 h" J) Q$ B2 `
[root@ceph-3 ~]#
! A+ ]1 C2 V; K# V2 }$ N* q& p/ R
- r9 y6 D4 r: |4 }1 x! n9 }[root@ceph-3 ~]# ceph fs ls: U4 W+ h/ _+ o1 X4 r" y5 H) Y
name: cephfs, metadata pool: metedata, data pools: [cephfsdata ]( \! _/ g& t- N6 g! ~
[root@ceph-3 ~]# ceph fs reset cephfs -
6 y, Y0 D/ i+ X--admin-daemon     -f                 -n                 -v                 --watch-info
0 d4 i1 }5 @7 z& P# `$ Y1 K. M0 A1 j--admin-socket     --format           --name             --verbose          --watch-sec
  b  \+ Y2 ~  G! s% o-c                 -h                 -o                 --version          --watch-warn
  r9 ?2 [1 Q1 i# a0 ~, e; t--cluster          --help             --out-file         -w                 * _. _, W2 z0 g
--concise          -i                 -s                 --watch            7 h2 T8 f% F6 y& V: ~/ _. Q6 P# `9 {
--conf             --id               --status           --watch-debug      
$ q6 V# f+ k$ n; e3 w--connect-timeout  --in-file          --user             --watch-error      
* \. S; o- o  l+ Z  l[root@ceph-3 ~]# ceph fs reset cephfs -& v4 _' v4 L! n- p& u
--admin-daemon     -f                 -n                 -v                 --watch-info8 C' L$ A' c" I4 u, \; f
--admin-socket     --format           --name             --verbose          --watch-sec3 M8 b: [) S* g0 \9 F% O3 {1 x% s
-c                 -h                 -o                 --version          --watch-warn( T; W1 X( J7 l
--cluster          --help             --out-file         -w                 & U) ^$ a7 w, t/ S  F" w+ J
--concise          -i                 -s                 --watch            
+ E. H. l' |" u/ @2 Y6 `. P( T/ E--conf             --id               --status           --watch-debug      
; x5 w) X! x: ~" |8 G--connect-timeout  --in-file          --user             --watch-error      
! w, ?- E# r% c[root@ceph-3 ~]# ceph fs reset cephfs --yes-i-really-mean-it ( A1 K* p1 [' @! }0 \# f
                                                    #disaster recovery only: reset to a single-MDS map
" r: h& h1 H, ]" X" D[root@ceph-3 ~]# ceph fs reset cephfs --yes-i-really-mean-it & Q2 H( ]9 T$ U( c+ w
[root@ceph-3 ~]# ceph fs ls
8 Q8 c, Q. H8 Oname: cephfs, metadata pool: metedata, data pools: [cephfsdata ]
; t4 B3 {& Q$ D' T' l% m[root@ceph-3 ~]# ceph fs status ' |6 M1 N; E( H
cephfs - 1 clients  K; s% w4 T% S& q9 z
======
3 B% O. B2 ~" S9 y. y7 i3 {& h+------+--------+--------+----------+-------+-------+9 c( Y) P  D) x- p. X8 f
| Rank | State  |  MDS   | Activity |  dns  |  inos |
" e  }7 i7 }" l- V+------+--------+--------+----------+-------+-------+
# A. W0 h9 J& Q* _|  0   | rejoin | ceph-3 |          |    1  |    3  |4 U8 x' v& D+ p* g7 }0 e
+------+--------+--------+----------+-------+-------+
7 l  Q6 H( e6 q+------------+----------+-------+-------++ e) O" k( w! h' k! Q, P
|    Pool    |   type   |  used | avail |
! y- x0 o% s% U. v  m+------------+----------+-------+-------+
7 b6 n9 \* Y0 ~+ m|  metedata  | metadata | 6023  |  277G |
  i7 [" z6 T6 M2 u| cephfsdata |   data   | 44.5k |  277G |
# Y1 Y1 R1 W  z+------------+----------+-------+-------+
/ r- E  x% V( u$ k  R0 Y
7 B* r; k. J; T9 o2 E+-------------+
; d+ M% d( U7 ^" K| Standby MDS |
; X; Q' B$ p% W" Y6 t$ a3 m+-------------+, |3 _3 ^$ z1 Z/ U- K! M2 U
|    ceph-2   |
! d* ^$ R" W) H. w3 f|    ceph-1   |0 E6 G- U% B" c( I# V" J" h
+-------------+
* d$ Q) J4 M. n# O! r0 _MDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
; }  q. G' o3 I( A2 f" A; K[root@ceph-3 ~]# ceph fs status * s8 ^/ B' @; ^
cephfs - 1 clients) j& }* |1 L* h. k2 w8 m
======
% o' Q! m  m0 r  A4 q+------+--------+--------+---------------+-------+-------+
. K6 ~) }; m- T9 K& o" q$ C) R| Rank | State  |  MDS   |    Activity   |  dns  |  inos |; e2 x5 k+ d& o
+------+--------+--------+---------------+-------+-------+4 g6 c% `" y6 s4 v; J  e8 L
|  0   | active | ceph-3 | Reqs:    0 /s |    1  |    3  |
, q/ ]4 M9 k3 E# t+------+--------+--------+---------------+-------+-------+
" u% P& p, X. `2 a. d: l+------------+----------+-------+-------+
9 M* y) J) v: L6 i|    Pool    |   type   |  used | avail |
% N* N7 }, X2 K) ?+------------+----------+-------+-------+
( i) S0 D+ F7 K6 E|  metedata  | metadata | 6135  |  277G |
5 n- }0 O6 v; x6 R. q: C) ?2 S| cephfsdata |   data   | 44.5k |  277G |/ k) D- D& P5 n9 c
+------------+----------+-------+-------+, C1 I9 R/ b2 ~4 J# b! |" `

. ?: T& w8 R) Y$ o( \/ T2 G+-------------+0 f3 W; j* d7 P
| Standby MDS |4 A0 z8 f- y9 z8 W% ~5 e# u* H
+-------------+. F0 G2 G% J4 t
|    ceph-2   |( }) E4 f2 H4 t6 E' \# `
|    ceph-1   |  x8 J5 w: |& r6 {
+-------------+
8 Y( G& s$ ~* b8 t# ~) ~7 g- |& e' nMDS version: ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
2 p, k4 ~1 x* J9 {* D[root@ceph-3 ~]# - H6 X0 T) J! z" N
* y, d+ K0 v+ _$ l% X
+ e9 k5 W  U1 V2 u0 D$ c

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
 楼主| 发表于 2022-8-11 09:25:06 | 显示全部楼层
解决:
0 h# D2 L) C2 F针对原因一:
' k7 n$ o$ l7 I( t& Q/ e+ C" L* V. g
此次情况比较特殊,ceph health detail命令竟然看不到stuck的osd,解决如下:' X- P4 J& H* U7 N4 S7 s% u
把 使用率比较高的osd降权重,手动触发osd的数据迁移。
. _) y6 O5 I: T4 ^重新ceph health detail命令,此时会发现stuck的osd,重启相应的osd,mds状态恢复。8 r& i8 L. s% R; e: I, f
针对原因二(收效甚微):
, R' l( L7 Y( h7 M( M% r
0 V1 Z* H+ C5 v( N: w# o; h8 c调小mds mds_cache_memory_limit到40G 以下
! P4 l- V: p( J9 z9 o4 g# ceph tell mds.\* injectargs '--mds_cache_memory_limit=40000000000'- O3 \# x* y* d1 n( ^
在/etc/ceph/ceph.conf修改,传到所有ceph节点5 f+ n! U7 o+ A3 u# }
针对原因三(主要原因):5 U1 G2 P7 h7 f9 K0 ?* p

8 k3 x1 c( q* x4 w5 M/ o方法一 mds多活热备调低敏感度,有两个操作:
4 S' _/ Y2 L* [6 x( i: ]8 M' A
1 x! [) H7 \: _4 \* W# M调高mds_beacon_grace=300,调高超时时间。
2 Y3 U: t0 G) l) @/ v6 Z, v* x
  写到/etc/ceph/ceph.conf里,传到所有mds节点,然后重启mds& p7 S2 Z7 [6 U. E( H* P. E/ V

: j  S9 Q/ T) l/ @9 h4 B1 xmds采用热备模式,替换原来冷备模式,加快mds加入速度。
' H+ n' k7 W8 U8 X3 j0 ]6 a! Y$ c) q6 f; k% k( D
  ceph fs set <fs name> allow_standby_replay true
" c* m1 W4 c8 U& D2 @) q( q4 x& B) B& U. y0 ]4 p
方法二 mds多活热备静态目录方法,即给目录划分不同的mds
4 }6 F* u7 P7 p: K' t8 x# x
% i; ^1 J; `5 n- |; [3 `命令:. K) `  Y8 E  B7 [* S
setfattr -n ceph.dir.pin -v 2 path/to/dir
; J; T5 o( {( jmkdir -p a/b
! q, j3 K$ {- G. w# U- t  _# “a” and “a/b” both start without an export pin set0 ^6 f4 }( M" w2 M- s
setfattr -n ceph.dir.pin -v 1 a/1 {1 ]3 f/ L% ~' ]# |
# a and b are now pinned to rank 1' p' y# D' M: B
setfattr -n ceph.dir.pin -v 0 a/b
: S, j7 `9 m, m1 X5 n# a/b is now pinned to rank 0 and a/ and the rest of its children are still pinned to rank 14 C5 D5 V# Y1 G) }8 r) n; p
& X  W4 u  G% D: d; ]( @

! J& d' a/ f$ `  n
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-11 22:57 , Processed in 0.033159 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表