记一次ceph存储同步问题noout,nobackfill,norecover flag(s) set Degraded data r

admin

检查ceph 状态：
[root@compute01 ~]# ceph -s
  cluster:
    id:     a6d0359e-8295-4e62-8620-54273462e79f
) P4 Q' s9 F: J5 N" O  v" p    health: HEALTH_WARN
! g- \: a" w4 y! C9 g9 ]- W' F! _: e            noout,nobackfill,norecover flag(s) set
5 X8 c  q0 {1 B* s1 {! s. K            Degraded data redundancy: 3793/96483 objects degraded (3.931%), 876 pgs degraded, 512 pgs undersized
4 i+ Z0 m- `% ~5 p& {5 u            6 pgs not deep-scrubbed in time
/ L1 l, n8 k7 s5 U* Q! p            clock skew detected on mon.compute02, mon.compute03

7 Y, ^. l# ^: H- W4 I( _( u  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 22m)
/ @1 J! v1 I& ~    mgr: compute02(active, since 4d), standbys: compute03, compute01
    osd: 6 osds: 6 up (since 22m), 6 in (since 2w); 512 remapped pgs
3 n; i( k! ?4 }- P3 u3 v         flags noout,nobackfill,norecover
5 h: n! c8 F; e7 C% H
# _- t4 u' L# u# n: U  data:
    pools:   5 pools, 1408 pgs
    objects: 32.65k objects, 132 GiB
9 y0 r; r# \& O3 G9 G, w' ^- Z    usage:   381 GiB used, 4.9 TiB / 5.2 TiB avail
    pgs:     3793/96483 objects degraded (3.931%)
             450 active+recovery_wait+undersized+degraded+remapped
             444 active+clean
: b( |8 g, g- b5 \& [) u2 t7 k/ J, Q2 q" U             362 active+recovery_wait+degraded
1 r8 i+ `0 b! S             87  active+recovery_wait
             62  active+undersized+degraded+remapped+backfill_wait
4 c9 F  f) w; e. l  z. H' U             2   active+recovering+degraded
             1   active+recovering

5 [5 f+ S4 X$ L: C  io:
    client:   156 KiB/s rd, 78 KiB/s wr, 14 op/s rd, 8 op/s wr
% A$ J" e, k" x0 [7 ~3 ^# W. x+ ]4 O
. I, C- W# q. W" [0 U  L! }0 h
) R& z5 {1 H& E$ y) N设置ceph osd 开启recover和覆盖同步配置：
6 k! g1 d3 u# e; x/ [
6 Y# d5 k) V5 l4 f/ q2 r
5 d8 O* X5 y, S* G$ a3 b, v[root@compute01 ~]# ceph osd unset norecover
) S% u! r1 L5 k; s( R' V% b4 S- mnorecover is unset
[root@compute01 ~]# ceph osd unset nobackfill
nobackfill is unset

查看状态：
3 |3 H; [$ `: H[root@compute01 ~]# ceph -s
6 O, f  ^! N% I9 m" g3 n# R% o" O! C7 _) C  cluster:
    id:     a6d0359e-8295-4e62-8620-54273462e79f
    health: HEALTH_WARN
' K. W) s; D' x6 O# W1 F            noout flag(s) set
( [/ ?3 C: l$ X( d            Degraded data redundancy: 3537/96486 objects degraded (3.666%), 705 pgs degraded, 512 pgs undersized
            6 pgs not deep-scrubbed in time
            clock skew detected on mon.compute02, mon.compute03

  services:
$ U6 J0 v" U( @1 G' T    mon: 3 daemons, quorum compute01,compute02,compute03 (age 23m)
$ T  U. d  E* w. z  w1 \    mgr: compute02(active, since 4d), standbys: compute03, compute01
    osd: 6 osds: 6 up (since 23m), 6 in (since 2w); 512 remapped pgs
         flags noout
+ Q# C& p6 g: ]
  data:
% c1 N+ Y* ]  S1 i! |    pools:   5 pools, 1408 pgs
- C9 d: ?* n/ f" i1 r2 p7 h0 \+ f    objects: 32.65k objects, 132 GiB
  F( W0 f# W3 V6 o    usage:   382 GiB used, 4.9 TiB / 5.2 TiB avail
    pgs:     3537/96486 objects degraded (3.666%)
             649 active+clean
             450 active+recovery_wait+undersized+degraded+remapped
9 K" W8 u) u8 s# W6 m& t             191 active+recovery_wait+degraded
0 y' J7 _0 R- K             62  active+undersized+degraded+remapped+backfill_wait
             54  active+recovery_wait
             2   active+recovering+degraded

  io:
2 n6 F, Q3 E: r( ~- {: p0 ]5 J3 F    client:   204 KiB/s rd, 797 KiB/s wr, 147 op/s rd, 42 op/s wr
    recovery: 89 MiB/s, 21 objects/s
, _" r1 l& X" D' ?/ X, }+ k4 R 上面看到数据开始在同步了，覆盖了。

查看osd是否都正常：
9 \/ J4 a: u' I! u9 a[root@compute01 ~]# ceph osd tree
ID  CLASS WEIGHT  TYPE NAME              STATUS REWEIGHT PRI-AFF
! n: X' e& e% T0 g# Y3 s-11             0 host ssd-compute05                             
; t; c( Y; J% F* P: n. x -3       6.00000 root ssd                                       
-4       2.00000     host ssd-compute01                        
  0   ssd 1.00000         osd.0              up  1.00000 1.00000
, L* u, R0 S5 @( N  1   ssd 1.00000         osd.1              up  1.00000 1.00000
-5       2.00000     host ssd-compute02                        
/ v5 C1 @; `6 O  {5 R  2   ssd 1.00000         osd.2              up  1.00000 1.00000
  3   ssd 1.00000         osd.3              up  1.00000 1.00000
- O+ D" T0 T0 {( @* J" H/ _" O3 H: O  _ -6       2.00000     host ssd-compute03                        
  5   ssd 1.00000         osd.5              up  1.00000 1.00000
2 O$ _- q6 o+ T  R: q( w# l, ?& O  6   ssd 1.00000         osd.6              up  1.00000 1.00000
" ^2 u  l* _0 q% n -1             0 root default                                   
8 o1 w7 u" V6 M9 V; F7 K. [3 M
[root@compute01 ~]# ceph -s
' U+ d" F4 _5 J% O8 B  cluster:
    id:     a6d0359e-8295-4e62-8620-54273462e79f
    health: HEALTH_WARN
& i% k; l. `, _8 }, X            noout flag(s) set
            Reduced data availability: 3 pgs peering
            Degraded data redundancy: 2173/96486 objects degraded (2.252%), 125 pgs degraded, 133 pgs undersized
1 {' z' d4 i! e2 w0 @+ q3 y            6 pgs not deep-scrubbed in time
            clock skew detected on mon.compute02, mon.compute03
3 s" a9 B- ^! q: j5 p
! T" Z7 E* y' D6 y  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 24m)
( i4 E, h1 X6 ~# X" u3 F) M    mgr: compute02(active, since 4d), standbys: compute03, compute01
    osd: 6 osds: 6 up (since 24m), 6 in (since 2w); 104 remapped pgs
         flags noout

  data:
& X" S' e) \) C3 A7 W* `    pools:   5 pools, 1408 pgs
$ p7 z) G; Z2 [$ ?* [- s    objects: 32.65k objects, 132 GiB
    usage:   383 GiB used, 4.9 TiB / 5.2 TiB avail
: D2 H3 @3 `; U" z    pgs:     0.568% pgs not active
" G7 m) ~& n) c             2173/96486 objects degraded (2.252%)
7 t; E% G- w$ @' O( s7 @- ^             1267 active+clean
             62   active+undersized+degraded+remapped+backfill_wait
             62   active+recovery_wait+undersized+degraded+remapped
             8    active+undersized+remapped
8 {' j$ H4 M! u" k. Y             8    peering
. [0 b% s' D7 I$ |7 J$ i             1    active+recovering+undersized+degraded+remapped
( H0 P4 p( `- R' E$ ]
4 M- M- |$ b% v! M9 R# n  io:
: _1 K: C8 G, E' H    client:   407 KiB/s rd, 792 KiB/s wr, 34 op/s rd, 52 op/s wr
    recovery: 72 MiB/s, 2 keys/s, 22 objects/s
$ B- Q8 O/ D. B" e: f9 K' n) g

  j$ c2 v' c" e# I! q% ?! j查看时间：
[root@compute01 ~]# date
Mon Dec 20 10:32:23 CST 2021
% x6 D( X1 l. C# E[root@compute01 ~]# date
' G2 u2 Z4 K1 O, q6 P  c( XMon Dec 20 10:32:38 CST 2021
[root@compute01 ~]# date
( o1 l" S4 B$ ^1 x6 y8 pMon Dec 20 10:33:00 CST 2021

" ~4 _* _, q3 @3 E2 J[root@compute01 ~]# chronyc sources
- z/ J2 x! p+ j210 Number of sources = 2
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
, q2 D8 f, d) `! a: a- P8 c===============================================================================
^? compute01                     0   9     0     -     +0ns[   +0ns] +/-    0ns
; o- h4 K- n+ p: ?8 [+ _^? compute02                     0   9     0     -     +0ns[   +0ns] +/-    0ns
[root@compute01 ~]# vim /etc/chrony.conf

6 P& y" s0 B. Q: y6 X[root@compute01 ~]# systemctl restart chronyd.service
; ?) _& Q! b1 p4 u8 C% j[root@compute01 ~]# chronyc sources
2 L2 D- g/ B& f! f) x210 Number of sources = 2
9 R; m5 o. o- h2 A1 f; KMS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* compute01                    10   6    17     3  +2667ns[-7752ns] +/-   28us
( f3 k6 r$ d1 c2 Z" K! t9 l% w^? compute02                     0   6     0     -     +0ns[   +0ns] +/-    0ns
[root@compute01 ~]#
[root@compute01 ~]#
[root@compute01 ~]# vim /etc/chrony.conf
+ e1 L1 A( t5 _+ }6 Y[root@compute01 ~]# date
Mon Dec 20 10:36:29 CST 2021
3 Y7 @9 A2 p6 j6 M, U) s4 i% h

时间同步即可：
9 ]% ~, ^* C! r$ U6 x. f4 o/ A0 e
[root@compute01 ~]# ceph -s
; D0 Y% k! w7 r+ t$ a& c  cluster:
    id:     a6d0359e-8295-4e62-8620-54273462e79f
    health: HEALTH_WARN
6 {( ~% a) G! T+ [( f            noout flag(s) set
6 Z9 }( q) L  B- X+ V* T            1 pgs not deep-scrubbed in time

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 48s)
( B: N+ k2 n/ q    mgr: compute02(active, since 4d), standbys: compute03, compute01
4 a* d/ _& g7 `  ^    osd: 6 osds: 6 up (since 28m), 6 in (since 2w)
         flags noout
8 c  d6 [  W  n
1 U/ }0 T& @4 w8 d9 |  data:
( M' Z% T& ]) f    pools:   5 pools, 1408 pgs
    objects: 32.65k objects, 132 GiB
    usage:   384 GiB used, 4.9 TiB / 5.2 TiB avail
+ |. H3 w( ]* U9 M2 U    pgs:     1407 active+clean
2 ]7 E1 Q3 a9 e2 \; d             1    active+clean+scrubbing+deep
- {# W! b4 X1 P" u/ P
1 \2 w, n5 r* }  y  io:
3 E' `8 P1 S6 V( T) S3 @* u    client:   20 KiB/s rd, 499 KiB/s wr, 4 op/s rd, 18 op/s wr
' d3 x' Y, b4 j) @: r. p
查看ceph存储是否正常：
/ E* m9 l5 I0 u" h) y
[root@compute01 ~]# ceph -s
  cluster:
; Y; r2 [% ?! \+ h+ C( n+ N$ w6 _    id:     a6d0359e-8295-4e62-8620-54273462e79f
    health: HEALTH_WARN
            noout flag(s) set
# p4 m8 |- j2 B0 t            1 pgs not deep-scrubbed in time
) e/ {1 W; p0 w) I5 \$ z
8 P9 H* A1 M" W. r  services:
) ]: q5 ~# w4 B    mon: 3 daemons, quorum compute01,compute02,compute03 (age 60s)
$ N; Q$ r- D2 H7 u    mgr: compute02(active, since 4d), standbys: compute03, compute01
* W* D# c" W, F% }! V) ]% q/ v+ v    osd: 6 osds: 6 up (since 28m), 6 in (since 2w)
         flags noout

  data:
/ \% O2 b" {4 ^: u* Q; S    pools:   5 pools, 1408 pgs
    objects: 32.65k objects, 132 GiB
. I7 {9 q3 X5 ]    usage:   384 GiB used, 4.9 TiB / 5.2 TiB avail
) X& h; i  K# _    pgs:     1407 active+clean
3 t  W2 A3 `7 r* f( K  M) n             1    active+clean+scrubbing+deep

2 ^3 s0 L+ Z' D1 B% g/ |/ g  io:
) @" G( {6 h6 U$ N/ a5 w0 ^    client:   120 KiB/s rd, 595 KiB/s wr, 13 op/s rd, 33 op/s wr

+ H, C- W' P1 u! {, D
[root@compute01 ~]# ceph osd unset noout
noout is unset
  d5 E$ d. b% ]$ D. z7 x& h6 E
检查状态：
! I  c3 U* G9 d2 M: @: k+ X% h[root@compute01 ~]# ceph -s
4 g0 c- G7 S$ W  cluster:
* o- V, F/ k$ |/ z    id:     a6d0359e-8295-4e62-8620-54273462e79f
    health: HEALTH_OK
5 B2 c# x1 z' n
/ M) \2 `1 a! w& b( x3 c  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 100s)
    mgr: compute02(active, since 4d), standbys: compute03, compute01
    osd: 6 osds: 6 up (since 29m), 6 in (since 2w)
8 Q8 H- `) z; V& s( E
7 k! j+ |( t; g2 v  data:
) f; M- }$ S& p7 d1 _    pools:   5 pools, 1408 pgs
3 j# n' D( ~# Y0 d3 F  D* j    objects: 32.65k objects, 132 GiB
+ j/ E3 v6 k$ q    usage:   384 GiB used, 4.9 TiB / 5.2 TiB avail
    pgs:     1408 active+clean
) ]- b' I% P& o* U9 t  f; A6 w
# E3 s. T: J' z' V1 L  io:
    client:   539 KiB/s rd, 285 KiB/s wr, 191 op/s rd, 26 op/s wr
) k! b) `9 i8 T, e( D
[root@compute01 ~]# ceph -s
: f* F$ I! D8 K9 O! u  cluster:
7 ?0 @1 {! d( P+ W! e: Z    id:     a6d0359e-8295-4e62-8620-54273462e79f
    health: HEALTH_OK

5 J0 t# d9 p) t. g  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 102s)
    mgr: compute02(active, since 4d), standbys: compute03, compute01
    osd: 6 osds: 6 up (since 29m), 6 in (since 2w)

6 |# u6 [) E& y" [" Z9 V, \  data:
    pools:   5 pools, 1408 pgs
    objects: 32.65k objects, 132 GiB
# O4 ~" I) \9 C# |  }5 `5 |" H* X    usage:   384 GiB used, 4.9 TiB / 5.2 TiB avail
3 F6 X( |, e4 T" k) D) h    pgs:     1407 active+clean
# }. w, F" M& [! p: v/ A             1    active+clean+scrubbing+deep

" T* P) |: _; Y) u# h" S  io:
    client:   110 KiB/s rd, 196 KiB/s wr, 12 op/s rd, 18 op/s wr
3 @7 a1 B& V* X/ Y  }
[root@compute01 ~]# ceph -s
. r( J% `' }' D. j9 x% _/ c9 I  cluster:
5 a' v$ D9 b* }* o# F  }! w3 i    id:     a6d0359e-8295-4e62-8620-54273462e79f
8 K+ E6 v9 k  X7 {+ x4 y/ z    health: HEALTH_OK
. F0 i' J( I- }
2 W# l. _2 ^7 L& Y/ I( ^4 L& x  services:
  V: t3 V9 G/ j; O" c    mon: 3 daemons, quorum compute01,compute02,compute03 (age 105s)
& C2 G3 o8 p: q7 v9 j7 [( B    mgr: compute02(active, since 4d), standbys: compute03, compute01
$ [9 N( d% j, |, l( l) Y    osd: 6 osds: 6 up (since 29m), 6 in (since 2w)
  S( Y4 l: b! I% Z
  data:
* [6 L. m$ w6 v    pools:   5 pools, 1408 pgs
1 n; r3 T$ N  u. L: T( |    objects: 32.65k objects, 132 GiB
' F/ q! K$ T% M2 l2 G) X) q& Q    usage:   384 GiB used, 4.9 TiB / 5.2 TiB avail
    pgs:     1407 active+clean
  i, r* }7 U6 G' Q             1    active+clean+scrubbing+deep
8 j9 S& x+ ]7 F5 n. u5 f
  io:
    client:   74 KiB/s rd, 193 KiB/s wr, 10 op/s rd, 18 op/s wr

[root@compute01 ~]# ceph -s
  cluster:
7 e. g1 t2 b4 N, R8 y% Q! _: @, P    id:     a6d0359e-8295-4e62-8620-54273462e79f
' e7 c# `" O& Q  M    health: HEALTH_OK
. o8 }! f+ `7 p! {6 y$ U
1 L4 X- t7 r4 k3 M1 p+ \  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 3m)
    mgr: compute02(active, since 4d), standbys: compute03, compute01
    osd: 6 osds: 6 up (since 30m), 6 in (since 2w)
$ B: _! [( J' P; w2 ~' s- Z
  data:
# o, ~0 B; \6 b: B! Z    pools:   5 pools, 1408 pgs
    objects: 32.65k objects, 132 GiB
    usage:   384 GiB used, 4.9 TiB / 5.2 TiB avail
  E# ?7 W2 y1 _" J# W; J" _    pgs:     1408 active+clean
* c5 @  N# w  E; O6 Y* d0 B: ]
  io:
$ T8 b. n! F2 ?9 a. D! \    client:   840 KiB/s rd, 1.4 MiB/s wr, 63 op/s rd, 43 op/s wr

- d& Z) U4 I5 o) d
5 D2 o' B: k, U# d4 Q, b
9 p. Z/ q" K3 \: i4 ~- x! T0 @
6 M( h  C, h# [3 I) M总结，此过程中出现的问题，一是时间同步服务器有问题，二是ceph 设置了不剔除osd，不同步，不分配问题。

" t! c- y2 _% }5 W

admin

ceph存储同步数据需要时间同步，如果时间不同步，需要调整时间同步服务器，检查为什么不能同步时间。   以前chrony服务配置的时候，不用设置allow ，现在可能需要设置。或者以前也需要设置，只是没有注意这个细节性的地方。
9 P7 T& N- [' B& f同步时间服务器，第一修改配置，第二检查配置allow 允许的范围，第三其他节点同步时间。第四启动服务。