pgs not deep-scrubbed in time pgs not scrubbed in time

admin

  health: HEALTH_WARN
            103 pgs not deep-scrubbed in time
" m+ s: W* G% R* {            103 pgs not scrubbed in time

; p7 T5 F; O1 ^, S3 Q" `5 }
) V  O. d% v5 {& E2 A等待数据同步。。。

  v! x9 R# i7 E' j- ?2 A
9 j! P& R% ^9 F4 e, u$ hHEALTH_WARN 103 pgs not deep-scrubbed in time; 103 pgs not scrubbed in time
PG_NOT_DEEP_SCRUBBED 103 pgs not deep-scrubbed in time
* G/ l1 P) i* N* w    pg 6.e0b not deep-scrubbed since 2021-06-28 18:28:08.588650
    pg 6.de0 not deep-scrubbed since 2021-07-03 12:33:00.513828
8 t# l& ]: g( P    pg 6.db1 not deep-scrubbed since 2021-07-03 12:35:03.062884
* D5 [) a  W7 H( s; \    pg 6.ca1 not deep-scrubbed since 2021-07-03 02:42:46.861830
    pg 6.c89 not deep-scrubbed since 2021-07-02 13:49:34.855029
  k- R; V. v6 `! A! s3 T6 ?5 i    pg 6.c5c not deep-scrubbed since 2021-07-02 07:37:48.003940
4 a. U' l6 H" K4 k5 N1 e    pg 6.b92 not deep-scrubbed since 2021-06-29 22:12:59.141939
    pg 6.b58 not deep-scrubbed since 2021-07-03 12:35:28.447157
    pg 6.ac0 not deep-scrubbed since 2021-07-03 14:38:09.864147
0 B7 z$ I, D0 W" c  I) N    pg 6.9e2 not deep-scrubbed since 2021-07-02 11:47:49.647491
    pg 6.9d7 not deep-scrubbed since 2021-07-01 20:02:08.874218
3 }+ d. c% g3 c. T: w. [3 F    pg 6.87f not deep-scrubbed since 2021-06-28 10:51:12.497479
3 j8 M: h+ ~8 N- \6 r: y, I    pg 6.855 not deep-scrubbed since 2021-07-02 18:06:53.818531
- i8 U; w$ f2 l    pg 6.80b not deep-scrubbed since 2021-07-01 04:19:22.306237
    pg 6.7e9 not deep-scrubbed since 2021-06-27 18:10:07.455727
    pg 6.6fd not deep-scrubbed since 2021-06-29 08:38:20.258996
    pg 6.293 not deep-scrubbed since 2021-07-02 08:47:37.504747
    pg 6.200 not deep-scrubbed since 2021-07-01 04:01:00.278643
2 f( \- ?/ T# O6 H    pg 10.1f3 not deep-scrubbed since 2021-07-01 14:25:22.693480
    pg 10.1f2 not deep-scrubbed since 2021-06-28 03:59:40.764659
    pg 10.1f1 not deep-scrubbed since 2021-06-30 11:16:00.963498
    pg 10.1f7 not deep-scrubbed since 2021-07-04 07:16:44.250403
* R7 ]( u# k$ x) q6 X    pg 10.1f5 not deep-scrubbed since 2021-07-02 12:42:02.569107
. V9 p7 E' o0 F/ c0 ?    pg 6.1f4 not deep-scrubbed since 2021-07-03 00:05:20.473485
    pg 10.1fc not deep-scrubbed since 2021-06-30 20:56:06.196431
0 @, W% f1 J6 s& L* j' h) i9 d    pg 6.1eb not deep-scrubbed since 2021-06-29 20:04:12.262456
    pg 10.1d6 not deep-scrubbed since 2021-06-30 02:07:46.129699
    pg 10.1d4 not deep-scrubbed since 2021-07-01 11:32:42.272646
    pg 10.1d8 not deep-scrubbed since 2021-06-30 07:37:44.085923
  j5 b6 S6 B& f4 }    pg 6.1cd not deep-scrubbed since 2021-07-01 04:30:00.444931
' _3 C! i: c8 G- S+ |& F5 @    pg 10.1cc not deep-scrubbed since 2021-06-30 06:09:15.315974
    pg 10.1ba not deep-scrubbed since 2021-06-29 17:21:02.111359
1 i& b  Q0 D# U) K# \5 j    pg 10.1bf not deep-scrubbed since 2021-07-03 19:57:35.604662
    pg 10.1a3 not deep-scrubbed since 2021-07-02 11:23:44.217536
% i: h" B- Y  Z# k4 C2 X    pg 10.1af not deep-scrubbed since 2021-06-29 22:10:39.023429
+ ]4 G- p; I$ h' e, Z3 M% @2 q    pg 10.199 not deep-scrubbed since 2021-07-03 02:32:12.180803
    pg 10.18e not deep-scrubbed since 2021-06-27 17:12:51.643297
    pg 10.174 not deep-scrubbed since 2021-06-27 19:41:07.816913
    pg 10.17f not deep-scrubbed since 2021-07-02 14:56:42.657188
' O" x$ H: d. W# q    pg 10.162 not deep-scrubbed since 2021-07-02 05:46:27.640847
    pg 10.167 not deep-scrubbed since 2021-07-03 16:35:19.520049
    pg 10.165 not deep-scrubbed since 2021-06-28 13:47:19.985230
+ g* m/ N8 ?% T* |3 R' x    pg 10.16a not deep-scrubbed since 2021-06-30 21:08:30.189430
1 h! G2 T; ?0 i# J    pg 6.14e not deep-scrubbed since 2021-07-01 06:22:40.787080
    pg 10.142 not deep-scrubbed since 2021-06-29 13:21:30.898408
! W. |8 w. d" E' o8 }* Z    pg 10.14b not deep-scrubbed since 2021-06-29 14:32:37.920806
. Y& ]& b0 L3 p1 Z9 F# s: m    pg 10.149 not deep-scrubbed since 2021-07-04 08:08:29.218563
6 Q0 w8 M! E  k+ s1 s2 A    pg 10.148 not deep-scrubbed since 2021-06-28 01:51:39.088487
) @4 J2 h8 y: [9 h" [" J    pg 10.13b not deep-scrubbed since 2021-06-29 09:57:53.164065
    pg 10.120 not deep-scrubbed since 2021-07-04 11:04:45.623218
. E; l( u- v8 ~: N! E    53 more pgs...
PG_NOT_SCRUBBED 103 pgs not scrubbed in time
    pg 6.e0b not scrubbed since 2021-07-04 01:48:58.845213
    pg 6.de0 not scrubbed since 2021-07-03 12:33:00.513828
- `* t5 @2 ?) B  y    pg 6.db1 not scrubbed since 2021-07-03 12:35:03.062884
1 w/ w7 @; D8 ^# a    pg 6.ca1 not scrubbed since 2021-07-04 04:44:11.654762
) f5 O. o* u# |; }# ~    pg 6.c89 not scrubbed since 2021-07-03 19:05:06.046447
    pg 6.c5c not scrubbed since 2021-07-03 18:40:17.262641
: b2 T9 V3 z. t2 T" H$ `0 }    pg 6.b92 not scrubbed since 2021-07-03 19:01:09.546897
    pg 6.b58 not scrubbed since 2021-07-03 12:35:28.447157
3 A1 o' F; ^8 z- |3 p    pg 6.ac0 not scrubbed since 2021-07-03 14:38:09.864147
8 I0 }( G! F& a8 F0 ]9 z' T: W3 x    pg 6.9e2 not scrubbed since 2021-07-03 18:07:46.295197
    pg 6.9d7 not scrubbed since 2021-07-04 08:32:17.053636
    pg 6.87f not scrubbed since 2021-07-03 19:25:45.463598
    pg 6.855 not scrubbed since 2021-07-04 00:45:05.669044
1 _- \/ f- m1 o: G, o9 b% O    pg 6.80b not scrubbed since 2021-07-04 01:33:12.799049
: m( N9 F# @# x6 f, \    pg 6.7e9 not scrubbed since 2021-07-03 11:40:19.348110
. l0 e) q7 \1 j7 x; m0 z    pg 6.6fd not scrubbed since 2021-07-04 08:25:15.393349
    pg 6.293 not scrubbed since 2021-07-03 16:34:39.774991
$ B' j4 G) x$ f% O    pg 6.200 not scrubbed since 2021-07-03 16:30:05.437802
    pg 10.1f3 not scrubbed since 2021-07-04 05:10:37.906556
    pg 10.1f2 not scrubbed since 2021-07-04 02:39:29.157905
    pg 10.1f1 not scrubbed since 2021-07-04 07:44:20.778922
    pg 10.1f7 not scrubbed since 2021-07-04 07:16:44.250403
5 f) @+ `0 [9 {+ e) p. _    pg 10.1f5 not scrubbed since 2021-07-03 16:59:54.742894
    pg 6.1f4 not scrubbed since 2021-07-04 07:28:38.982395
    pg 10.1fc not scrubbed since 2021-07-04 08:41:39.397538
    pg 6.1eb not scrubbed since 2021-07-03 04:04:14.994707
+ k! w2 l5 G. _; Q- ~! ~    pg 10.1d6 not scrubbed since 2021-07-03 17:13:33.936182
8 H2 S+ z' b( T    pg 10.1d4 not scrubbed since 2021-07-03 18:25:08.967808
    pg 10.1d8 not scrubbed since 2021-07-04 05:01:44.498562
0 s3 P7 r0 H- X6 n: X, b    pg 6.1cd not scrubbed since 2021-07-03 12:44:39.288892
    pg 10.1cc not scrubbed since 2021-07-03 20:34:45.036754
    pg 10.1ba not scrubbed since 2021-07-03 13:38:42.666479
    pg 10.1bf not scrubbed since 2021-07-03 19:57:35.604662
    pg 10.1a3 not scrubbed since 2021-07-03 18:53:20.861765
    pg 10.1af not scrubbed since 2021-07-03 16:12:21.661524
3 |" C% |. p' o2 o' {    pg 10.199 not scrubbed since 2021-07-04 12:13:53.644669
+ _/ n" B% G0 c    pg 10.18e not scrubbed since 2021-07-04 06:43:43.854868
    pg 10.174 not scrubbed since 2021-07-03 13:33:39.370896
    pg 10.17f not scrubbed since 2021-07-03 22:03:32.376883
( Y* h4 {0 g& V# F7 j    pg 10.162 not scrubbed since 2021-07-04 08:28:30.284614
    pg 10.167 not scrubbed since 2021-07-03 16:35:19.520049
4 t( u" T* l7 z6 B  ^8 ?+ k- F    pg 10.165 not scrubbed since 2021-07-03 20:28:48.875738
    pg 10.16a not scrubbed since 2021-07-03 09:45:40.821290
    pg 6.14e not scrubbed since 2021-07-03 21:27:11.736701
+ G7 t/ R* a- ]2 a+ ?    pg 10.142 not scrubbed since 2021-07-03 13:40:31.085534
) R: \* k7 R; D6 L    pg 10.14b not scrubbed since 2021-07-03 05:29:35.857549
    pg 10.149 not scrubbed since 2021-07-04 08:08:29.218563
    pg 10.148 not scrubbed since 2021-07-03 10:59:57.327024
2 D# x5 ?% `3 F* J4 f5 {    pg 10.13b not scrubbed since 2021-07-04 10:51:52.980119
    pg 10.120 not scrubbed since 2021-07-04 11:04:45.623218
' F* J; g: d+ S- \9 n    53 more pgs...

; e7 D; G4 M- `

admin

ceph pg dump pgs | awk '{print $1" "$23}' | column -t
) j5 U. K; A* n- o  _1 |- C/ [' l统计：
" {# L- s7 G# `3 C........
4 E' ~' S4 W/ B4 z, o8 a: j" m6.fea    18:52:37.325860
6.feb    10:27:17.560151
6.fec    14:01:48.356228
6.fed    22:58:24.674774
6.fee    12:59:54.665857
6.fef    02:57:40.071398
; A9 C& \. r6 e1 P+ z! }+ a+ W6.ff0    13:53:59.057123
' H/ u; _+ Z* B6.ff1    22:44:29.222198
5 N4 r* ]4 o7 ^& o2 Q6.ff2    08:39:08.836682
6.ff3    22:15:22.663800
% Y. G* s0 Y5 h" t- Y6.ff4    10:53:18.285518
" m1 s; z% m2 g4 p& Z6 \% V6.ff5    00:35:30.314441
6.ff6    12:46:47.548084
  A4 J/ w4 S! Q' [# d2 P: w6.ff7    01:50:14.514644
- w$ V+ `# w3 l2 w6.ff8    21:52:26.449523
6.ff9    06:27:41.330282
6.ffa    15:22:14.880443
6.ffb    12:11:48.388809
- m# v( }2 U# P: w* {* A6.ffc    15:05:19.534203
& @: `$ {+ Z7 I  G6.ffd    06:06:45.223006
6.ffe    13:00:10.003672
1 o. P; w; k; j8 U: l6.fff    07:47:00.681640

admin

等待同步完成之后，自然恢复正常状态。
/ D" `$ c* k: Q2 W( y# ceph status  
  cluster:
    id:     29046cc0-0682-496b-98b1-912e59964282
3 I2 A& X$ U: D2 \    health: HEALTH_OK

  services:
# L3 @& v1 X0 X) u4 u: w    mon: 3 daemons, quorum hostceph1,hostceph2,hostceph3 (age 4m)
+ C  O- Z2 l+ v/ b7 ~1 L; A    mgr: hostceph1(active, since 6m), standbys: hostceph2, hostceph3
9 l: D  t+ s8 m& o" p# t: l    osd: 18 osds: 18 up (since 3m), 18 in (since 66m)

6 R: d  C) u- a  data:
: A! D, \  D- i) Q  b6 \3 ~    pools:   6 pools, 7552 pgs
    objects: 352.18k objects, 1.3 TiB
& U! l2 H4 p9 ]% d! K' o- U    usage:   4.6 TiB used, 63 TiB / 67 TiB avail
. H, R7 I* i, h/ e4 ?    pgs:     7552 active+clean
3 d& P) p( S8 z* ~% u/ z( E. o7 G
  io:
: v2 r: l, p: H6 {* A* S2 n    client:   62 KiB/s rd, 73 op/s rd, 0 op/s wr