pgs not deep-scrubbed in time异常处理

admin

发现出现异常warn信息，虽然不影响整个集群正常使用，但强迫症患者还是忍不了，下面是过程。查看具体报错信息
7 k! X6 \6 U4 W( D( ^( P6 cHEALTH_WARN 2 pgs not deep-scrubbed in time
PG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time
    pg 18.41 not deep-scrubbed since 2022-12-07 20:15:50.550606
    pg 5.16d not deep-scrubbed since 2022-12-07 22:21:58.141071
* x7 q+ w) K: Z. e  x
8 H% W3 K# x  d+ x- M" m
. R0 K5 v1 Z3 z& n- ^[root@controller1 ~]# ceph pg deep-scrub 18.41
  N- n, u7 y6 o# ainstructing pg 18.41 on osd.6 to deep-scrub
+ M" V" t0 k, N3 y& g2 x' L: e1 W/ O[root@controller1 ~]# ceph pg deep-scrub 5.16
  {& X: F3 d( t" s3 Y8 winstructing pg 5.16 on osd.13 to deep-scrub


) M; B5 ~- Y; a/ t3 Y  l5 u
1 z) W9 G3 A6 c$ @% J+ ]9 s[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
, G6 G0 A7 s  e  ?7 e    "osd_deep_scrub_interval": "604800.000000",
5 E( k1 I7 |! e1 i' Y+ |[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
8 w/ I/ a7 J& Z  Z[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
    "osd_deep_scrub_interval": "3628800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
7 a* V& I$ Y  M2 H' ]! N: D恢复正常后，再改回来：
[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
    "osd_deep_scrub_interval": "604800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
+ l9 y* ?* U! p+ M[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
    "osd_deep_scrub_interval": "3628800.000000",
. {' t. ]* O( c! S+ j( H[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800^C
5 r; B5 c2 I: V
[root@controller1 ~]#
9 e, O# ]  M5 v  m/ L, E( `0 P[root@controller1 ~]# ceph -s
  cluster:
    id:     9d22e36a-2bdd-4d2d-8394-48af75ead777
    health: HEALTH_OK
" ]" f2 u  q( ^3 \
. J' x1 k2 @+ K/ X# B* T4 n  x  services:
) ^: E$ W& G7 e# o0 f4 i    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 5M)
) g% B: B; m$ h    mgr: ceph1(active, since 19M), standbys: ceph2,ceph3
    osd: 40 osds: 40 up (since 3w), 40 in (since 12M)
    rgw: 3 daemons active (host09, host10, host11)
- r" L3 l) l* N( J2 b
  task status:
- }8 @) i& f% m/ C" A8 X
  data:
2 x2 a! X5 ^. z* y% E  p    pools:   16 pools, 3072 pgs
- J, S! u% l$ P. h# b    objects: 4.20M objects, 16 TiB
' r+ v* b/ q# m9 n6 S/ m    usage:   40 TiB used, 107 TiB / 148 TiB avail
    pgs:     3067 active+clean
8 R( B, g" u8 R) b/ A# m) Q6 e$ c0 J             5    active+clean+scrubbing+deep
: u- A/ c/ b% v2 K6 v8 m
# Y% E' T0 T! S0 e( `) C3 h  io:
    client:   403 KiB/s rd, 9.5 MiB/s wr, 514 op/s rd, 466 op/s wr

5 d% T; n& N8 Y; d[root@ ~]# ceph config set global osd_deep_scrub_interval 604800
; d" ~5 e3 f2 b[root@ ~]#
6 e2 Q( m3 z1 \. f6 `4 q
6 ~1 Y) I$ X3 {9 D9 A
2 ]! L# [( |( _3 Y" x1 Osystemctl restart ceph-osd@6.service

admin

根源
- t' _5 B  l7 n% ~! nRHCS 4 has introduced 2 new parameters on scrubbing interval warning, and default values are
Raw
( q( h5 j, S! b+ Y% w"mon_warn_pg_not_deep_scrubbed_ratio": "0.750000",
"mon_warn_pg_not_scrubbed_ratio": "0.500000",
( |. b) Y3 }5 VThese ratios are based on the following parameters, and default values are
7 W$ h  l4 C* \' s$ M7 lRaw
"osd_scrub_max_interval": "604800.000000",
0 z$ _  O5 n2 g' M; }6 g! U"osd_deep_scrub_interval": "604800.000000"
When pgs are not scrubbed / deep scrubbed for the configured ratio of the interval, warnings of "pgs not scrubbed / deep-scrubbed in time" will show in Ceph status.
* \; z$ y& h5 D+ Q$ ?' l) ^! R; G
1 o' ^0 E/ q8 R6 W# u) p2 XWhen setting "mon_warn_pg_not_deep_scrubbed_ratio" or "mon_warn_pg_not_scrubbed_ratio" values to 0, warnings will be disabled. Please evaluate the cluster's past usage carefully before setting.

  i/ S* y6 |! S% f; q* r: OSometimes, we might get a false warning even though the osd_deep_scrub_interval has been increased. This happens when the scrubbing parameters are not applied globally because these settings are used by both OSDs and MONs.
The OSDs use them to determine when to run scrub, and the MONs/MGRs use them to check if they need to show the warning.
You can set it globally as:

Raw
  ?, Y6 |* Y$ y: I3 Y# ceph config set global osd_deep_scrub_interval 3628800
诊断步骤
Checking scrubbing related configurations by admin socket
Raw
/ O" Y0 s4 A* w5 ~7 ]ceph --admin-daemon /var/run/ceph/<admin_socket_name>.asok config show | grep scrub

admin

You can set the deep scrub period to 2 week, to stretch the deep scrub window. Insted of

osd_deep_scrub_interval = 604800
3 C5 `3 _# ^2 W. |1 P5 @3 ?use:

osd_deep_scrub_interval = 1209600