pgs not deep-scrubbed in time异常处理

admin

发现出现异常warn信息，虽然不影响整个集群正常使用，但强迫症患者还是忍不了，下面是过程。查看具体报错信息
8 ~( d4 i7 G2 U/ b! R9 W2 a+ D, DHEALTH_WARN 2 pgs not deep-scrubbed in time
' p6 h3 y0 i9 k$ c: A0 JPG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time
- J/ G- v- r8 s) `( O9 U6 |! j9 ^5 W    pg 18.41 not deep-scrubbed since 2022-12-07 20:15:50.550606
8 n4 f( a9 e- x; c9 `) }    pg 5.16d not deep-scrubbed since 2022-12-07 22:21:58.141071
2 Q2 k  C4 T1 e! K
- g' u! ~( ~; x6 K% ?
[root@controller1 ~]# ceph pg deep-scrub 18.41
1 a+ S9 K( W! v5 uinstructing pg 18.41 on osd.6 to deep-scrub
& e/ {3 q* M% K3 e9 G% ?[root@controller1 ~]# ceph pg deep-scrub 5.16
instructing pg 5.16 on osd.13 to deep-scrub



+ v( k9 n6 k4 c; H8 e7 W  Z/ {2 N9 ^[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
" ]6 s2 u0 J" n: S    "osd_deep_scrub_interval": "604800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
9 j) ?$ h9 W7 b& ~: t* Q0 X[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
- D4 A; }" V, b6 K$ t! l7 d    "osd_deep_scrub_interval": "3628800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
恢复正常后，再改回来：
  h, B) g7 D0 H, L5 `[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
1 F) V- b- z9 e    "osd_deep_scrub_interval": "604800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
; U) C" ]2 V9 ^) g[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
    "osd_deep_scrub_interval": "3628800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800^C

[root@controller1 ~]#
0 k1 G# c2 z  A# k7 a5 {- N% h[root@controller1 ~]# ceph -s
8 ]0 v9 q3 L2 J. k1 z  cluster:
% a- i6 H4 Q" X# h3 c    id:     9d22e36a-2bdd-4d2d-8394-48af75ead777
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 5M)
    mgr: ceph1(active, since 19M), standbys: ceph2,ceph3
$ U; n1 s) Y$ z9 R$ J5 g    osd: 40 osds: 40 up (since 3w), 40 in (since 12M)
    rgw: 3 daemons active (host09, host10, host11)

! p: b8 ]' a4 C+ h; K) u  task status:
5 O9 R* Q9 b9 s
3 v6 E& I8 p, D8 k% ~  data:
    pools:   16 pools, 3072 pgs
    objects: 4.20M objects, 16 TiB
! ]; J5 E0 B# p4 t# `8 i    usage:   40 TiB used, 107 TiB / 148 TiB avail
' K& A3 ~5 Z! Z    pgs:     3067 active+clean
             5    active+clean+scrubbing+deep
9 J0 O) J( Z" A2 L$ }, Z4 Q
  io:
    client:   403 KiB/s rd, 9.5 MiB/s wr, 514 op/s rd, 466 op/s wr
. Q$ z" A3 P2 Y% D7 h/ C) [
[root@ ~]# ceph config set global osd_deep_scrub_interval 604800
[root@ ~]#
' k2 G& F, f2 o' h+ {

systemctl restart ceph-osd@6.service

8 `( J; R* u5 i2 M- E
- h, T3 P+ c6 V. a9 D  l

admin

根源
RHCS 4 has introduced 2 new parameters on scrubbing interval warning, and default values are
) `# S7 n$ H( gRaw
2 l! p; Y) e7 \" f: u* v# l& k"mon_warn_pg_not_deep_scrubbed_ratio": "0.750000",
/ Y7 z) [8 H! E/ m"mon_warn_pg_not_scrubbed_ratio": "0.500000",
These ratios are based on the following parameters, and default values are
4 E7 x1 L9 @. B: D9 w9 GRaw
+ g( s0 U; e7 \% e. e' M"osd_scrub_max_interval": "604800.000000",
"osd_deep_scrub_interval": "604800.000000"
. x+ \: u0 V; F7 k9 ZWhen pgs are not scrubbed / deep scrubbed for the configured ratio of the interval, warnings of "pgs not scrubbed / deep-scrubbed in time" will show in Ceph status.
7 O. T3 C* T; }3 N. u% ?2 B3 W
4 }5 E. M$ I& K( k5 z' rWhen setting "mon_warn_pg_not_deep_scrubbed_ratio" or "mon_warn_pg_not_scrubbed_ratio" values to 0, warnings will be disabled. Please evaluate the cluster's past usage carefully before setting.

' m3 A+ v7 t) R$ O( S4 J/ xSometimes, we might get a false warning even though the osd_deep_scrub_interval has been increased. This happens when the scrubbing parameters are not applied globally because these settings are used by both OSDs and MONs.
8 T. b  w# J& w& {' b2 N/ zThe OSDs use them to determine when to run scrub, and the MONs/MGRs use them to check if they need to show the warning.
You can set it globally as:

Raw
7 S. [, n! u& Y% B$ G( _/ L9 o# ceph config set global osd_deep_scrub_interval 3628800
诊断步骤
Checking scrubbing related configurations by admin socket
Raw
ceph --admin-daemon /var/run/ceph/<admin_socket_name>.asok config show | grep scrub

admin

You can set the deep scrub period to 2 week, to stretch the deep scrub window. Insted of

; @5 a) F! g; d1 G8 W+ p$ @. m, l7 u osd_deep_scrub_interval = 604800
use:

osd_deep_scrub_interval = 1209600