pgs not deep-scrubbed in time异常处理

admin

发现出现异常warn信息，虽然不影响整个集群正常使用，但强迫症患者还是忍不了，下面是过程。查看具体报错信息
HEALTH_WARN 2 pgs not deep-scrubbed in time
PG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time
; i5 D( O- O" r9 L/ U3 h/ K    pg 18.41 not deep-scrubbed since 2022-12-07 20:15:50.550606
    pg 5.16d not deep-scrubbed since 2022-12-07 22:21:58.141071
" H: C0 K5 h' O; {) y2 ~7 k  v; k

[root@controller1 ~]# ceph pg deep-scrub 18.41
instructing pg 18.41 on osd.6 to deep-scrub
. Y4 p3 @4 ~; K. {! o) L[root@controller1 ~]# ceph pg deep-scrub 5.16
7 i& {8 ?% Z/ I9 O8 rinstructing pg 5.16 on osd.13 to deep-scrub
6 X. B5 u' c2 n( b6 n0 ~$ `  w6 `
; V1 x: V' O4 X/ U- ]

; p' u( V; }& L5 S6 P' L' e" B7 u[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
0 ~! L3 C1 ?9 r/ @5 ~3 E8 g    "osd_deep_scrub_interval": "604800.000000",
- x& b" B: r: P5 q[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
! U' a/ p2 T7 A! s7 C[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
    "osd_deep_scrub_interval": "3628800.000000",
, N$ p) R1 v1 b8 a& ][root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
. {( n- w$ y4 `恢复正常后，再改回来：
! ]/ X4 P1 |" X[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
# ]/ ]) A) v1 x    "osd_deep_scrub_interval": "604800.000000",
' I+ J. j- j- d[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
: M8 o" g* Y+ Z[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
, r+ n' o% @# W    "osd_deep_scrub_interval": "3628800.000000",
[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800^C

  {: l, W/ d5 N: B/ i[root@controller1 ~]#
9 G/ `0 _% Y) p% M" W# x+ Q[root@controller1 ~]# ceph -s
  cluster:
* `# ^2 I4 h$ A: k0 |    id:     9d22e36a-2bdd-4d2d-8394-48af75ead777
7 ]4 N6 y8 j+ X    health: HEALTH_OK

" n) F/ l" @- r. }  O) z; L  services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 5M)
! G; g; P9 ~% l3 t' D  r  N$ F! _; C( F    mgr: ceph1(active, since 19M), standbys: ceph2,ceph3
    osd: 40 osds: 40 up (since 3w), 40 in (since 12M)
# `6 }* p- \0 W* w" K    rgw: 3 daemons active (host09, host10, host11)
& }$ y- [; ^; |
2 r$ u) g5 T- _* }  task status:
# ]- R3 y8 Y& p5 C' b
  data:
    pools:   16 pools, 3072 pgs
    objects: 4.20M objects, 16 TiB
    usage:   40 TiB used, 107 TiB / 148 TiB avail
; W2 m- X4 B6 N- M$ R* B" N    pgs:     3067 active+clean
* f6 D8 |$ ^( O# X: t5 j" a             5    active+clean+scrubbing+deep
+ S" R8 l) |. A+ V0 H/ |, N& P: b
  io:
$ `# b. B5 v5 z4 _' u, [    client:   403 KiB/s rd, 9.5 MiB/s wr, 514 op/s rd, 466 op/s wr

[root@ ~]# ceph config set global osd_deep_scrub_interval 604800
8 V9 y; o. \& J5 m[root@ ~]#

' d" @/ R, A8 y1 n/ x$ d2 H6 R
systemctl restart ceph-osd@6.service

4 |1 b# t7 n+ m8 R% Z( O

5 v" H7 z4 U, x4 O5 T. |

admin

根源
RHCS 4 has introduced 2 new parameters on scrubbing interval warning, and default values are
! G; Q. o# N: W$ @  O0 SRaw
"mon_warn_pg_not_deep_scrubbed_ratio": "0.750000",
"mon_warn_pg_not_scrubbed_ratio": "0.500000",
& L' p0 B' v( N* E% g+ iThese ratios are based on the following parameters, and default values are
8 G0 T( t# P! f: |Raw
"osd_scrub_max_interval": "604800.000000",
5 L4 G- S2 w9 ~$ h4 M/ X% i" O$ N$ U: O"osd_deep_scrub_interval": "604800.000000"
When pgs are not scrubbed / deep scrubbed for the configured ratio of the interval, warnings of "pgs not scrubbed / deep-scrubbed in time" will show in Ceph status.
, H2 [3 D3 L2 @' z" r6 a
& k7 Z. C; p/ g' ?! ]$ R: ~' UWhen setting "mon_warn_pg_not_deep_scrubbed_ratio" or "mon_warn_pg_not_scrubbed_ratio" values to 0, warnings will be disabled. Please evaluate the cluster's past usage carefully before setting.

Sometimes, we might get a false warning even though the osd_deep_scrub_interval has been increased. This happens when the scrubbing parameters are not applied globally because these settings are used by both OSDs and MONs.
The OSDs use them to determine when to run scrub, and the MONs/MGRs use them to check if they need to show the warning.
" b4 S, n% k# D! M8 }! q, ~7 ~You can set it globally as:

Raw
# ceph config set global osd_deep_scrub_interval 3628800
诊断步骤
" k; T% I3 X2 _, S9 `Checking scrubbing related configurations by admin socket
. P7 ?# j! e) e1 j2 x7 _Raw
ceph --admin-daemon /var/run/ceph/<admin_socket_name>.asok config show | grep scrub

admin

You can set the deep scrub period to 2 week, to stretch the deep scrub window. Insted of

osd_deep_scrub_interval = 604800
% m+ {' k" v% R# P6 a% ~use:

osd_deep_scrub_interval = 1209600