ceph存储backfills值和recover速度设置

admin

ceph daemon osd.2 config show|grep osd_max_backfills
, O) R8 p* v6 Bceph daemon osd.2 config show|grep osd_recovery_max_active
! s$ R, l) `3 f# I5 ~
ceph daemon osd.2 config show|grep osd_recovery_sleep

! {* D# F! Q0 V
ceph tell osd.* injectargs "--osd_recovery_sleep_hdd=0"
ceph tell osd.* injectargs "--osd_max_backfills=10"
" H- N. q3 N' Z, |ceph tell osd.* injectargs "--osd_recovery_max_active=15"
& e# r# }+ r+ y) y: I  }
5 N. i) [& q5 ?' a$ c* ^
同步速度就会上去
8 U4 @% E& l3 }+ o

admin

官方建议的PG使用数量：
( w( Y) c" N$ Q. X" ~% B/ o$ X
6 o& K( y+ u, N9 e集群中小于5个OSD，则设置PG的数量为128。
8 \8 n$ }6 h& p1 y2 G& [8 r8 p集群有5-10个OSD时，设置PG的数量为512。
% {' C% D0 f0 Q& |0 v3 E2 I5 ^( u集群中有10-50个OSD时，设置PG的数量为1024.
$ |# Y/ D+ J. j5 G当集群中超过50个OSD时，需要权衡PG的数量，有一种公式：

​ (osd_number * 100)

; j+ Y' y2 z" p" a+ Upg_num = ------------------------------
+ r& ~+ ~2 C* \: z  c% G! c+ v
​ pool size
! k% v, K; T) Y- V: \
4 H' G0 U% P" o& g5 ]3 L例如集群有90个OSD，Pool设置的副本数为3：(90*100)÷3=3000
" N- X' A5 |7 ^7 q, [
PGP的数量要和PG的数量保持一致。
: l( r. w7 p1 e- G
# Y4 x$ g* v0 b7 ?2 W( Z1 f注意这个PG数量是集群所能承担的总PG数量，每个资源池分配多个PG，还需要通过总的PG数乘以资源池占整个集群数据比例，最终拿到一个资源池应该分配多少个PG的数量。

admin

控制数据恢复及回填速度
6 k" K5 F# O  ^  z: v4 S9 B5 o9 J# 将用于数据恢复操作的优先级提到最高级别
# `1 U& e' b( f9 P# i4 {5 hceph tell osd.* injectargs "--osd_recovery_op_priority=63"
; X: H0 ^9 T5 p' |6 K( `
# 将用于client I/O操作的优先级降到3
, [) `- }7 I" Z6 [ceph tell osd.* injectargs "--osd_client_op_priority=3"
3 B1 ~, R% U+ o: j  S
3 C# ?, x. n  ~& Z" R# 将每个osd上用于回填并发操作数由默认的1调整到50
. Q2 h  |! k, \, J$ O  H+ Mceph tell osd.* injectargs "--osd_max_backfills=50"

2 ]% j% R' M6 }* ]1 w. j8 ^# 将每个osd上用于恢复的并发操作数由默认的3调整到50
ceph tell osd.* injectargs "--osd_recovery_max_active=50"
/ M/ D/ B/ Y% u: W0 O) M' X/ w9 X
# 将每个osd上用于执行恢复的线程数由默认的1调整到10
# ?. [. O+ S. G8 a+ W- ]7 _ceph tell osd.* injectargs "--osd_recovery_threads=10"

admin

[root@hostceph1 ~]# ceph daemon osd.2 config show|grep osd_max_backfills
    "osd_max_backfills": "1",
' M# D+ j! Y, k3 f[root@hostceph1 ~]# ceph daemon osd.2 config show|grep osd_recovery_max_active
    "osd_recovery_max_active": "3",
' |8 r+ o9 o# B! D4 w% \[root@hostceph1 ~]# ceph daemon osd.2 config show|grep osd_recovery_sleep
    "osd_recovery_sleep": "0.000000",
9 v% D$ @4 @  C- ?9 N    "osd_recovery_sleep_hdd": "0.100000",
8 r8 Z' [6 F% k; S& [    "osd_recovery_sleep_hybrid": "0.025000",
& [9 E% j4 X( m$ A    "osd_recovery_sleep_ssd": "0.000000",
( c1 r: M2 Z3 w+ ?" b) M8 f[root@hostceph1 ~]# ceph daemon osd.2 config show|grep max_backfills
% \+ K' S" Q0 |5 C, c    "osd_max_backfills": "1",
[root@hostceph1 ~]# ceph daemon osd.2 config show|grep recovery_max_active
' Y5 H4 I7 S3 X    "osd_recovery_max_active": "3",
$ p0 @4 F9 {0 f) n& Q# Z+ c$ E9 y[root@hostceph1 ~]# ceph daemon osd.2 config show|grep recovery_threads
[root@hostceph1 ~]# ceph tell osd.* injectargs "--osd_recovery_sleep_hdd=0"
osd.0: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.1: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.2: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.3: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
7 \0 A) m% u. `2 A1 [4 e' l6 qosd.4: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
4 H! ?( O& E; n* P3 r- Oosd.5: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.6: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.7: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.8: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
  |, r) J/ u, u. _7 e5 d3 wosd.9: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.10: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
' E: `5 _9 ?- X( S  sosd.11: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.12: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.13: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.14: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
8 ~) D& [$ U) w3 L# Tosd.15: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.16: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.17: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
& ^2 ~0 f+ \$ w( t* n* vosd.18: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.19: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.20: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.21: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
+ g# q0 K+ K% ~% {& \! Zosd.22: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
! c, ?9 z4 j3 |$ zosd.23: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.24: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
9 O1 s. m& b) Q8 O$ p# Q9 Q& cosd.25: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
$ @1 g7 I. R/ ?# `osd.26: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.27: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
+ S* N& o5 w! t# E4 R* Hosd.28: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
1 T& P0 S9 U% u+ \osd.29: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.30: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.31: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
8 H3 n0 Y& z# d1 u% W2 q1 @osd.32: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
osd.33: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
' \2 V6 [% r* k[root@hostceph1 ~]# ceph tell osd.* injectargs "--osd_max_backfills=10"
osd.0: osd_max_backfills = '10'
osd.1: osd_max_backfills = '10'
) ^+ m$ a1 _: E- O' yosd.2: osd_max_backfills = '10'
8 v1 U5 K3 {# p" T) _- S+ xosd.3: osd_max_backfills = '10'
osd.4: osd_max_backfills = '10'
osd.5: osd_max_backfills = '10'
osd.6: osd_max_backfills = '10'
( S1 s8 [0 [) k* oosd.7: osd_max_backfills = '10'
7 Z2 w0 v; E5 P! E1 L" dosd.8: osd_max_backfills = '10'
osd.9: osd_max_backfills = '10'
osd.10: osd_max_backfills = '10'
osd.11: osd_max_backfills = '10'
osd.12: osd_max_backfills = '10'
osd.13: osd_max_backfills = '10'
& w3 ~" d. o4 `osd.14: osd_max_backfills = '10'
osd.15: osd_max_backfills = '10'
osd.16: osd_max_backfills = '10'
osd.17: osd_max_backfills = '10'
: J  M" F, g: C, aosd.18: osd_max_backfills = '10'
osd.19: osd_max_backfills = '10'
# a$ k, W, G% Q8 |8 f5 _6 w0 Bosd.20: osd_max_backfills = '10'
osd.21: osd_max_backfills = '10'
4 b+ N* l! e9 G9 p: D/ I0 P( F0 Dosd.22: osd_max_backfills = '10'
" ]7 o5 j: a4 r! i& l9 j/ vosd.23: osd_max_backfills = '10'
0 U; m; l: \1 c; _# Fosd.24: osd_max_backfills = '10'
osd.25: osd_max_backfills = '10'
osd.26: osd_max_backfills = '10'
4 w" y) c. z. `( `' Qosd.27: osd_max_backfills = '10'
8 ^/ Q) s; I! O0 Tosd.28: osd_max_backfills = '10'
& A, u" F& L  x) z* |osd.29: osd_max_backfills = '10'
! q. S/ v+ {# ?6 D3 G; Nosd.30: osd_max_backfills = '10'
osd.31: osd_max_backfills = '10'
osd.32: osd_max_backfills = '10'
osd.33: osd_max_backfills = '10'
( \! r3 {& p+ S1 f' E$ \# z; @[root@hostceph1 ~]# ceph tell osd.* injectargs "--osd_recovery_max_active=10"
osd.0: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.1: osd_recovery_max_active = '10' (not observed, change may require restart)
7 l6 r0 N& B0 N8 Qosd.2: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.3: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.4: osd_recovery_max_active = '10' (not observed, change may require restart)
- b% x2 ~# h3 E3 Hosd.5: osd_recovery_max_active = '10' (not observed, change may require restart)
$ T* g4 j% K/ ?% S  W7 hosd.6: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.7: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.8: osd_recovery_max_active = '10' (not observed, change may require restart)
5 q8 a3 q6 E2 |: Q: P! i/ S& ~osd.9: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.10: osd_recovery_max_active = '10' (not observed, change may require restart)
: A- i7 y" }# R' i$ ^osd.11: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.12: osd_recovery_max_active = '10' (not observed, change may require restart)
) l4 X2 t" K5 q7 i/ yosd.13: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.14: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.15: osd_recovery_max_active = '10' (not observed, change may require restart)
. m! c* U0 f+ G! Xosd.16: osd_recovery_max_active = '10' (not observed, change may require restart)
  m2 x  r3 `7 ]& @' m, g/ j" I, Uosd.17: osd_recovery_max_active = '10' (not observed, change may require restart)
8 O$ n2 w4 Y& K5 bosd.18: osd_recovery_max_active = '10' (not observed, change may require restart)
( t+ N% U+ C9 [8 josd.19: osd_recovery_max_active = '10' (not observed, change may require restart)
( x, k! I" V& s- n/ Mosd.20: osd_recovery_max_active = '10' (not observed, change may require restart)
# w! c- I! O# t6 d0 Rosd.21: osd_recovery_max_active = '10' (not observed, change may require restart)
- P& H! t" m1 D! A& K1 cosd.22: osd_recovery_max_active = '10' (not observed, change may require restart)
) d2 |6 u1 Q# a+ M6 Vosd.23: osd_recovery_max_active = '10' (not observed, change may require restart)
( J( C0 T( }, ~: Z6 eosd.24: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.25: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.26: osd_recovery_max_active = '10' (not observed, change may require restart)
: w- Y( s" h$ @; c# j& mosd.27: osd_recovery_max_active = '10' (not observed, change may require restart)
4 o$ h' U" u/ g: E$ }9 r- kosd.28: osd_recovery_max_active = '10' (not observed, change may require restart)
) r4 C& H4 Z9 Z' bosd.29: osd_recovery_max_active = '10' (not observed, change may require restart)
osd.30: osd_recovery_max_active = '10' (not observed, change may require restart)
4 _- E' D1 Z  u: Nosd.31: osd_recovery_max_active = '10' (not observed, change may require restart)
7 Y; @) U5 f, ?' ~osd.32: osd_recovery_max_active = '10' (not observed, change may require restart)
& g& O! N9 A; y1 V% E, Y8 josd.33: osd_recovery_max_active = '10' (not observed, change may require restart)
[root@hostceph1 ~]# ceph -s
+ o6 O6 Y- J3 }8 [' l* F9 `7 p$ [) \  cluster:
    id:     29046cc0-0682-496b-98b1-912e59964282
    health: HEALTH_WARN
            Reduced data availability: 24 pgs inactive
& o: W" R0 Q5 {- _            24 pgs not deep-scrubbed in time
            24 pgs not scrubbed in time

7 C, g: Y$ q$ _- P" O  services:
    mon: 3 daemons, quorum hostceph1,hostceph2,hostceph3 (age 3M)
2 `& ^! Z- `* O. r3 y    mgr: hostceph2(active, since 3M), standbys: hostceph1, hostceph3
# y) I! \$ ?. B2 C    osd: 34 osds: 34 up (since 3M), 34 in (since 3M)

  data:
6 ?0 m/ w9 @4 I$ w3 o4 E    pools:   9 pools, 5760 pgs
! B8 f$ L0 b2 j) `5 Q0 t    objects: 3.11M objects, 12 TiB
* a8 N3 Q+ b. C% y7 p3 k1 j    usage:   33 TiB used, 92 TiB / 126 TiB avail
    pgs:     0.417% pgs unknown
/ X5 @( x3 M; g" o* `! |5 K- `             5735 active+clean
             24   unknown
             1    active+clean+scrubbing+deep

  io:
+ |' ~" W* P# a2 W1 b+ \    client:   212 KiB/s rd, 20 MiB/s wr, 258 op/s rd, 647 op/s wr
- O5 D/ T# c0 A, d: C7 N! e4 I* t
% Q: s! n! M! h& x) n[root@hostceph1 ~]# ceph -s
( [$ V0 Z. W+ w4 h0 j9 j  cluster:
    id:     29046cc0-0682-496b-98b1-912e59964282
    health: HEALTH_WARN
            Reduced data availability: 24 pgs inactive
% y1 H  H6 F) s) e            24 pgs not deep-scrubbed in time
            24 pgs not scrubbed in time
2 t3 K3 T0 ]5 F2 P( t
  services:
    mon: 3 daemons, quorum hostceph1,hostceph2,hostceph3 (age 3M)
    mgr: hostceph2(active, since 3M), standbys: hostceph1, hostceph3
) t4 h/ O( T  a) Y0 m! m8 H    osd: 34 osds: 34 up (since 3M), 34 in (since 3M)
1 L# e, A) K6 O+ [0 f6 A" G
2 h9 x* u1 Y5 y3 @( v( T3 w, m- n  data:
    pools:   9 pools, 5760 pgs
' @# n) W8 h* N- K! }$ h. P    objects: 3.11M objects, 12 TiB
* d8 h" H0 t' l    usage:   33 TiB used, 92 TiB / 126 TiB avail
: k7 n1 s# a7 z) f% Z) s! f    pgs:     0.417% pgs unknown
             5735 active+clean
             24   unknown
             1    active+clean+scrubbing+deep
8 N  A( Z8 q4 w
) z2 i# v% ~/ \" `  io:
' e; ^0 I# u' }4 p  T% ]  e, F    client:   9.7 KiB/s rd, 20 MiB/s wr, 12 op/s rd, 625 op/s wr