ceph osd 报错 osd.6 179 unable to obtain rotating service keys; retrying

admin

1 osd.6 179 unable to obtain rotating service keys; retrying

- l/ M: u$ b2 ] systemctl status ceph-osd@6.service
; e4 N% f# s& E6 m● ceph-osd@6.service - Ceph object storage daemon osd.6
* u; ?" z7 C  a* X1 A8 R% X+ |# i     Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago
" W$ {0 e) H8 ^% l    Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
   Main PID: 4254 (ceph-osd)
+ C7 C9 d! T8 J7 ]7 n2 J      Tasks: 62
     Memory: 26.4M
3 q) L  e! X/ w* e3 x) w" d        CPU: 1.689s
0 Z% d0 s" r, Z9 F! u! j     CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
             └─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph

Jan 17 16:09:24 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
Jan 17 16:09:24 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
Jan 17 16:09:28 compute02 ceph-osd[4254]: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true
Jan 17 16:09:58 compute02 ceph-osd[4254]: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
! D# ^) C. }$ L* KJan 17 16:10:28 compute02 ceph-osd[4254]: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
' P9 s+ z! s2 D% P: L8 oJan 17 16:10:58 compute02 ceph-osd[4254]: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
) V6 N" J  t6 f' |* F1 @Jan 17 16:11:28 compute02 ceph-osd[4254]: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying


% s8 k8 |4 }4 c7 O% G+ |. E


5 h  {* ^& \! {7 t  f' z; r导致问题：ceph状态异常：
root@controller:~# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME            STATUS  REWEIGHT  PRI-AFF
-1         9.00000  root default                                 
& _6 U9 g6 k% Z; ^) f5 t-4         3.00000      host compute01                           
2 f2 r! d. x8 I( ]1 P- T+ v+ u4 p 3    hdd  1.00000          osd.3            up   1.00000  1.00000
" r1 O$ U: ~+ I" ~) ^( ] 4    hdd  1.00000          osd.4            up   1.00000  1.00000
& M0 E) [* y! h 5    hdd  1.00000          osd.5            up   1.00000  1.00000
: q2 N7 m- [+ y. G9 {4 `# Y-5         3.00000      host compute02                           
6    hdd  1.00000          osd.6          down         0  1.00000
' Q2 t: G( }, F! _  w* e7 \ 7    hdd  1.00000          osd.7          down         0  1.00000
8    hdd  1.00000          osd.8            up   1.00000  1.00000
-3         3.00000      host controller                           
0    hdd  1.00000          osd.0            up   1.00000  1.00000
1    hdd  1.00000          osd.1            up   1.00000  1.00000
- Q( B, |# s# |: B# P2 T 2    hdd  1.00000          osd.2            up   1.00000  1.00000


( J: q4 H; p% H& m. z这种问题是因为时间不同步导致的：
" P5 x5 O# y( ]
调整时间配置chrony
# A7 C$ d9 F  w1 c一般是检查/etc/chrony/chrony.conf文件中同步的配置。
pool controller iburst
3 `- Y  D# d3 d; S8 S) ]
& ]( @2 }5 J0 i. O% u另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：
vim /etc/chrony/chrony.conf
allow 192.168.8.0/24

, j, _' H- m% h" o
所有的都重启下chrony的服务。
, p0 o$ X: m4 c. \. M* M5 Hroot@controller:~# systemctl restart chrony.service
! m: B* o1 O! e" l9 K/ R
# n! `: j; m- k4 z4 I6 r5 V
计算节点检查：
chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^? 192.168.8.65                  0   8     0     -     +0ns[   +0ns] +/-    0ns
) o$ i6 n" J* C, c8 n* groot@compute02:~# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
8 s+ H+ V# |2 A1 z/ e/ N^? 192.168.8.65                  0   8     0     -     +0ns[   +0ns] +/-    0ns
3 g$ E( B6 m+ D5 z" p2 v8 lroot@compute02:~# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony

计算节点重启chrony的服务：

root@compute02:~# systemctl restart chrony.service
检查状态：

8 d8 u; [9 z6 `root@compute02:~# chronyc sources
/ \# n# c: z! z, |! _( @3 m7 k4 d. sMS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
( m" {$ V  d+ Z. T4 l4 M& v) [^? 192.168.8.65                  2   6     3     1  +8087ms[+8087ms] +/-   13ms


  @) q" Q5 n6 J" E/ x
9 ~, [4 u1 _1 qroot@compute02:~# systemctl status ceph-osd@6.service
● ceph-osd@6.service - Ceph object storage daemon osd.6
     Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago
3 K' E; X1 v( U1 a2 k* g/ I7 y6 x" u   Main PID: 6049 (ceph-osd)
9 D; e$ w' J# ?8 N      Tasks: 62
) t  u* P: u- u  v3 ^/ N% Y& D) \     Memory: 84.6M
6 X1 i  k& ]+ H! Z: O2 G        CPU: 15.598s
5 [, P- s( X' m     CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
; b  {% c+ f( c8 d1 B* F+ ^$ E             └─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph

Jan 17 16:15:08 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
Jan 17 16:15:08 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
Jan 17 16:15:11 compute02 ceph-osd[6049]: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true
& D4 }+ C% q2 C2 t6 cJan 17 16:15:41 compute02 ceph-osd[6049]: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
$ t% z* N+ {5 g) w, A" qJan 17 16:16:11 compute02 ceph-osd[6049]: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:16:41 compute02 ceph-osd[6049]: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
' o0 R! A8 Y$ }: j1 dJan 17 16:17:11 compute02 ceph-osd[6049]: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:17:14 compute02 ceph-osd[6049]: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>
# t. X% L# r! w4 q$ croot@compute02:~#
root@compute02:~# systemctl restart ceph-osd@6.service
root@compute02:~# systemctl status ceph-osd@6.service
1 f9 ]6 V5 t' d● ceph-osd@6.service - Ceph object storage daemon osd.6
3 t4 s7 K# {, D9 N. C6 R     Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
- l: d- P7 K$ M! D. L# ?     Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago
    Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
   Main PID: 7795 (ceph-osd)
      Tasks: 14
) ]% _; b, t6 R$ A2 ?/ I     Memory: 11.7M
* X" d2 q, \7 I$ e7 y; @6 S# g8 Z% U        CPU: 339ms
0 F& x8 @9 ?1 o0 m6 v- a# B     CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
             └─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph

7 h8 o( o' d4 `9 U" ^5 e' iJan 17 16:25:44 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
Jan 17 16:25:44 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
# [5 n$ i; d! L( G9 o6 c


检查日志：
  X+ B* O& Q9 Y' I+ _6 K. z( v) l: H tail -f /var/log/ceph/ceph-osd.6.log
; r; V! @' w9 W8 x! r8 J2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [3,0] -> [3,0,6], acting [3,0] -> [3,0,6], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1,5] -> [1,5,6], acting [1,5] -> [1,5,6], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
$ K2 I- e% {2 f  e- p+ W! `( B2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
: }, f; i5 |3 u! u2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [5,0] -> [5,0,6], acting [5,0] -> [5,0,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
' s& l; J- R4 Y$ ^( t1 T- b2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [2,4] -> [2,4,6], acting [2,4] -> [2,4,6], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [4,1] -> [6,4,1], acting [4,1] -> [6,4,1], acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary
6 [, k# v0 J% r; T3 ?! @( d3 Y

3 k4 I; x) |$ Q/ w: w% }& C
恢复正常。
! ?+ d6 b0 n3 U+ x& K* i. J+ }