ceph osd 报错 osd.6 179 unable to obtain rotating service keys; retrying

admin

1 osd.6 179 unable to obtain rotating service keys; retrying
) p: z7 ]  S# O% V
$ C. g- Y' R7 A+ S# y. v( T systemctl status ceph-osd@6.service
8 u  `7 `- C( Z: L& s● ceph-osd@6.service - Ceph object storage daemon osd.6
2 R4 K6 @) x- O; F8 q- Y# \     Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
+ W% ]7 j+ C7 l0 Q     Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago
7 _3 }4 _- W1 @7 S    Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
   Main PID: 4254 (ceph-osd)
: w; l! E! Z, ]% V5 Y3 h      Tasks: 62
, R7 P: c$ C3 S     Memory: 26.4M
7 J0 p! P6 u( D" t        CPU: 1.689s
$ n0 ]- `& Y/ C1 C' S+ R     CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
3 V, [+ m8 |; O4 i* C             └─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
) W  h8 S6 t4 `" y, ~! L2 }( G7 ^
Jan 17 16:09:24 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
* I: e8 r/ Z1 u2 sJan 17 16:09:24 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
& M5 r& |4 X8 M" f* AJan 17 16:09:28 compute02 ceph-osd[4254]: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true
, u3 _* n, _3 m4 M: a2 }1 ]Jan 17 16:09:58 compute02 ceph-osd[4254]: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:10:28 compute02 ceph-osd[4254]: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:10:58 compute02 ceph-osd[4254]: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:11:28 compute02 ceph-osd[4254]: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying





; y$ f9 w: L& N. @3 b/ k导致问题：ceph状态异常：
root@controller:~# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME            STATUS  REWEIGHT  PRI-AFF
-1         9.00000  root default                                 
, m& P# q$ H$ C# N+ m& o: y- \-4         3.00000      host compute01                           
5 W# Q# v5 R8 k+ m 3    hdd  1.00000          osd.3            up   1.00000  1.00000
: ~! L# M% D* N  N# L 4    hdd  1.00000          osd.4            up   1.00000  1.00000
5    hdd  1.00000          osd.5            up   1.00000  1.00000
-5         3.00000      host compute02                           
2 @/ P6 e' A- I* ]- |: Y 6    hdd  1.00000          osd.6          down         0  1.00000
4 j/ }" B7 B9 v 7    hdd  1.00000          osd.7          down         0  1.00000
! X/ N4 b: W) d6 e! h9 `% k 8    hdd  1.00000          osd.8            up   1.00000  1.00000
-3         3.00000      host controller                           
0    hdd  1.00000          osd.0            up   1.00000  1.00000
1    hdd  1.00000          osd.1            up   1.00000  1.00000
2    hdd  1.00000          osd.2            up   1.00000  1.00000

* X2 K. ?8 \  o5 p6 m* c6 H
1 E3 t+ Z, Z2 e! o这种问题是因为时间不同步导致的：
) V( ]2 |" ^/ b  D. M( o# R
调整时间配置chrony
+ h; @, b0 g, w* E7 y1 g% q5 _% X8 E一般是检查/etc/chrony/chrony.conf文件中同步的配置。
. g  N& u+ A9 w5 Q9 E# h7 Wpool controller iburst
6 w) u# U$ F5 L' s$ l7 r: r! I6 L
另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：
vim /etc/chrony/chrony.conf
allow 192.168.8.0/24
4 H6 `. f( ^2 K! V

所有的都重启下chrony的服务。
0 K  o' \1 T$ u! S' M! v. oroot@controller:~# systemctl restart chrony.service


- T2 U3 G: J2 ]: h. {5 I计算节点检查：
chronyc sources
" P4 x. B/ ~5 LMS Name/IP address         Stratum Poll Reach LastRx Last sample               
4 k  Q" q% K" y1 c" N===============================================================================
" g  x: U& a7 E  }, e8 \, d^? 192.168.8.65                  0   8     0     -     +0ns[   +0ns] +/-    0ns
root@compute02:~# chronyc sources
# ]& I0 a8 y: g+ M9 xMS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
5 l' m7 b8 `' L7 m8 l, |^? 192.168.8.65                  0   8     0     -     +0ns[   +0ns] +/-    0ns
root@compute02:~# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony
% V1 U$ ^9 R. T3 X; O
计算节点重启chrony的服务：
" [8 Y( s/ k+ a9 x8 t/ @3 P6 L
* X4 J& [2 p5 b0 p, troot@compute02:~# systemctl restart chrony.service
9 y! {( b1 I8 {7 X& Z# ?& S. Z检查状态：

root@compute02:~# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
0 g  O  Y  d4 A5 m; W  a( C===============================================================================
- U" {  N7 G4 [. j9 y: L% V^? 192.168.8.65                  2   6     3     1  +8087ms[+8087ms] +/-   13ms
& d/ A4 C* q. M4 b4 i. @
" K5 B+ j% U( o7 i7 ~  M

root@compute02:~# systemctl status ceph-osd@6.service
7 k/ e% L4 w( _9 T● ceph-osd@6.service - Ceph object storage daemon osd.6
8 l# j5 B6 D3 Z4 M     Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago
   Main PID: 6049 (ceph-osd)
      Tasks: 62
     Memory: 84.6M
        CPU: 15.598s
     CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
" h" s. f8 ]2 e, n9 e. \' m             └─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
7 f. D1 j1 N. H  Q+ s6 k
" Z7 h5 {2 r- n$ E, e6 K& u: j& f' HJan 17 16:15:08 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
Jan 17 16:15:08 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
- X0 Y- Y& G3 h  _( V, V6 }  \Jan 17 16:15:11 compute02 ceph-osd[6049]: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true
Jan 17 16:15:41 compute02 ceph-osd[6049]: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:16:11 compute02 ceph-osd[6049]: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:16:41 compute02 ceph-osd[6049]: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
* s6 D/ a" I8 v7 ?% z/ f* D3 CJan 17 16:17:11 compute02 ceph-osd[6049]: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
% M3 \3 L  i; }) A& ^Jan 17 16:17:14 compute02 ceph-osd[6049]: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>
root@compute02:~#
8 h" i6 w( f: i! }3 Hroot@compute02:~# systemctl restart ceph-osd@6.service
" ]# B: {2 X( A% l1 Proot@compute02:~# systemctl status ceph-osd@6.service
& U: ?& i6 }4 _3 U2 X) r● ceph-osd@6.service - Ceph object storage daemon osd.6
     Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago
    Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
' y- j( q' k; _& v1 \$ M; q9 Y   Main PID: 7795 (ceph-osd)
. r  d1 g9 C" e7 I      Tasks: 14
! E) i( I3 I" N/ w: _7 z     Memory: 11.7M
4 A7 k! |  P  S3 z        CPU: 339ms
     CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
3 D0 a. T# n1 M2 q# f& ^/ h. h- j             └─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
; E+ m' |" F5 \# d( C
Jan 17 16:25:44 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
. W0 l+ v( \0 G8 jJan 17 16:25:44 compute02 systemd[1]: Started Ceph object storage daemon osd.6.


8 h: I  a# k- M1 t
检查日志：
( v* @: m  ?4 ^& S1 S& g# D tail -f /var/log/ceph/ceph-osd.6.log
2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [3,0] -> [3,0,6], acting [3,0] -> [3,0,6], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
& |8 h. T" s: `, F/ A' ^2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1,5] -> [1,5,6], acting [1,5] -> [1,5,6], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
1 P$ p1 \) S7 W2 P+ Z2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
- G$ q8 \# v6 `$ D* k) f( x2 }2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
( y3 h# E; q2 {) q# p2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [5,0] -> [5,0,6], acting [5,0] -> [5,0,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
7 i* s2 X1 J, E) y" q2 c2 {2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
7 F, Y" M2 H3 x) I2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [2,4] -> [2,4,6], acting [2,4] -> [2,4,6], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f59153fb640  1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [4,1] -> [6,4,1], acting [4,1] -> [6,4,1], acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 4540138320759226367
0 t" |( R3 \9 z- u) W2 a% |2024-01-17T16:25:50.363+0800 7f5916bfe640  1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary

5 h& ]+ @' D8 U$ q- Q0 t) d

% Y! F. A: s+ X* B恢复正常。
: x  M6 E- B8 D- Q0 M