|
|
1 osd.6 179 unable to obtain rotating service keys; retrying
, ]: C+ I0 M; v7 o& I6 L* G, i1 Y. M# \& f
systemctl status ceph-osd@6.service
; R1 y2 E$ M5 `1 X● ceph-osd@6.service - Ceph object storage daemon osd.6
2 Y. ~+ H% a$ @, C, p2 i* M+ A Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled) p+ U3 w" `3 a! J
Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago9 C) A! O, g9 C9 @1 X. Z8 ^7 }
Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)( J/ Z8 l" H2 O! O- V4 x
Main PID: 4254 (ceph-osd)$ w: ~8 @. k' h' d0 {" _' R0 J. m
Tasks: 62
2 P7 m: h/ {* L) Y R# P* }6 H3 z/ U Memory: 26.4M- v' i; u0 j' {7 X [( F0 W+ K& m
CPU: 1.689s
; j/ G/ z0 W% G- Y% ? CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service5 x& o# T& K% A7 J" X0 R
└─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph$ I/ X& u! c1 m& z% k" m& }
/ p% n) v r2 L( oJan 17 16:09:24 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...# d" {- a3 K3 A9 I: I
Jan 17 16:09:24 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
9 z8 B9 \" \0 @1 Q" @Jan 17 16:09:28 compute02 ceph-osd[4254]: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true. i* \* F# P% ]9 |
Jan 17 16:09:58 compute02 ceph-osd[4254]: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying. I. ]% f: o' r; y6 `
Jan 17 16:10:28 compute02 ceph-osd[4254]: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying! J; j) H- h) }2 N, ]
Jan 17 16:10:58 compute02 ceph-osd[4254]: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying- d# [, ]2 H1 R4 |* Z; O9 F7 p/ g
Jan 17 16:11:28 compute02 ceph-osd[4254]: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying: K$ d+ e+ a! e
" \# f9 `5 d/ O$ q$ F
5 g) i- _6 Q2 e# N% b1 { W: E" T9 v
5 g' R% X4 Z- w" @' u7 Y( ?
' ?* w0 Z! v3 L1 P5 o# [导致问题：ceph状态异常：
h, x& r) C, F* R3 uroot@controller:~# ceph osd tree2 e' a# J* o6 d/ }" f p
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
8 F. `( n! p1 M5 w$ {2 [-1 9.00000 root default
3 ?. l% ]3 x" e1 a$ g-4 3.00000 host compute01
* G& a% n# [* n 3 hdd 1.00000 osd.3 up 1.00000 1.00000
6 C: F: k% X3 w7 U1 t5 D( G 4 hdd 1.00000 osd.4 up 1.00000 1.00000* }: g7 |7 p; ?3 S8 X7 Y
5 hdd 1.00000 osd.5 up 1.00000 1.00000% R: u- h; i2 H; W) i
-5 3.00000 host compute02 v1 n$ u/ Z: X, v
6 hdd 1.00000 osd.6 down 0 1.00000
9 i0 K6 r9 B: d( A1 L+ x) L1 O9 @ 7 hdd 1.00000 osd.7 down 0 1.00000' f- W4 s4 }' M6 G
8 hdd 1.00000 osd.8 up 1.00000 1.00000
0 y- ?- T" U8 z" i/ v! E, j-3 3.00000 host controller ; a* `3 Q% Y/ p* |
0 hdd 1.00000 osd.0 up 1.00000 1.00000" O7 s# M$ l: Q% a" J7 O
1 hdd 1.00000 osd.1 up 1.00000 1.00000
/ |7 G1 }* r: G: D' | W) o% z! X( i 2 hdd 1.00000 osd.2 up 1.00000 1.00000/ d* x, r' d! i9 `6 d- u f) h
* J! ?3 j, Z3 s. n7 P+ M+ v
0 Y2 E' r- F) n. q! H- M这种问题是因为时间不同步导致的：; i$ i" |9 n/ Y |: k* q
; W7 J% R/ `8 J* C' ^' b调整时间配置chrony: ^. ]* X0 E9 ^8 }( o! h
一般是检查/etc/chrony/chrony.conf文件中同步的配置。0 L7 C" Z+ n0 L' h
pool controller iburst% j6 }" M$ E1 D/ ?( n
9 ^" M' Y. F D4 J' J
另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：4 F1 \' h# Y4 E1 |- _. v% I. ]
vim /etc/chrony/chrony.conf " W4 T1 r, K$ u3 l3 \
allow 192.168.8.0/24+ g3 \& h0 o3 A p2 \
5 [9 [$ | I/ U0 \, A, ]& j! \, T' m) z2 h, j) C S
所有的都重启下chrony的服务。
+ A' q& M& r& C4 H/ x: U( y) Jroot@controller:~# systemctl restart chrony.service
7 w0 _& |8 m5 g Z. g: G
3 w2 q* d& w+ s/ y6 C# b
: a6 z8 s. N9 t* B计算节点检查： G3 p/ ~9 m' Q8 d( {& E; k
chronyc sources- R: g& y6 q& U; W: [, X# w
MS Name/IP address Stratum Poll Reach LastRx Last sample
9 U: a; b% a: ~+ \: l===============================================================================. m1 U& P3 R. Q) \8 Z& ~
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns' `+ q' r8 s8 w7 ^# O' s
root@compute02:~# chronyc sources
0 v, j. c$ |$ h4 a; O, HMS Name/IP address Stratum Poll Reach LastRx Last sample 0 c3 V# Q! }2 Y; E$ L: t' O- K6 V' U
=============================================================================== d& e' J- j+ f) }" O1 V
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns
# L% h$ o- \# _$ H. t' iroot@compute02:~# systemctl enable chrony.service8 z) n2 j1 w. A6 G. ^/ s) B
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
/ L/ O8 T( C" O6 [7 P% Q) nExecuting: /lib/systemd/systemd-sysv-install enable chrony
# Q- }* F& v! M* ]
9 v& ?+ ^9 D0 P8 [计算节点重启chrony的服务：
) s) L2 k: b0 y, {6 B6 f
$ t0 c2 I5 F0 g, g3 I) |. Groot@compute02:~# systemctl restart chrony.service
4 o( h, u& m8 P: d检查状态：( y; d7 t; ~2 I5 J
' {% ~0 n( R* L# ]3 G/ v
root@compute02:~# chronyc sources
7 ^0 `; g* G% p' ~MS Name/IP address Stratum Poll Reach LastRx Last sample 9 a* ]$ `( Y* @. H+ `4 V o2 s7 d) e
===============================================================================
/ G' }( P1 n1 E, d( W^? 192.168.8.65 2 6 3 1 +8087ms[+8087ms] +/- 13ms
$ z% g8 O e- J/ ]/ T( f4 Z+ @9 e1 S0 m6 a5 D4 s
( m) C9 C' T n: [5 A: k$ Q* s
; t1 b& Q! L" o" e7 \root@compute02:~# systemctl status ceph-osd@6.service 3 `0 f( l( X9 U) r' h7 e1 L% w) d
● ceph-osd@6.service - Ceph object storage daemon osd.6
( S2 j1 f1 c2 l. @8 J( [3 e Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
& f5 w* `7 l7 n! G* }$ Y Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago
1 o2 B l+ m. c+ f0 T- X Main PID: 6049 (ceph-osd)
\. ^- s% [* ?( G Tasks: 62$ D1 p- c5 ]0 p4 Q' b
Memory: 84.6M- Z; W3 h1 \" z/ r
CPU: 15.598s E4 c+ z3 ~% V& i
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service. Y7 Q; t: s. C" c8 s6 Z, Y6 Q V
└─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
+ y. R. @/ v7 B! M r
' G6 J3 L) R3 ]7 PJan 17 16:15:08 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...7 Y) ?% K( w2 g6 X
Jan 17 16:15:08 compute02 systemd[1]: Started Ceph object storage daemon osd.6.5 Q+ c- p0 Z+ x- {4 f9 \% J+ e2 s
Jan 17 16:15:11 compute02 ceph-osd[6049]: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true
9 V$ g- K5 M r4 Z8 U8 t( bJan 17 16:15:41 compute02 ceph-osd[6049]: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
. `/ u+ E9 u5 [. {* PJan 17 16:16:11 compute02 ceph-osd[6049]: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying; h3 J; R0 Z( A8 P3 L
Jan 17 16:16:41 compute02 ceph-osd[6049]: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
# t& m; p2 `3 n) U0 |! C" k, ^Jan 17 16:17:11 compute02 ceph-osd[6049]: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
$ {- a* |: W" ^ b8 DJan 17 16:17:14 compute02 ceph-osd[6049]: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>
5 T( s2 e8 _3 Xroot@compute02:~#
9 ?; f3 r @# Q5 broot@compute02:~# systemctl restart ceph-osd@6.service
% C8 R& q4 {& b# Troot@compute02:~# systemctl status ceph-osd@6.service
3 J' {2 J, P9 C● ceph-osd@6.service - Ceph object storage daemon osd.6
0 |6 e) V6 \) [, A5 g4 Z Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)- N: n( k# [. }
Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago
2 u, a1 N% v4 D% n4 j p0 H Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
% O& T% z+ C6 F9 a' E- c8 u Main PID: 7795 (ceph-osd)7 o8 F1 G6 C. [$ }! s
Tasks: 14
; F4 u% @8 o7 b$ t$ X4 r Memory: 11.7M* J/ Y/ U. t0 }( _# a5 {8 q6 Z
CPU: 339ms
- \! H" H9 I/ h2 Y! @) G CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
- y! P& m* A! Z- [& o. K) J7 c* } └─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph# C2 ?. w1 q) G# o1 H- W6 y# s
. w2 \' |9 N5 c: R! Z) p
Jan 17 16:25:44 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
6 ?- ? Y1 M& E' fJan 17 16:25:44 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
3 t( h, d. A0 Q4 _* a% a% h) q
' p) y4 \; R# Z2 n$ t
7 \ X+ h; ?" u4 ^/ g, e: l% ~- K, m+ _7 a) ?+ Z! k% }
检查日志：
# |; `5 o. g0 B tail -f /var/log/ceph/ceph-osd.6.log ' @% U+ n a7 v) [
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [3,0] -> [3,0,6], acting [3,0] -> [3,0,6], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
- e% E/ ~. T5 w' ~! k2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1,5] -> [1,5,6], acting [1,5] -> [1,5,6], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 45401383207592263678 g8 R' M. C9 w S9 ~
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
: r9 h& h% [& |2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
& u0 Z8 r. [/ L1 @0 }6 |" Z2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [5,0] -> [5,0,6], acting [5,0] -> [5,0,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
3 g" x% ]" M3 F) j7 P! E a, q( x0 Y2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
# ^& p) l, \% Y* c$ O( ]2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [2,4] -> [2,4,6], acting [2,4] -> [2,4,6], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 45401383207592263673 N' \1 E$ ^( k. ]7 u3 J. }
2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
. K3 J/ ~8 l' Y4 E1 ^5 k2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [4,1] -> [6,4,1], acting [4,1] -> [6,4,1], acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 4540138320759226367
9 S x z' Y' P- s2 w8 g" A2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary
. T7 A9 T+ `$ l- V' [4 d0 Z* F2 `& W X) y0 y* `2 o
# \, r/ a3 I( S2 F
0 B) c( }9 S5 w5 ~5 ]恢复正常。1 [$ f+ o; N& H9 f
|
|
发表于 2024-1-17 17:00:00