|
|
1 osd.6 179 unable to obtain rotating service keys; retrying% {$ C2 e) m7 G' d3 @1 M7 ? N6 l
+ Q2 J6 x1 z7 |) M systemctl status ceph-osd@6.service : H/ ^4 F& m2 ^& p) P+ l
● ceph-osd@6.service - Ceph object storage daemon osd.6
5 D* m+ \6 a; C; ]9 a9 d4 S Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)6 P- l1 ?7 s7 k( R% m
Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago
) l6 G8 [5 ? G1 m' W9 T( p0 F! A- q Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)1 D, i8 S( Y% }7 ]* C o0 m/ ^1 p
Main PID: 4254 (ceph-osd)) q# c4 B& W5 q
Tasks: 62
# j# [' r* [2 D$ Z. ?* e* N& k Memory: 26.4M
9 |3 u2 ?% ]" j2 z9 [, @! c6 p CPU: 1.689s* j n, V6 H5 j% ^/ G) J5 B; A
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service% |" h1 i0 V4 E
└─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
! d# |! @& z! _1 O
2 U: e" s' y0 N: C! ~% BJan 17 16:09:24 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
/ @" k5 j3 N& {" C/ N- q4 UJan 17 16:09:24 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
+ {! z! S2 d, l" N; u% |Jan 17 16:09:28 compute02 ceph-osd[4254]: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true
6 C4 Q6 q) B. QJan 17 16:09:58 compute02 ceph-osd[4254]: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying' \) i4 P/ R; ~% p4 O
Jan 17 16:10:28 compute02 ceph-osd[4254]: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying7 D( R% D3 ^% `# k4 m. s
Jan 17 16:10:58 compute02 ceph-osd[4254]: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying5 W, y9 b8 P y0 a. x; c" [) ]' {
Jan 17 16:11:28 compute02 ceph-osd[4254]: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying1 j1 P0 y Q* I* O8 C- C
9 i1 }0 `. \, Q% `0 r$ l
& a! `- F |! G) q2 |; g1 j8 A Y$ ?" |/ \
4 g! f( i( s0 j: R) d4 E
# A4 K( W# y4 c; i( w* E
导致问题：ceph状态异常：
: S5 j) N8 R$ n& r4 froot@controller:~# ceph osd tree
( B. U4 l! P3 E! `4 O2 SID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
[& v' E; j+ p" n-1 9.00000 root default
4 s/ w X g: z. X& U8 }: L4 e: L-4 3.00000 host compute01 ! c4 k: x5 X9 A; r5 s
3 hdd 1.00000 osd.3 up 1.00000 1.00000+ X1 U/ T ?' z0 U% }' w, S+ ]8 A ~8 l
4 hdd 1.00000 osd.4 up 1.00000 1.000002 s* M% E6 h8 k# ?# g3 X3 X: A
5 hdd 1.00000 osd.5 up 1.00000 1.00000
" W) X& D+ E( o% ]-5 3.00000 host compute02 5 u9 A- w/ R+ m% n% W$ \
6 hdd 1.00000 osd.6 down 0 1.00000
4 D6 u' B4 r+ p/ i2 e 7 hdd 1.00000 osd.7 down 0 1.00000/ z( _2 v( a5 P6 I4 d9 d# ]: E
8 hdd 1.00000 osd.8 up 1.00000 1.00000
" \% C( v" u9 {# C6 l5 \( G' ?+ a-3 3.00000 host controller 8 e$ S; h) ^5 C7 r
0 hdd 1.00000 osd.0 up 1.00000 1.000002 n" `; Z) F# @: G: N
1 hdd 1.00000 osd.1 up 1.00000 1.00000
+ @' @( ^+ J7 Z7 H/ ?+ k 2 hdd 1.00000 osd.2 up 1.00000 1.00000
0 v% W) ]' y/ a% B0 h x. n; f! ]0 T4 E7 y* j
1 u# W$ Z2 d, n$ h' y( a5 S9 ^这种问题是因为时间不同步导致的：
5 h5 E; e0 B$ _# Q- ^2 K
6 O1 G; C0 ]7 S调整时间配置chrony6 n) k$ u/ h6 Y2 C- f( `3 C" J
一般是检查/etc/chrony/chrony.conf文件中同步的配置。0 W+ a* f+ A3 u2 {! w
pool controller iburst4 Q1 R4 U" g1 \3 i0 M* y
, W+ y" V; t# ?1 C& Q. T* h$ K另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：
& u1 r2 N% U' h" i) k) Ovim /etc/chrony/chrony.conf 1 X$ M& U. |; T: x! A& N, I9 S
allow 192.168.8.0/24
3 `2 V0 G/ D* l3 o9 k9 S5 o( j( J) l4 r$ O
* Y9 u) x. H( U0 i+ A' h所有的都重启下chrony的服务。/ C& [. W9 t) I
root@controller:~# systemctl restart chrony.service 5 X4 x' V9 u$ Q8 `) j
9 E; m6 i3 X. m# o: D; c: c0 u9 K' c g- `8 @, h1 e: q
计算节点检查：
3 `6 `& n4 x& z0 @ chronyc sources1 A7 K5 g' V" ~7 ~& r3 N, F
MS Name/IP address Stratum Poll Reach LastRx Last sample 2 G$ {( ~8 _1 J
===============================================================================& Y: c- ~' X6 n
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns4 L) E6 I, Q) k
root@compute02:~# chronyc sources
0 S% \' y; G7 |/ p' v$ LMS Name/IP address Stratum Poll Reach LastRx Last sample
' Y9 q$ c; E% o+ c===============================================================================
& W7 j7 I. c1 Z2 I( |; d^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns. s" W: U3 s: V% }! @7 h, i, _
root@compute02:~# systemctl enable chrony.service
1 k( A1 _; t" {, `Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.' S' a/ L6 z6 s& P! K0 ^. f
Executing: /lib/systemd/systemd-sysv-install enable chrony
8 y5 ^* E6 U2 H; Y: S7 n/ {
~0 h8 E% `( [: `" J+ B9 [, M计算节点重启chrony的服务：7 n* p' i/ o& I4 ?% v
9 s" d" ?$ \) @
root@compute02:~# systemctl restart chrony.service6 u& Z2 L5 V. ]- g8 M/ d+ N0 f, Y
检查状态：: ~+ U+ C5 [7 k# u+ M
) E4 i& \9 t+ k2 Y. m
root@compute02:~# chronyc sources# F1 G& L+ W/ d
MS Name/IP address Stratum Poll Reach LastRx Last sample
, o$ J7 J( G$ O+ U/ _- |& \) k===============================================================================8 N% N3 I8 y% w
^? 192.168.8.65 2 6 3 1 +8087ms[+8087ms] +/- 13ms0 ^& v& ?2 D) A; D; U7 w
& B$ V9 ` h9 e! Q4 V5 ]# X9 z0 ]3 n+ k
; i: K4 y. q' Y3 uroot@compute02:~# systemctl status ceph-osd@6.service
5 I7 U/ u6 s* w5 v● ceph-osd@6.service - Ceph object storage daemon osd.6
I& Z) ?; W7 H! P! Q, [& c: ] Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)( l' d4 f+ U$ w1 _2 C) E: F8 a* b
Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago
4 T5 E. t: A+ N" A4 S% B Main PID: 6049 (ceph-osd)
# t) o' @( q' R- [( X" L' {7 Z Tasks: 62
, k) h: X0 [; h) E Memory: 84.6M
$ E; T: k. U; c CPU: 15.598s
( U& ]: t; \2 u r/ c2 N9 w CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service' Z5 W; u; C! X- X
└─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph( }4 T& ~# a$ t
( ~3 J+ s% n: ^' R+ ?/ Y
Jan 17 16:15:08 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
7 @( _3 \: \& j" L/ \0 mJan 17 16:15:08 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
0 v& x, D6 k0 w0 s7 HJan 17 16:15:11 compute02 ceph-osd[6049]: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true3 X4 p6 j% j L' H+ J! V
Jan 17 16:15:41 compute02 ceph-osd[6049]: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying& R# t7 \) u4 Q# X
Jan 17 16:16:11 compute02 ceph-osd[6049]: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying' i8 Z3 |4 Y$ K
Jan 17 16:16:41 compute02 ceph-osd[6049]: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
7 X5 Z8 ^' H L" P' TJan 17 16:17:11 compute02 ceph-osd[6049]: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
7 _# s8 K2 I8 a# VJan 17 16:17:14 compute02 ceph-osd[6049]: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>
, p+ Q" [. h9 L/ e) c" ^3 sroot@compute02:~#
; K8 H) B# G8 H+ U' }( ^root@compute02:~# systemctl restart ceph-osd@6.service 9 N8 f& c7 q' s( t
root@compute02:~# systemctl status ceph-osd@6.service
2 ^7 H3 _# E8 T● ceph-osd@6.service - Ceph object storage daemon osd.6( r) ]4 V- d. i6 Q4 E @/ _7 V
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
8 G5 p Y5 y' I! k. S Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago+ o4 g3 ? H. [& U+ P
Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)/ g0 P1 @" `( c- D, C5 W: S% G, n
Main PID: 7795 (ceph-osd)% d) \! ^1 L2 J1 I* n
Tasks: 14
& D/ o: j: N, L8 ]7 H: l Memory: 11.7M$ `, L" \5 X8 }) S
CPU: 339ms5 a4 N$ V# b# B4 R
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service) w( K4 T; T& Q2 R
└─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
! U2 x- E8 U" H" h- V% @; T, _ L8 V
Jan 17 16:25:44 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...* T) f, L) x' u4 d
Jan 17 16:25:44 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
9 d1 D' K' s# X8 C ]2 Y$ \" D' b2 H4 q& f/ d" f" J- Z' k+ P
+ `* Q" R6 D) \+ F; N: l1 ^
0 q1 k/ X! x1 ? q# _ f- F, v检查日志：
8 w4 l6 }! \% _# y tail -f /var/log/ceph/ceph-osd.6.log U% z$ p8 ~! X3 K; T7 W/ B
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [3,0] -> [3,0,6], acting [3,0] -> [3,0,6], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
H/ s1 n7 i* L5 f' b- x2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1,5] -> [1,5,6], acting [1,5] -> [1,5,6], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367, y9 A( S8 b- Z% e I) f/ t
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
; |& i, r# G! ]' D2 n- |. U2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
% g3 P5 d2 ~; @/ Z0 X4 T* I. @! p6 `2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [5,0] -> [5,0,6], acting [5,0] -> [5,0,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 45401383207592263672 U# C. \4 q; G
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray, q" V/ k& |% Y8 B
2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [2,4] -> [2,4,6], acting [2,4] -> [2,4,6], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
+ D" W/ b' h) V) J+ p2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
1 R7 U7 ~8 Z6 |2 d4 r2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [4,1] -> [6,4,1], acting [4,1] -> [6,4,1], acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 45401383207592263674 a4 A8 s7 ]0 j* d7 ]* N
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary2 u* [* ?" D% H, ]4 Q& [5 h+ f
! J' H$ Q, B3 J" J3 i3 b5 F. J0 B# v4 t; v
& i) D' p. e$ l2 a/ ]恢复正常。
* c$ u: s. z' H1 \ |
|
发表于 2024-1-17 17:00:00