|
|
1 osd.6 179 unable to obtain rotating service keys; retrying- o! I" X& J, f+ S% |( C
& t1 R8 y* R( v systemctl status ceph-osd@6.service 0 N! w) x; p, |9 R2 O9 t1 M& n5 y% N
● ceph-osd@6.service - Ceph object storage daemon osd.67 R6 W9 ^% q/ F' X% v
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
3 L, e8 \ Z5 w Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago$ j* L5 p; I {+ E# [5 K; }8 C7 k
Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
: U- \. F' e" R: j& P. @( G q1 `* O Main PID: 4254 (ceph-osd)
. q2 U) M% p; u# O z Tasks: 62
2 y# I4 v' u M' P( T; ? Memory: 26.4M
/ ~. p3 r K8 i; w CPU: 1.689s
9 [% K: n0 k+ s- l2 [ CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service; Z% I$ |9 G8 ]3 a
└─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
( u1 F1 k8 u; R4 S; [
2 [7 X7 N8 M3 C# t: s& H5 mJan 17 16:09:24 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
. d: P8 F3 J: uJan 17 16:09:24 compute02 systemd[1]: Started Ceph object storage daemon osd.6.- y" p- Z/ o# r" J0 u+ F
Jan 17 16:09:28 compute02 ceph-osd[4254]: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true
4 ~6 ]7 @. U" bJan 17 16:09:58 compute02 ceph-osd[4254]: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
5 w$ Z p. X+ rJan 17 16:10:28 compute02 ceph-osd[4254]: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
5 @' R/ q4 j1 `3 r. H; H- zJan 17 16:10:58 compute02 ceph-osd[4254]: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
' ]# t* ]* G" K( y7 ^9 B& X& KJan 17 16:11:28 compute02 ceph-osd[4254]: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
5 r1 O6 g! x! [) U5 B# F4 F
- K d) ?1 i& F+ {0 Y9 r! n, T3 i1 e& r; O/ A ]
1 c$ i, s) Q3 I- x7 l4 C" ^ N( i' x4 o' X6 i6 I1 I* R# T
+ v6 j: o9 F; X) y4 X8 j- n, ~导致问题：ceph状态异常：4 z, v4 `. M- L
root@controller:~# ceph osd tree7 C/ }3 E A+ D' h6 m) y4 ^" a
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF+ U! Z5 L! B; S
-1 9.00000 root default
u: @4 i' x1 }-4 3.00000 host compute01 " J; x* T, J( f1 I, r: {4 \
3 hdd 1.00000 osd.3 up 1.00000 1.00000
' }& p' Y( A: v9 E6 |, k: r7 ` 4 hdd 1.00000 osd.4 up 1.00000 1.000004 Z% v/ M, i! z l3 o9 M
5 hdd 1.00000 osd.5 up 1.00000 1.000009 ?! d8 k8 K1 c! n( ^# Y3 B
-5 3.00000 host compute02
' q. P; M+ q3 ` 6 hdd 1.00000 osd.6 down 0 1.000003 f4 O' A( W: ~1 n! F
7 hdd 1.00000 osd.7 down 0 1.00000
( D% ? m( T7 E& ]4 \+ J8 y 8 hdd 1.00000 osd.8 up 1.00000 1.000003 l$ }! a6 r' R: h6 ?6 o0 l: d
-3 3.00000 host controller + a8 p* H0 g, { E. ~3 q
0 hdd 1.00000 osd.0 up 1.00000 1.00000* O8 O! p" k" C, f' p
1 hdd 1.00000 osd.1 up 1.00000 1.00000; o: I( x0 \: A! T$ |
2 hdd 1.00000 osd.2 up 1.00000 1.00000
( G. a w0 W, B; ]* k/ G) {0 s8 u5 ^# G S' o& o' {
1 j* Q6 W4 l0 A; d& O/ L6 O这种问题是因为时间不同步导致的：
+ }/ H- H. n* g& F2 C% K8 }
+ x6 j; o% I. n7 ~调整时间配置chrony9 A+ F z6 h6 x
一般是检查/etc/chrony/chrony.conf文件中同步的配置。/ Y; e g; E: X% I( G6 i8 H
pool controller iburst8 K9 _, U, }, g
/ i% Q% \3 v& U另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：
) g! `- Z( y6 _2 v( m- ]+ ]vim /etc/chrony/chrony.conf
2 m2 g- @5 N1 Z; n0 xallow 192.168.8.0/24% _9 v; ]/ K3 @- U
; A% m6 X# ^# h2 D9 I
[. y$ t5 T* H4 r- O% S, J& V9 Z
所有的都重启下chrony的服务。
- y* \* [' E; groot@controller:~# systemctl restart chrony.service
9 j* |; O' e# \$ s+ _& m7 _
! b* O6 Y+ A) q$ ]* w& d( }8 _+ K1 r0 F, l6 [8 w
计算节点检查：
3 i5 X, O: W# X8 ]8 h' w chronyc sources
/ w" x' N: |1 w2 kMS Name/IP address Stratum Poll Reach LastRx Last sample
3 W9 k4 Q$ ~& q: e, h===============================================================================3 R9 l. O/ t* w; R) a' e
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns i1 R2 z& K! t+ \: ^- |+ S
root@compute02:~# chronyc sources
) \7 } B; x9 NMS Name/IP address Stratum Poll Reach LastRx Last sample + ]. {1 o6 t2 U6 q$ k$ Z
===============================================================================
- H5 N$ ^- f5 g^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns
8 H% `1 Q }0 v _% l: {# m1 H, Mroot@compute02:~# systemctl enable chrony.service) S% e! P4 r5 W, V8 k
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.* `) \. a: {0 V$ d
Executing: /lib/systemd/systemd-sysv-install enable chrony, H3 T3 M, e G8 D I, X
2 }4 ^: L8 E9 X
计算节点重启chrony的服务：6 s0 J: t4 h& |7 c. S; R9 d
% x/ {, E6 i# Y/ b5 t0 w9 ]1 q8 |root@compute02:~# systemctl restart chrony.service
3 J, H' Z4 D- X& a& k8 i检查状态：' r" I6 u4 w3 R
/ N: A6 P6 Z& s |* `
root@compute02:~# chronyc sources/ s2 L m$ P3 n$ ^: n3 {4 U
MS Name/IP address Stratum Poll Reach LastRx Last sample 7 x+ M7 Q4 O: U8 K& } i8 ], ]3 ^
===============================================================================# }/ V% j6 c1 T( G& F. w: k) N/ y
^? 192.168.8.65 2 6 3 1 +8087ms[+8087ms] +/- 13ms
9 q4 Y/ N# r+ ^6 y+ }) c3 [& f% a
4 }& {" \- h- J/ p0 `6 W
. G& W' Y! ]" r% X% I+ K0 e, Oroot@compute02:~# systemctl status ceph-osd@6.service 6 [+ B [, @) H+ L1 z
● ceph-osd@6.service - Ceph object storage daemon osd.6# i( |- F X5 U& x% p
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled). ^/ w* a' J# Z6 m7 T8 S1 }
Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago
7 E7 J# {# _7 y7 |0 l Main PID: 6049 (ceph-osd)
4 D7 ]' F3 ]) L2 ~+ n Tasks: 622 s9 O- m0 ~# B$ X
Memory: 84.6M
1 J6 t9 e- c" { CPU: 15.598s
' q# I* F- X: |$ W+ F CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
' T# T1 t6 ^, S$ _ M* Z$ R- F └─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph9 ]5 z# c9 s5 h
0 C' G5 w# {* |5 V
Jan 17 16:15:08 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...- Z) b( e1 Y8 ?: x
Jan 17 16:15:08 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
& x- r" W( d5 q; kJan 17 16:15:11 compute02 ceph-osd[6049]: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true
4 g: m5 @; I- rJan 17 16:15:41 compute02 ceph-osd[6049]: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying* Y6 C) d6 g! \5 o& I
Jan 17 16:16:11 compute02 ceph-osd[6049]: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying1 x* S$ @; x* I& W
Jan 17 16:16:41 compute02 ceph-osd[6049]: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying* P- |/ M' N1 a# l7 P
Jan 17 16:17:11 compute02 ceph-osd[6049]: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
9 [6 A( |7 V4 M6 E( E' R }; g' j* [Jan 17 16:17:14 compute02 ceph-osd[6049]: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>: B+ a$ o* e4 g0 W$ P D8 A
root@compute02:~#
- L) b" c5 A1 Z- }$ F5 lroot@compute02:~# systemctl restart ceph-osd@6.service
H d4 x; u3 Q$ a1 x# h$ [( w9 @ Sroot@compute02:~# systemctl status ceph-osd@6.service ! v3 w1 ?, p. y8 S
● ceph-osd@6.service - Ceph object storage daemon osd.6; K( C6 C8 N- |4 H2 M1 i/ ]; K+ s) c# B
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
* a* R \( }- f% F) Q Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago
& p a% V3 C9 Y6 x3 M+ ? Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
, J0 m3 i; {: @" Q8 A: f Main PID: 7795 (ceph-osd)3 M) \, \# p1 f7 h1 [2 i
Tasks: 142 L* b2 y A0 F" p3 g) p5 Q) A
Memory: 11.7M7 m' x5 D- t5 P% H9 L
CPU: 339ms# l, l. {7 g9 |; v9 F1 ?7 p
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service, P% S4 V. I+ Z; h4 B
└─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph9 s, {7 S1 e2 l1 E8 P; `
" }: q$ Y ]& k; n5 m4 m( S0 r5 eJan 17 16:25:44 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...
9 ]8 `: h& x( k$ S+ KJan 17 16:25:44 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
! D7 s" {+ g2 z Y4 J: Y
7 j* h+ {: G$ j6 M" B8 E- A- `& J- o
. T* W; g4 d$ N( D* e/ p检查日志：! s1 e9 z$ P, j/ k, | z# T
tail -f /var/log/ceph/ceph-osd.6.log
0 z5 }; a) c) e @) r2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [3,0] -> [3,0,6], acting [3,0] -> [3,0,6], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
4 v8 c1 N; B! o( V2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1,5] -> [1,5,6], acting [1,5] -> [1,5,6], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367# d1 I8 L2 G# X
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
& }3 B1 J+ s. \2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
! I4 o' Y; z. l: W$ {2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [5,0] -> [5,0,6], acting [5,0] -> [5,0,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367" B& t8 U! U' V( w
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
' p1 t1 r9 U6 c2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [2,4] -> [2,4,6], acting [2,4] -> [2,4,6], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367/ ]3 s$ C T e! d
2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
( }( [( j: @% ]- [: I2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [4,1] -> [6,4,1], acting [4,1] -> [6,4,1], acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 4540138320759226367
( g: j* {8 _" {; b/ r2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary
6 D, J; C+ T5 X: y9 A' m2 m' `& P: E' Y! w' ?
% v/ T$ n) u0 ^: Q: U2 U8 N; {: q+ ~" W: O) j% U5 V$ U
恢复正常。' L+ `7 b8 |& H( @( @
|
|
发表于 2024-1-17 17:00:00