- 积分
- 16843
在线时间 小时
最后登录1970-1-1
|
马上注册，结交更多好友，享用更多功能，让你轻松玩转社区。
您需要 登录 才可以下载或查看，没有账号？开始注册
x
1 osd.6 179 unable to obtain rotating service keys; retrying0 ]! E1 ?4 j$ D* h$ r2 G
" g5 k/ ~4 d3 i4 b/ e systemctl status ceph-osd@6.service & m: E. \ E4 h% E6 G+ e% i
● ceph-osd@6.service - Ceph object storage daemon osd.6/ p) `4 C7 K) M0 T7 X
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)8 P* z/ f5 i: o2 U. r1 t4 Y9 y8 W
Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago; `6 H& D/ ^$ f, g9 n
Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS) |$ c! ~) e" L: }" Z5 d
Main PID: 4254 (ceph-osd)4 g4 y' z6 |6 Q
Tasks: 62
6 ?, z S9 P7 D Memory: 26.4M
, I1 t& U G0 z: T8 g. y CPU: 1.689s* J3 R! D) I3 a
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
4 c/ X3 e' g2 s( u; D4 K) J; f └─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph# n) e! l0 Q4 \% T: T8 e
: x$ m2 _1 ]- R; hJan 17 16:09:24 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...7 ~$ a) }" n# `2 |* A# E
Jan 17 16:09:24 compute02 systemd[1]: Started Ceph object storage daemon osd.6.% i8 q4 j/ S2 Q% x4 t W
Jan 17 16:09:28 compute02 ceph-osd[4254]: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true
2 N% i4 D3 T$ a2 E6 M8 m! \' }Jan 17 16:09:58 compute02 ceph-osd[4254]: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
+ L/ Q; a' s) e1 UJan 17 16:10:28 compute02 ceph-osd[4254]: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
2 }( G" X G# |6 J) s2 sJan 17 16:10:58 compute02 ceph-osd[4254]: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
3 K/ }/ A% Q$ q! bJan 17 16:11:28 compute02 ceph-osd[4254]: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying: N5 J6 b* a' y) e" H K7 x
9 r) J4 Y# B% M( v) c9 @& y, @; b: O5 I% _
# E0 F# c ?- |7 M0 `3 ~7 r- c1 m Y+ \3 g6 O
8 w* Q6 n2 ?. |3 }0 g6 h! Q \导致问题：ceph状态异常：
2 N; n1 W6 ]. z% wroot@controller:~# ceph osd tree
! s/ S3 r# ^6 d% j9 D5 cID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
* y- Z9 J6 g% c4 i: s8 P-1 9.00000 root default
8 J" ~& C: ]; A0 ~+ Z-4 3.00000 host compute01 : C4 E( M# _, t9 g4 p |, z- c
3 hdd 1.00000 osd.3 up 1.00000 1.00000
* I, q- b8 E/ z3 @ 4 hdd 1.00000 osd.4 up 1.00000 1.00000
( P; }/ ~; Y. c/ X 5 hdd 1.00000 osd.5 up 1.00000 1.000005 K) [$ Y8 t+ o, v/ y/ o1 }
-5 3.00000 host compute02
) m; S1 x$ R, ~$ t$ @- {! y 6 hdd 1.00000 osd.6 down 0 1.00000
3 _3 y9 f1 d3 i' u! t 7 hdd 1.00000 osd.7 down 0 1.00000* @8 h, `1 Z1 b" _( l- `' o
8 hdd 1.00000 osd.8 up 1.00000 1.00000
/ ]4 }/ K B) p6 f# G( J-3 3.00000 host controller
$ R5 [: C6 \9 Z7 Y8 a/ f7 Z8 @! \1 f 0 hdd 1.00000 osd.0 up 1.00000 1.00000+ m6 L% `! f) v6 Q
1 hdd 1.00000 osd.1 up 1.00000 1.00000
8 f5 D I4 j7 S7 U% S2 g, n& t 2 hdd 1.00000 osd.2 up 1.00000 1.00000( F; h, g) m, e: U
- p- J7 h6 t L- K% y$ d5 d' `: G' i4 b @2 G; m
这种问题是因为时间不同步导致的：
& I) f/ ~2 i' j0 A# A
9 D$ L; Z! o$ m& Y调整时间配置chrony/ C5 r& Q0 h7 z: ?
一般是检查/etc/chrony/chrony.conf文件中同步的配置。
- k0 I9 H' C; x; j" G/ F' e$ `pool controller iburst
4 w/ [4 w' G. ~/ O! b* k
# b7 Y; j% u3 E( r% f% k! ?8 O3 a另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：
' J; G G% R' c6 t, }" W) J. S$ qvim /etc/chrony/chrony.conf 8 E j5 U) E, x9 o3 y/ P
allow 192.168.8.0/24; p0 C+ F7 i0 f7 d3 M+ ^( X
* B1 c. J! H- A3 F8 i
0 u# B5 ~ `. T2 j5 t* O S所有的都重启下chrony的服务。( t) }) n& A: ~; J( ^: p% A. b
root@controller:~# systemctl restart chrony.service 9 u8 b. u3 V+ [7 L1 a* c
+ j8 A' B% t: k9 [/ F3 |3 g" E+ ^
) u6 r( r" J: `# G计算节点检查：
8 Y# s4 S" m7 S0 ]2 l" ]9 U# ^8 h0 F chronyc sources4 [2 V- ?. Z1 y
MS Name/IP address Stratum Poll Reach LastRx Last sample ( e k* Y5 `; s; k9 n
===============================================================================3 @! z, i4 l8 ]! O% o( x5 Z0 N
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns
9 f9 z) b7 o& q$ r9 `) }root@compute02:~# chronyc sources
0 z2 o: k" N# GMS Name/IP address Stratum Poll Reach LastRx Last sample
' I8 D5 I. r4 H& @===============================================================================) v' W0 m. C' X! N2 v7 O
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns9 ^2 K5 o% T; I) S9 X
root@compute02:~# systemctl enable chrony.service
9 b, `7 S* k# @Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
, P# ?( d9 ~. s7 R) M0 ^! N" Q5 aExecuting: /lib/systemd/systemd-sysv-install enable chrony6 T" V, c/ u! r. d9 {8 T
+ T$ n8 ^0 ^! x& o6 f计算节点重启chrony的服务：
/ ~" X4 C7 @* {6 a! k% J
% g/ V+ [7 c( z! Croot@compute02:~# systemctl restart chrony.service9 p5 P& p3 z& d$ L, [! B
检查状态：) U, t- f6 M/ D7 o- F" L
7 }1 ^9 X1 ]5 y: k8 T
root@compute02:~# chronyc sources/ P7 B4 S; o" E6 v2 @8 [# d
MS Name/IP address Stratum Poll Reach LastRx Last sample $ S! e C8 b: z( W1 H& z& T
===============================================================================" I7 a G: V; G: M' w
^? 192.168.8.65 2 6 3 1 +8087ms[+8087ms] +/- 13ms
7 W7 M; |% \/ A
6 J- x4 f9 r5 Z2 c5 S
- Z) c2 m8 n. h- p6 }, @6 V- o' m F9 x% D l6 o- K/ c* f: R
root@compute02:~# systemctl status ceph-osd@6.service
$ Y1 v& ^3 l! G- i4 y1 v9 n3 ]● ceph-osd@6.service - Ceph object storage daemon osd.6
2 y' T4 z( L- w! z, t: \ Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)! ~: y) `) L" h( ~8 u
Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago0 A9 U9 z: `( M' r* T: ?
Main PID: 6049 (ceph-osd)
5 ^# R3 I+ `5 O6 F: l" n Tasks: 62
, i, S/ W% p6 c! g4 n, L5 W" o( C Memory: 84.6M
9 T& a: X7 Q6 a& F8 d6 ^% r8 c CPU: 15.598s8 s% s9 ~: s! j4 S, ]1 N
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service/ H; A& S! R) k; y# ~
└─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph7 }/ c3 T: \7 @# k/ Q" M
* T% c* S, h! ]* F( R9 ]: i4 p G* t9 dJan 17 16:15:08 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...6 z* m- D- @- z, [: v
Jan 17 16:15:08 compute02 systemd[1]: Started Ceph object storage daemon osd.6.
( \* X4 U' e, e. LJan 17 16:15:11 compute02 ceph-osd[6049]: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true
* x$ i+ a: [. N$ K7 QJan 17 16:15:41 compute02 ceph-osd[6049]: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
: ^" p" B- S( a+ ]Jan 17 16:16:11 compute02 ceph-osd[6049]: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
+ q+ x! k6 q5 h; v0 oJan 17 16:16:41 compute02 ceph-osd[6049]: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
9 X% g9 A3 ?3 D- }9 H! dJan 17 16:17:11 compute02 ceph-osd[6049]: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
7 W2 I! ^4 W/ BJan 17 16:17:14 compute02 ceph-osd[6049]: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>& m) J. s6 Z& H, N
root@compute02:~#
) l- W. B" r& o% a ~1 Sroot@compute02:~# systemctl restart ceph-osd@6.service
- R0 O7 L* M. Oroot@compute02:~# systemctl status ceph-osd@6.service
" J+ i/ w& j+ Y z& a& {: K* Y● ceph-osd@6.service - Ceph object storage daemon osd.6
6 r1 f* w* X$ ^ ?: O Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)3 l* `$ v+ {5 G, N
Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago
" r* v2 c+ z! V8 m N Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
5 O+ ^, }: |$ d$ }( `( ^ Main PID: 7795 (ceph-osd)
# g7 s. E# l9 d; _ A( x Tasks: 14
: A& N5 c H+ b* w. J" W Memory: 11.7M8 s7 g/ G! `8 M' @- V( O% l
CPU: 339ms
# G7 t1 W& B8 U$ ?. z: g7 m CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
9 E% [5 h- l' L. f9 m, K └─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
: D; X$ z3 h9 K$ ?3 z2 h2 G$ d0 m
6 y% {$ V/ ^' N3 q5 IJan 17 16:25:44 compute02 systemd[1]: Starting Ceph object storage daemon osd.6...7 Q& @" a4 M' e4 t, k
Jan 17 16:25:44 compute02 systemd[1]: Started Ceph object storage daemon osd.6.; G' B" {( @0 @" _9 i6 _: Q$ b+ g& M
/ n& ]- W# v N2 {8 j
4 f1 Q5 Q; T& d3 \8 i/ |
; E, H h: w' ^" L) @" i
检查日志：
" o4 }) y7 ?5 z6 U% u tail -f /var/log/ceph/ceph-osd.6.log # M& [/ ^" E m$ }$ y
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [3,0] -> [3,0,6], acting [3,0] -> [3,0,6], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367* F* y( d- L2 |
2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1,5] -> [1,5,6], acting [1,5] -> [1,5,6], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
" U* D) X. \* t$ f3 G& f. J2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.42( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [3,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
! b+ Q" R# b/ Z4 `: }2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[5.1d( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [1,5,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
$ q7 ]1 W& m5 f& N) ^' v2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [5,0] -> [5,0,6], acting [5,0] -> [5,0,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367, N0 |7 X ]/ v: y2 p0 w* R
2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[2.56( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [5,0,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
9 Q! ?0 H }% H/ {% [% `( ?4 m; J2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [2,4] -> [2,4,6], acting [2,4] -> [2,4,6], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 45401383207592263673 a) A; Y. I5 m' Z* Q1 N' }
2024-01-17T16:25:50.363+0800 7f59153fb640 1 osd.6 pg_epoch: 224 pg[2.6d( empty local-lis/les=216/217 n=0 ec=68/68 lis/c=216/216 les/c/f=217/217/0 sis=224) [2,4,6] r=2 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
( A, u& m0 C. G) ^2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [4,1] -> [6,4,1], acting [4,1] -> [6,4,1], acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 4540138320759226367
. k( ^$ b/ j1 H3 B. t# S; B1 H6 ~! i2024-01-17T16:25:50.363+0800 7f5916bfe640 1 osd.6 pg_epoch: 224 pg[5.1f( empty local-lis/les=216/217 n=0 ec=83/83 lis/c=216/216 les/c/f=217/217/0 sis=224) [6,4,1] r=0 lpr=224 pi=[216,224)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary* S, T& D) P9 H7 @) d6 @; z
- T& r5 C; e) n9 c' N* s8 w: X; y5 W4 I0 ~
2 g8 p$ v) _% b. W- g9 o: d7 Y& n \
恢复正常。
" ?* E1 @! g4 q' C% m |
|
/4 
|返回首页|Archiver|小黑屋|易陆发现技术论坛
( 蜀ICP备2026014127号-1 )
窥视卡
雷达卡
发表于 2024-1-17 17:00:00
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
照妖镜