找回密码
 注册
查看: 3792|回复: 6

ceph osd down message日志

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2021-7-20 12:55:06 | 显示全部楼层 |阅读模式
Jul 20 12:47:56 compute03 ceph-osd: 2021-07-20 12:47:56.933 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
5 v. ^5 ~( ], |2 h3 Z; MJul 20 12:47:57 compute03 ceph-osd: 2021-07-20 12:47:57.950 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)" B. S' I/ y% R( K
Jul 20 12:47:58 compute03 ceph-osd: 2021-07-20 12:47:58.986 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
$ l9 W  \+ o2 E2 O. F0 WJul 20 12:47:59 compute03 ceph-osd: 2021-07-20 12:47:59.978 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)5 p( h9 X6 k5 \8 ]: r
Jul 20 12:48:00 compute03 ceph-osd: 2021-07-20 12:48:00.955 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
- C7 B% M  a/ m1 XJul 20 12:48:01 compute03 ceph-osd: 2021-07-20 12:48:01.925 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)  q6 ?2 r' ?, j' \1 n( _
Jul 20 12:48:02 compute03 ceph-osd: 2021-07-20 12:48:02.949 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
. V- v% Q" @5 J' ]Jul 20 12:48:03 compute03 ceph-osd: 2021-07-20 12:48:03.971 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)+ ]0 j& W% L1 M1 I8 Y0 K  l6 {

1 V% r9 c& K6 V, c
; v( I4 g  c. \通过stop NetworkMange服务,和firewalld服务:5 X, q3 H( s  D3 o
[root@compute03 ceph]# systemctl status firewalld.service
1 k' Y* u+ d9 O0 w: f5 X● firewalld.service - firewalld - dynamic firewall daemon
" Z8 {2 o. P) w% c' y4 i9 h   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)5 f4 z( J& x/ P6 E9 O
   Active: inactive (dead)
! Z, m$ {& n. _     Docs: man:firewalld(1): {0 g$ C* O% G% |. O+ }' P
[root@compute03 ceph]# systemctl disable firewalld.service
+ g# W* m) [* m" b; [: f& f[root@compute03 ceph]# systemctl status firewalld.service 0 d+ c% ]7 c2 z) l- i, B9 b; D
● firewalld.service - firewalld - dynamic firewall daemon
5 @/ T; I; c6 ~" w4 C/ E! H   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)$ U' t( Q0 w+ \5 W) g2 k5 |8 ~" e
   Active: inactive (dead)
, r' u/ s; R9 P/ _. z% Y/ D$ G( c     Docs: man:firewalld(1)
, `, D4 Y) i% {3 a; H  K8 m[root@compute03 ceph]#
1 i  o, U4 f/ f. c4 _
; z# A+ ?7 R9 K- y- ?% ]) ^: a[root@compute02 ceph]# systemctl disable firewalld.service
7 c' [9 _+ ]5 b3 l# S[root@compute02 ceph]# systemctl stop firewalld.service
9 @+ ^0 v& H; g- x  R* K[root@compute02 ceph]# ceph osd tree
  K6 B* @  `, W, \6 _! y. K5 V- NID CLASS WEIGHT TYPE NAME    STATUS REWEIGHT PRI-AFF
% U! D  E& l4 a9 S, s-1            0 root default                        
8 \" w+ B& O# L! b$ _ 0   hdd      0 osd.0            up  1.00000 1.00000 ; n7 N, c& p+ u& Q
1   hdd      0 osd.1            up  1.00000 1.00000
( V2 B( @8 \& f1 t 2   hdd      0 osd.2            up  1.00000 1.00000 : O1 x; Y7 o0 q3 Z$ _! k

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 12:56:55 | 显示全部楼层
Jul 20 12:50:11 compute03 ceph-osd: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
+ W7 [6 d0 h5 ~7 ^Jul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
0 j, ?$ ^, y( R$ K2 ]Jul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)2 Z2 |8 s% b" u8 ^9 l4 _
Jul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)2 D3 I; q$ r5 M/ Z9 ]& u1 t1 |. Z
Jul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
( o( a# S# J! s" \6 b( G/ O4 AJul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)7 ]0 Q/ P4 M. T0 n- d& d7 `
Jul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)6 W, w( Y5 e/ t" }$ [
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0' D9 w; k- s! H! L& m
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
# ?/ ^3 P4 ?! C# o( l% HJul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 12:59:55 | 显示全部楼层
ceph-osd@2.service - Ceph object storage daemon osd.23 @$ c9 a6 _: t% b- t
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled)
) J! J/ K0 s  H, b; S( K7 w   Active: inactive (dead) since Tue 2021-07-20 12:50:15 CST; 6min ago
9 i% D6 K, \! ]! B  Process: 4680 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
" H6 {: j; M6 `! z/ ` Main PID: 4680 (code=exited, status=0/SUCCESS)# Q2 [4 Y0 |# Q1 d1 w+ t9 p' ^' U
8 p, h$ j" c( ?; a% T4 H
Jul 20 12:50:11 compute03 ceph-osd[4680]: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
' m% }! I9 g* I8 F, f+ ~2 N! GJul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)! t( ~: P2 h# D& F
Jul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
+ X" n7 t) i$ M; C0 _: |Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
8 w( x& P* ?. ~, j4 C( oJul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)1 T. u$ B/ S: c/ D; W# e
Jul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
/ C0 L9 z' l8 s" E7 gJul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
+ \- ]2 C3 q- J5 qJul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthr...) ) UID: 0
& |, ]1 d. h% h( D; e2 FJul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
$ m4 }6 u$ `9 o" MJul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***/ s+ d" d4 h1 N2 d( K" X& T) `
Hint: Some lines were ellipsized, use -l to show in full.

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 13:09:18 | 显示全部楼层
Jul 20 13:03:22 compute03 ceph-osd: 2021-07-20 13:03:22.222 7f308763f700 -1 osd.2 79 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 13:02:43.327130 (oldest deadline 2021-07-20 13:03:03.327130)
3 R6 z* u1 O$ Z# Y3 G; P! dJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
) o1 {; S) [* K& T2 w$ f. bJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Got signal Interrupt ***6 f; p, c% V% R, @. _9 Z+ x$ z8 f( Y
Jul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) ***

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 13:43:56 | 显示全部楼层
back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
9 s4 U1 z  o6 D( W8 SJul 20 13:35:13 compute03 ceph-osd: 2021-07-20 13:35:13.237 7f3703776700 -1 osd.2 87 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
4 N4 B& B$ I( h# W/ A  b' C^C
7 D8 E, m9 v3 o+ B( |% u6 l[root@compute03 ~]# service iptables stop3 t& T  ~% N6 l) F3 e7 s- y
Redirecting to /bin/systemctl stop iptables.service
+ U' K. {7 F5 p. G4 Q5 y% GFailed to stop iptables.service: Unit iptables.service not loaded.2 [; P) }! I2 v' }# I1 Y, Y
[root@compute03 ~]# getenforce " \+ m  x7 [* R1 Y! e8 {; i
Permissive

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 13:44:38 | 显示全部楼层
关闭selinux8 N6 T# l; |% h/ K3 r
永久性关闭(这样需要重启服务器后生效)
  p4 B2 N$ \" ]5 Z% r # sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
% J+ J' E( F" v: `2 t
* c3 R" _  t: w0 C+ B: d3 i( j! K8 ^, T+ M1 x* K% w! m% t
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
' c  W+ s  {7 c

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 14:23:29 | 显示全部楼层
最后发现是有张网卡的ip地址配置成一模一样的,造成ip地址冲突。排查网络ip问题,也是问题原因。
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 01:05 , Processed in 0.019793 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表