找回密码
 注册
查看: 3793|回复: 6

ceph osd down message日志

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2021-7-20 12:55:06 | 显示全部楼层 |阅读模式
Jul 20 12:47:56 compute03 ceph-osd: 2021-07-20 12:47:56.933 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167); r0 `5 R( ]5 j# M* C( ^
Jul 20 12:47:57 compute03 ceph-osd: 2021-07-20 12:47:57.950 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
; B1 _$ w- v4 s3 R! OJul 20 12:47:58 compute03 ceph-osd: 2021-07-20 12:47:58.986 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)9 k# x" K6 U$ K+ Z
Jul 20 12:47:59 compute03 ceph-osd: 2021-07-20 12:47:59.978 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)4 W6 @" U  ^+ B% e  `" \
Jul 20 12:48:00 compute03 ceph-osd: 2021-07-20 12:48:00.955 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)* M1 k/ u4 [4 `( u7 P' U# s$ p  A: a
Jul 20 12:48:01 compute03 ceph-osd: 2021-07-20 12:48:01.925 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
: Z; Q$ _7 H# o. r8 q; C2 TJul 20 12:48:02 compute03 ceph-osd: 2021-07-20 12:48:02.949 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)* _, b) Q& v; S0 y" ~
Jul 20 12:48:03 compute03 ceph-osd: 2021-07-20 12:48:03.971 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)# z2 C  C+ F9 m
7 D  z6 d) s3 F

1 V2 H7 [+ w+ _3 Q7 J通过stop NetworkMange服务,和firewalld服务:+ e/ a# ~5 ]' H1 B' w
[root@compute03 ceph]# systemctl status firewalld.service ) W0 Z$ F5 C) K: @4 q1 [
● firewalld.service - firewalld - dynamic firewall daemon
7 o4 _! C1 W! Q4 j# T   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
- ?& r. I- {' b+ c3 {   Active: inactive (dead)
' _2 x0 H  ]# r5 J5 a% x$ R" b     Docs: man:firewalld(1)
4 i# D$ k/ J4 n( T/ i' B[root@compute03 ceph]# systemctl disable firewalld.service
$ @! L5 o. u' m: y: V9 d[root@compute03 ceph]# systemctl status firewalld.service
3 s% L% @  _6 m# j● firewalld.service - firewalld - dynamic firewall daemon# J5 h. Q* j) s- K9 G/ H
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)/ M8 M1 n# H- [9 P& u- }
   Active: inactive (dead)$ |' X8 ]  t3 i& c0 x+ q6 n
     Docs: man:firewalld(1)/ n$ |7 ^( g2 ?! j
[root@compute03 ceph]# 4 v/ H/ O0 l; W; }# F8 {

$ i4 x/ I' I! e8 Z2 ~+ {  A[root@compute02 ceph]# systemctl disable firewalld.service
% r: Z1 R5 }1 F; \+ q$ u. x[root@compute02 ceph]# systemctl stop firewalld.service
$ q. v1 I2 ]& N, {[root@compute02 ceph]# ceph osd tree2 Z' Q2 r: Y" G
ID CLASS WEIGHT TYPE NAME    STATUS REWEIGHT PRI-AFF ) K/ ^2 T# a( v/ Q8 r2 i
-1            0 root default                         7 b7 a& ~7 ?' x* v% i  I
0   hdd      0 osd.0            up  1.00000 1.00000
) x8 n% B1 h0 I; w) q3 x/ q( ~4 H 1   hdd      0 osd.1            up  1.00000 1.00000
! ]1 y% F+ ^& e, c 2   hdd      0 osd.2            up  1.00000 1.00000
% m8 t4 F/ k1 {8 f( F* g6 a

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 12:56:55 | 显示全部楼层
Jul 20 12:50:11 compute03 ceph-osd: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
: |/ k. R! P& j' n" }% gJul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
- U- k+ Z: @6 b2 V5 Y4 tJul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
/ Q$ ]. J0 P# L, J% y- b/ WJul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)1 F6 X" M3 A7 H. _" ]
Jul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
3 @+ @" ~; N- {) g0 ^Jul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
( m- S' T: j7 L* P3 R' U2 d0 HJul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)$ |5 f' |6 d% W" |7 ~) h" u, U! T
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
! ^- X  g* {; O; JJul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
( C7 l1 q4 c1 x; QJul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 12:59:55 | 显示全部楼层
ceph-osd@2.service - Ceph object storage daemon osd.2! D( f0 \& K0 x0 J
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled)! ~* b4 \) p7 P* U( i$ R
   Active: inactive (dead) since Tue 2021-07-20 12:50:15 CST; 6min ago
; T( y0 {3 e; x1 T* L$ Y. J  Process: 4680 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
* @# u+ i) L* K+ L! K& H Main PID: 4680 (code=exited, status=0/SUCCESS)! v% _; N. P! x+ f4 e$ y1 W

* `6 S. \3 C* T( X9 Z2 bJul 20 12:50:11 compute03 ceph-osd[4680]: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)$ T3 Y% p$ j  T, g( O" b' C
Jul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
) y+ l, q: r$ X8 n5 q$ IJul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551): ~4 d( G* S5 H+ E
Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804), H' \& Q' K2 j
Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
5 v' g# F  r; F8 _; }. CJul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)" C4 `& H$ J: F: p" _2 R6 L
Jul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)2 c* S7 A& b3 r5 S% t) e- F( J( R
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthr...) ) UID: 0: V9 d9 ~# Z# L
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
3 V3 a3 I1 f; e7 x" \9 U5 GJul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***
4 D/ k- n3 ]( {+ VHint: Some lines were ellipsized, use -l to show in full.

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 13:09:18 | 显示全部楼层
Jul 20 13:03:22 compute03 ceph-osd: 2021-07-20 13:03:22.222 7f308763f700 -1 osd.2 79 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 13:02:43.327130 (oldest deadline 2021-07-20 13:03:03.327130)
0 x+ N* g0 f6 Q: a9 ~" sJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
$ u7 C# W6 Z4 w' l" F- J. xJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Got signal Interrupt ***8 y' F7 ?: N  v+ W0 ]( @* Q
Jul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) ***

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 13:43:56 | 显示全部楼层
back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512). l' @: y2 x+ `7 C  k, o
Jul 20 13:35:13 compute03 ceph-osd: 2021-07-20 13:35:13.237 7f3703776700 -1 osd.2 87 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
6 J' {& @5 D2 x* f0 v0 |7 v^C8 }! Y; b  T, d) X8 `
[root@compute03 ~]# service iptables stop
3 c7 @2 V; C. r% q- j% m  a' d& B! NRedirecting to /bin/systemctl stop iptables.service: N6 v8 b9 B1 F/ {- p" L
Failed to stop iptables.service: Unit iptables.service not loaded.! O  ^, _% X4 B! R
[root@compute03 ~]# getenforce
8 |$ B7 ~8 X$ ~) p2 @/ y! [; {! GPermissive

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 13:44:38 | 显示全部楼层
关闭selinux0 R: n: a! T6 v" n0 e1 Z
永久性关闭(这样需要重启服务器后生效)
4 `. p* K7 f/ O7 T( E( t$ c # sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
% L+ U& f+ Z7 |  J+ F
& h+ I9 I9 l+ r, B( ~1 ~" \+ W  A$ S* q0 M: l  f0 t, F. t: ~
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
3 x, Y* ~$ f* z' [9 Y1 g; V$ W

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-7-20 14:23:29 | 显示全部楼层
最后发现是有张网卡的ip地址配置成一模一样的,造成ip地址冲突。排查网络ip问题,也是问题原因。
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 02:19 , Processed in 0.022882 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表