ceph osd down message日志

admin

Jul 20 12:47:56 compute03 ceph-osd: 2021-07-20 12:47:56.933 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
4 a: a5 ?  t1 R  N% L8 SJul 20 12:47:57 compute03 ceph-osd: 2021-07-20 12:47:57.950 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
) S7 t' ^6 ~- T  G* F2 _" i/ m: gJul 20 12:47:58 compute03 ceph-osd: 2021-07-20 12:47:58.986 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:47:59 compute03 ceph-osd: 2021-07-20 12:47:59.978 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:48:00 compute03 ceph-osd: 2021-07-20 12:48:00.955 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:48:01 compute03 ceph-osd: 2021-07-20 12:48:01.925 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
5 A! i& O+ H5 v6 `0 zJul 20 12:48:02 compute03 ceph-osd: 2021-07-20 12:48:02.949 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
* m" ~( p0 L& sJul 20 12:48:03 compute03 ceph-osd: 2021-07-20 12:48:03.971 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)


# j$ ?) G% K% S6 E; `通过stop NetworkMange服务，和firewalld服务：
3 O. I3 L) |/ j$ W& `# @/ j1 t[root@compute03 ceph]# systemctl status firewalld.service
) k9 J+ J9 b2 Z  x) o● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
" {6 R' a0 w- T   Active: inactive (dead)
     Docs: man:firewalld(1)
4 ]. g, K. L8 q[root@compute03 ceph]# systemctl disable firewalld.service
[root@compute03 ceph]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
9 ]. d/ F  `  j   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
& ^' M: t, {) c) B; {$ M   Active: inactive (dead)
3 x& d, X0 B6 X( v4 G3 I     Docs: man:firewalld(1)
[root@compute03 ceph]#

( b) o7 D! P  l% ?! r[root@compute02 ceph]# systemctl disable firewalld.service
: I: @1 S1 q  T' Z- L[root@compute02 ceph]# systemctl stop firewalld.service
[root@compute02 ceph]# ceph osd tree
ID CLASS WEIGHT TYPE NAME    STATUS REWEIGHT PRI-AFF
! ?/ b0 p: S+ T0 t8 |-1            0 root default                        
0   hdd      0 osd.0            up  1.00000 1.00000
1   hdd      0 osd.1            up  1.00000 1.00000
2   hdd      0 osd.2            up  1.00000 1.00000

admin

Jul 20 12:50:11 compute03 ceph-osd: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
  ]) a* w1 @, v0 v' d3 k, ^- Z% uJul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
Jul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
Jul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
# x7 N! t0 \, m0 Z2 eJul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
8 X* w, ]5 e' A+ W3 k$ }Jul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
- ~% Y  S  Q! _+ mJul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
( v- {( v2 x& D* gJul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
& G1 q4 w, q% FJul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***

admin

● ceph-osd@2.service - Ceph object storage daemon osd.2
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled)
. C3 D: d/ \/ w* ~3 ~2 V1 r2 g' h   Active: inactive (dead) since Tue 2021-07-20 12:50:15 CST; 6min ago
: c! }+ C: L/ T7 h* j7 `  Process: 4680 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
9 A( [4 Q* ]9 u) d0 ? Main PID: 4680 (code=exited, status=0/SUCCESS)
" A  `1 V/ A! d& n1 w0 g8 O
( w# l" V$ `9 `- d$ v4 iJul 20 12:50:11 compute03 ceph-osd[4680]: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
4 q6 `+ }5 @  @$ v1 t* JJul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
Jul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
4 T# W0 D/ _! e0 u1 eJul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
8 i! X$ V2 T3 d( ~! E! z: _/ C* SJul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
& G3 _+ ?9 ]4 e; i5 g$ q4 _Jul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthr...) ) UID: 0
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***
Hint: Some lines were ellipsized, use -l to show in full.

admin

Jul 20 13:03:22 compute03 ceph-osd: 2021-07-20 13:03:22.222 7f308763f700 -1 osd.2 79 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 13:02:43.327130 (oldest deadline 2021-07-20 13:03:03.327130)
5 C( d7 j4 Y% Z9 n2 n1 E( P9 mJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
" l. m  _7 C) m5 d8 S% AJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Got signal Interrupt ***
/ O: |: f; Z: v0 X1 s$ v0 @9 W; YJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) ***

admin

back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
/ e+ \' T; `& V; k. \- ]Jul 20 13:35:13 compute03 ceph-osd: 2021-07-20 13:35:13.237 7f3703776700 -1 osd.2 87 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
^C
[root@compute03 ~]# service iptables stop
Redirecting to /bin/systemctl stop iptables.service
Failed to stop iptables.service: Unit iptables.service not loaded.
  V) p& r( z0 Y  q% {[root@compute03 ~]# getenforce
' J$ D3 a9 A$ p' `4 kPermissive

admin

关闭selinux
永久性关闭（这样需要重启服务器后生效）
. R2 H3 H/ Q2 N4 ]% n # sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
: S4 b6 e0 b" s7 p- D
: v+ j0 ?; ], t7 K, t% m
/ f9 u1 M3 B' |, a% Q sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
! a. O, M5 C# N

admin

最后发现是有张网卡的ip地址配置成一模一样的，造成ip地址冲突。排查网络ip问题，也是问题原因。