log_channel(cluster) log [ERR].. soid 1:0a2506d3:::rbd_data....head candidate ha

admin

ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
1 y9 G# x; v. ^9 ~" N* e, r. t2 A
blk_update_request: I/O error, dev sdi, sector 72908960


Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Sense Key : Medium Error [current]
& E: X' h; x0 p7 ?4 ?; Y/ eJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Add. Sense: No additional sense information
( c# B% ]( x* }( @! U2 JJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
0 C. i( y7 R# I- n: W% oJul 16 10:32:18 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
0 \' Z7 Q- E" j. EJul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.121742 7f0b5a5ef700 -1 osd.67 790 shutdown
+ G' ?5 f- S4 _- ^$ q& m2 V- [Jul 16 10:32:19 cn01 systemd-logind: New session 103926 of user root.
" \, h" C! [& eJul 16 10:32:19 cn01 systemd: Started Session 103926 of user root.
6 P- C7 a  U0 `  j' SJul 16 10:32:19 cn01 systemd: Starting Session 103926 of user root.
. C; ~& I+ ^2 h" v- JJul 16 10:32:22 cn01 systemd: Stopped Ceph object storage daemon.
, B( x( ~1 C6 K# F9 @+ u5 X9 m

* u7 g8 N# A0 Y" [; g  e- q8 j
( @: T! I* d9 r8 A) E通过日志分析，应该是磁盘损坏了。
, D7 t  c. X& D( L( g更换磁盘。
% Y: [  b6 X$ _* V) ?8 v$ s; F" k

admin

Jul 16 10:35:41 cn01 systemd: Starting Ceph object storage daemon...
Jul 16 10:35:41 cn01 systemd: Started Ceph object storage daemon.
9 I2 o* h: l/ T2 d+ |) q4 F2 s1 ~0 j" sJul 16 10:35:42 cn01 ceph-osd: starting osd.67 at :/0 osd_data /var/lib/ceph/osd/ceph-67 /var/lib/ceph/osd/ceph-67/journal
Jul 16 10:35:42 cn01 ceph-osd: 2023-07-16 10:35:42.085939 7fe6fa918f80 -1 leveldb: Compacting leveldb store...
Jul 16 10:35:43 cn01 ceph-osd: 2023-07-16 10:35:43.212904 7fe6fa918f80 -1 leveldb: Finished compacting leveldb store
Jul 16 10:35:47 cn01 ceph-osd: 2023-07-16 10:35:47.177395 7fe6fa918f80 -1 osd.67 790 log_to_monitors {default=true}

admin

Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2 }! J  f6 y& [$ |Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
0 p1 w, ~1 G7 F! CJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
  s: D  O1 `7 e! m( K) C" R5 IJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
+ I) W- j3 |7 N1 e& hJul 16 11:16:55 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:16:56 cn01 ceph-osd: 2023-07-16 11:16:56.623469 7fe6a75bc700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
; E2 ?$ O6 S  zJul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12266 (742792048s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
5 G* p4 \' P. c$ ~) p- G  dJul 16 11:17:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12267 (742792048s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
: p9 `' W8 X- \$ KJul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12270 (742792055s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12271 (742792055s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:02 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
7 c9 j" a, _. X! Q4 ]* bJul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12274 (742792062s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
- y* b7 X' b* n4 GJul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12275 (742792062s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:09 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12278 (742792068s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12279 (742792069s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
* C0 H" W& e* D! ZJul 16 11:18:16 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
% q( ]3 A0 v- t7 {- c2 M6 i6 \9 MJul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12282 (742792075s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
' H- ?* ^, B( cJul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12283 (742792075s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:22 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12286 (742792082s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
* t- C  Y$ @8 z# ~$ H4 E6 C- ?Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
% a, E' x7 n- y: H, O# vJul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
* j9 [5 q9 b) F# `) l0 ^Jul 16 11:18:29 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12287 (742792082s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
2 Y; s8 o/ O2 F& [' {  H. JJul 16 11:18:36 cn01 kernel: megaraid_sas 0000:18:00.0: 12290 (742792089s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
2 b2 Y8 H! {, V8 R; |Jul 16 11:18:36 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0