log_channel(cluster) log [ERR].. soid 1:0a2506d3:::rbd_data....head candidate ha

admin

ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
3 D3 U" N) _3 v+ D7 S7 e6 {
& c1 Q: K3 }7 dblk_update_request: I/O error, dev sdi, sector 72908960

5 K( e7 P* Y3 K4 H" I* C0 l5 p
Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Sense Key : Medium Error [current]
% ]# E0 @! L- T! UJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Add. Sense: No additional sense information
+ g  {1 r; S6 FJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 10:32:18 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
! o* q) Q  r4 SJul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.121742 7f0b5a5ef700 -1 osd.67 790 shutdown
8 _" }7 |( M+ n3 Q7 IJul 16 10:32:19 cn01 systemd-logind: New session 103926 of user root.
2 w" Z* P* h# D6 ~+ dJul 16 10:32:19 cn01 systemd: Started Session 103926 of user root.
Jul 16 10:32:19 cn01 systemd: Starting Session 103926 of user root.
Jul 16 10:32:22 cn01 systemd: Stopped Ceph object storage daemon.
" B% d( F8 o' v3 H; v


3 N5 o9 U, g' u$ E: ^0 d通过日志分析，应该是磁盘损坏了。
更换磁盘。

admin

Jul 16 10:35:41 cn01 systemd: Starting Ceph object storage daemon...
% E, \5 H5 V. Z+ DJul 16 10:35:41 cn01 systemd: Started Ceph object storage daemon.
4 Q/ b. U1 b. _: [8 [. }9 YJul 16 10:35:42 cn01 ceph-osd: starting osd.67 at :/0 osd_data /var/lib/ceph/osd/ceph-67 /var/lib/ceph/osd/ceph-67/journal
Jul 16 10:35:42 cn01 ceph-osd: 2023-07-16 10:35:42.085939 7fe6fa918f80 -1 leveldb: Compacting leveldb store...
5 T/ B0 |' f; S3 f6 a  JJul 16 10:35:43 cn01 ceph-osd: 2023-07-16 10:35:43.212904 7fe6fa918f80 -1 leveldb: Finished compacting leveldb store
" [8 k4 A- I6 `7 b5 \Jul 16 10:35:47 cn01 ceph-osd: 2023-07-16 10:35:47.177395 7fe6fa918f80 -1 osd.67 790 log_to_monitors {default=true}

admin

Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
/ {8 R& L7 E+ M0 d6 ~* Z2 MJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
7 C. w' L, s8 W- T: AJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
& D9 d/ {6 p- T  ^" C. [Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 11:16:55 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
# k0 @" w( i% F5 A3 r' _3 j. w5 dJul 16 11:16:56 cn01 ceph-osd: 2023-07-16 11:16:56.623469 7fe6a75bc700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
Jul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12266 (742792048s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
5 t. |$ h# q0 h  t6 \! q# |2 UJul 16 11:17:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
- O7 I' x) X1 j" \* SJul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12267 (742792048s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12270 (742792055s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12271 (742792055s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
/ F4 \  S" P; o! A% ^; PJul 16 11:18:02 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
- s! ^! Z% p7 u# H3 W% {Jul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12274 (742792062s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
) b" ]" o0 E9 w! HJul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12275 (742792062s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:09 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
" S2 N$ t% ]: |: TJul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12278 (742792068s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12279 (742792069s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
$ V1 v+ V2 B' k. l8 UJul 16 11:18:16 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
! t( z; R3 d& d6 [Jul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12282 (742792075s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12283 (742792075s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:22 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
9 O- w' d% o$ w" b+ N2 gJul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12286 (742792082s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
1 C) w. W$ B/ U3 dJul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
7 d" y9 E# M$ u7 u  I" EJul 16 11:18:29 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12287 (742792082s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:36 cn01 kernel: megaraid_sas 0000:18:00.0: 12290 (742792089s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:36 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0