Fix “ext4-fs error ext4_lookup deleted inode referenced” error in Ubuntu

admin

The other day I started my Ubuntu virtual machine and encountered with the following error.

ext4-fs error ext4_lookup deleted inode referenced

$ v( S7 {' F2 E  @; o0 H: u$ \                               
登录/注册后可看大图

ext4-fs error ext4_lookup deleted inode referenced

It didn't boot and kept showing me the above error continuously. After a couple web searches, I found the solution for the above error is so obvious. Yes, I guess the file system EXT4 for the partition /dev/sda1 is corrupted. So, I decided to log in to the single user mode and perform manual file system check.

' F, p  C3 E$ \& e

To log in to the single user mode, reboot your system and select Advanced options for Ubuntu from the GRUB boot menu.

Check the following screenshot.

: }3 V* t2 m% m4 z7 f' ^5 ~) |                               
登录/注册后可看大图

Ubuntu grub menu

Then, select Recovery mode.

                               
登录/注册后可看大图

Choose Ubuntu recovery mode

" D9 h! j8 g  [7 U

---

On latest Ubuntu versions, you can log in to the single user mode as described in the following link.

• How To Boot Into Rescue Mode Or Emergency Mode In Ubuntu 18.04
  

---

At the single user mode prompt, run:

fsck /dev/sda1

                               
登录/注册后可看大图

Run fsck command in Ubuntu single user mode

You'll be now asked to answer a couple of questions. Type "Y" until you get the problem solved.

                               
登录/注册后可看大图

Fix "ext4-fs error ext4_lookup deleted inode referenced" error in Ubuntu

Finally, you'll see the FILE SYSTEM WAS MODIFIED message. Type reboot and press ENTER.

" O0 [/ {- o  I5 Y                               
登录/注册后可看大图

Ubuntu file system is modified and fixed

( d7 Y( j" V% V; @8 x' a- G

That's it. Now, the system will boot without any problem. Be mindful that some of your files might be lost already. If you have anything important on your hard drive, connect your hard drive as secondary hdd in any system and do the backup before running the fsck command. Also note that this method will not work if your hard drive is already dead or nearly in dead condition. Keeping an external backup is always a recommended.

Thanks for stopping by!

: p; _) l( f7 E# @- y7 f

admin

解决EXT4-fs error (device dm-0): ext4_lookup 问题
2019年1月8日 作者 sharpbai
0 e0 g  x2 r% V$ w3 U- Z: g$ o问题
定期检查家用存储服务器的状态，发现dmesg输出信息中包含着和存储相关的报错信息


[2320875.914609] md: data-check of RAID array md0
[2340593.337728] EXT4-fs error (device dm-0): ext4_lookup:1575: inode #2: comm updatedb.mlocat: deleted inode referenced: 11
[2365918.220935] EXT4-fs (dm-0): error count since last fsck: 27
[2365918.220942] EXT4-fs (dm-0): initial error at time 1544633747: ext4_lookup:1575: inode 2
[2365918.220948] EXT4-fs (dm-0): last error at time 1546727139: ext4_lookup:1575: inode 2
[2403445.347062] EXT4-fs error (device dm-0): ext4_lookup:1575: inode #2: comm ls: deleted inode referenced: 11
$ g, W9 Z& {4 [) d7 _& q  S# z[2403446.810209] EXT4-fs error (device dm-0): ext4_lookup:1575: inode #2: comm ls: deleted inode referenced: 12

5 Y6 D- e* X* [$ y8 \1
: G$ n3 l& C# s# `% r( O" F& G2
3
! s% {* @. ?& `4
5
% t  ~- k: J7 Z2 \7 w1 A5 r- P+ `6
7
8
[2320875.914609] md: data-check of RAID array md0
) _  x! |: @. z[2340593.337728] EXT4-fs error (device dm-0): ext4_lookup:1575: inode #2: comm updatedb.mlocat: deleted inode referenced: 11
[2365918.220935] EXT4-fs (dm-0): error count since last fsck: 27
, z) C; c4 K7 V% k: W0 z  c[2365918.220942] EXT4-fs (dm-0): initial error at time 1544633747: ext4_lookup:1575: inode 2
[2365918.220948] EXT4-fs (dm-0): last error at time 1546727139: ext4_lookup:1575: inode 2
; W' `5 X2 f- U5 ~[2403445.347062] EXT4-fs error (device dm-0): ext4_lookup:1575: inode #2: comm ls: deleted inode referenced: 11
[2403446.810209] EXT4-fs error (device dm-0): ext4_lookup:1575: inode #2: comm ls: deleted inode referenced: 12
' u& ~8 o" M: x- a, J  K
' W. C" ~* o' f; |排查
存储问题不可小视。首先要确定下问题是什么。看起来是ext4文件系统存在文件节点错误，而且一直没有修复。接下来要确定到底是哪个设备出现了问题。通过日志看到是dm-0设备。但是dm-0设备是哪个存储设备呢?

经过查找，得知通过dmsetup ls可以得知device mapper设备与存储卷的对应关系。通过输出卷的名字，得知是搭建的软RAID1设备/dev/md0所对应的存储卷。

首先检查RAID1阵列是否存在问题，通过运行cat /proc/mdstat，发现阵列在重新检查中，并未出现不同步或降级问题。于是可以确认问题出在ext4文件系统本身。
1 \8 A: y) P8 l: b
5 O. e7 h6 m2 G( g) b解决
对于ext4分区文件系统中文件节点的错误，fsck一般可以解决。于是接下来首先umount问题卷，接下来fsck /dev/md0，一路按y确认修复操作。再通过mount命令挂载，等待了30分钟后，dmesg不再有文件系统错误出现，问题解决。

5 V8 [; M' ]( a" Y) ~- c5 ?# H复盘
好端端的ext4文件系统怎么就出现错误了呢。分析了下原因，应该有两点。一点是/dev/md0这个RAID1阵列在前段时间出现了其中一块盘跪掉然后替换的情况，期间出现了几次强制断电重启，可能因此留下了文件系统错误。

另外一点，由于这个RAID1阵列并非系统内置设备，为了防止系统不正常启动，所以挂载动作没有放在/etc/fstab中，而是从启动命令中添加了挂载命令语句。两者的差别在于，默认情况下，通过fstab挂载的设备会首先运行fsck检查，再挂载，因此每次一有小问题就会修复，问题不会累积扩大。但是手工挂载并没有检查文件系统，就更容易出问题。
3 C0 v  U: ]8 D! Y9 W; T7 o7 G5 `
预防
- x3 |+ v8 C7 C所以接下来又做了一点，在系统启动的挂载脚本执行之前，添加了一行/sbin/fsck -a UUID=xxxxxxxxxxx，这样每次就会先自动检查磁盘，再挂载，防止问题发生。