|
|
[root@compute1 ~]# , Z" Z& _ K( P& E
Message from syslogd@compute1 at Dec 13 17:56:10 ...
: T7 K2 r, s' B1 e7 ?0 J kernel:NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ksoftirqd/16:89]; C! L$ @* E1 j8 r1 x
# n0 Q# T3 P4 P! @2 u! Y
Message from syslogd@compute1 at Dec 13 17:56:22 ...
$ [+ T. f+ M# {# _ kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:2:918]
2 ~2 [# i7 q# L" T8 X4 \/ B1 m
3 ]0 \* z; S3 H8 v9 D: i" p0 t/ hMessage from syslogd@compute1 at Dec 13 17:57:05 ...
- F3 t- g5 E' a: m kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:11804]6 n, z- ]; z6 y: X7 ^
. }7 w7 x% Q4 N4 u! Y8 x
Message from syslogd@compute1 at Dec 13 17:57:17 ...+ U$ }1 M2 L' Z) ~0 N
kernel:NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ksoftirqd/16:89], E1 E2 j5 n; `) Z% b8 F8 R) r6 Y
# i j. S% T) L: q6 [. n) P7 B% @$ n% F: U; n
kernel:NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s!7 R5 p! f8 E& _$ i/ J3 X, C9 [+ o9 y
解决：* W* s4 w+ G1 S
/ _$ @, W% g8 B8 c9 S
echo 30 > /proc/sys/kernel/watchdog_thresh( E: D8 N. }$ t/ ^! Q
#追加到配置文件去4 ]0 w9 @5 s X Z0 [/ f
tail -l /proc/sys/kernel/watchdog_thresh
3 p p6 J0 i( H$ A6 z$ t! e) G; \#查看确认
^& H7 v+ P/ l, Asysctl -w kernel.watchdog_thresh=30
$ l6 ~! x/ i' h/ O#临时生效6 Q: {0 |/ u3 b; \9 U2 r
6 [% d" N* Q& E1 m: a# U* h, p. Q0 n0 v) N8 g. c, f2 M- H7 F# l, R( X
解决办法:6 N) h b( J: ~) K1 t# [! s$ |
( g7 [: s" v; l
#追加到配置文件中
3 g3 S. r% p3 e, g3 B
* `9 f7 f, R, k& r! recho 30 > /proc/sys/kernel/watchdog_thresh
0 f% x. |5 H* ^" }! i" p4 @
g2 R. X. g' J" p$ |; {3 o#查看
. H$ q8 l8 y: `9 D. [
3 z* M# N8 d& y& q[root@git-node1 data]# tail -1 /proc/sys/kernel/watchdog_thresh
8 _0 X" j. S7 a5 v3 E0 [30
2 n* m# Y+ U; D4 P9 @8 Y8 O1 X# Y
. j1 L0 ~6 F! l2 `! `#临时生效2 }/ Y! P7 f1 H4 K& W O
1 g% Y- E. _, K) e
sysctl -w kernel.watchdog_thresh=30
, N6 W( h$ m0 r! h
- y0 t* M# N( C2 ^
/ n8 G* R+ v8 ^& r1 i3 d/ `, W% `) P
#内核软死锁（soft lockup）bug原因分析
, w! A. t4 |7 k3 `- {
9 Q' i& o# |8 R# i, F5 R/ P; D. O) GSoft lockup名称解释：所谓，soft lockup就是说，这个bug没有让系统彻底死机，但是若干个进程（或者kernel thread）被锁死在了某个状态（一般在内核区域），很多情况下这个是由于内核锁的使用的问题。
: q+ h, v" |8 X5 U' T$ c
' D/ E- ]. `$ R, A" }: p' d' |3 }6 t# A. F: r" Z. ?: b( `
: f5 y1 W V% {; M, U, j: k: Z
vim /etc/sysctl.conf
; M$ Q+ f/ j! ^4 I5 F
9 A: O, e! I8 Q/ [- Z, gkernel.watchdog_thresh=307 g& u. Y' ^. U5 g; u! C
# P8 l+ ^( h' z: L
: T' J* q, F- f5 d I8 U
|
|
发表于 2021-12-18 21:11:27