马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。
您需要 登录 才可以下载或查看,没有账号?开始注册
x
$ R, @2 B$ t5 t \, z' X
Jan 14 04:06:25 localhost kernel: [9821120.248545] ------------[ cut here ]------------
- e, {" [6 `- M1 X9 h5 WJan 14 04:06:25 localhost kernel: [9821120.248573] WARNING: CPU: 2 PID: 13942 at fs/dcache.c:362 d_shrink_del+0x2c/0x6e()
$ q1 y3 u" t) D% H$ ]Jan 14 04:06:25 localhost kernel: [9821120.248584] Modules linked in: deflate ctr twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic serpent_sse2_x86_64 ablk_helper cryptd glue_helper lrw serpent_generic xts gf128mul blowfish_ge: A, |+ Z# e: v9 z4 f. X
neric blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic sha256_generic crypto_null xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo 8021q mrp garp stp llc cls_u32 sch_htb xt_nat xt_LOG xt_comm
3 y( E% Y s# C% K3 s9 B/ e5 zent nf_conntrack_netlink iptable_nat nf_nat_ipv4 ip6table_filter ip6table_raw ip6_tables iptable_filter nf_conntrack_ipv4 nf_defrag_ipv4 xt_CT nfnetlink_cthelper nfnetlink iptable_raw nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_h323 nf_conntrack_h323 nf_
! |. c G/ ~$ o8 U6 ^nat_sip nf_conntrack_sip nf_nat_proto_gre nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative ipv6 fuse i2c_piix4 i2
' n4 c+ u( D5 C' K$ oJan 14 04:06:25 localhost kernel: [9821120.249030] CPU: 2 PID: 13942 Comm: sh Not tainted 3.13.11-1-amd64-vyos #10 H, X/ z+ a/ J5 D) m+ Q1 v
Jan 14 04:06:25 localhost kernel: [9821120.249043] Hardware name: Alcatel-Lucent ACCS NFV r1.0 HA, BIOS 0.5.1 01/01/2011) g( {$ a: K) o& ?
Jan 14 04:06:25 localhost kernel: [9821120.249054] 0000000000000000 ffffffff814f58a3 ffffffff813eb0b1 0000000000000000
$ K6 K2 H; e Y5 Y1 ~- UJan 14 04:06:25 localhost kernel: [9821120.249125] ffffffff8104a614 ffff880215609380 ffffffff81133665 ffffffff813edfd2+ b; B" h. E4 ]
Jan 14 04:06:25 localhost kernel: [9821120.249143] ffff880080cc66c0 ffff880080cc6718 ffff880231b1dce0 ffff880231b1dca0! G T0 a7 D% V% N3 r; M( e
Jan 14 04:06:25 localhost kernel: [9821120.249161] Call Trace:$ a; D7 g. I3 _) E3 ^
Jan 14 04:06:25 localhost kernel: [9821120.249183] [<ffffffff813eb0b1>] ? dump_stack+0x41/0x58' o: k J+ `* Q" d) ?. D$ E
Jan 14 04:06:25 localhost kernel: [9821120.249208] [<ffffffff8104a614>] ? warn_slowpath_common+0x79/0x922 e6 W. X4 O: h$ {
Jan 14 04:06:25 localhost kernel: [9821120.249230] [<ffffffff81133665>] ? d_shrink_del+0x2c/0x6e! H; {6 X4 z# V; W3 X
Jan 14 04:06:25 localhost kernel: [9821120.249250] [<ffffffff813edfd2>] ? _raw_spin_unlock+0x5/0x6
% }" b9 V7 P1 T, X( i; ^1 m! \Jan 14 04:06:25 localhost kernel: [9821120.249273] [<ffffffff81133665>] ? d_shrink_del+0x2c/0x6e. u) H- N' K) {) W% C3 q+ k
Jan 14 04:06:25 localhost kernel: [9821120.249295] [<ffffffff81134667>] ? shrink_dentry_list+0x51/0xc6
" b9 X! N( l$ AJan 14 04:06:25 localhost kernel: [9821120.249315] [<ffffffff813edfd2>] ? _raw_spin_unlock+0x5/0x6
I* F9 ~- E" }" |Jan 14 04:06:25 localhost kernel: [9821120.249339] [<ffffffff811347a5>] ? shrinJan 14 03:30:34 localhost kernel: [ 0.000000] Linux version 3.13.11-1-amd64-vyos (jenkins@squeeze64devel) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Wed Aug 12 02:08:05 UTC 2015. m; V7 k% z. \
Jan 14 03:30:34 localhost kernel: [ 0.000000] ACPI: RSDP 00000000000f70d0 000014 (v00 BOCHS )/ x) J8 Z+ T: B9 Y; q4 s7 \
Jan 14 03:30:34 localhost kernel: [ 0.000000] ACPI: RSDT 00000000bffff70a 00002C (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)% ?; W' C G$ n1 p6 Q
Jan 14 03:30:34 localhost kernel: [ 0.000000] ACPI: FACP 00000000bffff606 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)$ _1 ]( j: G2 v$ i
Jan 14 03:30:34 localhost kernel: [ 0.000000] ACPI: DSDT 00000000bfffe040 0015C6 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
, G" E4 |" p8 NJan 14 03:30:34 localhost kernel: [ 0.000000] ACPI: FACS 00000000bfffe000 0000409 N' P0 f3 R. R$ V( u A
Jan 14 03:30:34 localhost kernel: [ 0.000000] ACPI: APIC 00000000bffff67a 000090 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
$ x7 A2 u( A, y1 s( @Jan 14 03:30:34 localhost kernel: [ 0.000000] Zone ranges:/ y9 A) p x0 p" d9 P
Jan 14 03:30:34 localhost kernel: [ 0.000000] DMA [mem 0x00001000-0x00ffffff]- m" H& r1 N3 P5 \$ V
Jan 14 03:30:34 localhost kernel: [ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
+ g ?2 z+ q8 |9 l7 I2 VJan 14 03:30:34 localhost kernel: [ 0.000000] Normal [mem 0x100000000-0x23fffffff]
: S* Q1 e. r' k# B/ z% e# g4 K6 W# ~Jan 14 03:30:34 localhost kernel: [ 0.000000] Movable zone start for each node
5 e$ ] W0 d6 ]$ ]Jan 14 03:30:34 localhost kernel: [ 0.000000] Early memory node ranges. {7 R- h: e8 s9 D% t; V5 p
Jan 14 03:30:34 localhost kernel: [ 0.000000] node 0: [mem 0x00001000-0x0009efff]
/ K$ A- n J" m4 C8 _1 QJan 14 03:30:34 localhost kernel: [ 0.000000] node 0: [mem 0x00100000-0xbfff9fff], D! `0 e; s, b7 F& P! _
Jan 14 03:30:34 localhost kernel: [ 0.000000] node 0: [mem 0x100000000-0x23fffffff]
. j" T: S4 Z& D( }3 d; MJan 14 03:30:34 localhost kernel: [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2064258
# v" [+ \8 |$ nJan 14 03:30:34 localhost kernel: [ 0.000000] Policy zone: Normal
/ j# J4 c5 D6 C* o ~ l8 bJan 14 03:30:34 localhost kernel: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/1.1.7/vmlinuz boot=live quiet vyatta-union=/boot/1.1.7 console=ttyS0,9600 console=tty0
& b4 G5 e$ \) Q' Q+ c0 e8 dJan 14 03:30:34 localhost kernel: [ 0.000000] Memory: 8167636K/8388192K available (4043K kernel code, 594K rwdata, 1652K rodata, 976K init, 700K bss, 220556K reserved)
- s; E" M; Y$ Y! u2 X0 _Jan 14 03:30:34 localhost kernel: [ 0.040964] ACPI: All ACPI Tables successfully acquired
& H# [9 D2 Z `0 G+ q% Q; LJan 14 03:30:34 localhost kernel: [ 0.050000] Failed to access perfctr msr (MSR c1 is 0), b; S6 c- J. Y/ N' u- Y
Jan 14 03:30:34 localhost kernel: [ 0.142534] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20131115/hwxface-580)
% {( m+ K, z7 I) E) A4 vJan 14 03:30:34 localhost kernel: [ 0.142541] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20131115/hwxface-580)
, ]3 O* U5 Z6 U* U3 `3 K1 @Jan 14 03:30:34 localhost kernel: [ 0.142545] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20131115/hwxface-580)# ?" j$ d( h& p
Jan 14 03:30:34 localhost kernel: [ 0.142549] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20131115/hwxface-580)
& f0 _) L8 C: d# OJan 14 03:30:34 localhost kernel: [ 0.150892] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.' ^. r. K1 j$ U( S' m; h. K$ B" l
Jan 14 03:30:34 localhost kernel: [ 0.443990] ACPI: Enabled 3 GPEs in block 00 to 0F# H8 j( T+ d4 i5 k: W
Jan 14 03:30:34 localhost kernel: [ 0.444104] SCSI subsystem initialized9 c( y9 ~% q7 M
Jan 14 03:30:34 localhost kernel: [ 0.532368] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 117 @- c$ L& r0 L( U9 L1 b
Jan 14 03:30:34 localhost kernel: [ 1.028215] type=2000 audit(1578972622.313:1): initialized& ~; y0 u9 L$ u- p0 H( a
Jan 14 03:30:34 localhost kernel: [ 1.029615] bounce pool size: 64 pages
3 A1 ^& ^6 g* g8 y) z, ?) W SJan 14 03:30:34 localhost kernel: [ 1.138358] Key type dns_resolver registered
- p( p$ b; _* DJan 14 03:30:34 localhost kernel: [ 1.244013] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 107 y o& S3 d/ y% s* L
Jan 14 03:30:34 localhost kernel: [ 1.357808] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10. y% V E2 s: |9 K0 B
Jan 14 03:30:34 localhost kernel: [ 2.040051] raid6: sse2x1 4639 MB/s- J/ c0 N) F) O3 U" v
Jan 14 03:30:34 localhost kernel: [ 2.210027] raid6: sse2x2 6362 MB/s
4 c c1 Y! I6 fJan 14 03:30:34 localhost kernel: [ 2.380032] raid6: sse2x4 7357 MB/s U' e+ n- A% E3 ^5 u' }
Jan 14 03:30:34 localhost kernel: [ 2.380037] raid6: using algorithm sse2x4 (7357 MB/s)9 g$ ^/ N& H" ~ G9 @% b) u3 x
Jan 14 03:30:34 localhost kernel: [ 2.380040] raid6: using intx1 recovery algorithm3 @: |, @) C. K @: r
Jan 14 03:30:34 localhost kernel: [ 8.178340] EXT4-fs (vda1): warning: checktime reached, running e2fsck is recommended9 b4 V, e p6 ^: C. B
Jan 14 03:30:34 localhost kernel: [ 9.087041] random: debconf-communi urandom read with 25 bits of entropy available
4 C0 ?; t6 l; R3 r$ z9 o/ T% cJan 14 03:30:34 localhost kernel: [ 10.625650] platform microcode: Direct firmware load failed with error -26 S3 R" [4 |4 x3 c/ r! u. A9 N
Jan 14 03:30:34 localhost kernel: [ 10.625654] platform microcode: Falling back to user helper6 `* r) E6 s0 O4 }8 @8 u3 B
Jan 14 03:30:34 localhost kernel: [ 10.652304] platform microcode: Direct firmware load failed with error -2
3 O/ A( O8 |: h# ?9 kJan 14 03:30:34 localhost kernel: [ 10.652309] platform microcode: Falling back to user helper
; j7 O" c& Y! j9 L. N& O6 r& o# cJan 14 03:30:34 localhost kernel: [ 10.661767] platform microcode: Direct firmware load failed with error -2; q8 R5 L& O: {9 m7 S' t
Jan 14 03:30:34 localhost kernel: [ 10.661771] platform microcode: Falling back to user helper, W! ]7 k+ q7 W2 Q7 ~3 ~" m! e
Jan 14 03:30:34 localhost kernel: [ 10.669655] platform microcode: Direct firmware load failed with error -2
- Q6 A( E5 M* j. `1 a2 y2 g7 oJan 14 03:30:34 localhost kernel: [ 10.669658] platform microcode: Falling back to user helper! K- N- r& H Q0 z h
Jan 14 03:30:34 localhost nrpe[2151]: Starting up daemon! p+ v v+ u' y; m6 }0 J' k, K+ }
Jan 14 03:30:34 localhost nrpe[2151]: Network server bind failure (99: Cannot assign requested address)% h$ H5 ^) I: {/ G Y' Q2 `3 ^
Jan 14 03:30:34 localhost zebra[2157]: Zebra 0.99.20.1 starting: vty@02 q. L: f, a& j& I0 r
Jan 14 03:30:34 localhost ripd[2168]: RIPd 0.99.20.1 starting: vty@0( M# u* i% [/ K( e3 {$ r) I2 q
Jan 14 03:30:34 localhost ripngd[2170]: RIPNGd 0.99.20.1 starting: vty@0
. s+ F: W/ u) ^: H& vJan 14 03:30:34 localhost ospfd[2172]: OSPFd 0.99.20.1 starting: vty@0. M5 q3 P, k0 B: M" @/ d
Jan 14 03:30:34 localhost ospf6d[2174]: OSPF6d (Quagga-0.99.20.1 ospf6d-0.9.7r) starts: vty@0
6 d: x2 H9 f. SJan 14 03:30:34 localhost bgpd[2176]: BGPd 0.99.20.1 starting: vty@0, bgp@<all>:179
' `: k3 B0 _2 E3 x' [$ [Jan 14 03:30:37 localhost zebra[2157]: interface eth0 index 2 changed <UP,BROADCAST,RUNNING,MULTICAST>.) }/ _8 |( ^* F; Q' m7 g
Jan 14 03:30:38 localhost kernel: [ 17.283749] sshd (2567): /proc/2567/oom_adj is deprecated, please use /proc/2567/oom_score_adj instead.
0 j* F* _4 w; E0 L$ k( B; n$ G
1 Z Y6 c7 r v9 \6 @
% X8 z" V9 F) {2 S* P$ T* ?3 r0 e0 ^! L, t# J
现象:
& e6 ]4 g% e D三个措施。9 L( z5 ?+ g& f/ ?2 P+ O% C9 \
1、增大内存
9 ~& z# V* v" y9 W; u: J从12GB增大到16GB。6 E/ F# |" @8 W8 Y0 i
virsh setmaxmem vgfsxxx 16GB --config: f8 ?3 B% U/ M* j# T8 }
运行启动后
, }5 V8 j! W- o6 }% ^8 B1 Fvirsh setmem vgfsxxx 16GB- \% r1 y8 \8 V& X" Z
这个没有用,跑了几次测试后,仍然掉服务。 2、调整lwfsd的服务优先级
* G3 f5 d( }7 _设置lwfsd的服务优先级为“-17”+ I/ ^/ K0 Q F" S: w
PID=`ps | grep lwfs | grep -v grep | awk '{print $1}'`
# G& s7 }, ?# ~$ d/ @8 ?echo -17 > /proc/$PID/oom_adj$ f9 I( C% n* h7 B6 n Q% v
echo -17 > /proc/$PID/task/$PID/oom_adj- v# {# \0 K+ t8 H, u% U7 x; j
这个好像有用。 3、修改内存分配策略
* _* U: Z/ i. F; L并且echo "2" >/proc/sys/vm/overcommit_memory,使得分配内存时,必须存在足够的空间用于映射。* o$ Q0 |3 X4 m L2 Z- u
这个好像也有一定的用处。再跑跑试试。 / Y/ k F* s+ Y* B3 K% r: N
|