虚机内存溢出问题，分析

admin

Jan 14 04:06:25 localhost kernel: [9821120.248545] ------------[ cut here ]------------
Jan 14 04:06:25 localhost kernel: [9821120.248573] WARNING: CPU: 2 PID: 13942 at fs/dcache.c:362 d_shrink_del+0x2c/0x6e()
# }  Z' d1 z& G3 EJan 14 04:06:25 localhost kernel: [9821120.248584] Modules linked in: deflate ctr twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic serpent_sse2_x86_64 ablk_helper cryptd glue_helper lrw serpent_generic xts gf128mul blowfish_ge
neric blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic sha256_generic crypto_null xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo 8021q mrp garp stp llc cls_u32 sch_htb xt_nat xt_LOG xt_comm
  p" \# r( ]& |2 lent nf_conntrack_netlink iptable_nat nf_nat_ipv4 ip6table_filter ip6table_raw ip6_tables iptable_filter nf_conntrack_ipv4 nf_defrag_ipv4 xt_CT nfnetlink_cthelper nfnetlink iptable_raw nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_h323 nf_conntrack_h323 nf_
nat_sip nf_conntrack_sip nf_nat_proto_gre nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative ipv6 fuse i2c_piix4 i2
) J; ]5 A2 @" }" t# |3 h' \% sJan 14 04:06:25 localhost kernel: [9821120.249030] CPU: 2 PID: 13942 Comm: sh Not tainted 3.13.11-1-amd64-vyos #1
Jan 14 04:06:25 localhost kernel: [9821120.249043] Hardware name: Alcatel-Lucent ACCS NFV r1.0 HA, BIOS 0.5.1 01/01/2011
Jan 14 04:06:25 localhost kernel: [9821120.249054]  0000000000000000 ffffffff814f58a3 ffffffff813eb0b1 0000000000000000
Jan 14 04:06:25 localhost kernel: [9821120.249125]  ffffffff8104a614 ffff880215609380 ffffffff81133665 ffffffff813edfd2
Jan 14 04:06:25 localhost kernel: [9821120.249143]  ffff880080cc66c0 ffff880080cc6718 ffff880231b1dce0 ffff880231b1dca0
+ U2 C6 G4 H. hJan 14 04:06:25 localhost kernel: [9821120.249161] Call Trace:
Jan 14 04:06:25 localhost kernel: [9821120.249183]  [<ffffffff813eb0b1>] ? dump_stack+0x41/0x58
& M# P. c: B8 n  ~6 M% @Jan 14 04:06:25 localhost kernel: [9821120.249208]  [<ffffffff8104a614>] ? warn_slowpath_common+0x79/0x92
7 q- u3 H7 J3 z$ LJan 14 04:06:25 localhost kernel: [9821120.249230]  [<ffffffff81133665>] ? d_shrink_del+0x2c/0x6e
% e: m& e" V0 }Jan 14 04:06:25 localhost kernel: [9821120.249250]  [<ffffffff813edfd2>] ? _raw_spin_unlock+0x5/0x6
Jan 14 04:06:25 localhost kernel: [9821120.249273]  [<ffffffff81133665>] ? d_shrink_del+0x2c/0x6e
5 v: M5 j3 j0 C' J' BJan 14 04:06:25 localhost kernel: [9821120.249295]  [<ffffffff81134667>] ? shrink_dentry_list+0x51/0xc6
3 O# r. l5 R+ x6 O2 A9 tJan 14 04:06:25 localhost kernel: [9821120.249315]  [<ffffffff813edfd2>] ? _raw_spin_unlock+0x5/0x6
Jan 14 04:06:25 localhost kernel: [9821120.249339]  [<ffffffff811347a5>] ? shrinJan 14 03:30:34 localhost kernel: [    0.000000] Linux version 3.13.11-1-amd64-vyos (jenkins@squeeze64devel) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Wed Aug 12 02:08:05 UTC 2015
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: RSDP 00000000000f70d0 000014 (v00 BOCHS )
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: RSDT 00000000bffff70a 00002C (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: FACP 00000000bffff606 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: DSDT 00000000bfffe040 0015C6 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
1 i/ D( R0 p" L3 q( b' jJan 14 03:30:34 localhost kernel: [    0.000000] ACPI: FACS 00000000bfffe000 000040
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: APIC 00000000bffff67a 000090 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
5 B: t& O/ n) g7 V8 h. a/ XJan 14 03:30:34 localhost kernel: [    0.000000] Zone ranges:
Jan 14 03:30:34 localhost kernel: [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
; a. F: M! r. a# WJan 14 03:30:34 localhost kernel: [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
1 m% u4 M* |4 W9 J+ y3 B8 u( JJan 14 03:30:34 localhost kernel: [    0.000000]   Normal   [mem 0x100000000-0x23fffffff]
/ d" T! B0 W. Q# O% D. qJan 14 03:30:34 localhost kernel: [    0.000000] Movable zone start for each node
' g7 p3 K$ f+ P; ~. E2 y, @Jan 14 03:30:34 localhost kernel: [    0.000000] Early memory node ranges
( ^& u" v- E1 _5 g2 S$ gJan 14 03:30:34 localhost kernel: [    0.000000]   node   0: [mem 0x00001000-0x0009efff]
( z9 Z4 l5 B1 V/ H, H/ o7 BJan 14 03:30:34 localhost kernel: [    0.000000]   node   0: [mem 0x00100000-0xbfff9fff]
Jan 14 03:30:34 localhost kernel: [    0.000000]   node   0: [mem 0x100000000-0x23fffffff]
Jan 14 03:30:34 localhost kernel: [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2064258
' _* F% r  F0 Z3 K% Q( s$ z$ F$ LJan 14 03:30:34 localhost kernel: [    0.000000] Policy zone: Normal
" `& I" L- [, v' K& M" v  VJan 14 03:30:34 localhost kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/1.1.7/vmlinuz boot=live quiet vyatta-union=/boot/1.1.7 console=ttyS0,9600 console=tty0
5 L8 S5 W$ S  C' I! EJan 14 03:30:34 localhost kernel: [    0.000000] Memory: 8167636K/8388192K available (4043K kernel code, 594K rwdata, 1652K rodata, 976K init, 700K bss, 220556K reserved)
Jan 14 03:30:34 localhost kernel: [    0.040964] ACPI: All ACPI Tables successfully acquired
Jan 14 03:30:34 localhost kernel: [    0.050000] Failed to access perfctr msr (MSR c1 is 0)
Jan 14 03:30:34 localhost kernel: [    0.142534] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20131115/hwxface-580)
Jan 14 03:30:34 localhost kernel: [    0.142541] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20131115/hwxface-580)
3 m; E! D2 t2 M5 n1 |- l9 o$ |3 YJan 14 03:30:34 localhost kernel: [    0.142545] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20131115/hwxface-580)
Jan 14 03:30:34 localhost kernel: [    0.142549] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20131115/hwxface-580)
% c* P# h* v6 B9 I% L& Y4 H1 p4 JJan 14 03:30:34 localhost kernel: [    0.150892] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
; v) Y& o" @! n/ |  P$ ]! O" rJan 14 03:30:34 localhost kernel: [    0.443990] ACPI: Enabled 3 GPEs in block 00 to 0F
Jan 14 03:30:34 localhost kernel: [    0.444104] SCSI subsystem initialized
Jan 14 03:30:34 localhost kernel: [    0.532368] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
Jan 14 03:30:34 localhost kernel: [    1.028215] type=2000 audit(1578972622.313:1): initialized
. W6 C0 z0 Z% C4 ?+ J7 L. kJan 14 03:30:34 localhost kernel: [    1.029615] bounce pool size: 64 pages
Jan 14 03:30:34 localhost kernel: [    1.138358] Key type dns_resolver registered
Jan 14 03:30:34 localhost kernel: [    1.244013] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
Jan 14 03:30:34 localhost kernel: [    1.357808] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
$ B3 ]4 R6 \8 v; p3 IJan 14 03:30:34 localhost kernel: [    2.040051] raid6: sse2x1    4639 MB/s
: S  w, r9 O3 PJan 14 03:30:34 localhost kernel: [    2.210027] raid6: sse2x2    6362 MB/s
  C( d7 u& v1 P: {- r; OJan 14 03:30:34 localhost kernel: [    2.380032] raid6: sse2x4    7357 MB/s
Jan 14 03:30:34 localhost kernel: [    2.380037] raid6: using algorithm sse2x4 (7357 MB/s)
$ N4 ]- N7 ?/ lJan 14 03:30:34 localhost kernel: [    2.380040] raid6: using intx1 recovery algorithm
Jan 14 03:30:34 localhost kernel: [    8.178340] EXT4-fs (vda1): warning: checktime reached, running e2fsck is recommended
% _" `- W1 p" _7 P) R" S0 Q! RJan 14 03:30:34 localhost kernel: [    9.087041] random: debconf-communi urandom read with 25 bits of entropy available
Jan 14 03:30:34 localhost kernel: [   10.625650] platform microcode: Direct firmware load failed with error -2
Jan 14 03:30:34 localhost kernel: [   10.625654] platform microcode: Falling back to user helper
! k, K% w5 B% L, r  ]9 J9 MJan 14 03:30:34 localhost kernel: [   10.652304] platform microcode: Direct firmware load failed with error -2
" T5 J0 [3 m+ L4 M+ f0 N( P' Y' {Jan 14 03:30:34 localhost kernel: [   10.652309] platform microcode: Falling back to user helper
Jan 14 03:30:34 localhost kernel: [   10.661767] platform microcode: Direct firmware load failed with error -2
8 \5 V, u& D. J. o( K  [4 O( ~Jan 14 03:30:34 localhost kernel: [   10.661771] platform microcode: Falling back to user helper
1 h% S" s9 U% M) H2 ^Jan 14 03:30:34 localhost kernel: [   10.669655] platform microcode: Direct firmware load failed with error -2
5 T/ i* S' l1 S$ J& ZJan 14 03:30:34 localhost kernel: [   10.669658] platform microcode: Falling back to user helper
0 }$ c, m: ^6 O2 S0 W# NJan 14 03:30:34 localhost nrpe[2151]: Starting up daemon
Jan 14 03:30:34 localhost nrpe[2151]: Network server bind failure (99: Cannot assign requested address)
: u2 J( h6 v6 p% a9 |  |: VJan 14 03:30:34 localhost zebra[2157]: Zebra 0.99.20.1 starting: vty@0
Jan 14 03:30:34 localhost ripd[2168]: RIPd 0.99.20.1 starting: vty@0
! L7 F% r0 H( B' Q3 ?. H' pJan 14 03:30:34 localhost ripngd[2170]: RIPNGd 0.99.20.1 starting: vty@0
Jan 14 03:30:34 localhost ospfd[2172]: OSPFd 0.99.20.1 starting: vty@0
Jan 14 03:30:34 localhost ospf6d[2174]: OSPF6d (Quagga-0.99.20.1 ospf6d-0.9.7r) starts: vty@0
' Y2 z! E6 G: t  s& R1 n3 ?$ Z+ Q  Y; qJan 14 03:30:34 localhost bgpd[2176]: BGPd 0.99.20.1 starting: vty@0, bgp@<all>:179
  e; j- T! E/ x4 |Jan 14 03:30:37 localhost zebra[2157]: interface eth0 index 2 changed <UP,BROADCAST,RUNNING,MULTICAST>.
Jan 14 03:30:38 localhost kernel: [   17.283749] sshd (2567): /proc/2567/oom_adj is deprecated, please use /proc/2567/oom_score_adj instead.

5 q5 C, k* F) e7 [0 y

9 u8 K. y" Q$ Y# C2 ]4 c6 e现象：
& x2 v* w6 L% R: @0 c

三个措施。
  f; c$ Q1 V5 v2 c; D1、增大内存
从12GB增大到16GB。
! C! v  L! ?( i2 ~virsh setmaxmem vgfsxxx 16GB --config
5 [3 i( R; L, j7 ^8 W6 B: k运行启动后
virsh setmem vgfsxxx 16GB
这个没有用，跑了几次测试后，仍然掉服务。

2、调整lwfsd的服务优先级
% A  z- g. h$ k设置lwfsd的服务优先级为“-17”
PID=`ps | grep lwfs | grep -v grep | awk '{print $1}'`
) e; }# z4 a& Mecho -17 > /proc/$PID/oom_adj
echo -17 > /proc/$PID/task/$PID/oom_adj
. m- s) R2 i4 t这个好像有用。

3、修改内存分配策略
! h# c/ {5 j0 H( W) Z并且echo "2" >/proc/sys/vm/overcommit_memory，使得分配内存时，必须存在足够的空间用于映射。
这个好像也有一定的用处。再跑跑试试。

3 O" d( \8 T  \