虚机内存溢出问题，分析

admin

Jan 14 04:06:25 localhost kernel: [9821120.248545] ------------[ cut here ]------------
Jan 14 04:06:25 localhost kernel: [9821120.248573] WARNING: CPU: 2 PID: 13942 at fs/dcache.c:362 d_shrink_del+0x2c/0x6e()
Jan 14 04:06:25 localhost kernel: [9821120.248584] Modules linked in: deflate ctr twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic serpent_sse2_x86_64 ablk_helper cryptd glue_helper lrw serpent_generic xts gf128mul blowfish_ge
6 o4 o% c/ k9 P) X" _3 |neric blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic sha256_generic crypto_null xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo 8021q mrp garp stp llc cls_u32 sch_htb xt_nat xt_LOG xt_comm
# ^, I& U0 m* F8 G" ?- O5 Z% d: Oent nf_conntrack_netlink iptable_nat nf_nat_ipv4 ip6table_filter ip6table_raw ip6_tables iptable_filter nf_conntrack_ipv4 nf_defrag_ipv4 xt_CT nfnetlink_cthelper nfnetlink iptable_raw nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_h323 nf_conntrack_h323 nf_
$ o" R! b8 Y9 i8 z6 p) d/ gnat_sip nf_conntrack_sip nf_nat_proto_gre nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative ipv6 fuse i2c_piix4 i2
Jan 14 04:06:25 localhost kernel: [9821120.249030] CPU: 2 PID: 13942 Comm: sh Not tainted 3.13.11-1-amd64-vyos #1
Jan 14 04:06:25 localhost kernel: [9821120.249043] Hardware name: Alcatel-Lucent ACCS NFV r1.0 HA, BIOS 0.5.1 01/01/2011
Jan 14 04:06:25 localhost kernel: [9821120.249054]  0000000000000000 ffffffff814f58a3 ffffffff813eb0b1 0000000000000000
/ H7 ^; G) y7 a, O+ o$ e; HJan 14 04:06:25 localhost kernel: [9821120.249125]  ffffffff8104a614 ffff880215609380 ffffffff81133665 ffffffff813edfd2
Jan 14 04:06:25 localhost kernel: [9821120.249143]  ffff880080cc66c0 ffff880080cc6718 ffff880231b1dce0 ffff880231b1dca0
3 h9 ?1 \6 \+ w4 oJan 14 04:06:25 localhost kernel: [9821120.249161] Call Trace:
  q! C4 @" t! z$ ]$ CJan 14 04:06:25 localhost kernel: [9821120.249183]  [<ffffffff813eb0b1>] ? dump_stack+0x41/0x58
Jan 14 04:06:25 localhost kernel: [9821120.249208]  [<ffffffff8104a614>] ? warn_slowpath_common+0x79/0x92
Jan 14 04:06:25 localhost kernel: [9821120.249230]  [<ffffffff81133665>] ? d_shrink_del+0x2c/0x6e
Jan 14 04:06:25 localhost kernel: [9821120.249250]  [<ffffffff813edfd2>] ? _raw_spin_unlock+0x5/0x6
Jan 14 04:06:25 localhost kernel: [9821120.249273]  [<ffffffff81133665>] ? d_shrink_del+0x2c/0x6e
Jan 14 04:06:25 localhost kernel: [9821120.249295]  [<ffffffff81134667>] ? shrink_dentry_list+0x51/0xc6
1 C* @! T5 j& S2 k6 C9 aJan 14 04:06:25 localhost kernel: [9821120.249315]  [<ffffffff813edfd2>] ? _raw_spin_unlock+0x5/0x6
/ K8 n' K& Z$ ^/ U) O+ c9 g' |0 Y- {, HJan 14 04:06:25 localhost kernel: [9821120.249339]  [<ffffffff811347a5>] ? shrinJan 14 03:30:34 localhost kernel: [    0.000000] Linux version 3.13.11-1-amd64-vyos (jenkins@squeeze64devel) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Wed Aug 12 02:08:05 UTC 2015
/ a! L6 j! {* X2 a5 fJan 14 03:30:34 localhost kernel: [    0.000000] ACPI: RSDP 00000000000f70d0 000014 (v00 BOCHS )
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: RSDT 00000000bffff70a 00002C (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
Jan 14 03:30:34 localhost kernel: [    0.000000] ACPI: FACP 00000000bffff606 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
( K4 ~) w7 k  u' l; f6 o! t5 {' XJan 14 03:30:34 localhost kernel: [    0.000000] ACPI: DSDT 00000000bfffe040 0015C6 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
2 l; z) X, R% KJan 14 03:30:34 localhost kernel: [    0.000000] ACPI: FACS 00000000bfffe000 000040
+ W; ~: \4 }5 M4 n* uJan 14 03:30:34 localhost kernel: [    0.000000] ACPI: APIC 00000000bffff67a 000090 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
Jan 14 03:30:34 localhost kernel: [    0.000000] Zone ranges:
Jan 14 03:30:34 localhost kernel: [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
. Y1 t; J- b0 u$ F% O9 XJan 14 03:30:34 localhost kernel: [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
, H( I3 y# K% n4 k; Z  nJan 14 03:30:34 localhost kernel: [    0.000000]   Normal   [mem 0x100000000-0x23fffffff]
' {2 @$ J- j5 w" QJan 14 03:30:34 localhost kernel: [    0.000000] Movable zone start for each node
" I! X0 g" b! c0 S; vJan 14 03:30:34 localhost kernel: [    0.000000] Early memory node ranges
& e- g# M: x6 v/ e' \Jan 14 03:30:34 localhost kernel: [    0.000000]   node   0: [mem 0x00001000-0x0009efff]
: Q8 {8 Y/ ?: l6 `# [Jan 14 03:30:34 localhost kernel: [    0.000000]   node   0: [mem 0x00100000-0xbfff9fff]
Jan 14 03:30:34 localhost kernel: [    0.000000]   node   0: [mem 0x100000000-0x23fffffff]
3 {0 T; |2 ?+ T, z: C4 ~Jan 14 03:30:34 localhost kernel: [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2064258
Jan 14 03:30:34 localhost kernel: [    0.000000] Policy zone: Normal
, [) O' S  g+ c! cJan 14 03:30:34 localhost kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/1.1.7/vmlinuz boot=live quiet vyatta-union=/boot/1.1.7 console=ttyS0,9600 console=tty0
Jan 14 03:30:34 localhost kernel: [    0.000000] Memory: 8167636K/8388192K available (4043K kernel code, 594K rwdata, 1652K rodata, 976K init, 700K bss, 220556K reserved)
Jan 14 03:30:34 localhost kernel: [    0.040964] ACPI: All ACPI Tables successfully acquired
8 l0 T- ?6 a3 J  g: A" a+ PJan 14 03:30:34 localhost kernel: [    0.050000] Failed to access perfctr msr (MSR c1 is 0)
' s  f) c/ d& ?' _$ k& Y( u" \0 ]- nJan 14 03:30:34 localhost kernel: [    0.142534] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20131115/hwxface-580)
! h; H+ }$ a. _( o3 x8 N+ LJan 14 03:30:34 localhost kernel: [    0.142541] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20131115/hwxface-580)
1 f  n1 N) }  x% e& V# C. M, @+ KJan 14 03:30:34 localhost kernel: [    0.142545] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20131115/hwxface-580)
4 F) N  b' B$ d2 PJan 14 03:30:34 localhost kernel: [    0.142549] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20131115/hwxface-580)
Jan 14 03:30:34 localhost kernel: [    0.150892] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
9 d" ^  p' `- Z4 AJan 14 03:30:34 localhost kernel: [    0.443990] ACPI: Enabled 3 GPEs in block 00 to 0F
Jan 14 03:30:34 localhost kernel: [    0.444104] SCSI subsystem initialized
8 R5 n# @! [" u* ~8 G6 tJan 14 03:30:34 localhost kernel: [    0.532368] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
Jan 14 03:30:34 localhost kernel: [    1.028215] type=2000 audit(1578972622.313:1): initialized
' |6 A6 d/ x( J$ o4 `9 C, UJan 14 03:30:34 localhost kernel: [    1.029615] bounce pool size: 64 pages
( s5 ], h$ v( L4 AJan 14 03:30:34 localhost kernel: [    1.138358] Key type dns_resolver registered
Jan 14 03:30:34 localhost kernel: [    1.244013] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
Jan 14 03:30:34 localhost kernel: [    1.357808] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
9 E/ \0 I8 s/ _3 ?8 MJan 14 03:30:34 localhost kernel: [    2.040051] raid6: sse2x1    4639 MB/s
* f) \* R7 g; a' d- a% PJan 14 03:30:34 localhost kernel: [    2.210027] raid6: sse2x2    6362 MB/s
" W8 p% y  z$ ~6 |( VJan 14 03:30:34 localhost kernel: [    2.380032] raid6: sse2x4    7357 MB/s
+ o9 @: V1 f! P% O) uJan 14 03:30:34 localhost kernel: [    2.380037] raid6: using algorithm sse2x4 (7357 MB/s)
. s- P4 d& {% I: C  X5 gJan 14 03:30:34 localhost kernel: [    2.380040] raid6: using intx1 recovery algorithm
6 D3 ~7 p* C. {5 L7 _Jan 14 03:30:34 localhost kernel: [    8.178340] EXT4-fs (vda1): warning: checktime reached, running e2fsck is recommended
Jan 14 03:30:34 localhost kernel: [    9.087041] random: debconf-communi urandom read with 25 bits of entropy available
Jan 14 03:30:34 localhost kernel: [   10.625650] platform microcode: Direct firmware load failed with error -2
" G1 S0 t1 |* LJan 14 03:30:34 localhost kernel: [   10.625654] platform microcode: Falling back to user helper
Jan 14 03:30:34 localhost kernel: [   10.652304] platform microcode: Direct firmware load failed with error -2
8 [* Q+ K/ R, E! _4 o; \9 RJan 14 03:30:34 localhost kernel: [   10.652309] platform microcode: Falling back to user helper
Jan 14 03:30:34 localhost kernel: [   10.661767] platform microcode: Direct firmware load failed with error -2
9 M3 U2 O" [8 G. }4 h7 C3 D* \/ ]) QJan 14 03:30:34 localhost kernel: [   10.661771] platform microcode: Falling back to user helper
Jan 14 03:30:34 localhost kernel: [   10.669655] platform microcode: Direct firmware load failed with error -2
9 z- w- f( ?8 L! P+ R1 x+ MJan 14 03:30:34 localhost kernel: [   10.669658] platform microcode: Falling back to user helper
/ Z' M1 C6 m) Z% a" c7 o& |Jan 14 03:30:34 localhost nrpe[2151]: Starting up daemon
. M& u6 d" C; i; VJan 14 03:30:34 localhost nrpe[2151]: Network server bind failure (99: Cannot assign requested address)
5 Y( u: X9 X+ H0 p& x8 dJan 14 03:30:34 localhost zebra[2157]: Zebra 0.99.20.1 starting: vty@0
9 E* j/ ^2 d; }6 T: LJan 14 03:30:34 localhost ripd[2168]: RIPd 0.99.20.1 starting: vty@0
3 ^( X$ e; N; ]Jan 14 03:30:34 localhost ripngd[2170]: RIPNGd 0.99.20.1 starting: vty@0
Jan 14 03:30:34 localhost ospfd[2172]: OSPFd 0.99.20.1 starting: vty@0
3 q6 U3 I; s, Z+ d- P3 H4 e3 G- LJan 14 03:30:34 localhost ospf6d[2174]: OSPF6d (Quagga-0.99.20.1 ospf6d-0.9.7r) starts: vty@0
4 Q, p% }% ]  ]) Q2 |Jan 14 03:30:34 localhost bgpd[2176]: BGPd 0.99.20.1 starting: vty@0, bgp@<all>:179
) N3 i) y0 n- l4 G! _! m! qJan 14 03:30:37 localhost zebra[2157]: interface eth0 index 2 changed <UP,BROADCAST,RUNNING,MULTICAST>.
/ I7 j+ |4 L5 lJan 14 03:30:38 localhost kernel: [   17.283749] sshd (2567): /proc/2567/oom_adj is deprecated, please use /proc/2567/oom_score_adj instead.

+ L. M4 u5 u  T

; U+ N+ @, P+ M7 `6 h现象：
. j; T9 f# L$ U! j8 D2 }+ b

三个措施。
. p" C/ B1 C  P* q1 Q, Q1、增大内存
3 f* j+ p, X# X从12GB增大到16GB。
virsh setmaxmem vgfsxxx 16GB --config
* f1 [* u4 M8 x8 q" A运行启动后
virsh setmem vgfsxxx 16GB
这个没有用，跑了几次测试后，仍然掉服务。

2、调整lwfsd的服务优先级
设置lwfsd的服务优先级为“-17”
PID=`ps | grep lwfs | grep -v grep | awk '{print $1}'`
echo -17 > /proc/$PID/oom_adj
echo -17 > /proc/$PID/task/$PID/oom_adj
这个好像有用。

3、修改内存分配策略
并且echo "2" >/proc/sys/vm/overcommit_memory，使得分配内存时，必须存在足够的空间用于映射。
这个好像也有一定的用处。再跑跑试试。