* [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 7:11 ` Xuzhichuang
0 siblings, 0 replies; 20+ messages in thread
From: Xuzhichuang @ 2015-07-14 7:11 UTC (permalink / raw)
To: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, Songjiangtao (mygirlsjt),
Zhangwei (FF), Qiuxishi
Hi, all
Description of problem:
Recently, one of my Linux system invoked oom-killer, but the system still has much memory, I don't know why the system still invoked oom-killer, anybody can help me to see it, thanks.
Linux kernel version: 3.0.58
Following is the message:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138769] iostat cpuset=/ mems_allowed=0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138773] Pid: 18117, comm: iostat Tainted: P W NX 3.0.58-0.6.6-xen #1
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138775] Call Trace:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138800] [<ffffffff800088be>] dump_trace+0x6e/0x1a0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138810] [<ffffffff803f773d>] dump_stack+0x69/0x6f
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138821] [<ffffffff800dbced>] dump_header+0x9d/0x120
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138826] [<ffffffff800dc505>] oom_kill_process+0x95/0x1a0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138830] [<ffffffff800dc746>] out_of_memory+0x136/0x220
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138834] [<ffffffff800e0fda>] __alloc_pages_slowpath+0x7ba/0x810
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138838] [<ffffffff800e1219>] __alloc_pages_nodemask+0x1e9/0x200
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138845] [<ffffffff8011ae38>] cache_grow+0x348/0x450
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138850] [<ffffffff8011b243>] cache_alloc_refill+0x303/0x4d0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138854] [<ffffffff8011ba70>] __kmalloc+0x1b0/0x290
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138862] [<ffffffff8014c1da>] seq_read+0x13a/0x3b0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138869] [<ffffffff8018a762>] proc_reg_read+0x92/0xe0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138877] [<ffffffff80129877>] vfs_read+0xc7/0x130
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138881] [<ffffffff801299e3>] sys_read+0x53/0xa0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138887] [<ffffffff80402d73>] system_call_fastpath+0x16/0x1b
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138922] [<00007f935f57f4c0>] 0x7f935f57f4bf
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138923] Mem-Info:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138925] DMA per-cpu:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138927] CPU 0: hi: 0, btch: 1 usd: 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138929] CPU 1: hi: 0, btch: 1 usd: 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138930] DMA32 per-cpu:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138932] CPU 0: hi: 155, btch: 38 usd: 11
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138933] CPU 1: hi: 155, btch: 38 usd: 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138936] active_anon:227111 inactive_anon:10382 isolated_anon:0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138937] active_file:203 inactive_file:189 isolated_file:47
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138938] unevictable:95395 dirty:0 writeback:0 unstable:0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138939] free:247834 slab_reclaimable:18187 slab_unreclaimable:53853
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138940] mapped:11485 shmem:11167 pagetables:0 bounce:0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap = 0kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147221] 788480 pages RAM
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147224] 93755 pages reserved
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147225] 143842 pages shared
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147226] 420430 pages non-shared
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147228] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147242] [ 784] 0 784 4745 725 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147255] [ 3063] 0 3063 3221 290 0 0 0 fma-sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147258] [ 3065] 0 3065 1002 157 0 0 0 acpid
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147262] [ 3155] 0 3155 969 71 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147265] [ 3158] 0 3158 4990 345 1 0 0 syslog-ng
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147268] [ 3173] 0 3173 2455 329 0 0 0 irqbalance
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147271] [ 3176] 0 3176 1069 143 1 0 0 klogd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147273] [ 3183] 0 3183 3221 290 0 0 0 pma-sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147277] [ 3186] 100 3186 5357 136 1 0 0 dbus-daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147280] [ 3223] 0 3223 21330 376 1 0 0 ovsdb-server
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147283] [ 3268] 0 3268 1490 68 1 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147286] [ 3271] 0 3271 1523 115 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147289] [ 3273] 0 3273 969 72 0 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147292] [ 3317] 0 3317 28331 4059 0 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147295] [ 3320] 0 3320 21593 962 1 0 0 ovs-vswitchd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147298] [ 3322] 0 3322 1858 387 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147301] [ 3341] 103 3341 9408 1502 0 0 0 hald
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147304] [ 3348] 0 3348 4886 129 1 0 0 ovs-brcompatd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147307] [ 3350] 0 3350 1688 203 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147310] [ 3352] 0 3352 38107 2004 1 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147315] [ 3356] 0 3356 26731 336 0 0 0 console-kit-dae
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147318] [ 3357] 0 3357 78512 3688 0 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147321] [ 3364] 0 3364 4576 187 0 0 0 hald-runner
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147324] [ 3433] 0 3433 1622 176 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147327] [ 3444] 0 3444 27681 463 0 -17 -1000 evs_service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147330] [ 3488] 0 3488 1679 236 1 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147333] [ 3628] 0 3628 17899 2152 1 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147336] [ 3631] 0 3631 969 72 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147339] [ 3683] 0 3683 3200 185 0 0 0 calcload
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147342] [ 3723] 0 3723 8701 215 0 0 0 hald-addon-inpu
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147346] [ 3746] 0 3746 969 71 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147348] [ 3866] 103 3866 9133 168 1 0 0 hald-addon-acpi
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147351] [ 4780] 0 4780 12000 497 0 0 0 cdemud
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147355] [ 4834] 0 4834 5751 181 1 0 0 cron
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147358] [ 4911] 0 4911 2859 257 1 0 0 get_os_statisti
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147361] [ 4934] 0 4934 2990 397 1 0 0 cdemud.sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147365] [ 4982] 0 4982 2826 196 1 0 0 process-monitor
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147368] [ 4996] 0 4996 61223 17993 1 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147372] [ 5036] 0 5036 969 71 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147374] [ 5040] 0 5040 2092 1087 0 0 0 haveged
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147377] [ 5153] 0 5153 4023 1288 0 0 0 get_os_statisti
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147380] [ 5291] 0 5291 6291 693 0 0 0 perl
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147385] [ 5478] 74 5478 6979 254 0 0 0 ntpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147387] [ 5527] 0 5527 4727 132 1 0 0 rpcbind
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147390] [ 5538] 0 5538 6811 1377 0 0 0 oxenstored
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147395] [ 5566] 0 5566 8364 232 1 0 0 xenconsoled
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147398] [ 5582] 0 5582 15059 232 0 -17 -1000 sshd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147401] [ 5586] 0 5586 326104 12283 0 0 0 libvirtd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147404] [ 5606] 0 5606 1823 362 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147408] [ 5651] 0 5651 32557 438 0 0 0 PVDriverUpg
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147411] [ 5658] 0 5658 30160 468 0 0 0 xen-numad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147414] [ 5717] 0 5717 969 72 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147417] [ 5942] 0 5942 13742 354 0 0 0 httpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147420] [ 5945] 0 5945 13739 361 1 0 0 httpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147423] [ 5985] 0 5985 2991 383 0 0 0 xen-numad.sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147425] [ 5991] 0 5991 697122 30365 1 0 0 httpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147428] [ 6012] 0 6012 13090 439 0 0 0 uvpvifd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147432] [ 6014] 0 6014 2579 197 0 0 0 uvpalarmd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147435] [ 6214] 0 6214 3176 199 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147438] [ 6215] 0 6215 3176 206 1 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147441] [ 6216] 0 6216 3176 199 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147445] [ 6218] 0 6218 3176 199 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147448] [ 6220] 0 6220 3176 206 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147451] [31686] 0 31686 17093 363 0 0 0 login
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147454] [10759] 0 10759 5395 397 1 0 0 bash
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147458] [ 5759] 0 5759 1051 116 0 0 0 iscsid
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147460] [ 5760] 0 5760 1174 662 0 -17 -1000 iscsid
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147490] [ 6718] 0 6718 1055916 1712 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147494] [ 8699] 0 8699 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147498] [25676] 0 25676 1056926 2722 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147501] [25722] 0 25722 42174 2554 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147504] [27918] 0 27918 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147508] [27966] 0 27966 42174 2549 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147522] [31700] 0 31700 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147526] [31803] 0 31803 1055961 1757 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147529] [31822] 0 31822 1055746 1542 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147532] [31906] 0 31906 53096 5219 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147536] [ 4572] 0 4572 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147540] [ 4733] 0 4733 1055961 1757 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147544] [ 4879] 0 4879 53353 5192 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147551] [10962] 0 10962 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147554] [ 1512] 0 1512 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147557] [ 1561] 0 1561 69030 4872 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147561] [24739] 0 24739 1056168 1964 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147565] [ 9314] 0 9314 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147568] [ 5568] 0 5568 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147573] [17503] 0 17503 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147578] [27798] 0 27798 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147582] [19334] 0 19334 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147587] [32578] 0 32578 1055911 1707 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147594] [ 4288] 0 4288 1055963 1759 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147597] [16309] 0 16309 1056924 2720 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147600] [16360] 0 16360 107523 10407 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147604] [ 4269] 0 4269 1056927 2723 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147607] [ 4290] 0 4290 1057296 3039 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147610] [ 4362] 0 4362 65453 2542 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147618] [ 6618] 0 6618 1056426 2222 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147621] [ 6689] 0 6689 84452 4283 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147624] [ 9083] 0 9083 1055909 1705 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147628] [22568] 0 22568 1055911 1707 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147631] [22666] 0 22666 1055909 1705 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147634] [23156] 0 23156 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147638] [23231] 0 23231 1055914 1710 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147641] [25296] 0 25296 1056796 2592 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147644] [25328] 0 25328 40073 1955 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147647] [26886] 0 26886 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147650] [26928] 0 26928 1058194 3990 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147653] [27003] 0 27003 51716 3660 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147657] [32414] 0 32414 1057167 2454 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147660] [ 1211] 0 1211 1057167 2454 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147664] [ 3879] 0 3879 1057668 2955 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147667] [ 3916] 0 3916 63722 5126 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147670] [ 1588] 0 1588 1055916 1712 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147674] [24458] 0 24458 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147677] [24511] 0 24511 42157 2492 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147680] [25952] 0 25952 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147683] [26072] 0 26072 42431 2543 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147687] [15980] 0 15980 1057442 2729 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147690] [16020] 0 16020 1056794 2081 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147692] [16132] 0 16132 44991 2567 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147704] [ 4539] 0 4539 1056925 2755 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147707] [ 4548] 0 4548 1055774 1570 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147710] [ 4637] 0 4637 71793 7409 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147717] [16448] 0 16448 19077 429 0 0 0 uvp-memoc
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147720] [ 8385] 0 8385 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147723] [27258] 0 27258 1055916 1712 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147727] [ 9074] 0 9074 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147730] [22593] 0 22593 1056925 2721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147733] [22617] 0 22617 1055774 1570 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147736] [22743] 0 22743 43967 2523 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147740] [16681] 0 16681 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147744] [12010] 0 12010 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147766] [19734] 0 19734 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147773] [ 7168] 0 7168 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147776] [ 7182] 0 7182 1056168 1964 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147779] [ 7264] 0 7264 54773 3259 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147784] [15043] 0 15043 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147787] [15071] 0 15071 1057513 3309 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147790] [15147] 0 15147 52469 3645 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147794] [ 1836] 0 1836 1056925 2721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147797] [ 1865] 0 1865 1056474 2270 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147800] [ 1955] 0 1955 52996 3283 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147804] [ 3177] 0 3177 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147806] [ 3197] 0 3197 1057732 3475 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147809] [ 3340] 0 3340 43711 2155 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147812] [13314] 0 13314 1056796 2592 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147815] [13382] 0 13382 54976 3682 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147818] [18756] 0 18756 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147821] [24317] 0 24317 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147824] [24350] 0 24350 73082 5655 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147828] [21288] 0 21288 1055891 1687 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147831] [21322] 0 21322 1056168 1964 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147834] [21466] 0 21466 43693 1748 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147855] [14876] 0 14876 1056117 1913 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147887] [ 2346] 0 2346 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147890] [ 4434] 0 4434 1056876 2672 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147894] [ 4500] 0 4500 1056505 2301 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147897] [ 4568] 0 4568 110024 5565 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147901] [ 3082] 0 3082 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147904] [ 9846] 0 9846 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147907] [ 9894] 0 9894 44973 2120 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147911] [20691] 0 20691 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147915] [32615] 0 32615 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147918] [23616] 0 23616 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147921] [25228] 0 25228 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147924] [25298] 0 25298 72583 5630 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147928] [ 6560] 0 6560 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147931] [ 6811] 0 6811 72538 5581 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147934] [ 7705] 0 7705 1057167 2454 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147939] [22734] 0 22734 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147942] [22788] 0 22788 72533 5593 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147946] [28038] 0 28038 1056730 2017 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147949] [28110] 0 28110 62999 4507 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147952] [17404] 0 17404 1056985 2272 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147956] [20920] 0 20920 1056117 1913 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147959] [28581] 0 28581 1056117 1913 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147963] [31105] 0 31105 1056117 1913 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147967] [20771] 0 20771 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147970] [27980] 0 27980 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147973] [28056] 0 28056 54085 4320 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147977] [21019] 0 21019 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147981] [ 2907] 0 2907 5326 315 1 0 0 cma-sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147984] [ 2963] 0 2963 41618 2289 0 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147987] [29772] 0 29772 28845 4232 1 -16 -941 multipathd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147990] [30125] 0 30125 1523 82 1 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148000] [32716] 0 32716 214492 431 0 0 0 systemvolume
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148003] [12364] 0 12364 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148006] [12581] 0 12581 42158 2523 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148010] [29098] 0 29098 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148013] [29126] 0 29126 53749 3226 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148016] [ 6830] 0 6830 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148018] [ 6872] 0 6872 53749 3291 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148021] [ 9454] 0 9454 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148024] [ 9550] 0 9550 52725 2260 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148037] [23513] 0 23513 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148040] [23588] 0 23588 63477 3622 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148043] [23661] 0 23661 4744 726 0 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148047] [14661] 0 14661 4744 722 0 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148060] [ 949] 0 949 1043 204 0 0 0 iostat
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148063] [ 955] 0 955 1011 202 0 0 0 sar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148065] [ 957] 0 957 1011 183 0 0 0 sadc
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148069] [13494] 0 13494 1055911 1707 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148072] [13959] 0 13959 1055932 1728 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148075] [14027] 0 14027 4744 663 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148078] [14029] 0 14029 4744 675 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148081] [14032] 0 14032 4744 676 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148083] [14045] 0 14045 19175 324 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148109] [14064] 0 14064 4962 296 1 -17 -1000 vif-bridge
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148112] [14119] 0 14119 43424 1710 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148115] [14168] 0 14168 43963 2219 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148118] [14314] 0 14314 2825 235 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148121] [14328] 0 14328 1011 145 1 0 0 ifconfig
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148124] [14365] 0 14365 43424 1678 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148127] [14405] 0 14405 1552 71 0 -17 -1000 ip
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148129] [14629] 0 14629 43424 1695 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148132] [14788] 0 14788 2825 236 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148135] [14790] 0 14790 3722 125 0 0 0 evsadm
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148137] [14791] 0 14791 1054 123 0 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148140] [14792] 0 14792 1054 122 0 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148143] [14973] 0 14973 2825 210 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148146] [14977] 0 14977 2825 246 0 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148149] [14981] 0 14981 4962 271 0 0 0 ntp
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148151] [15001] 0 15001 3761 195 0 0 0 ntpq
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148154] [15377] 0 15377 2825 227 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148157] [15382] 0 15382 3688 77 0 0 0 evsadm
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148159] [15383] 0 15383 1054 123 0 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148162] [16036] 0 16036 2825 224 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148165] [16037] 0 16037 2825 250 0 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148168] [16039] 0 16039 4930 263 0 0 0 vnad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148170] [16042] 0 16042 5324 355 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148173] [16128] 0 16128 43424 1683 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148180] [17196] 0 17196 2825 222 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148183] [17202] 0 17202 2825 219 0 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148186] [17204] 0 17204 4963 300 0 0 0 pmad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148189] [17219] 0 17219 1045 71 0 0 0 ps
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148192] [17325] 0 17325 10890 342 1 0 0 uvpalarm
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148195] [17838] 0 17838 1002 72 1 0 0 sleep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148198] [17882] 0 17882 2825 246 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148200] [17883] 0 17883 2825 224 1 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148203] [17885] 0 17885 4930 275 1 0 0 sysVolumeMonito
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148206] [17888] 0 17888 5325 318 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148209] [17909] 0 17909 43424 1660 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148212] [17975] 0 17975 43963 2247 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148215] [17983] 0 17983 5325 134 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148217] [17984] 0 17984 1045 72 1 0 0 ps
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148220] [17985] 0 17985 1124 117 1 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148223] [17986] 0 17986 1124 118 1 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148225] [17987] 0 17987 1124 119 1 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148228] [17991] 0 17991 43424 1660 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148231] [18052] 0 18052 1002 71 1 0 0 sleep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148233] [18066] 0 18066 40643 516 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148236] [18075] 0 18075 2929 299 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148239] [18117] 0 18117 1115 226 1 0 0 iostat
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148241] [18128] 0 18128 40619 337 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148244] [18178] 0 18178 12539 231 0 0 0 cron
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148247] [18182] 0 18182 2825 216 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148250] [18185] 0 18185 2825 243 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148252] [18186] 0 18186 3190 291 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148255] [18189] 0 18189 985 72 1 0 0 free
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148258] [18192] 0 18192 2825 255 1 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148260] [18203] 0 18203 4963 275 1 0 0 fmad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148263] [18267] 0 18267 2895 220 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148266] [18270] 0 18270 4023 1256 1 0 0 get_os_statisti
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148268] [18271] 0 18271 1007 72 1 0 0 cat
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148271] [18272] 0 18272 1045 71 0 0 0 ps
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148273] [18273] 0 18273 3190 87 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148276] Out of memory: Kill process 5991 (httpd) score 9 or sacrifice child
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148305] Killed process 14314 (sh) total-vm:11300kB, anon-rss:244kB, file-rss:696kB
Jul 10 12:33:03 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdo - tur checker reports path is down
Jul 10 12:33:03 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdk - tur checker reports path is down
Jul 10 12:33:03 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbf - tur checker reports path is down
Jul 10 12:33:04 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddk - tur checker reports path is down
Jul 10 12:33:06 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbq - tur checker reports path is down
Jul 10 12:33:06 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddh - tur checker reports path is down
Jul 10 12:33:08 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdo - tur checker reports path is down
Jul 10 12:33:08 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdk - tur checker reports path is down
Jul 10 12:33:08 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbf - tur checker reports path is down
Jul 10 12:33:09 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddk - tur checker reports path is down
Jul 10 12:33:10 BMS_CNA04 ovs-brcompatd: 322169|monitor|WARN|Monitor: send HEARTBEART_MESSGE again count = 181359600
Jul 10 12:33:11 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbq - tur checker reports path is down
Jul 10 12:33:11 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddh - tur checker reports path is down
Jul 10 12:33:13 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdo - tur checker reports path is down
Jul 10 12:33:13 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdk - tur checker reports path is down
Jul 10 12:33:13 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbf - tur checker reports path is down
^ permalink raw reply [flat|nested] 20+ messages in thread
* [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 7:11 ` Xuzhichuang
0 siblings, 0 replies; 20+ messages in thread
From: Xuzhichuang @ 2015-07-14 7:11 UTC (permalink / raw)
To: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, Songjiangtao (mygirlsjt),
Zhangwei (FF), Qiuxishi
Hi, all
Description of problem:
Recently, one of my Linux system invoked oom-killer, but the system still has much memory, I don't know why the system still invoked oom-killer, anybody can help me to see it, thanks.
Linux kernel version: 3.0.58
Following is the message:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138769] iostat cpuset=/ mems_allowed=0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138773] Pid: 18117, comm: iostat Tainted: P W NX 3.0.58-0.6.6-xen #1
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138775] Call Trace:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138800] [<ffffffff800088be>] dump_trace+0x6e/0x1a0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138810] [<ffffffff803f773d>] dump_stack+0x69/0x6f
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138821] [<ffffffff800dbced>] dump_header+0x9d/0x120
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138826] [<ffffffff800dc505>] oom_kill_process+0x95/0x1a0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138830] [<ffffffff800dc746>] out_of_memory+0x136/0x220
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138834] [<ffffffff800e0fda>] __alloc_pages_slowpath+0x7ba/0x810
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138838] [<ffffffff800e1219>] __alloc_pages_nodemask+0x1e9/0x200
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138845] [<ffffffff8011ae38>] cache_grow+0x348/0x450
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138850] [<ffffffff8011b243>] cache_alloc_refill+0x303/0x4d0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138854] [<ffffffff8011ba70>] __kmalloc+0x1b0/0x290
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138862] [<ffffffff8014c1da>] seq_read+0x13a/0x3b0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138869] [<ffffffff8018a762>] proc_reg_read+0x92/0xe0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138877] [<ffffffff80129877>] vfs_read+0xc7/0x130
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138881] [<ffffffff801299e3>] sys_read+0x53/0xa0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138887] [<ffffffff80402d73>] system_call_fastpath+0x16/0x1b
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138922] [<00007f935f57f4c0>] 0x7f935f57f4bf
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138923] Mem-Info:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138925] DMA per-cpu:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138927] CPU 0: hi: 0, btch: 1 usd: 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138929] CPU 1: hi: 0, btch: 1 usd: 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138930] DMA32 per-cpu:
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138932] CPU 0: hi: 155, btch: 38 usd: 11
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138933] CPU 1: hi: 155, btch: 38 usd: 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138936] active_anon:227111 inactive_anon:10382 isolated_anon:0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138937] active_file:203 inactive_file:189 isolated_file:47
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138938] unevictable:95395 dirty:0 writeback:0 unstable:0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138939] free:247834 slab_reclaimable:18187 slab_unreclaimable:53853
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138940] mapped:11485 shmem:11167 pagetables:0 bounce:0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap = 0kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147221] 788480 pages RAM
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147224] 93755 pages reserved
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147225] 143842 pages shared
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147226] 420430 pages non-shared
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147228] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147242] [ 784] 0 784 4745 725 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147255] [ 3063] 0 3063 3221 290 0 0 0 fma-sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147258] [ 3065] 0 3065 1002 157 0 0 0 acpid
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147262] [ 3155] 0 3155 969 71 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147265] [ 3158] 0 3158 4990 345 1 0 0 syslog-ng
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147268] [ 3173] 0 3173 2455 329 0 0 0 irqbalance
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147271] [ 3176] 0 3176 1069 143 1 0 0 klogd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147273] [ 3183] 0 3183 3221 290 0 0 0 pma-sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147277] [ 3186] 100 3186 5357 136 1 0 0 dbus-daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147280] [ 3223] 0 3223 21330 376 1 0 0 ovsdb-server
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147283] [ 3268] 0 3268 1490 68 1 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147286] [ 3271] 0 3271 1523 115 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147289] [ 3273] 0 3273 969 72 0 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147292] [ 3317] 0 3317 28331 4059 0 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147295] [ 3320] 0 3320 21593 962 1 0 0 ovs-vswitchd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147298] [ 3322] 0 3322 1858 387 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147301] [ 3341] 103 3341 9408 1502 0 0 0 hald
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147304] [ 3348] 0 3348 4886 129 1 0 0 ovs-brcompatd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147307] [ 3350] 0 3350 1688 203 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147310] [ 3352] 0 3352 38107 2004 1 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147315] [ 3356] 0 3356 26731 336 0 0 0 console-kit-dae
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147318] [ 3357] 0 3357 78512 3688 0 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147321] [ 3364] 0 3364 4576 187 0 0 0 hald-runner
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147324] [ 3433] 0 3433 1622 176 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147327] [ 3444] 0 3444 27681 463 0 -17 -1000 evs_service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147330] [ 3488] 0 3488 1679 236 1 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147333] [ 3628] 0 3628 17899 2152 1 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147336] [ 3631] 0 3631 969 72 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147339] [ 3683] 0 3683 3200 185 0 0 0 calcload
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147342] [ 3723] 0 3723 8701 215 0 0 0 hald-addon-inpu
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147346] [ 3746] 0 3746 969 71 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147348] [ 3866] 103 3866 9133 168 1 0 0 hald-addon-acpi
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147351] [ 4780] 0 4780 12000 497 0 0 0 cdemud
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147355] [ 4834] 0 4834 5751 181 1 0 0 cron
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147358] [ 4911] 0 4911 2859 257 1 0 0 get_os_statisti
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147361] [ 4934] 0 4934 2990 397 1 0 0 cdemud.sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147365] [ 4982] 0 4982 2826 196 1 0 0 process-monitor
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147368] [ 4996] 0 4996 61223 17993 1 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147372] [ 5036] 0 5036 969 71 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147374] [ 5040] 0 5040 2092 1087 0 0 0 haveged
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147377] [ 5153] 0 5153 4023 1288 0 0 0 get_os_statisti
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147380] [ 5291] 0 5291 6291 693 0 0 0 perl
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147385] [ 5478] 74 5478 6979 254 0 0 0 ntpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147387] [ 5527] 0 5527 4727 132 1 0 0 rpcbind
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147390] [ 5538] 0 5538 6811 1377 0 0 0 oxenstored
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147395] [ 5566] 0 5566 8364 232 1 0 0 xenconsoled
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147398] [ 5582] 0 5582 15059 232 0 -17 -1000 sshd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147401] [ 5586] 0 5586 326104 12283 0 0 0 libvirtd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147404] [ 5606] 0 5606 1823 362 0 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147408] [ 5651] 0 5651 32557 438 0 0 0 PVDriverUpg
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147411] [ 5658] 0 5658 30160 468 0 0 0 xen-numad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147414] [ 5717] 0 5717 969 72 1 0 0 startpar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147417] [ 5942] 0 5942 13742 354 0 0 0 httpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147420] [ 5945] 0 5945 13739 361 1 0 0 httpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147423] [ 5985] 0 5985 2991 383 0 0 0 xen-numad.sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147425] [ 5991] 0 5991 697122 30365 1 0 0 httpd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147428] [ 6012] 0 6012 13090 439 0 0 0 uvpvifd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147432] [ 6014] 0 6014 2579 197 0 0 0 uvpalarmd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147435] [ 6214] 0 6214 3176 199 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147438] [ 6215] 0 6215 3176 206 1 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147441] [ 6216] 0 6216 3176 199 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147445] [ 6218] 0 6218 3176 199 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147448] [ 6220] 0 6220 3176 206 0 0 0 mingetty
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147451] [31686] 0 31686 17093 363 0 0 0 login
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147454] [10759] 0 10759 5395 397 1 0 0 bash
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147458] [ 5759] 0 5759 1051 116 0 0 0 iscsid
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147460] [ 5760] 0 5760 1174 662 0 -17 -1000 iscsid
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147490] [ 6718] 0 6718 1055916 1712 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147494] [ 8699] 0 8699 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147498] [25676] 0 25676 1056926 2722 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147501] [25722] 0 25722 42174 2554 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147504] [27918] 0 27918 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147508] [27966] 0 27966 42174 2549 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147522] [31700] 0 31700 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147526] [31803] 0 31803 1055961 1757 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147529] [31822] 0 31822 1055746 1542 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147532] [31906] 0 31906 53096 5219 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147536] [ 4572] 0 4572 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147540] [ 4733] 0 4733 1055961 1757 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147544] [ 4879] 0 4879 53353 5192 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147551] [10962] 0 10962 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147554] [ 1512] 0 1512 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147557] [ 1561] 0 1561 69030 4872 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147561] [24739] 0 24739 1056168 1964 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147565] [ 9314] 0 9314 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147568] [ 5568] 0 5568 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147573] [17503] 0 17503 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147578] [27798] 0 27798 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147582] [19334] 0 19334 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147587] [32578] 0 32578 1055911 1707 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147594] [ 4288] 0 4288 1055963 1759 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147597] [16309] 0 16309 1056924 2720 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147600] [16360] 0 16360 107523 10407 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147604] [ 4269] 0 4269 1056927 2723 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147607] [ 4290] 0 4290 1057296 3039 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147610] [ 4362] 0 4362 65453 2542 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147618] [ 6618] 0 6618 1056426 2222 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147621] [ 6689] 0 6689 84452 4283 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147624] [ 9083] 0 9083 1055909 1705 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147628] [22568] 0 22568 1055911 1707 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147631] [22666] 0 22666 1055909 1705 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147634] [23156] 0 23156 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147638] [23231] 0 23231 1055914 1710 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147641] [25296] 0 25296 1056796 2592 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147644] [25328] 0 25328 40073 1955 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147647] [26886] 0 26886 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147650] [26928] 0 26928 1058194 3990 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147653] [27003] 0 27003 51716 3660 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147657] [32414] 0 32414 1057167 2454 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147660] [ 1211] 0 1211 1057167 2454 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147664] [ 3879] 0 3879 1057668 2955 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147667] [ 3916] 0 3916 63722 5126 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147670] [ 1588] 0 1588 1055916 1712 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147674] [24458] 0 24458 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147677] [24511] 0 24511 42157 2492 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147680] [25952] 0 25952 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147683] [26072] 0 26072 42431 2543 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147687] [15980] 0 15980 1057442 2729 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147690] [16020] 0 16020 1056794 2081 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147692] [16132] 0 16132 44991 2567 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147704] [ 4539] 0 4539 1056925 2755 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147707] [ 4548] 0 4548 1055774 1570 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147710] [ 4637] 0 4637 71793 7409 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147717] [16448] 0 16448 19077 429 0 0 0 uvp-memoc
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147720] [ 8385] 0 8385 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147723] [27258] 0 27258 1055916 1712 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147727] [ 9074] 0 9074 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147730] [22593] 0 22593 1056925 2721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147733] [22617] 0 22617 1055774 1570 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147736] [22743] 0 22743 43967 2523 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147740] [16681] 0 16681 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147744] [12010] 0 12010 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147766] [19734] 0 19734 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147773] [ 7168] 0 7168 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147776] [ 7182] 0 7182 1056168 1964 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147779] [ 7264] 0 7264 54773 3259 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147784] [15043] 0 15043 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147787] [15071] 0 15071 1057513 3309 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147790] [15147] 0 15147 52469 3645 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147794] [ 1836] 0 1836 1056925 2721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147797] [ 1865] 0 1865 1056474 2270 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147800] [ 1955] 0 1955 52996 3283 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147804] [ 3177] 0 3177 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147806] [ 3197] 0 3197 1057732 3475 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147809] [ 3340] 0 3340 43711 2155 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147812] [13314] 0 13314 1056796 2592 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147815] [13382] 0 13382 54976 3682 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147818] [18756] 0 18756 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147821] [24317] 0 24317 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147824] [24350] 0 24350 73082 5655 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147828] [21288] 0 21288 1055891 1687 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147831] [21322] 0 21322 1056168 1964 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147834] [21466] 0 21466 43693 1748 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147855] [14876] 0 14876 1056117 1913 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147887] [ 2346] 0 2346 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147890] [ 4434] 0 4434 1056876 2672 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147894] [ 4500] 0 4500 1056505 2301 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147897] [ 4568] 0 4568 110024 5565 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147901] [ 3082] 0 3082 1055916 1712 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147904] [ 9846] 0 9846 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147907] [ 9894] 0 9894 44973 2120 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147911] [20691] 0 20691 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147915] [32615] 0 32615 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147918] [23616] 0 23616 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147921] [25228] 0 25228 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147924] [25298] 0 25298 72583 5630 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147928] [ 6560] 0 6560 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147931] [ 6811] 0 6811 72538 5581 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147934] [ 7705] 0 7705 1057167 2454 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147939] [22734] 0 22734 1055929 1725 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147942] [22788] 0 22788 72533 5593 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147946] [28038] 0 28038 1056730 2017 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147949] [28110] 0 28110 62999 4507 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147952] [17404] 0 17404 1056985 2272 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147956] [20920] 0 20920 1056117 1913 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147959] [28581] 0 28581 1056117 1913 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147963] [31105] 0 31105 1056117 1913 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147967] [20771] 0 20771 1055925 1721 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147970] [27980] 0 27980 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147973] [28056] 0 28056 54085 4320 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147977] [21019] 0 21019 1055925 1721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147981] [ 2907] 0 2907 5326 315 1 0 0 cma-sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147984] [ 2963] 0 2963 41618 2289 0 0 0 python
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147987] [29772] 0 29772 28845 4232 1 -16 -941 multipathd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.147990] [30125] 0 30125 1523 82 1 0 0 monitor_daemon
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148000] [32716] 0 32716 214492 431 0 0 0 systemvolume
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148003] [12364] 0 12364 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148006] [12581] 0 12581 42158 2523 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148010] [29098] 0 29098 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148013] [29126] 0 29126 53749 3226 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148016] [ 6830] 0 6830 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148018] [ 6872] 0 6872 53749 3291 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148021] [ 9454] 0 9454 1056925 2721 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148024] [ 9550] 0 9550 52725 2260 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148037] [23513] 0 23513 1055929 1725 1 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148040] [23588] 0 23588 63477 3622 0 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148043] [23661] 0 23661 4744 726 0 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148047] [14661] 0 14661 4744 722 0 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148060] [ 949] 0 949 1043 204 0 0 0 iostat
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148063] [ 955] 0 955 1011 202 0 0 0 sar
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148065] [ 957] 0 957 1011 183 0 0 0 sadc
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148069] [13494] 0 13494 1055911 1707 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148072] [13959] 0 13959 1055932 1728 0 0 0 tapdisk2
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148075] [14027] 0 14027 4744 663 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148078] [14029] 0 14029 4744 675 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148081] [14032] 0 14032 4744 676 1 -17 -1000 udevd
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148083] [14045] 0 14045 19175 324 1 0 0 qemu-system-i38
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148109] [14064] 0 14064 4962 296 1 -17 -1000 vif-bridge
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148112] [14119] 0 14119 43424 1710 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148115] [14168] 0 14168 43963 2219 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148118] [14314] 0 14314 2825 235 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148121] [14328] 0 14328 1011 145 1 0 0 ifconfig
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148124] [14365] 0 14365 43424 1678 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148127] [14405] 0 14405 1552 71 0 -17 -1000 ip
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148129] [14629] 0 14629 43424 1695 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148132] [14788] 0 14788 2825 236 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148135] [14790] 0 14790 3722 125 0 0 0 evsadm
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148137] [14791] 0 14791 1054 123 0 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148140] [14792] 0 14792 1054 122 0 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148143] [14973] 0 14973 2825 210 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148146] [14977] 0 14977 2825 246 0 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148149] [14981] 0 14981 4962 271 0 0 0 ntp
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148151] [15001] 0 15001 3761 195 0 0 0 ntpq
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148154] [15377] 0 15377 2825 227 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148157] [15382] 0 15382 3688 77 0 0 0 evsadm
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148159] [15383] 0 15383 1054 123 0 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148162] [16036] 0 16036 2825 224 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148165] [16037] 0 16037 2825 250 0 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148168] [16039] 0 16039 4930 263 0 0 0 vnad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148170] [16042] 0 16042 5324 355 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148173] [16128] 0 16128 43424 1683 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148180] [17196] 0 17196 2825 222 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148183] [17202] 0 17202 2825 219 0 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148186] [17204] 0 17204 4963 300 0 0 0 pmad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148189] [17219] 0 17219 1045 71 0 0 0 ps
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148192] [17325] 0 17325 10890 342 1 0 0 uvpalarm
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148195] [17838] 0 17838 1002 72 1 0 0 sleep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148198] [17882] 0 17882 2825 246 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148200] [17883] 0 17883 2825 224 1 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148203] [17885] 0 17885 4930 275 1 0 0 sysVolumeMonito
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148206] [17888] 0 17888 5325 318 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148209] [17909] 0 17909 43424 1660 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148212] [17975] 0 17975 43963 2247 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148215] [17983] 0 17983 5325 134 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148217] [17984] 0 17984 1045 72 1 0 0 ps
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148220] [17985] 0 17985 1124 117 1 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148223] [17986] 0 17986 1124 118 1 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148225] [17987] 0 17987 1124 119 1 0 0 grep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148228] [17991] 0 17991 43424 1660 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148231] [18052] 0 18052 1002 71 1 0 0 sleep
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148233] [18066] 0 18066 40643 516 0 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148236] [18075] 0 18075 2929 299 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148239] [18117] 0 18117 1115 226 1 0 0 iostat
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148241] [18128] 0 18128 40619 337 1 0 0 NCclient
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148244] [18178] 0 18178 12539 231 0 0 0 cron
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148247] [18182] 0 18182 2825 216 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148250] [18185] 0 18185 2825 243 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148252] [18186] 0 18186 3190 291 1 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148255] [18189] 0 18189 985 72 1 0 0 free
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148258] [18192] 0 18192 2825 255 1 0 0 service
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148260] [18203] 0 18203 4963 275 1 0 0 fmad
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148263] [18267] 0 18267 2895 220 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148266] [18270] 0 18270 4023 1256 1 0 0 get_os_statisti
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148268] [18271] 0 18271 1007 72 1 0 0 cat
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148271] [18272] 0 18272 1045 71 0 0 0 ps
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148273] [18273] 0 18273 3190 87 0 0 0 sh
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148276] Out of memory: Kill process 5991 (httpd) score 9 or sacrifice child
Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.148305] Killed process 14314 (sh) total-vm:11300kB, anon-rss:244kB, file-rss:696kB
Jul 10 12:33:03 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdo - tur checker reports path is down
Jul 10 12:33:03 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdk - tur checker reports path is down
Jul 10 12:33:03 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbf - tur checker reports path is down
Jul 10 12:33:04 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddk - tur checker reports path is down
Jul 10 12:33:06 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbq - tur checker reports path is down
Jul 10 12:33:06 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddh - tur checker reports path is down
Jul 10 12:33:08 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdo - tur checker reports path is down
Jul 10 12:33:08 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdk - tur checker reports path is down
Jul 10 12:33:08 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbf - tur checker reports path is down
Jul 10 12:33:09 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddk - tur checker reports path is down
Jul 10 12:33:10 BMS_CNA04 ovs-brcompatd: 322169|monitor|WARN|Monitor: send HEARTBEART_MESSGE again count = 181359600
Jul 10 12:33:11 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbq - tur checker reports path is down
Jul 10 12:33:11 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sddh - tur checker reports path is down
Jul 10 12:33:13 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdo - tur checker reports path is down
Jul 10 12:33:13 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdk - tur checker reports path is down
Jul 10 12:33:13 BMS_CNA04 multipathd: 361051721000d0afe22e0594200000002: sdbf - tur checker reports path is down
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 7:11 ` Xuzhichuang
@ 2015-07-14 8:15 ` Michal Hocko
-1 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2015-07-14 8:15 UTC (permalink / raw)
To: Xuzhichuang
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> Hi, all
>
> Description of problem:
>
> Recently, one of my Linux system invoked oom-killer, but the system
> still has much memory, I don't know why the system still invoked
> oom-killer, anybody can help me to see it, thanks.
>
> Linux kernel version: 3.0.58
>
> Following is the message:
>
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
[...]
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
You are well above watermarks but note that you have basically no
pages on the file LRU and you have _no swap_ so the anon memory is
unreclaimable. There is still around 72M of reclaimable slab but that
could be hard to reclaim due to internal fragmentation. The allocation
request is GFP_KERNEL so the slab shrinkers shouldn't back off due to
__GFP_FS restrictions.
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
Moreover your allocation request was oreder 2 and you do not have much
memory there because most of the free memory is in order-0-2.
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap = 0kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
^^^^^^^^^^^^^^^^
So I am not surprised about the oom killer much.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 8:15 ` Michal Hocko
0 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2015-07-14 8:15 UTC (permalink / raw)
To: Xuzhichuang
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> Hi, all
>
> Description of problem:
>
> Recently, one of my Linux system invoked oom-killer, but the system
> still has much memory, I don't know why the system still invoked
> oom-killer, anybody can help me to see it, thanks.
>
> Linux kernel version: 3.0.58
>
> Following is the message:
>
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
[...]
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
You are well above watermarks but note that you have basically no
pages on the file LRU and you have _no swap_ so the anon memory is
unreclaimable. There is still around 72M of reclaimable slab but that
could be hard to reclaim due to internal fragmentation. The allocation
request is GFP_KERNEL so the slab shrinkers shouldn't back off due to
__GFP_FS restrictions.
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
Moreover your allocation request was oreder 2 and you do not have much
memory there because most of the free memory is in order-0-2.
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap = 0kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
^^^^^^^^^^^^^^^^
So I am not surprised about the oom killer much.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 8:15 ` Michal Hocko
@ 2015-07-14 8:42 ` Xishi Qiu
-1 siblings, 0 replies; 20+ messages in thread
From: Xishi Qiu @ 2015-07-14 8:42 UTC (permalink / raw)
To: Michal Hocko
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On 2015/7/14 16:15, Michal Hocko wrote:
> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
>> Hi, all
>>
>> Description of problem:
>>
>> Recently, one of my Linux system invoked oom-killer, but the system
>> still has much memory, I don't know why the system still invoked
>> oom-killer, anybody can help me to see it, thanks.
>>
>> Linux kernel version: 3.0.58
>>
>> Following is the message:
>>
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
> [...]
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
>
> You are well above watermarks but note that you have basically no
> pages on the file LRU and you have _no swap_ so the anon memory is
> unreclaimable. There is still around 72M of reclaimable slab but that
> could be hard to reclaim due to internal fragmentation. The allocation
> request is GFP_KERNEL so the slab shrinkers shouldn't back off due to
> __GFP_FS restrictions.
>
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>
> Moreover your allocation request was oreder 2 and you do not have much
> memory there because most of the free memory is in order-0-2.
>
Hi Michal,
order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
so you mean this large buddy block was reclaimed during the moment of oom and
print, right?
Thanks,
Xishi Qiu
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap = 0kB
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
> ^^^^^^^^^^^^^^^^
> So I am not surprised about the oom killer much.
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 8:42 ` Xishi Qiu
0 siblings, 0 replies; 20+ messages in thread
From: Xishi Qiu @ 2015-07-14 8:42 UTC (permalink / raw)
To: Michal Hocko
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On 2015/7/14 16:15, Michal Hocko wrote:
> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
>> Hi, all
>>
>> Description of problem:
>>
>> Recently, one of my Linux system invoked oom-killer, but the system
>> still has much memory, I don't know why the system still invoked
>> oom-killer, anybody can help me to see it, thanks.
>>
>> Linux kernel version: 3.0.58
>>
>> Following is the message:
>>
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
> [...]
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
>
> You are well above watermarks but note that you have basically no
> pages on the file LRU and you have _no swap_ so the anon memory is
> unreclaimable. There is still around 72M of reclaimable slab but that
> could be hard to reclaim due to internal fragmentation. The allocation
> request is GFP_KERNEL so the slab shrinkers shouldn't back off due to
> __GFP_FS restrictions.
>
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>
> Moreover your allocation request was oreder 2 and you do not have much
> memory there because most of the free memory is in order-0-2.
>
Hi Michal,
order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
so you mean this large buddy block was reclaimed during the moment of oom and
print, right?
Thanks,
Xishi Qiu
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap = 0kB
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
> ^^^^^^^^^^^^^^^^
> So I am not surprised about the oom killer much.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 8:42 ` Xishi Qiu
@ 2015-07-14 9:00 ` Michal Hocko
-1 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2015-07-14 9:00 UTC (permalink / raw)
To: Xishi Qiu
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
> On 2015/7/14 16:15, Michal Hocko wrote:
>
> > On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
[...]
> >> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> >
> > Moreover your allocation request was oreder 2 and you do not have much
> > memory there because most of the free memory is in order-0-2.
> >
>
> Hi Michal,
>
> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
> so you mean this large buddy block was reclaimed during the moment of oom and
> print, right?
Not really. Those high order blocks are inaccessible for your GFP_KERNEL
allocation. See __zone_watermark_ok.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 9:00 ` Michal Hocko
0 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2015-07-14 9:00 UTC (permalink / raw)
To: Xishi Qiu
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
> On 2015/7/14 16:15, Michal Hocko wrote:
>
> > On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
[...]
> >> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> >
> > Moreover your allocation request was oreder 2 and you do not have much
> > memory there because most of the free memory is in order-0-2.
> >
>
> Hi Michal,
>
> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
> so you mean this large buddy block was reclaimed during the moment of oom and
> print, right?
Not really. Those high order blocks are inaccessible for your GFP_KERNEL
allocation. See __zone_watermark_ok.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 9:00 ` Michal Hocko
@ 2015-07-14 9:14 ` Xishi Qiu
-1 siblings, 0 replies; 20+ messages in thread
From: Xishi Qiu @ 2015-07-14 9:14 UTC (permalink / raw)
To: Michal Hocko
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On 2015/7/14 17:00, Michal Hocko wrote:
> On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
>> On 2015/7/14 16:15, Michal Hocko wrote:
>>
>>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> [...]
>>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>>>
>>> Moreover your allocation request was oreder 2 and you do not have much
>>> memory there because most of the free memory is in order-0-2.
>>>
>>
>> Hi Michal,
>>
>> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
>> so you mean this large buddy block was reclaimed during the moment of oom and
>> print, right?
>
> Not really. Those high order blocks are inaccessible for your GFP_KERNEL
> allocation. See __zone_watermark_ok.
>
I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
Thanks,
Xishi Qiu
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 9:14 ` Xishi Qiu
0 siblings, 0 replies; 20+ messages in thread
From: Xishi Qiu @ 2015-07-14 9:14 UTC (permalink / raw)
To: Michal Hocko
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On 2015/7/14 17:00, Michal Hocko wrote:
> On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
>> On 2015/7/14 16:15, Michal Hocko wrote:
>>
>>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> [...]
>>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>>>
>>> Moreover your allocation request was oreder 2 and you do not have much
>>> memory there because most of the free memory is in order-0-2.
>>>
>>
>> Hi Michal,
>>
>> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
>> so you mean this large buddy block was reclaimed during the moment of oom and
>> print, right?
>
> Not really. Those high order blocks are inaccessible for your GFP_KERNEL
> allocation. See __zone_watermark_ok.
>
I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
Thanks,
Xishi Qiu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 9:14 ` Xishi Qiu
@ 2015-07-14 9:28 ` Michal Hocko
-1 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2015-07-14 9:28 UTC (permalink / raw)
To: Xishi Qiu
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On Tue 14-07-15 17:14:40, Xishi Qiu wrote:
> On 2015/7/14 17:00, Michal Hocko wrote:
>
> > On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
> >> On 2015/7/14 16:15, Michal Hocko wrote:
> >>
> >>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> > [...]
> >>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> >>>
> >>> Moreover your allocation request was oreder 2 and you do not have much
> >>> memory there because most of the free memory is in order-0-2.
> >>>
> >>
> >> Hi Michal,
> >>
> >> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
> >> so you mean this large buddy block was reclaimed during the moment of oom and
> >> print, right?
> >
> > Not really. Those high order blocks are inaccessible for your GFP_KERNEL
> > allocation. See __zone_watermark_ok.
> >
>
> I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
No. The watermark is calculated per order. And you have almost all the
free memory in the lower orders. From a quick glance it seems that even
order-1 allocations wouldn't fit into min watermark.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 9:28 ` Michal Hocko
0 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2015-07-14 9:28 UTC (permalink / raw)
To: Xishi Qiu
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On Tue 14-07-15 17:14:40, Xishi Qiu wrote:
> On 2015/7/14 17:00, Michal Hocko wrote:
>
> > On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
> >> On 2015/7/14 16:15, Michal Hocko wrote:
> >>
> >>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> > [...]
> >>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> >>>
> >>> Moreover your allocation request was oreder 2 and you do not have much
> >>> memory there because most of the free memory is in order-0-2.
> >>>
> >>
> >> Hi Michal,
> >>
> >> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
> >> so you mean this large buddy block was reclaimed during the moment of oom and
> >> print, right?
> >
> > Not really. Those high order blocks are inaccessible for your GFP_KERNEL
> > allocation. See __zone_watermark_ok.
> >
>
> I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
No. The watermark is calculated per order. And you have almost all the
free memory in the lower orders. From a quick glance it seems that even
order-1 allocations wouldn't fit into min watermark.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 9:28 ` Michal Hocko
@ 2015-07-14 9:50 ` Xishi Qiu
-1 siblings, 0 replies; 20+ messages in thread
From: Xishi Qiu @ 2015-07-14 9:50 UTC (permalink / raw)
To: Michal Hocko
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On 2015/7/14 17:28, Michal Hocko wrote:
> On Tue 14-07-15 17:14:40, Xishi Qiu wrote:
>> On 2015/7/14 17:00, Michal Hocko wrote:
>>
>>> On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
>>>> On 2015/7/14 16:15, Michal Hocko wrote:
>>>>
>>>>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
>>> [...]
>>>>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>>>>>
>>>>> Moreover your allocation request was oreder 2 and you do not have much
>>>>> memory there because most of the free memory is in order-0-2.
>>>>>
>>>>
>>>> Hi Michal,
>>>>
>>>> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
>>>> so you mean this large buddy block was reclaimed during the moment of oom and
>>>> print, right?
>>>
>>> Not really. Those high order blocks are inaccessible for your GFP_KERNEL
>>> allocation. See __zone_watermark_ok.
>>>
>>
>> I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
>
> No. The watermark is calculated per order. And you have almost all the
> free memory in the lower orders. From a quick glance it seems that even
> order-1 allocations wouldn't fit into min watermark.
So we can't alloc memory because it doesn't fit the watermark, and
the large buddy blocks maybe from reserved, or maybe not, right?
Thanks,
Xishi Qiu
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-14 9:50 ` Xishi Qiu
0 siblings, 0 replies; 20+ messages in thread
From: Xishi Qiu @ 2015-07-14 9:50 UTC (permalink / raw)
To: Michal Hocko
Cc: Xuzhichuang, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF)
On 2015/7/14 17:28, Michal Hocko wrote:
> On Tue 14-07-15 17:14:40, Xishi Qiu wrote:
>> On 2015/7/14 17:00, Michal Hocko wrote:
>>
>>> On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
>>>> On 2015/7/14 16:15, Michal Hocko wrote:
>>>>
>>>>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
>>> [...]
>>>>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>>>>>
>>>>> Moreover your allocation request was oreder 2 and you do not have much
>>>>> memory there because most of the free memory is in order-0-2.
>>>>>
>>>>
>>>> Hi Michal,
>>>>
>>>> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB,
>>>> so you mean this large buddy block was reclaimed during the moment of oom and
>>>> print, right?
>>>
>>> Not really. Those high order blocks are inaccessible for your GFP_KERNEL
>>> allocation. See __zone_watermark_ok.
>>>
>>
>> I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
>
> No. The watermark is calculated per order. And you have almost all the
> free memory in the lower orders. From a quick glance it seems that even
> order-1 allocations wouldn't fit into min watermark.
So we can't alloc memory because it doesn't fit the watermark, and
the large buddy blocks maybe from reserved, or maybe not, right?
Thanks,
Xishi Qiu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-14 7:11 ` Xuzhichuang
@ 2015-07-15 0:09 ` David Rientjes
-1 siblings, 0 replies; 20+ messages in thread
From: David Rientjes @ 2015-07-15 0:09 UTC (permalink / raw)
To: Xuzhichuang
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
[-- Attachment #1: Type: TEXT/PLAIN, Size: 5696 bytes --]
On Tue, 14 Jul 2015, Xuzhichuang wrote:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138769] iostat cpuset=/ mems_allowed=0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138773] Pid: 18117, comm: iostat Tainted: P W NX 3.0.58-0.6.6-xen #1
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138775] Call Trace:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138800] [<ffffffff800088be>] dump_trace+0x6e/0x1a0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138810] [<ffffffff803f773d>] dump_stack+0x69/0x6f
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138821] [<ffffffff800dbced>] dump_header+0x9d/0x120
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138826] [<ffffffff800dc505>] oom_kill_process+0x95/0x1a0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138830] [<ffffffff800dc746>] out_of_memory+0x136/0x220
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138834] [<ffffffff800e0fda>] __alloc_pages_slowpath+0x7ba/0x810
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138838] [<ffffffff800e1219>] __alloc_pages_nodemask+0x1e9/0x200
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138845] [<ffffffff8011ae38>] cache_grow+0x348/0x450
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138850] [<ffffffff8011b243>] cache_alloc_refill+0x303/0x4d0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138854] [<ffffffff8011ba70>] __kmalloc+0x1b0/0x290
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138862] [<ffffffff8014c1da>] seq_read+0x13a/0x3b0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138869] [<ffffffff8018a762>] proc_reg_read+0x92/0xe0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138877] [<ffffffff80129877>] vfs_read+0xc7/0x130
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138881] [<ffffffff801299e3>] sys_read+0x53/0xa0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138887] [<ffffffff80402d73>] system_call_fastpath+0x16/0x1b
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138922] [<00007f935f57f4c0>] 0x7f935f57f4bf
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138923] Mem-Info:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138925] DMA per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138927] CPU 0: hi: 0, btch: 1 usd: 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138929] CPU 1: hi: 0, btch: 1 usd: 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138930] DMA32 per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138932] CPU 0: hi: 155, btch: 38 usd: 11
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138933] CPU 1: hi: 155, btch: 38 usd: 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138936] active_anon:227111 inactive_anon:10382 isolated_anon:0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138937] active_file:203 inactive_file:189 isolated_file:47
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138938] unevictable:95395 dirty:0 writeback:0 unstable:0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138939] free:247834 slab_reclaimable:18187 slab_unreclaimable:53853
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138940] mapped:11485 shmem:11167 pagetables:0 bounce:0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
The problem is most of your memory for ZONE_DMA32 is available only in
sizes of order-0 and order-1 and the slab allocator is trying to allocate
order-2 memory with no possibility of fallback to a smaller order.
You're running on a 3.0.58 kernel, but the watermark calculation should be
the same in recent kernels.
If you follow the logic of __zone_watermark_ok(), which uses the same
watermarks as printed above, the min watermark for this zone is 1751 pages
and the total zone free pages is 247588. Discounting order-0 memory,
there are only 59075 pages free with a min watermark of 875 pages.
Discounting order-1 memory, there are 157 pages free with a min watermark
of 437 pages. This is where your allocation fails. Even though the zone
has 672KB of memory available, the per-order watermark fails.
The only option you have to avoid this other than changing your workload
is to alter lowmem_reserve_ratio, see Documentation/sysctl/vm.txt. You
have 916KB of memory in ZONE_DMA that could be used for this allocation if
it wasn't reserved for DMA allocations.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-15 0:09 ` David Rientjes
0 siblings, 0 replies; 20+ messages in thread
From: David Rientjes @ 2015-07-15 0:09 UTC (permalink / raw)
To: Xuzhichuang
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
[-- Attachment #1: Type: TEXT/PLAIN, Size: 5696 bytes --]
On Tue, 14 Jul 2015, Xuzhichuang wrote:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138769] iostat cpuset=/ mems_allowed=0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138773] Pid: 18117, comm: iostat Tainted: P W NX 3.0.58-0.6.6-xen #1
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138775] Call Trace:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138800] [<ffffffff800088be>] dump_trace+0x6e/0x1a0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138810] [<ffffffff803f773d>] dump_stack+0x69/0x6f
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138821] [<ffffffff800dbced>] dump_header+0x9d/0x120
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138826] [<ffffffff800dc505>] oom_kill_process+0x95/0x1a0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138830] [<ffffffff800dc746>] out_of_memory+0x136/0x220
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138834] [<ffffffff800e0fda>] __alloc_pages_slowpath+0x7ba/0x810
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138838] [<ffffffff800e1219>] __alloc_pages_nodemask+0x1e9/0x200
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138845] [<ffffffff8011ae38>] cache_grow+0x348/0x450
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138850] [<ffffffff8011b243>] cache_alloc_refill+0x303/0x4d0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138854] [<ffffffff8011ba70>] __kmalloc+0x1b0/0x290
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138862] [<ffffffff8014c1da>] seq_read+0x13a/0x3b0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138869] [<ffffffff8018a762>] proc_reg_read+0x92/0xe0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138877] [<ffffffff80129877>] vfs_read+0xc7/0x130
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138881] [<ffffffff801299e3>] sys_read+0x53/0xa0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138887] [<ffffffff80402d73>] system_call_fastpath+0x16/0x1b
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138922] [<00007f935f57f4c0>] 0x7f935f57f4bf
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138923] Mem-Info:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138925] DMA per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138927] CPU 0: hi: 0, btch: 1 usd: 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138929] CPU 1: hi: 0, btch: 1 usd: 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138930] DMA32 per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138932] CPU 0: hi: 155, btch: 38 usd: 11
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138933] CPU 1: hi: 155, btch: 38 usd: 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138936] active_anon:227111 inactive_anon:10382 isolated_anon:0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138937] active_file:203 inactive_file:189 isolated_file:47
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138938] unevictable:95395 dirty:0 writeback:0 unstable:0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138939] free:247834 slab_reclaimable:18187 slab_unreclaimable:53853
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138940] mapped:11485 shmem:11167 pagetables:0 bounce:0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
The problem is most of your memory for ZONE_DMA32 is available only in
sizes of order-0 and order-1 and the slab allocator is trying to allocate
order-2 memory with no possibility of fallback to a smaller order.
You're running on a 3.0.58 kernel, but the watermark calculation should be
the same in recent kernels.
If you follow the logic of __zone_watermark_ok(), which uses the same
watermarks as printed above, the min watermark for this zone is 1751 pages
and the total zone free pages is 247588. Discounting order-0 memory,
there are only 59075 pages free with a min watermark of 875 pages.
Discounting order-1 memory, there are 157 pages free with a min watermark
of 437 pages. This is where your allocation fails. Even though the zone
has 672KB of memory available, the per-order watermark fails.
The only option you have to avoid this other than changing your workload
is to alter lowmem_reserve_ratio, see Documentation/sysctl/vm.txt. You
have 916KB of memory in ZONE_DMA that could be used for this allocation if
it wasn't reserved for DMA allocations.
^ permalink raw reply [flat|nested] 20+ messages in thread
* 答复: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-15 0:09 ` David Rientjes
@ 2015-07-15 1:46 ` Xuzhichuang
-1 siblings, 0 replies; 20+ messages in thread
From: Xuzhichuang @ 2015-07-15 1:46 UTC (permalink / raw)
To: David Rientjes
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 6945 bytes --]
Hi,
Thanks for your replying.
According to the OOM message, OOM killer is invoked by the function seq_read, I found two patches in the latest kernel which can be avoid or fixed this problem.
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=058504edd02667eef8fac9be27ab3ea74332e9b4
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=5cec38ac866bfb8775638e71a86e4d8cac30caae
As the patches said, it changed the seq_file code fallback to vmalloc allocations if kmalloc failed, instead of OOM kill processes.
-----邮件原件-----
å‘件人: David Rientjes [mailto:rientjes@google.com]
å‘é€æ—¶é—´: 2015å¹´7月15æ—¥ 8:10
收件人: Xuzhichuang
抄é€: linux-mm@kvack.org; linux-kernel@vger.kernel.org; Songjiangtao (mygirlsjt); Zhangwei (FF); Qiuxishi
主题: Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
On Tue, 14 Jul 2015, Xuzhichuang wrote:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked
> oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138769] iostat cpuset=/
> mems_allowed=0 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138773] Pid: 18117, comm: iostat Tainted: PÂ Â Â Â Â Â Â WÂ NX 3.0.58-0.6.6-xen #1 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138775] Call Trace:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138800]Â
> [<ffffffff800088be>] dump_trace+0x6e/0x1a0 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138810]Â [<ffffffff803f773d>] dump_stack+0x69/0x6f
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138821]Â
> [<ffffffff800dbced>] dump_header+0x9d/0x120 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138826]Â [<ffffffff800dc505>]
> oom_kill_process+0x95/0x1a0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138830]Â [<ffffffff800dc746>] out_of_memory+0x136/0x220 Jul
> 10 12:33:03 BMS_CNA04 kernel: [18136514.138834]Â [<ffffffff800e0fda>]
> __alloc_pages_slowpath+0x7ba/0x810
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138838]Â
> [<ffffffff800e1219>] __alloc_pages_nodemask+0x1e9/0x200
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138845]Â
> [<ffffffff8011ae38>] cache_grow+0x348/0x450 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138850]Â [<ffffffff8011b243>]
> cache_alloc_refill+0x303/0x4d0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138854]Â [<ffffffff8011ba70>] __kmalloc+0x1b0/0x290 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138862]Â [<ffffffff8014c1da>]
> seq_read+0x13a/0x3b0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138869]Â [<ffffffff8018a762>] proc_reg_read+0x92/0xe0 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138877]Â [<ffffffff80129877>]
> vfs_read+0xc7/0x130 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138881]Â [<ffffffff801299e3>] sys_read+0x53/0xa0 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138887]Â [<ffffffff80402d73>] system_call_fastpath+0x16/0x1b Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138922]Â [<00007f935f57f4c0>] 0x7f935f57f4bf Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138923] Mem-Info:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138925] DMA per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138927] CPUÂ Â Â 0: hi:Â Â Â
> 0, btch:Â Â 1 usd:Â Â 0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138929] CPUÂ Â Â 1: hi:Â Â Â 0, btch:Â Â 1 usd:Â Â 0 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138930] DMA32 per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138932] CPUÂ Â Â 0: hi:Â
> 155, btch:Â 38 usd:Â 11 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138933] CPUÂ Â Â 1: hi:Â 155, btch:Â 38 usd:Â Â 0 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138936] active_anon:227111
> inactive_anon:10382 isolated_anon:0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138937]Â active_file:203 inactive_file:189 isolated_file:47
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138938] Â unevictable:95395
> dirty:0 writeback:0 unstable:0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138939]Â free:247834 slab_reclaimable:18187
> slab_unreclaimable:53853 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138940]Â mapped:11485 shmem:11167 pagetables:0 bounce:0 Jul
> 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB
> min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB
> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB
> mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014
> 3014 3014 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32
> free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB
> inactive_anon:41528kB active_file:812kB inactive_file:756kB
> unevictable:381580kB isolated(anon):0kB isolated(file):188kB
> present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB
> mapped:45940kB shmem:44668kB slab_reclaimable:72748kB
> slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192
> all_unreclaimable? no Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138960] lowmem_reserve[]: 0 0 0 0 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB
> 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB
> 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB
> 0*2048kB 0*4096kB = 990396kB
The problem is most of your memory for ZONE_DMA32 is available only in sizes of order-0 and order-1 and the slab allocator is trying to allocate
order-2 memory with no possibility of fallback to a smaller order.
You're running on a 3.0.58 kernel, but the watermark calculation should be the same in recent kernels.
If you follow the logic of __zone_watermark_ok(), which uses the same watermarks as printed above, the min watermark for this zone is 1751 pages and the total zone free pages is 247588. Discounting order-0 memory, there are only 59075 pages free with a min watermark of 875 pages.
Discounting order-1 memory, there are 157 pages free with a min watermark of 437 pages. This is where your allocation fails. Even though the zone has 672KB of memory available, the per-order watermark fails.
The only option you have to avoid this other than changing your workload is to alter lowmem_reserve_ratio, see Documentation/sysctl/vm.txt. You have 916KB of memory in ZONE_DMA that could be used for this allocation if it wasn't reserved for DMA allocations.
ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥
^ permalink raw reply [flat|nested] 20+ messages in thread
* 答复: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-15 1:46 ` Xuzhichuang
0 siblings, 0 replies; 20+ messages in thread
From: Xuzhichuang @ 2015-07-15 1:46 UTC (permalink / raw)
To: David Rientjes
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 6978 bytes --]
Hi,
Thanks for your replying.
According to the OOM message, OOM killer is invoked by the function seq_read, I found two patches in the latest kernel which can be avoid or fixed this problem.
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=058504edd02667eef8fac9be27ab3ea74332e9b4
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=5cec38ac866bfb8775638e71a86e4d8cac30caae
As the patches said, it changed the seq_file code fallback to vmalloc allocations if kmalloc failed, instead of OOM kill processes.
-----邮件原件-----
å‘件人: David Rientjes [mailto:rientjes@google.com]
å‘é€æ—¶é—´: 2015å¹´7月15æ—¥ 8:10
收件人: Xuzhichuang
抄é€: linux-mm@kvack.org; linux-kernel@vger.kernel.org; Songjiangtao (mygirlsjt); Zhangwei (FF); Qiuxishi
主题: Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory
On Tue, 14 Jul 2015, Xuzhichuang wrote:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked
> oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138769] iostat cpuset=/
> mems_allowed=0 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138773] Pid: 18117, comm: iostat Tainted: PÂ Â Â Â Â Â Â WÂ NX 3.0.58-0.6.6-xen #1 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138775] Call Trace:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138800]Â
> [<ffffffff800088be>] dump_trace+0x6e/0x1a0 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138810]Â [<ffffffff803f773d>] dump_stack+0x69/0x6f
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138821]Â
> [<ffffffff800dbced>] dump_header+0x9d/0x120 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138826]Â [<ffffffff800dc505>]
> oom_kill_process+0x95/0x1a0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138830]Â [<ffffffff800dc746>] out_of_memory+0x136/0x220 Jul
> 10 12:33:03 BMS_CNA04 kernel: [18136514.138834]Â [<ffffffff800e0fda>]
> __alloc_pages_slowpath+0x7ba/0x810
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138838]Â
> [<ffffffff800e1219>] __alloc_pages_nodemask+0x1e9/0x200
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138845]Â
> [<ffffffff8011ae38>] cache_grow+0x348/0x450 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138850]Â [<ffffffff8011b243>]
> cache_alloc_refill+0x303/0x4d0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138854]Â [<ffffffff8011ba70>] __kmalloc+0x1b0/0x290 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138862]Â [<ffffffff8014c1da>]
> seq_read+0x13a/0x3b0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138869]Â [<ffffffff8018a762>] proc_reg_read+0x92/0xe0 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138877]Â [<ffffffff80129877>]
> vfs_read+0xc7/0x130 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138881]Â [<ffffffff801299e3>] sys_read+0x53/0xa0 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138887]Â [<ffffffff80402d73>] system_call_fastpath+0x16/0x1b Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138922]Â [<00007f935f57f4c0>] 0x7f935f57f4bf Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138923] Mem-Info:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138925] DMA per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138927] CPUÂ Â Â 0: hi:Â Â Â
> 0, btch:Â Â 1 usd:Â Â 0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138929] CPUÂ Â Â 1: hi:Â Â Â 0, btch:Â Â 1 usd:Â Â 0 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138930] DMA32 per-cpu:
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138932] CPUÂ Â Â 0: hi:Â
> 155, btch:Â 38 usd:Â 11 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138933] CPUÂ Â Â 1: hi:Â 155, btch:Â 38 usd:Â Â 0 Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138936] active_anon:227111
> inactive_anon:10382 isolated_anon:0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138937]Â active_file:203 inactive_file:189 isolated_file:47
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138938] Â unevictable:95395
> dirty:0 writeback:0 unstable:0 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138939]Â free:247834 slab_reclaimable:18187
> slab_unreclaimable:53853 Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138940]Â mapped:11485 shmem:11167 pagetables:0 bounce:0 Jul
> 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB
> min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB
> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB
> mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014
> 3014 3014 Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32
> free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB
> inactive_anon:41528kB active_file:812kB inactive_file:756kB
> unevictable:381580kB isolated(anon):0kB isolated(file):188kB
> present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB
> mapped:45940kB shmem:44668kB slab_reclaimable:72748kB
> slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192
> all_unreclaimable? no Jul 10 12:33:03 BMS_CNA04 kernel:
> [18136514.138960] lowmem_reserve[]: 0 0 0 0 Jul 10 12:33:03 BMS_CNA04
> kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB
> 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB Jul 10
> 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB
> 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB
> 0*2048kB 0*4096kB = 990396kB
The problem is most of your memory for ZONE_DMA32 is available only in sizes of order-0 and order-1 and the slab allocator is trying to allocate
order-2 memory with no possibility of fallback to a smaller order.
You're running on a 3.0.58 kernel, but the watermark calculation should be the same in recent kernels.
If you follow the logic of __zone_watermark_ok(), which uses the same watermarks as printed above, the min watermark for this zone is 1751 pages and the total zone free pages is 247588. Discounting order-0 memory, there are only 59075 pages free with a min watermark of 875 pages.
Discounting order-1 memory, there are 157 pages free with a min watermark of 437 pages. This is where your allocation fails. Even though the zone has 672KB of memory available, the per-order watermark fails.
The only option you have to avoid this other than changing your workload is to alter lowmem_reserve_ratio, see Documentation/sysctl/vm.txt. You have 916KB of memory in ZONE_DMA that could be used for this allocation if it wasn't reserved for DMA allocations.
N‹§²æìr¸›zǧu©ž²Æ {\b†éì¹»\x1c®&Þ–)îÆi¢žØ^n‡r¶‰šŽŠÝ¢j$½§$¢¸\x05¢¹¨è§~Š'.)îÄÃ,yèm¶ŸÿÃ\f%Š{±šj+ƒðèž×¦j)Z†·Ÿ
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 答复: [BUG REPORT] OOM Killer is invoked while the system still has much memory
2015-07-15 1:46 ` Xuzhichuang
@ 2015-07-15 1:57 ` David Rientjes
-1 siblings, 0 replies; 20+ messages in thread
From: David Rientjes @ 2015-07-15 1:57 UTC (permalink / raw)
To: Xuzhichuang
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
On Wed, 15 Jul 2015, Xuzhichuang wrote:
> Hi,
>
> Thanks for your replying.
>
> According to the OOM message, OOM killer is invoked by the function seq_read, I found two patches in the latest kernel which can be avoid or fixed this problem.
>
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=058504edd02667eef8fac9be27ab3ea74332e9b4
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=5cec38ac866bfb8775638e71a86e4d8cac30caae
>
> As the patches said, it changed the seq_file code fallback to vmalloc allocations if kmalloc failed, instead of OOM kill processes.
>
Yes, we use those two patches as well internally. You may want to give
them a try if this is the only source of oom killer issues, but keep in
mind that other subsystems like the tcp layer will often do high-order
allocations as well. If you can free up some of that ZONE_DMA memory that
is unneeded with lowmem_reserve_ratio, you might get a little more room.
Good luck!
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 答复: [BUG REPORT] OOM Killer is invoked while the system still has much memory
@ 2015-07-15 1:57 ` David Rientjes
0 siblings, 0 replies; 20+ messages in thread
From: David Rientjes @ 2015-07-15 1:57 UTC (permalink / raw)
To: Xuzhichuang
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Songjiangtao (mygirlsjt), Zhangwei (FF), Qiuxishi
On Wed, 15 Jul 2015, Xuzhichuang wrote:
> Hi,
>
> Thanks for your replying.
>
> According to the OOM message, OOM killer is invoked by the function seq_read, I found two patches in the latest kernel which can be avoid or fixed this problem.
>
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=058504edd02667eef8fac9be27ab3ea74332e9b4
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/seq_file.c?id=5cec38ac866bfb8775638e71a86e4d8cac30caae
>
> As the patches said, it changed the seq_file code fallback to vmalloc allocations if kmalloc failed, instead of OOM kill processes.
>
Yes, we use those two patches as well internally. You may want to give
them a try if this is the only source of oom killer issues, but keep in
mind that other subsystems like the tcp layer will often do high-order
allocations as well. If you can free up some of that ZONE_DMA memory that
is unneeded with lowmem_reserve_ratio, you might get a little more room.
Good luck!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2015-07-15 1:57 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-14 7:11 [BUG REPORT] OOM Killer is invoked while the system still has much memory Xuzhichuang
2015-07-14 7:11 ` Xuzhichuang
2015-07-14 8:15 ` Michal Hocko
2015-07-14 8:15 ` Michal Hocko
2015-07-14 8:42 ` Xishi Qiu
2015-07-14 8:42 ` Xishi Qiu
2015-07-14 9:00 ` Michal Hocko
2015-07-14 9:00 ` Michal Hocko
2015-07-14 9:14 ` Xishi Qiu
2015-07-14 9:14 ` Xishi Qiu
2015-07-14 9:28 ` Michal Hocko
2015-07-14 9:28 ` Michal Hocko
2015-07-14 9:50 ` Xishi Qiu
2015-07-14 9:50 ` Xishi Qiu
2015-07-15 0:09 ` David Rientjes
2015-07-15 0:09 ` David Rientjes
2015-07-15 1:46 ` 答复: " Xuzhichuang
2015-07-15 1:46 ` Xuzhichuang
2015-07-15 1:57 ` David Rientjes
2015-07-15 1:57 ` David Rientjes
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.