* memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
@ 2014-04-16 15:46 Richard Davies
2014-04-18 15:59 ` Michal Hocko
0 siblings, 1 reply; 84+ messages in thread
From: Richard Davies @ 2014-04-16 15:46 UTC (permalink / raw
To: cgroups, linux-mm
Hi all,
I have a simple reproducible test case in which untar in a memcg with a kmem
limit gets into trouble during heavy disk i/o (on ext3) and never properly
recovers. This is simplified from real world problems with heavy disk i/o
inside containers.
I feel there are probably two bugs here
- the disk i/o is not successfully managed within the kmem limit
- the cgroup never recovers, despite the untar i/o process exiting
I'm happy to help with any further debugging or try patches.
To replicate on Linux 3.14.0, run the following 6 commands:
# mkdir -p /sys/fs/cgroup/test/
# cat /sys/fs/cgroup/cpuset.cpus > /sys/fs/cgroup/test/cpuset.cpus
# cat /sys/fs/cgroup/cpuset.mems > /sys/fs/cgroup/test/cpuset.mems
# echo $((1<<26)) >/sys/fs/cgroup/test/memory.kmem.limit_in_bytes
# echo $$ > /sys/fs/cgroup/test/tasks
# tar xfz linux-3.14.1.tar.gz
Part way through the untar, the tar command exits after many errors:
...
linux-3.14.1/include/linux/ima.h: Can't create 'linux-3.14.1/include/linux/ima.h'
linux-3.14.1/include/linux/in.h: Can't create 'linux-3.14.1/include/linux/in.h'
linux-3.14.1/include/linux/in6.h: Can't create 'linux-3.14.1/include/linux/in6.h'
...
At the same time, many errors are logged in the kernel log, of the form:
14:58:45 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
14:58:45 kernel: cache: radix_tree_node(3:test2), object size: 560, buffer size: 568, default order: 2, min order: 0
14:58:45 kernel: node 0: slabs: 140, objs: 3920, free: 0
14:58:45 kernel: node 1: slabs: 464, objs: 12992, free: 0
and
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0
15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0
15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0
and
15:31:15 kernel: ENOMEM in journal_alloc_journal_head, retrying.
15:31:20 kernel: __slab_alloc: 6946855 callbacks suppressed
15:31:20 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8050)
15:31:20 kernel: cache: jbd2_journal_head(2:test), object size: 112, buffer size: 112, default order: 0, min order: 0
15:31:20 kernel: node 0: slabs: 80, objs: 2880, free: 0
15:31:20 kernel: node 1: slabs: 0, objs: 0, free: 0
After this, the bash prompt in the cgroup is unusable:
# ls
-bash: fork: Cannot allocate memory
But typically the system outside the cgroup continues to work.
All of the above happens every time that I run these 6 commands.
I am happy to help with extra information on kernel configuration, but I
hope that the above is sufficient for others to replicate. I'm also happy to
try suggestions and patches.
In addition, sometimes these 6 commands trigger more serious issues beyond
the cgroup. I have copied a long kernel log of such an incident below.
Thanks in advance for your help,
Richard.
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0
15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0
15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff811017ad>] ? rcu_eqs_enter+0x70/0x83
15:11:36 kernel: [<ffffffff811017ce>] ? rcu_user_enter+0xe/0x10
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 0
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 24
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 54
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 7
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 3
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 1
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 80
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 167
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 183
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 46
15:11:36 kernel: active_anon:20251 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7963301 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:7165 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3390912kB min:9572kB low:11964kB high:14356kB active_anon:7148kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4836kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12257092kB min:35416kB low:44268kB high:53124kB active_anon:28044kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:12072kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16189300kB min:45076kB low:56344kB high:67612kB active_anon:45812kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 49*4kB (UEM) 62*8kB (U) 30*16kB (U) 12*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 2*2048kB (MR) 826*4096kB (MR) = 3390740kB
15:11:36 kernel: Node 0 Normal: 112*4kB (UE) 57*8kB (UE) 12*16kB (UEM) 32*32kB (UEM) 15*64kB (UEM) 5*128kB (UM) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UM) 1*2048kB (M) 2990*4096kB (MR) = 12257160kB
15:11:36 kernel: Node 1 Normal: 95*4kB (UM) 8*8kB (U) 0*16kB 8*32kB (U) 6*64kB (UM) 10*128kB (UM) 2*256kB (EM) 2*512kB (UE) 0*1024kB 1*2048kB (M) 3951*4096kB (MR) = 16189244kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe
15:11:36 kernel: [ 4353] 104 4353 88747 10609 59 0 0 mysqld
15:11:36 kernel: [ 4354] -2 4354 1023 158 8 0 0 logger
15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres
15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres
15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 627 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 4353 (mysqld) score 0 or sacrifice child
15:11:36 kernel: Killed process 4353 (mysqld) total-vm:354988kB, anon-rss:35704kB, file-rss:6732kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 45
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 57
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 150
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 12
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 131
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 174
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 113
15:11:36 kernel: active_anon:11418 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7971790 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:5661 shmem:1531 pagetables:1478 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:6640kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16219896kB min:45076kB low:56344kB high:67612kB active_anon:15112kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:1780kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 55*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258116kB
15:11:36 kernel: Node 1 Normal: 245*4kB (UEM) 101*8kB (UEM) 21*16kB (UEM) 23*32kB (UEM) 19*64kB (UEM) 18*128kB (UEM) 9*256kB (M) 7*512kB (UEM) 2*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16220140kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe
15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres
15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres
15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 3988 (named) score 0 or sacrifice child
15:11:36 kernel: Killed process 3988 (named) total-vm:165728kB, anon-rss:15848kB, file-rss:2172kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 33
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 53
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 151
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 100
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 44
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 137
15:11:36 kernel: active_anon:11808 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7971490 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:6534 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10132kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16218696kB min:45076kB low:56344kB high:67612kB active_anon:16672kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 58*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258140kB
15:11:36 kernel: Node 1 Normal: 90*4kB (UEM) 64*8kB (UEM) 7*16kB (UE) 12*32kB (UEM) 15*64kB (UEM) 17*128kB (UEM) 7*256kB (M) 7*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16218776kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe
15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres
15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres
15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: [ 5197] -2 5197 12302 1440 25 0 0 mysqld
15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger
15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child
15:11:36 kernel: Killed process 4394 (postgres) total-vm:99840kB, anon-rss:1844kB, file-rss:1004kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 30
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 51
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 1
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 47
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173
15:11:36 kernel: active_anon:10570 inactive_anon:2937 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7972794 slab_reclaimable:19802 slab_unreclaimable:7927
15:11:36 kernel: mapped:6267 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3395828kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:2592kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:3960kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12270500kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:5904kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10520kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16208948kB min:45076kB low:56344kB high:67612kB active_anon:26032kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 137*4kB (UEM) 92*8kB (UM) 49*16kB (UM) 26*32kB (UEM) 12*64kB (UM) 11*128kB (UEM) 3*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3395796kB
15:11:36 kernel: Node 0 Normal: 337*4kB (UM) 138*8kB (UEM) 47*16kB (UEM) 47*32kB (UEM) 22*64kB (UEM) 10*128kB (UM) 2*256kB (UE) 3*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12270820kB
15:11:36 kernel: Node 1 Normal: 44*4kB (UEM) 70*8kB (UE) 9*16kB (UEM) 9*32kB (U) 6*64kB (U) 10*128kB (UE) 0*256kB 3*512kB (UEM) 1*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16209168kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183117 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe
15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
15:11:36 kernel: [ 4393] 103 4393 24775 452 36 0 0 postgres
15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: [ 5197] 104 5197 57326 4132 35 0 0 mysqld
15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger
15:11:36 kernel: Out of memory: Kill process 5197 (mysqld) score 0 or sacrifice child
15:11:36 kernel: Killed process 5197 (mysqld) total-vm:231548kB, anon-rss:17736kB, file-rss:5256kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 67
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 197
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 82
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 2
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 156
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 164
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173
15:11:36 kernel: active_anon:7704 inactive_anon:1830 isolated_anon:0
15:11:36 kernel: active_file:29305 inactive_file:151670 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7976547 slab_reclaimable:19802 slab_unreclaimable:7927
15:11:36 kernel: mapped:4180 shmem:220 pagetables:1478 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397200kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:1024kB active_file:15260kB inactive_file:79780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2208kB shmem:112kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12273064kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:3304kB active_file:63984kB inactive_file:434284kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:3924kB shmem:448kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16220024kB min:45076kB low:56344kB high:67612kB active_anon:14568kB inactive_anon:2992kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:320kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 117*8kB (UEM) 56*16kB (UEM) 29*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397532kB
15:11:36 kernel: Node 0 Normal: 314*4kB (UEM) 156*8kB (UEM) 55*16kB (UEM) 51*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273176kB
15:11:36 kernel: Node 1 Normal: 276*4kB (UEM) 126*8kB (UEM) 31*16kB (UEM) 28*32kB (UEM) 20*64kB (UM) 20*128kB (UEM) 7*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16220592kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181806 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe
15:11:36 kernel: [ 4357] 103 4357 24779 1271 35 0 0 postgres
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: [ 5202] -2 5202 2336 77 7 0 0 mysqld_safe
15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child
15:11:36 kernel: Killed process 4357 (postgres) total-vm:99116kB, anon-rss:1232kB, file-rss:3852kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 98
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 196
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 79
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 179
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 156
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:6997 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7976969 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12273056kB min:35416kB low:44268kB high:53124kB active_anon:13500kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16221640kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 357*4kB (UEM) 164*8kB (UEM) 64*16kB (UEM) 52*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273588kB
15:11:36 kernel: Node 1 Normal: 408*4kB (UEM) 141*8kB (UEM) 45*16kB (UEM) 28*32kB (UEM) 19*64kB (UM) 22*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16221912kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 1999 (httpd) score 0 or sacrifice child
15:11:36 kernel: Killed process 1999 (httpd) total-vm:329268kB, anon-rss:3272kB, file-rss:1208kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 75
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 159
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 180
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:6430 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7977670 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275340kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16222160kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 390*4kB (UEM) 183*8kB (UEM) 68*16kB (UEM) 56*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276112kB
15:11:36 kernel: Node 1 Normal: 431*4kB (UEM) 150*8kB (UEM) 46*16kB (UEM) 30*32kB (UEM) 19*64kB (UM) 23*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16222284kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 2001 (httpd) score 0 or sacrifice child
15:11:36 kernel: Killed process 2001 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1212kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 72
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 168
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 181
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:5804 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7978396 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275600kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16224804kB min:45076kB low:56344kB high:67612kB active_anon:10240kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 393*4kB (UEM) 180*8kB (UEM) 71*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276340kB
15:11:36 kernel: Node 1 Normal: 506*4kB (UEM) 161*8kB (UEM) 49*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16224960kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 2000 (httpd) score 0 or sacrifice child
15:11:36 kernel: Killed process 2000 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1208kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 68
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 183
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:5143 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10972kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 182*8kB (UEM) 50*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227520kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child
15:11:36 kernel: Killed process 1997 (elastic-poolio) total-vm:19684kB, anon-rss:276kB, file-rss:1212kB
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
15:11:36 kernel: Node 0 DMA32 per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
15:11:36 kernel: Node 0 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 159
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
15:11:36 kernel: Node 1 Normal per-cpu:
15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173
15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182
15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:5078 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10712kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 198*8kB (UEM) 55*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227728kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child
15:11:36 kernel: Killed process 1995 (httpd) total-vm:31888kB, anon-rss:952kB, file-rss:2344kB
15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
15:11:36 kernel: Killed process 4551 (apache2) total-vm:299432kB, anon-rss:1896kB, file-rss:760kB
15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
15:11:36 kernel: Killed process 4550 (apache2) total-vm:299440kB, anon-rss:1896kB, file-rss:756kB
15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
15:11:36 kernel: Killed process 4548 (apache2) total-vm:75736kB, anon-rss:1648kB, file-rss:572kB
15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
15:11:36 kernel: Killed process 4540 (apache2) total-vm:76004kB, anon-rss:1680kB, file-rss:1492kB
15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child
15:11:37 kernel: Killed process 4386 (bash) total-vm:19872kB, anon-rss:484kB, file-rss:1472kB
15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
2014-04-16 15:46 memcg with kmem limit doesn't recover after disk i/o causes limit to be hit Richard Davies
@ 2014-04-18 15:59 ` Michal Hocko
2014-04-18 17:57 ` Vladimir Davydov
0 siblings, 1 reply; 84+ messages in thread
From: Michal Hocko @ 2014-04-18 15:59 UTC (permalink / raw
To: Richard Davies; +Cc: cgroups, linux-mm, Vladimir Davydov
[CC Vladimir]
On Wed 16-04-14 16:46:50, Richard Davies wrote:
> Hi all,
>
> I have a simple reproducible test case in which untar in a memcg with a kmem
> limit gets into trouble during heavy disk i/o (on ext3) and never properly
> recovers. This is simplified from real world problems with heavy disk i/o
> inside containers.
>
> I feel there are probably two bugs here
> - the disk i/o is not successfully managed within the kmem limit
> - the cgroup never recovers, despite the untar i/o process exiting
>
> I'm happy to help with any further debugging or try patches.
>
>
> To replicate on Linux 3.14.0, run the following 6 commands:
>
> # mkdir -p /sys/fs/cgroup/test/
> # cat /sys/fs/cgroup/cpuset.cpus > /sys/fs/cgroup/test/cpuset.cpus
> # cat /sys/fs/cgroup/cpuset.mems > /sys/fs/cgroup/test/cpuset.mems
> # echo $((1<<26)) >/sys/fs/cgroup/test/memory.kmem.limit_in_bytes
> # echo $$ > /sys/fs/cgroup/test/tasks
> # tar xfz linux-3.14.1.tar.gz
>
> Part way through the untar, the tar command exits after many errors:
>
> ...
> linux-3.14.1/include/linux/ima.h: Can't create 'linux-3.14.1/include/linux/ima.h'
> linux-3.14.1/include/linux/in.h: Can't create 'linux-3.14.1/include/linux/in.h'
> linux-3.14.1/include/linux/in6.h: Can't create 'linux-3.14.1/include/linux/in6.h'
> ...
>
> At the same time, many errors are logged in the kernel log, of the form:
>
> 14:58:45 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
> 14:58:45 kernel: cache: radix_tree_node(3:test2), object size: 560, buffer size: 568, default order: 2, min order: 0
> 14:58:45 kernel: node 0: slabs: 140, objs: 3920, free: 0
> 14:58:45 kernel: node 1: slabs: 464, objs: 12992, free: 0
>
> and
>
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
> 15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0
> 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0
> 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0
>
> and
>
> 15:31:15 kernel: ENOMEM in journal_alloc_journal_head, retrying.
> 15:31:20 kernel: __slab_alloc: 6946855 callbacks suppressed
> 15:31:20 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8050)
> 15:31:20 kernel: cache: jbd2_journal_head(2:test), object size: 112, buffer size: 112, default order: 0, min order: 0
> 15:31:20 kernel: node 0: slabs: 80, objs: 2880, free: 0
> 15:31:20 kernel: node 1: slabs: 0, objs: 0, free: 0
>
>
> After this, the bash prompt in the cgroup is unusable:
>
> # ls
> -bash: fork: Cannot allocate memory
>
> But typically the system outside the cgroup continues to work.
>
>
> All of the above happens every time that I run these 6 commands.
>
> I am happy to help with extra information on kernel configuration, but I
> hope that the above is sufficient for others to replicate. I'm also happy to
> try suggestions and patches.
>
>
> In addition, sometimes these 6 commands trigger more serious issues beyond
> the cgroup. I have copied a long kernel log of such an incident below.
>
> Thanks in advance for your help,
>
> Richard.
>
>
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
> 15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0
> 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0
> 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
> 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0
> 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0
> 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff811017ad>] ? rcu_eqs_enter+0x70/0x83
> 15:11:36 kernel: [<ffffffff811017ce>] ? rcu_user_enter+0xe/0x10
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 24
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 54
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 7
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 1
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 80
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 167
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 183
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: active_anon:20251 inactive_anon:3312 isolated_anon:0
> 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
> 15:11:36 kernel: free:7963301 slab_reclaimable:19802 slab_unreclaimable:7992
> 15:11:36 kernel: mapped:7165 shmem:1531 pagetables:1575 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3390912kB min:9572kB low:11964kB high:14356kB active_anon:7148kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4836kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12257092kB min:35416kB low:44268kB high:53124kB active_anon:28044kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:12072kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16189300kB min:45076kB low:56344kB high:67612kB active_anon:45812kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 49*4kB (UEM) 62*8kB (U) 30*16kB (U) 12*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 2*2048kB (MR) 826*4096kB (MR) = 3390740kB
> 15:11:36 kernel: Node 0 Normal: 112*4kB (UE) 57*8kB (UE) 12*16kB (UEM) 32*32kB (UEM) 15*64kB (UEM) 5*128kB (UM) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UM) 1*2048kB (M) 2990*4096kB (MR) = 12257160kB
> 15:11:36 kernel: Node 1 Normal: 95*4kB (UM) 8*8kB (U) 0*16kB 8*32kB (U) 6*64kB (UM) 10*128kB (UM) 2*256kB (EM) 2*512kB (UE) 0*1024kB 1*2048kB (M) 3951*4096kB (MR) = 16189244kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 183020 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe
> 15:11:36 kernel: [ 4353] 104 4353 88747 10609 59 0 0 mysqld
> 15:11:36 kernel: [ 4354] -2 4354 1023 158 8 0 0 logger
> 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
> 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres
> 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres
> 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 627 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 4353 (mysqld) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4353 (mysqld) total-vm:354988kB, anon-rss:35704kB, file-rss:6732kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 45
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 57
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 150
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 12
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 131
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 174
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 113
> 15:11:36 kernel: active_anon:11418 inactive_anon:3312 isolated_anon:0
> 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
> 15:11:36 kernel: free:7971790 slab_reclaimable:19802 slab_unreclaimable:7992
> 15:11:36 kernel: mapped:5661 shmem:1531 pagetables:1478 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:6640kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16219896kB min:45076kB low:56344kB high:67612kB active_anon:15112kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:1780kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
> 15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 55*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258116kB
> 15:11:36 kernel: Node 1 Normal: 245*4kB (UEM) 101*8kB (UEM) 21*16kB (UEM) 23*32kB (UEM) 19*64kB (UEM) 18*128kB (UEM) 9*256kB (M) 7*512kB (UEM) 2*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16220140kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 183020 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe
> 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
> 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres
> 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres
> 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 3988 (named) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 3988 (named) total-vm:165728kB, anon-rss:15848kB, file-rss:2172kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 33
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 53
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 151
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 100
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 44
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 137
> 15:11:36 kernel: active_anon:11808 inactive_anon:3312 isolated_anon:0
> 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
> 15:11:36 kernel: free:7971490 slab_reclaimable:19802 slab_unreclaimable:7992
> 15:11:36 kernel: mapped:6534 shmem:1531 pagetables:1575 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10132kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16218696kB min:45076kB low:56344kB high:67612kB active_anon:16672kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
> 15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 58*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258140kB
> 15:11:36 kernel: Node 1 Normal: 90*4kB (UEM) 64*8kB (UEM) 7*16kB (UE) 12*32kB (UEM) 15*64kB (UEM) 17*128kB (UEM) 7*256kB (M) 7*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16218776kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 183020 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe
> 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
> 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres
> 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres
> 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: [ 5197] -2 5197 12302 1440 25 0 0 mysqld
> 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger
> 15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4394 (postgres) total-vm:99840kB, anon-rss:1844kB, file-rss:1004kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 30
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 51
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 1
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 47
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173
> 15:11:36 kernel: active_anon:10570 inactive_anon:2937 isolated_anon:0
> 15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
> 15:11:36 kernel: free:7972794 slab_reclaimable:19802 slab_unreclaimable:7927
> 15:11:36 kernel: mapped:6267 shmem:1531 pagetables:1575 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3395828kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:2592kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:3960kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12270500kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:5904kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10520kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16208948kB min:45076kB low:56344kB high:67612kB active_anon:26032kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 137*4kB (UEM) 92*8kB (UM) 49*16kB (UM) 26*32kB (UEM) 12*64kB (UM) 11*128kB (UEM) 3*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3395796kB
> 15:11:36 kernel: Node 0 Normal: 337*4kB (UM) 138*8kB (UEM) 47*16kB (UEM) 47*32kB (UEM) 22*64kB (UEM) 10*128kB (UM) 2*256kB (UE) 3*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12270820kB
> 15:11:36 kernel: Node 1 Normal: 44*4kB (UEM) 70*8kB (UE) 9*16kB (UEM) 9*32kB (U) 6*64kB (U) 10*128kB (UE) 0*256kB 3*512kB (UEM) 1*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16209168kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 183117 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe
> 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres
> 15:11:36 kernel: [ 4393] 103 4393 24775 452 36 0 0 postgres
> 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: [ 5197] 104 5197 57326 4132 35 0 0 mysqld
> 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger
> 15:11:36 kernel: Out of memory: Kill process 5197 (mysqld) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 5197 (mysqld) total-vm:231548kB, anon-rss:17736kB, file-rss:5256kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 67
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 197
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 82
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 2
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 156
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 164
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173
> 15:11:36 kernel: active_anon:7704 inactive_anon:1830 isolated_anon:0
> 15:11:36 kernel: active_file:29305 inactive_file:151670 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
> 15:11:36 kernel: free:7976547 slab_reclaimable:19802 slab_unreclaimable:7927
> 15:11:36 kernel: mapped:4180 shmem:220 pagetables:1478 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3397200kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:1024kB active_file:15260kB inactive_file:79780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2208kB shmem:112kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12273064kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:3304kB active_file:63984kB inactive_file:434284kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:3924kB shmem:448kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16220024kB min:45076kB low:56344kB high:67612kB active_anon:14568kB inactive_anon:2992kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:320kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 117*8kB (UEM) 56*16kB (UEM) 29*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397532kB
> 15:11:36 kernel: Node 0 Normal: 314*4kB (UEM) 156*8kB (UEM) 55*16kB (UEM) 51*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273176kB
> 15:11:36 kernel: Node 1 Normal: 276*4kB (UEM) 126*8kB (UEM) 31*16kB (UEM) 28*32kB (UEM) 20*64kB (UM) 20*128kB (UEM) 7*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16220592kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 181806 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe
> 15:11:36 kernel: [ 4357] 103 4357 24779 1271 35 0 0 postgres
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: [ 5202] -2 5202 2336 77 7 0 0 mysqld_safe
> 15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4357 (postgres) total-vm:99116kB, anon-rss:1232kB, file-rss:3852kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 98
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 196
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 79
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 179
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 156
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
> 15:11:36 kernel: active_anon:6997 inactive_anon:1812 isolated_anon:0
> 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
> 15:11:36 kernel: free:7976969 slab_reclaimable:19834 slab_unreclaimable:7983
> 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12273056kB min:35416kB low:44268kB high:53124kB active_anon:13500kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16221640kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
> 15:11:36 kernel: Node 0 Normal: 357*4kB (UEM) 164*8kB (UEM) 64*16kB (UEM) 52*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273588kB
> 15:11:36 kernel: Node 1 Normal: 408*4kB (UEM) 141*8kB (UEM) 45*16kB (UEM) 28*32kB (UEM) 19*64kB (UM) 22*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16221912kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 181749 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 1999 (httpd) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 1999 (httpd) total-vm:329268kB, anon-rss:3272kB, file-rss:1208kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 75
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 159
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 180
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
> 15:11:36 kernel: active_anon:6430 inactive_anon:1812 isolated_anon:0
> 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
> 15:11:36 kernel: free:7977670 slab_reclaimable:19834 slab_unreclaimable:7983
> 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12275340kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16222160kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
> 15:11:36 kernel: Node 0 Normal: 390*4kB (UEM) 183*8kB (UEM) 68*16kB (UEM) 56*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276112kB
> 15:11:36 kernel: Node 1 Normal: 431*4kB (UEM) 150*8kB (UEM) 46*16kB (UEM) 30*32kB (UEM) 19*64kB (UM) 23*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16222284kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 181749 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 2001 (httpd) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 2001 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1212kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 72
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 168
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 181
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
> 15:11:36 kernel: active_anon:5804 inactive_anon:1812 isolated_anon:0
> 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
> 15:11:36 kernel: free:7978396 slab_reclaimable:19834 slab_unreclaimable:7983
> 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12275600kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16224804kB min:45076kB low:56344kB high:67612kB active_anon:10240kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
> 15:11:36 kernel: Node 0 Normal: 393*4kB (UEM) 180*8kB (UEM) 71*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276340kB
> 15:11:36 kernel: Node 1 Normal: 506*4kB (UEM) 161*8kB (UEM) 49*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16224960kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 181749 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 2000 (httpd) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 2000 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1208kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 68
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 183
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
> 15:11:36 kernel: active_anon:5143 inactive_anon:1812 isolated_anon:0
> 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
> 15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
> 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10972kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
> 15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
> 15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 182*8kB (UEM) 50*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227520kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 181749 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 1997 (elastic-poolio) total-vm:19684kB, anon-rss:276kB, file-rss:1212kB
> 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0
> 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
> 15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
> 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009
> 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080
> 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28
> 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38
> 15:11:36 kernel: Call Trace:
> 15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
> 15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
> 15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
> 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
> 15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
> 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
> 15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
> 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
> 15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
> 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
> 15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
> 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
> 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
> 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
> 15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
> 15:11:36 kernel: Mem-Info:
> 15:11:36 kernel: Node 0 DMA per-cpu:
> 15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0
> 15:11:36 kernel: Node 0 DMA32 per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162
> 15:11:36 kernel: Node 0 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 159
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185
> 15:11:36 kernel: Node 1 Normal per-cpu:
> 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178
> 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46
> 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173
> 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115
> 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64
> 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182
> 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157
> 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
> 15:11:36 kernel: active_anon:5078 inactive_anon:1812 isolated_anon:0
> 15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
> 15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
> 15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
> 15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
> 15:11:36 kernel: free_cma:0
> 15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
> 15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
> 15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10712kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
> 15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> 15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
> 15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
> 15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 198*8kB (UEM) 55*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227728kB
> 15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> 15:11:36 kernel: 181749 total pagecache pages
> 15:11:36 kernel: 0 pages in swap cache
> 15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
> 15:11:36 kernel: Free swap = 16507900kB
> 15:11:36 kernel: Total swap = 16507900kB
> 15:11:36 kernel: 8388414 pages RAM
> 15:11:36 kernel: 0 pages HighMem/MovableOnly
> 15:11:36 kernel: 65682 pages reserved
> 15:11:36 kernel: 0 pages hwpoisoned
> 15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty
> 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty
> 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty
> 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup
> 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty
> 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup
> 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd
> 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd
> 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount
> 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid
> 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid
> 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq
> 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd
> 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd
> 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd
> 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd
> 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod
> 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd
> 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa
> 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic
> 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic
> 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init
> 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain
> 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init
> 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient
> 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd
> 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron
> 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd
> 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd
> 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd
> 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash
> 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2
> 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2
> 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2
> 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2
> 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron
> 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty
> 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd
> 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash
> 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail
> 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar
> 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep
> 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd
> 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty
> 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty
> 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
> 15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 1995 (httpd) total-vm:31888kB, anon-rss:952kB, file-rss:2344kB
> 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4551 (apache2) total-vm:299432kB, anon-rss:1896kB, file-rss:760kB
> 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4550 (apache2) total-vm:299440kB, anon-rss:1896kB, file-rss:756kB
> 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4548 (apache2) total-vm:75736kB, anon-rss:1648kB, file-rss:572kB
> 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child
> 15:11:36 kernel: Killed process 4540 (apache2) total-vm:76004kB, anon-rss:1680kB, file-rss:1492kB
> 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child
> 15:11:37 kernel: Killed process 4386 (bash) total-vm:19872kB, anon-rss:484kB, file-rss:1472kB
> 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
@ 2014-04-18 17:57 ` Vladimir Davydov
0 siblings, 0 replies; 84+ messages in thread
From: Vladimir Davydov @ 2014-04-18 17:57 UTC (permalink / raw
To: Richard Davies; +Cc: Michal Hocko, cgroups, linux-mm
Hi Richard,
18.04.2014 19:59, Michal Hocko:
> [CC Vladimir]
>
> On Wed 16-04-14 16:46:50, Richard Davies wrote:
>> Hi all,
>>
>> I have a simple reproducible test case in which untar in a memcg with a kmem
>> limit gets into trouble during heavy disk i/o (on ext3) and never properly
>> recovers. This is simplified from real world problems with heavy disk i/o
>> inside containers.
>>
>> I feel there are probably two bugs here
>> - the disk i/o is not successfully managed within the kmem limit
>> - the cgroup never recovers, despite the untar i/o process exiting
Unfortunately, work on per cgroup kmem limits is not completed yet.
Currently it lacks kmem reclaim on per cgroup memory pressure, which is
vital for using kmem limits in real life. Basically that means that if a
process inside a memory cgroup reaches its kmem limit, it will be
returned ENOMEM on any allocation attempt, and no attempt will be made
to reclaim old cached data.
In your case untar consumes all kmem available to the cgroup by
allocating memory for storing fs metadata (inodes, dentries). Those
metadata are left cached in memory after untar dies, because they can be
potentially used by other processes. As a result, any further attempt to
allocate kmem (e.g. to create a process) will fail. It should try to
reclaim the cached metadata instead, but this functionality is not
implemented yet.
In short, kmem limiting for memory cgroups is currently broken. Do not
use it. We are working on making it usable though.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
@ 2014-04-18 17:57 ` Vladimir Davydov
0 siblings, 0 replies; 84+ messages in thread
From: Vladimir Davydov @ 2014-04-18 17:57 UTC (permalink / raw
To: Richard Davies
Cc: Michal Hocko, cgroups-u79uwXL29TY76Z2rM5mHXA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg
Hi Richard,
18.04.2014 19:59, Michal Hocko:
> [CC Vladimir]
>
> On Wed 16-04-14 16:46:50, Richard Davies wrote:
>> Hi all,
>>
>> I have a simple reproducible test case in which untar in a memcg with a kmem
>> limit gets into trouble during heavy disk i/o (on ext3) and never properly
>> recovers. This is simplified from real world problems with heavy disk i/o
>> inside containers.
>>
>> I feel there are probably two bugs here
>> - the disk i/o is not successfully managed within the kmem limit
>> - the cgroup never recovers, despite the untar i/o process exiting
Unfortunately, work on per cgroup kmem limits is not completed yet.
Currently it lacks kmem reclaim on per cgroup memory pressure, which is
vital for using kmem limits in real life. Basically that means that if a
process inside a memory cgroup reaches its kmem limit, it will be
returned ENOMEM on any allocation attempt, and no attempt will be made
to reclaim old cached data.
In your case untar consumes all kmem available to the cgroup by
allocating memory for storing fs metadata (inodes, dentries). Those
metadata are left cached in memory after untar dies, because they can be
potentially used by other processes. As a result, any further attempt to
allocate kmem (e.g. to create a process) will fail. It should try to
reclaim the cached metadata instead, but this functionality is not
implemented yet.
In short, kmem limiting for memory cgroups is currently broken. Do not
use it. We are working on making it usable though.
Thanks.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
[not found] ` <5351679F.5040908-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
@ 2014-04-18 18:20 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-18 18:20 UTC (permalink / raw
To: Vladimir Davydov; +Cc: Richard Davies, cgroups, linux-mm
On Fri 18-04-14 21:57:51, Vladimir Davydov wrote:
> In short, kmem limiting for memory cgroups is currently broken. Do not
> use it. We are working on making it usable though.
Maybe we should make this explicit in both
Documentation/cgroups/memory.txt and config MEMCG_KMEM help text.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
@ 2014-04-18 18:20 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-18 18:20 UTC (permalink / raw
To: Vladimir Davydov
Cc: Richard Davies, cgroups-u79uwXL29TY76Z2rM5mHXA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg
On Fri 18-04-14 21:57:51, Vladimir Davydov wrote:
> In short, kmem limiting for memory cgroups is currently broken. Do not
> use it. We are working on making it usable though.
Maybe we should make this explicit in both
Documentation/cgroups/memory.txt and config MEMCG_KMEM help text.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
2014-04-18 18:20 ` Michal Hocko
(?)
@ 2014-04-18 18:37 ` Vladimir Davydov
-1 siblings, 0 replies; 84+ messages in thread
From: Vladimir Davydov @ 2014-04-18 18:37 UTC (permalink / raw
To: Michal Hocko; +Cc: Richard Davies, cgroups, linux-mm
18.04.2014 22:20, Michal Hocko:
> On Fri 18-04-14 21:57:51, Vladimir Davydov wrote:
>> In short, kmem limiting for memory cgroups is currently broken. Do not
>> use it. We are working on making it usable though.
>
> Maybe we should make this explicit in both
> Documentation/cgroups/memory.txt and config MEMCG_KMEM help text.
Yeah, definitely. I'll send the patch.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-18 17:57 ` Vladimir Davydov
(?)
(?)
@ 2014-04-20 14:28 ` Richard Davies
2014-04-20 18:35 ` Tim Hockin
` (2 more replies)
-1 siblings, 3 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-20 14:28 UTC (permalink / raw
To: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange
Cc: cgroups, linux-mm, containers
Vladimir Davydov wrote:
> Richard Davies wrote:
> > I have a simple reproducible test case in which untar in a memcg with a
> > kmem limit gets into trouble during heavy disk i/o (on ext3) and never
> > properly recovers. This is simplified from real world problems with
> > heavy disk i/o inside containers.
>
> Unfortunately, work on per cgroup kmem limits is not completed yet.
> Currently it lacks kmem reclaim on per cgroup memory pressure, which is
> vital for using kmem limits in real life.
...
> In short, kmem limiting for memory cgroups is currently broken. Do not
> use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other
processes outside that container.
kmem limits were the preferred mechanism in several previous discussions
about two years ago (I'm copying in participants from those previous
discussions and give links below). So I tried kmem first but found bugs.
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROC exists but is per-user, not per-container.
Perhaps there is an up-to-date task counter patchset or similar?
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302
Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878
Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
Re: [lxc-devel] process number limit
https://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg03309.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <5351679F.5040908-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
@ 2014-04-20 14:28 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-20 14:28 UTC (permalink / raw
To: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange
Cc: cgroups-u79uwXL29TY76Z2rM5mHXA,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg
Vladimir Davydov wrote:
> Richard Davies wrote:
> > I have a simple reproducible test case in which untar in a memcg with a
> > kmem limit gets into trouble during heavy disk i/o (on ext3) and never
> > properly recovers. This is simplified from real world problems with
> > heavy disk i/o inside containers.
>
> Unfortunately, work on per cgroup kmem limits is not completed yet.
> Currently it lacks kmem reclaim on per cgroup memory pressure, which is
> vital for using kmem limits in real life.
...
> In short, kmem limiting for memory cgroups is currently broken. Do not
> use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other
processes outside that container.
kmem limits were the preferred mechanism in several previous discussions
about two years ago (I'm copying in participants from those previous
discussions and give links below). So I tried kmem first but found bugs.
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROC exists but is per-user, not per-container.
Perhaps there is an up-to-date task counter patchset or similar?
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302
Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878
Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
Re: [lxc-devel] process number limit
https://www.mail-archive.com/lxc-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg03309.html
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies
@ 2014-04-20 18:35 ` Tim Hockin
[not found] ` <20140420142830.GC22077-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-22 18:39 ` Dwight Engen
2 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-20 18:35 UTC (permalink / raw
To: Richard Davies
Cc: Michal Hocko, Vladimir Davydov, Frederic Weisbecker,
Johannes Weiner, containers, Glauber Costa, Tejun Heo,
David Rientjes, linux-mm, Daniel Walsh, William Dauchy,
Max Kellermann, cgroups, Daniel Berrange
[-- Attachment #1: Type: text/plain, Size: 2325 bytes --]
I would still be in strong support of a cgroup replacement for NPROC rlimit.
On Apr 20, 2014 7:29 AM, "Richard Davies" <richard@arachsys.com> wrote:
> Vladimir Davydov wrote:
> > Richard Davies wrote:
> > > I have a simple reproducible test case in which untar in a memcg with a
> > > kmem limit gets into trouble during heavy disk i/o (on ext3) and never
> > > properly recovers. This is simplified from real world problems with
> > > heavy disk i/o inside containers.
> >
> > Unfortunately, work on per cgroup kmem limits is not completed yet.
> > Currently it lacks kmem reclaim on per cgroup memory pressure, which is
> > vital for using kmem limits in real life.
> ...
> > In short, kmem limiting for memory cgroups is currently broken. Do not
> > use it. We are working on making it usable though.
>
> Thanks for explaining the strange errors I got.
>
>
> My motivation is to prevent a fork bomb in a container from affecting other
> processes outside that container.
>
> kmem limits were the preferred mechanism in several previous discussions
> about two years ago (I'm copying in participants from those previous
> discussions and give links below). So I tried kmem first but found bugs.
>
>
> What is the best mechanism available today, until kmem limits mature?
>
> RLIMIT_NPROC exists but is per-user, not per-container.
>
> Perhaps there is an up-to-date task counter patchset or similar?
>
>
> Thank you all,
>
> Richard.
>
>
>
> Some references to previous discussions:
>
> Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for
> memcg: stripped down version
> http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
>
> Re: [PATCH 00/10] cgroups: Task counter subsystem v8
> http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
>
> [RFD] Merge task counter into memcg
> http://thread.gmane.org/gmane.linux.kernel/1280302
>
> Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
> http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
>
> [PATCH] new cgroup controller "fork"
> http://thread.gmane.org/gmane.linux.kernel/1210878
>
> Re: Process Limit cgroups
> http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
>
> Re: [lxc-devel] process number limit
> https://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg03309.html
>
[-- Attachment #2: Type: text/html, Size: 3477 bytes --]
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140420142830.GC22077-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-20 18:35 ` Tim Hockin
2014-04-22 18:39 ` Dwight Engen
1 sibling, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-20 18:35 UTC (permalink / raw
To: Richard Davies
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
David Rientjes, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, Johannes Weiner,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
I would still be in strong support of a cgroup replacement for NPROC rlimit.
On Apr 20, 2014 7:29 AM, "Richard Davies" <richard-li8END47hbdWk0Htik3J/w@public.gmane.org> wrote:
> Vladimir Davydov wrote:
> > Richard Davies wrote:
> > > I have a simple reproducible test case in which untar in a memcg with a
> > > kmem limit gets into trouble during heavy disk i/o (on ext3) and never
> > > properly recovers. This is simplified from real world problems with
> > > heavy disk i/o inside containers.
> >
> > Unfortunately, work on per cgroup kmem limits is not completed yet.
> > Currently it lacks kmem reclaim on per cgroup memory pressure, which is
> > vital for using kmem limits in real life.
> ...
> > In short, kmem limiting for memory cgroups is currently broken. Do not
> > use it. We are working on making it usable though.
>
> Thanks for explaining the strange errors I got.
>
>
> My motivation is to prevent a fork bomb in a container from affecting other
> processes outside that container.
>
> kmem limits were the preferred mechanism in several previous discussions
> about two years ago (I'm copying in participants from those previous
> discussions and give links below). So I tried kmem first but found bugs.
>
>
> What is the best mechanism available today, until kmem limits mature?
>
> RLIMIT_NPROC exists but is per-user, not per-container.
>
> Perhaps there is an up-to-date task counter patchset or similar?
>
>
> Thank you all,
>
> Richard.
>
>
>
> Some references to previous discussions:
>
> Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for
> memcg: stripped down version
> http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
>
> Re: [PATCH 00/10] cgroups: Task counter subsystem v8
> http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
>
> [RFD] Merge task counter into memcg
> http://thread.gmane.org/gmane.linux.kernel/1280302
>
> Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
> http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
>
> [PATCH] new cgroup controller "fork"
> http://thread.gmane.org/gmane.linux.kernel/1210878
>
> Re: Process Limit cgroups
> http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
>
> Re: [lxc-devel] process number limit
> https://www.mail-archive.com/lxc-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg03309.html
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies
2014-04-20 18:35 ` Tim Hockin
[not found] ` <20140420142830.GC22077-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-22 18:39 ` Dwight Engen
2014-04-22 20:05 ` Richard Davies
[not found] ` <20140422143943.20609800-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2 siblings, 2 replies; 84+ messages in thread
From: Dwight Engen @ 2014-04-22 18:39 UTC (permalink / raw
To: Richard Davies
Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange, cgroups, containers, linux-mm
On Sun, 20 Apr 2014 15:28:30 +0100
Richard Davies <richard@arachsys.com> wrote:
> Vladimir Davydov wrote:
> > Richard Davies wrote:
> > > I have a simple reproducible test case in which untar in a memcg
> > > with a kmem limit gets into trouble during heavy disk i/o (on
> > > ext3) and never properly recovers. This is simplified from real
> > > world problems with heavy disk i/o inside containers.
> >
> > Unfortunately, work on per cgroup kmem limits is not completed yet.
> > Currently it lacks kmem reclaim on per cgroup memory pressure,
> > which is vital for using kmem limits in real life.
> ...
> > In short, kmem limiting for memory cgroups is currently broken. Do
> > not use it. We are working on making it usable though.
>
> Thanks for explaining the strange errors I got.
>
>
> My motivation is to prevent a fork bomb in a container from affecting
> other processes outside that container.
>
> kmem limits were the preferred mechanism in several previous
> discussions about two years ago (I'm copying in participants from
> those previous discussions and give links below). So I tried kmem
> first but found bugs.
>
>
> What is the best mechanism available today, until kmem limits mature?
>
> RLIMIT_NPROC exists but is per-user, not per-container.
>
> Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's
fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would
find it useful.
> Thank you all,
>
> Richard.
>
>
>
> Some references to previous discussions:
>
> Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller
> for memcg: stripped down version
> http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
>
> Re: [PATCH 00/10] cgroups: Task counter subsystem v8
> http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
>
> [RFD] Merge task counter into memcg
> http://thread.gmane.org/gmane.linux.kernel/1280302
>
> Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
> http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
>
> [PATCH] new cgroup controller "fork"
> http://thread.gmane.org/gmane.linux.kernel/1210878
>
> Re: Process Limit cgroups
> http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
>
> Re: [lxc-devel] process number limit
> https://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg03309.html
> _______________________________________________
> Containers mailing list
> Containers@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140420142830.GC22077-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-20 18:35 ` Tim Hockin
@ 2014-04-22 18:39 ` Dwight Engen
1 sibling, 0 replies; 84+ messages in thread
From: Dwight Engen @ 2014-04-22 18:39 UTC (permalink / raw
To: Richard Davies
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On Sun, 20 Apr 2014 15:28:30 +0100
Richard Davies <richard-li8END47hbdWk0Htik3J/w@public.gmane.org> wrote:
> Vladimir Davydov wrote:
> > Richard Davies wrote:
> > > I have a simple reproducible test case in which untar in a memcg
> > > with a kmem limit gets into trouble during heavy disk i/o (on
> > > ext3) and never properly recovers. This is simplified from real
> > > world problems with heavy disk i/o inside containers.
> >
> > Unfortunately, work on per cgroup kmem limits is not completed yet.
> > Currently it lacks kmem reclaim on per cgroup memory pressure,
> > which is vital for using kmem limits in real life.
> ...
> > In short, kmem limiting for memory cgroups is currently broken. Do
> > not use it. We are working on making it usable though.
>
> Thanks for explaining the strange errors I got.
>
>
> My motivation is to prevent a fork bomb in a container from affecting
> other processes outside that container.
>
> kmem limits were the preferred mechanism in several previous
> discussions about two years ago (I'm copying in participants from
> those previous discussions and give links below). So I tried kmem
> first but found bugs.
>
>
> What is the best mechanism available today, until kmem limits mature?
>
> RLIMIT_NPROC exists but is per-user, not per-container.
>
> Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's
fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would
find it useful.
> Thank you all,
>
> Richard.
>
>
>
> Some references to previous discussions:
>
> Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller
> for memcg: stripped down version
> http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
>
> Re: [PATCH 00/10] cgroups: Task counter subsystem v8
> http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
>
> [RFD] Merge task counter into memcg
> http://thread.gmane.org/gmane.linux.kernel/1280302
>
> Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
> http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
>
> [PATCH] new cgroup controller "fork"
> http://thread.gmane.org/gmane.linux.kernel/1210878
>
> Re: Process Limit cgroups
> http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
>
> Re: [lxc-devel] process number limit
> https://www.mail-archive.com/lxc-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg03309.html
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-22 18:39 ` Dwight Engen
@ 2014-04-22 20:05 ` Richard Davies
2014-04-22 20:13 ` Tim Hockin
` (3 more replies)
[not found] ` <20140422143943.20609800-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
1 sibling, 4 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-22 20:05 UTC (permalink / raw
To: Dwight Engen
Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange, cgroups, containers, linux-mm
Dwight Engen wrote:
> Richard Davies wrote:
> > Vladimir Davydov wrote:
> > > In short, kmem limiting for memory cgroups is currently broken. Do
> > > not use it. We are working on making it usable though.
...
> > What is the best mechanism available today, until kmem limits mature?
> >
> > RLIMIT_NPROC exists but is per-user, not per-container.
> >
> > Perhaps there is an up-to-date task counter patchset or similar?
>
> I updated Frederic's task counter patches and included Max Kellermann's
> fork limiter here:
>
> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>
> I can send you a more recent patchset (against 3.13.10) if you would
> find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if
you have that too.
Thanks,
Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140422143943.20609800-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
@ 2014-04-22 20:05 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-22 20:05 UTC (permalink / raw
To: Dwight Engen
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
Dwight Engen wrote:
> Richard Davies wrote:
> > Vladimir Davydov wrote:
> > > In short, kmem limiting for memory cgroups is currently broken. Do
> > > not use it. We are working on making it usable though.
...
> > What is the best mechanism available today, until kmem limits mature?
> >
> > RLIMIT_NPROC exists but is per-user, not per-container.
> >
> > Perhaps there is an up-to-date task counter patchset or similar?
>
> I updated Frederic's task counter patches and included Max Kellermann's
> fork limiter here:
>
> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>
> I can send you a more recent patchset (against 3.13.10) if you would
> find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if
you have that too.
Thanks,
Richard.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-22 20:05 ` Richard Davies
@ 2014-04-22 20:13 ` Tim Hockin
2014-04-23 6:07 ` Marian Marinov
` (2 subsequent siblings)
3 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-22 20:13 UTC (permalink / raw
To: Richard Davies
Cc: Dwight Engen, Vladimir Davydov, Frederic Weisbecker,
David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann,
Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko,
Daniel Walsh, Daniel Berrange, cgroups, containers,
linux-mm@kvack.org
Who in kernel-land still needs to be convinced of the utility of this idea?
On Tue, Apr 22, 2014 at 1:05 PM, Richard Davies <richard@arachsys.com> wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>> > Vladimir Davydov wrote:
>> > > In short, kmem limiting for memory cgroups is currently broken. Do
>> > > not use it. We are working on making it usable though.
> ...
>> > What is the best mechanism available today, until kmem limits mature?
>> >
>> > RLIMIT_NPROC exists but is per-user, not per-container.
>> >
>> > Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
>
> Thanks,
>
> Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140422200531.GA19334-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-22 20:13 ` Tim Hockin
2014-04-23 6:07 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-22 20:13 UTC (permalink / raw
To: Richard Davies
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
David Rientjes, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
Who in kernel-land still needs to be convinced of the utility of this idea?
On Tue, Apr 22, 2014 at 1:05 PM, Richard Davies <richard-li8END47hbdWk0Htik3J/w@public.gmane.org> wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>> > Vladimir Davydov wrote:
>> > > In short, kmem limiting for memory cgroups is currently broken. Do
>> > > not use it. We are working on making it usable though.
> ...
>> > What is the best mechanism available today, until kmem limits mature?
>> >
>> > RLIMIT_NPROC exists but is per-user, not per-container.
>> >
>> > Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
>
> Thanks,
>
> Richard.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140422200531.GA19334-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-23 6:07 ` Marian Marinov
2014-04-23 6:07 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-23 6:07 UTC (permalink / raw
To: Richard Davies, Dwight Engen
Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange, cgroups, containers, linux-mm
On 04/22/2014 11:05 PM, Richard Davies wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>>> Vladimir Davydov wrote:
>>>> In short, kmem limiting for memory cgroups is currently broken. Do
>>>> not use it. We are working on making it usable though.
> ...
>>> What is the best mechanism available today, until kmem limits mature?
>>>
>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>
>>> Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Marian
>
> Thanks,
>
> Richard.
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140422200531.GA19334-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-22 20:13 ` Tim Hockin
@ 2014-04-23 6:07 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-23 6:07 UTC (permalink / raw
To: Richard Davies, Dwight Engen
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On 04/22/2014 11:05 PM, Richard Davies wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>>> Vladimir Davydov wrote:
>>>> In short, kmem limiting for memory cgroups is currently broken. Do
>>>> not use it. We are working on making it usable though.
> ...
>>> What is the best mechanism available today, until kmem limits mature?
>>>
>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>
>>> Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Marian
>
> Thanks,
>
> Richard.
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-23 6:07 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-23 6:07 UTC (permalink / raw
To: Richard Davies, Dwight Engen
Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange, cgroups-u79uwXL29TY76Z2rM5mHXA,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg
On 04/22/2014 11:05 PM, Richard Davies wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>>> Vladimir Davydov wrote:
>>>> In short, kmem limiting for memory cgroups is currently broken. Do
>>>> not use it. We are working on making it usable though.
> ...
>>> What is the best mechanism available today, until kmem limits mature?
>>>
>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>
>>> Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Marian
>
> Thanks,
>
> Richard.
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-23 6:07 ` Marian Marinov
(?)
(?)
@ 2014-04-23 12:49 ` Dwight Engen
2014-04-28 18:00 ` Serge Hallyn
` (3 more replies)
-1 siblings, 4 replies; 84+ messages in thread
From: Dwight Engen @ 2014-04-23 12:49 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker,
David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann,
Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko,
Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm
On Wed, 23 Apr 2014 09:07:28 +0300
Marian Marinov <mm@yuhu.biz> wrote:
> On 04/22/2014 11:05 PM, Richard Davies wrote:
> > Dwight Engen wrote:
> >> Richard Davies wrote:
> >>> Vladimir Davydov wrote:
> >>>> In short, kmem limiting for memory cgroups is currently broken.
> >>>> Do not use it. We are working on making it usable though.
> > ...
> >>> What is the best mechanism available today, until kmem limits
> >>> mature?
> >>>
> >>> RLIMIT_NPROC exists but is per-user, not per-container.
> >>>
> >>> Perhaps there is an up-to-date task counter patchset or similar?
> >>
> >> I updated Frederic's task counter patches and included Max
> >> Kellermann's fork limiter here:
> >>
> >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> >>
> >> I can send you a more recent patchset (against 3.13.10) if you
> >> would find it useful.
> >
> > Yes please, I would be interested in that. Ideally even against
> > 3.14.1 if you have that too.
>
> Dwight, do you have these patches in any public repo?
>
> I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> Marian
>
> >
> > Thanks,
> >
> > Richard.
> > --
> > To unsubscribe from this list: send the line "unsubscribe cgroups"
> > in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <535758A0.5000500-NV7Lj0SOnH0@public.gmane.org>
@ 2014-04-23 12:49 ` Dwight Engen
0 siblings, 0 replies; 84+ messages in thread
From: Dwight Engen @ 2014-04-23 12:49 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On Wed, 23 Apr 2014 09:07:28 +0300
Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> On 04/22/2014 11:05 PM, Richard Davies wrote:
> > Dwight Engen wrote:
> >> Richard Davies wrote:
> >>> Vladimir Davydov wrote:
> >>>> In short, kmem limiting for memory cgroups is currently broken.
> >>>> Do not use it. We are working on making it usable though.
> > ...
> >>> What is the best mechanism available today, until kmem limits
> >>> mature?
> >>>
> >>> RLIMIT_NPROC exists but is per-user, not per-container.
> >>>
> >>> Perhaps there is an up-to-date task counter patchset or similar?
> >>
> >> I updated Frederic's task counter patches and included Max
> >> Kellermann's fork limiter here:
> >>
> >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> >>
> >> I can send you a more recent patchset (against 3.13.10) if you
> >> would find it useful.
> >
> > Yes please, I would be interested in that. Ideally even against
> > 3.14.1 if you have that too.
>
> Dwight, do you have these patches in any public repo?
>
> I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> Marian
>
> >
> > Thanks,
> >
> > Richard.
> > --
> > To unsubscribe from this list: send the line "unsubscribe cgroups"
> > in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140423084942.560ae837-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
@ 2014-04-28 18:00 ` Serge Hallyn
2014-06-10 14:50 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Serge Hallyn @ 2014-04-28 18:00 UTC (permalink / raw
To: Dwight Engen
Cc: Marian Marinov, Richard Davies, Vladimir Davydov, Daniel Walsh,
Max Kellermann, Tim Hockin, Frederic Weisbecker, containers,
Johannes Weiner, Glauber Costa, Michal Hocko, linux-mm,
William Dauchy, David Rientjes, Tejun Heo, cgroups
Quoting Dwight Engen (dwight.engen@oracle.com):
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm@yuhu.biz> wrote:
>
> > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > Dwight Engen wrote:
> > >> Richard Davies wrote:
> > >>> Vladimir Davydov wrote:
> > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > >>>> Do not use it. We are working on making it usable though.
> > > ...
> > >>> What is the best mechanism available today, until kmem limits
> > >>> mature?
> > >>>
> > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > >>>
> > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > >>
> > >> I updated Frederic's task counter patches and included Max
> > >> Kellermann's fork limiter here:
> > >>
> > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > >>
> > >> I can send you a more recent patchset (against 3.13.10) if you
> > >> would find it useful.
> > >
> > > Yes please, I would be interested in that. Ideally even against
> > > 3.14.1 if you have that too.
> >
> > Dwight, do you have these patches in any public repo?
> >
> > I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
that a task limit would be a proper cgroup extension, and specifically
that approximating that with a kmem limit is not a reasonable substitute.
-serge
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-28 18:00 ` Serge Hallyn
0 siblings, 0 replies; 84+ messages in thread
From: Serge Hallyn @ 2014-04-28 18:00 UTC (permalink / raw
To: Dwight Engen
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
Michal Hocko, linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>
> > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > Dwight Engen wrote:
> > >> Richard Davies wrote:
> > >>> Vladimir Davydov wrote:
> > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > >>>> Do not use it. We are working on making it usable though.
> > > ...
> > >>> What is the best mechanism available today, until kmem limits
> > >>> mature?
> > >>>
> > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > >>>
> > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > >>
> > >> I updated Frederic's task counter patches and included Max
> > >> Kellermann's fork limiter here:
> > >>
> > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > >>
> > >> I can send you a more recent patchset (against 3.13.10) if you
> > >> would find it useful.
> > >
> > > Yes please, I would be interested in that. Ideally even against
> > > 3.14.1 if you have that too.
> >
> > Dwight, do you have these patches in any public repo?
> >
> > I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
that a task limit would be a proper cgroup extension, and specifically
that approximating that with a kmem limit is not a reasonable substitute.
-serge
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-28 18:00 ` Serge Hallyn
@ 2014-04-29 7:25 ` Michal Hocko
-1 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 7:25 UTC (permalink / raw
To: Serge Hallyn
Cc: Dwight Engen, Marian Marinov, Richard Davies, Vladimir Davydov,
Daniel Walsh, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Johannes Weiner, Glauber Costa, linux-mm,
William Dauchy, David Rientjes, Tejun Heo, cgroups
On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> Quoting Dwight Engen (dwight.engen@oracle.com):
> > On Wed, 23 Apr 2014 09:07:28 +0300
> > Marian Marinov <mm@yuhu.biz> wrote:
> >
> > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > Dwight Engen wrote:
> > > >> Richard Davies wrote:
> > > >>> Vladimir Davydov wrote:
> > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > >>>> Do not use it. We are working on making it usable though.
> > > > ...
> > > >>> What is the best mechanism available today, until kmem limits
> > > >>> mature?
> > > >>>
> > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > >>>
> > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > >>
> > > >> I updated Frederic's task counter patches and included Max
> > > >> Kellermann's fork limiter here:
> > > >>
> > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > >>
> > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > >> would find it useful.
> > > >
> > > > Yes please, I would be interested in that. Ideally even against
> > > > 3.14.1 if you have that too.
> > >
> > > Dwight, do you have these patches in any public repo?
> > >
> > > I would like to test them also.
> >
> > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> >
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>
> Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> that a task limit would be a proper cgroup extension, and specifically
> that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to
Vladimir, is not a reason for a new extension/controller. We are just
not yet there.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 7:25 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 7:25 UTC (permalink / raw
To: Serge Hallyn
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> > On Wed, 23 Apr 2014 09:07:28 +0300
> > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> >
> > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > Dwight Engen wrote:
> > > >> Richard Davies wrote:
> > > >>> Vladimir Davydov wrote:
> > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > >>>> Do not use it. We are working on making it usable though.
> > > > ...
> > > >>> What is the best mechanism available today, until kmem limits
> > > >>> mature?
> > > >>>
> > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > >>>
> > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > >>
> > > >> I updated Frederic's task counter patches and included Max
> > > >> Kellermann's fork limiter here:
> > > >>
> > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > >>
> > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > >> would find it useful.
> > > >
> > > > Yes please, I would be interested in that. Ideally even against
> > > > 3.14.1 if you have that too.
> > >
> > > Dwight, do you have these patches in any public repo?
> > >
> > > I would like to test them also.
> >
> > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> >
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>
> Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> that a task limit would be a proper cgroup extension, and specifically
> that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to
Vladimir, is not a reason for a new extension/controller. We are just
not yet there.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429072515.GB15058-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
@ 2014-04-29 13:03 ` Serge Hallyn
0 siblings, 0 replies; 84+ messages in thread
From: Serge Hallyn @ 2014-04-29 13:03 UTC (permalink / raw
To: Michal Hocko
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker, containers, Daniel Walsh,
cgroups, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
Quoting Michal Hocko (mhocko@suse.cz):
> On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > Quoting Dwight Engen (dwight.engen@oracle.com):
> > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > Marian Marinov <mm@yuhu.biz> wrote:
> > >
> > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > Dwight Engen wrote:
> > > > >> Richard Davies wrote:
> > > > >>> Vladimir Davydov wrote:
> > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > > >>>> Do not use it. We are working on making it usable though.
> > > > > ...
> > > > >>> What is the best mechanism available today, until kmem limits
> > > > >>> mature?
> > > > >>>
> > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > >>>
> > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > > >>
> > > > >> I updated Frederic's task counter patches and included Max
> > > > >> Kellermann's fork limiter here:
> > > > >>
> > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > >>
> > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > >> would find it useful.
> > > > >
> > > > > Yes please, I would be interested in that. Ideally even against
> > > > > 3.14.1 if you have that too.
> > > >
> > > > Dwight, do you have these patches in any public repo?
> > > >
> > > > I would like to test them also.
> > >
> > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > >
> > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >
> > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > that a task limit would be a proper cgroup extension, and specifically
> > that approximating that with a kmem limit is not a reasonable substitute.
>
> The current state of the kmem limit, which is improving a lot thanks to
> Vladimir, is not a reason for a new extension/controller. We are just
> not yet there.
It has nothing to do with the state of the limit. I simply don't
believe that emulating RLIMIT_NPROC by controlling stack size is a
good idea.
-serge
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429072515.GB15058-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
@ 2014-04-29 13:03 ` Serge Hallyn
0 siblings, 0 replies; 84+ messages in thread
From: Serge Hallyn @ 2014-04-29 13:03 UTC (permalink / raw
To: Michal Hocko
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, Glauber Costa, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
William Dauchy, Johannes Weiner, Tejun Heo,
cgroups-u79uwXL29TY76Z2rM5mHXA
Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
> On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> > >
> > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > Dwight Engen wrote:
> > > > >> Richard Davies wrote:
> > > > >>> Vladimir Davydov wrote:
> > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > > >>>> Do not use it. We are working on making it usable though.
> > > > > ...
> > > > >>> What is the best mechanism available today, until kmem limits
> > > > >>> mature?
> > > > >>>
> > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > >>>
> > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > > >>
> > > > >> I updated Frederic's task counter patches and included Max
> > > > >> Kellermann's fork limiter here:
> > > > >>
> > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > >>
> > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > >> would find it useful.
> > > > >
> > > > > Yes please, I would be interested in that. Ideally even against
> > > > > 3.14.1 if you have that too.
> > > >
> > > > Dwight, do you have these patches in any public repo?
> > > >
> > > > I would like to test them also.
> > >
> > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > >
> > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >
> > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > that a task limit would be a proper cgroup extension, and specifically
> > that approximating that with a kmem limit is not a reasonable substitute.
>
> The current state of the kmem limit, which is improving a lot thanks to
> Vladimir, is not a reason for a new extension/controller. We are just
> not yet there.
It has nothing to do with the state of the limit. I simply don't
believe that emulating RLIMIT_NPROC by controlling stack size is a
good idea.
-serge
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 13:03 ` Serge Hallyn
0 siblings, 0 replies; 84+ messages in thread
From: Serge Hallyn @ 2014-04-29 13:03 UTC (permalink / raw
To: Michal Hocko
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
> On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> > >
> > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > Dwight Engen wrote:
> > > > >> Richard Davies wrote:
> > > > >>> Vladimir Davydov wrote:
> > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > > >>>> Do not use it. We are working on making it usable though.
> > > > > ...
> > > > >>> What is the best mechanism available today, until kmem limits
> > > > >>> mature?
> > > > >>>
> > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > >>>
> > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > > >>
> > > > >> I updated Frederic's task counter patches and included Max
> > > > >> Kellermann's fork limiter here:
> > > > >>
> > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > >>
> > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > >> would find it useful.
> > > > >
> > > > > Yes please, I would be interested in that. Ideally even against
> > > > > 3.14.1 if you have that too.
> > > >
> > > > Dwight, do you have these patches in any public repo?
> > > >
> > > > I would like to test them also.
> > >
> > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > >
> > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >
> > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > that a task limit would be a proper cgroup extension, and specifically
> > that approximating that with a kmem limit is not a reasonable substitute.
>
> The current state of the kmem limit, which is improving a lot thanks to
> Vladimir, is not a reason for a new extension/controller. We are just
> not yet there.
It has nothing to do with the state of the limit. I simply don't
believe that emulating RLIMIT_NPROC by controlling stack size is a
good idea.
-serge
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 13:03 ` Serge Hallyn
@ 2014-04-29 13:57 ` Marian Marinov
-1 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-29 13:57 UTC (permalink / raw
To: Serge Hallyn, Michal Hocko
Cc: Richard Davies, Vladimir Davydov, Max Kellermann, Tim Hockin,
Frederic Weisbecker, containers, Daniel Walsh, cgroups,
Glauber Costa, linux-mm, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
On 04/29/2014 04:03 PM, Serge Hallyn wrote:
> Quoting Michal Hocko (mhocko@suse.cz):
>> On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
>>> Quoting Dwight Engen (dwight.engen@oracle.com):
>>>> On Wed, 23 Apr 2014 09:07:28 +0300
>>>> Marian Marinov <mm@yuhu.biz> wrote:
>>>>
>>>>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>>>>> Dwight Engen wrote:
>>>>>>> Richard Davies wrote:
>>>>>>>> Vladimir Davydov wrote:
>>>>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>>>>> Do not use it. We are working on making it usable though.
>>>>>> ...
>>>>>>>> What is the best mechanism available today, until kmem limits
>>>>>>>> mature?
>>>>>>>>
>>>>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>>>>
>>>>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>>>>
>>>>>>> I updated Frederic's task counter patches and included Max
>>>>>>> Kellermann's fork limiter here:
>>>>>>>
>>>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>>>>
>>>>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>>>>> would find it useful.
>>>>>>
>>>>>> Yes please, I would be interested in that. Ideally even against
>>>>>> 3.14.1 if you have that too.
>>>>>
>>>>> Dwight, do you have these patches in any public repo?
>>>>>
>>>>> I would like to test them also.
>>>>
>>>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>>>>
>>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>>>
>>> Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
>>> that a task limit would be a proper cgroup extension, and specifically
>>> that approximating that with a kmem limit is not a reasonable substitute.
>>
>> The current state of the kmem limit, which is improving a lot thanks to
>> Vladimir, is not a reason for a new extension/controller. We are just
>> not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
>
> -serge
I think that having a limit on the number of processes allowed in a cgroup is a lot better then relaying on the kmem limit.
The problem that task-limit tries to solve is degradation of system performance caused by too many processes in a
certain cgroup. I'm currently testing the patches with 3.12.16.
-hackman
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 13:03 ` Serge Hallyn
(?)
(?)
@ 2014-04-29 13:57 ` Marian Marinov
-1 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-29 13:57 UTC (permalink / raw
To: Serge Hallyn, Michal Hocko
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, Glauber Costa, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
William Dauchy, Johannes Weiner, Tejun Heo,
cgroups-u79uwXL29TY76Z2rM5mHXA
On 04/29/2014 04:03 PM, Serge Hallyn wrote:
> Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
>> On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
>>> Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
>>>> On Wed, 23 Apr 2014 09:07:28 +0300
>>>> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>>>>
>>>>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>>>>> Dwight Engen wrote:
>>>>>>> Richard Davies wrote:
>>>>>>>> Vladimir Davydov wrote:
>>>>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>>>>> Do not use it. We are working on making it usable though.
>>>>>> ...
>>>>>>>> What is the best mechanism available today, until kmem limits
>>>>>>>> mature?
>>>>>>>>
>>>>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>>>>
>>>>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>>>>
>>>>>>> I updated Frederic's task counter patches and included Max
>>>>>>> Kellermann's fork limiter here:
>>>>>>>
>>>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>>>>
>>>>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>>>>> would find it useful.
>>>>>>
>>>>>> Yes please, I would be interested in that. Ideally even against
>>>>>> 3.14.1 if you have that too.
>>>>>
>>>>> Dwight, do you have these patches in any public repo?
>>>>>
>>>>> I would like to test them also.
>>>>
>>>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>>>>
>>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>>>
>>> Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
>>> that a task limit would be a proper cgroup extension, and specifically
>>> that approximating that with a kmem limit is not a reasonable substitute.
>>
>> The current state of the kmem limit, which is improving a lot thanks to
>> Vladimir, is not a reason for a new extension/controller. We are just
>> not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
>
> -serge
I think that having a limit on the number of processes allowed in a cgroup is a lot better then relaying on the kmem limit.
The problem that task-limit tries to solve is degradation of system performance caused by too many processes in a
certain cgroup. I'm currently testing the patches with 3.12.16.
-hackman
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 13:57 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-29 13:57 UTC (permalink / raw
To: Serge Hallyn, Michal Hocko
Cc: Richard Davies, Vladimir Davydov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
On 04/29/2014 04:03 PM, Serge Hallyn wrote:
> Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
>> On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
>>> Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
>>>> On Wed, 23 Apr 2014 09:07:28 +0300
>>>> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>>>>
>>>>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>>>>> Dwight Engen wrote:
>>>>>>> Richard Davies wrote:
>>>>>>>> Vladimir Davydov wrote:
>>>>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>>>>> Do not use it. We are working on making it usable though.
>>>>>> ...
>>>>>>>> What is the best mechanism available today, until kmem limits
>>>>>>>> mature?
>>>>>>>>
>>>>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>>>>
>>>>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>>>>
>>>>>>> I updated Frederic's task counter patches and included Max
>>>>>>> Kellermann's fork limiter here:
>>>>>>>
>>>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>>>>
>>>>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>>>>> would find it useful.
>>>>>>
>>>>>> Yes please, I would be interested in that. Ideally even against
>>>>>> 3.14.1 if you have that too.
>>>>>
>>>>> Dwight, do you have these patches in any public repo?
>>>>>
>>>>> I would like to test them also.
>>>>
>>>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>>>>
>>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>>>
>>> Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
>>> that a task limit would be a proper cgroup extension, and specifically
>>> that approximating that with a kmem limit is not a reasonable substitute.
>>
>> The current state of the kmem limit, which is improving a lot thanks to
>> Vladimir, is not a reason for a new extension/controller. We are just
>> not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
>
> -serge
I think that having a limit on the number of processes allowed in a cgroup is a lot better then relaying on the kmem limit.
The problem that task-limit tries to solve is degradation of system performance caused by too many processes in a
certain cgroup. I'm currently testing the patches with 3.12.16.
-hackman
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 13:03 ` Serge Hallyn
` (3 preceding siblings ...)
(?)
@ 2014-04-29 14:04 ` Tim Hockin
-1 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 14:04 UTC (permalink / raw
To: Serge Hallyn
Cc: Michal Hocko, Vladimir Davydov, Marian Marinov,
Frederic Weisbecker, Johannes Weiner, containers, Tim Hockin,
Glauber Costa, Tejun Heo, David Rientjes, linux-mm, Daniel Walsh,
William Dauchy, Max Kellermann, cgroups, Richard Davies
[-- Attachment #1: Type: text/plain, Size: 2464 bytes --]
Thank you. These are two different things. They may have a relationship
but they ate not the same, and pretending they are is a bad experience.
On Apr 29, 2014 6:04 AM, "Serge Hallyn" <serge.hallyn@ubuntu.com> wrote:
> Quoting Michal Hocko (mhocko@suse.cz):
> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > > Quoting Dwight Engen (dwight.engen@oracle.com):
> > > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > > Marian Marinov <mm@yuhu.biz> wrote:
> > > >
> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > > Dwight Engen wrote:
> > > > > >> Richard Davies wrote:
> > > > > >>> Vladimir Davydov wrote:
> > > > > >>>> In short, kmem limiting for memory cgroups is currently
> broken.
> > > > > >>>> Do not use it. We are working on making it usable though.
> > > > > > ...
> > > > > >>> What is the best mechanism available today, until kmem limits
> > > > > >>> mature?
> > > > > >>>
> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > > >>>
> > > > > >>> Perhaps there is an up-to-date task counter patchset or
> similar?
> > > > > >>
> > > > > >> I updated Frederic's task counter patches and included Max
> > > > > >> Kellermann's fork limiter here:
> > > > > >>
> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > > >>
> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > > >> would find it useful.
> > > > > >
> > > > > > Yes please, I would be interested in that. Ideally even against
> > > > > > 3.14.1 if you have that too.
> > > > >
> > > > > Dwight, do you have these patches in any public repo?
> > > > >
> > > > > I would like to test them also.
> > > >
> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > > >
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> > >
> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > > that a task limit would be a proper cgroup extension, and specifically
> > > that approximating that with a kmem limit is not a reasonable
> substitute.
> >
> > The current state of the kmem limit, which is improving a lot thanks to
> > Vladimir, is not a reason for a new extension/controller. We are just
> > not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
>
> -serge
>
[-- Attachment #2: Type: text/html, Size: 3834 bytes --]
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 13:03 ` Serge Hallyn
` (2 preceding siblings ...)
(?)
@ 2014-04-29 14:04 ` Tim Hockin
-1 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 14:04 UTC (permalink / raw
To: Serge Hallyn
Cc: Vladimir Davydov, Richard Davies, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
David Rientjes, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, Johannes Weiner,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA, Daniel Walsh
Thank you. These are two different things. They may have a relationship
but they ate not the same, and pretending they are is a bad experience.
On Apr 29, 2014 6:04 AM, "Serge Hallyn" <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org> wrote:
> Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > > Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> > > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> > > >
> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > > Dwight Engen wrote:
> > > > > >> Richard Davies wrote:
> > > > > >>> Vladimir Davydov wrote:
> > > > > >>>> In short, kmem limiting for memory cgroups is currently
> broken.
> > > > > >>>> Do not use it. We are working on making it usable though.
> > > > > > ...
> > > > > >>> What is the best mechanism available today, until kmem limits
> > > > > >>> mature?
> > > > > >>>
> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > > >>>
> > > > > >>> Perhaps there is an up-to-date task counter patchset or
> similar?
> > > > > >>
> > > > > >> I updated Frederic's task counter patches and included Max
> > > > > >> Kellermann's fork limiter here:
> > > > > >>
> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > > >>
> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > > >> would find it useful.
> > > > > >
> > > > > > Yes please, I would be interested in that. Ideally even against
> > > > > > 3.14.1 if you have that too.
> > > > >
> > > > > Dwight, do you have these patches in any public repo?
> > > > >
> > > > > I would like to test them also.
> > > >
> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > > >
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> > >
> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > > that a task limit would be a proper cgroup extension, and specifically
> > > that approximating that with a kmem limit is not a reasonable
> substitute.
> >
> > The current state of the kmem limit, which is improving a lot thanks to
> > Vladimir, is not a reason for a new extension/controller. We are just
> > not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
>
> -serge
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 13:03 ` Serge Hallyn
@ 2014-04-29 15:43 ` Michal Hocko
-1 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 15:43 UTC (permalink / raw
To: Serge Hallyn
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker, containers, Daniel Walsh,
cgroups, Glauber Costa, linux-mm, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
On Tue 29-04-14 13:03:53, Serge Hallyn wrote:
> Quoting Michal Hocko (mhocko@suse.cz):
> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > > Quoting Dwight Engen (dwight.engen@oracle.com):
> > > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > > Marian Marinov <mm@yuhu.biz> wrote:
> > > >
> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > > Dwight Engen wrote:
> > > > > >> Richard Davies wrote:
> > > > > >>> Vladimir Davydov wrote:
> > > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > > > >>>> Do not use it. We are working on making it usable though.
> > > > > > ...
> > > > > >>> What is the best mechanism available today, until kmem limits
> > > > > >>> mature?
> > > > > >>>
> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > > >>>
> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > > > >>
> > > > > >> I updated Frederic's task counter patches and included Max
> > > > > >> Kellermann's fork limiter here:
> > > > > >>
> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > > >>
> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > > >> would find it useful.
> > > > > >
> > > > > > Yes please, I would be interested in that. Ideally even against
> > > > > > 3.14.1 if you have that too.
> > > > >
> > > > > Dwight, do you have these patches in any public repo?
> > > > >
> > > > > I would like to test them also.
> > > >
> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > > >
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> > >
> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > > that a task limit would be a proper cgroup extension, and specifically
> > > that approximating that with a kmem limit is not a reasonable substitute.
> >
> > The current state of the kmem limit, which is improving a lot thanks to
> > Vladimir, is not a reason for a new extension/controller. We are just
> > not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
I was not the one who decided that the kmem extension of memory
controller should cover also the task number as a side effect but still
the decision sounds plausible to me because the kmem approach is more
generic.
Btw. if this is a problem them please go ahead and continue the original
discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the
other people involved.
I do not see any new arguments here, except that the kmem implementation
is not ready yet.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 13:03 ` Serge Hallyn
` (4 preceding siblings ...)
(?)
@ 2014-04-29 15:43 ` Michal Hocko
-1 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 15:43 UTC (permalink / raw
To: Serge Hallyn
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, Glauber Costa, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
William Dauchy, Johannes Weiner, Tejun Heo,
cgroups-u79uwXL29TY76Z2rM5mHXA
On Tue 29-04-14 13:03:53, Serge Hallyn wrote:
> Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > > Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> > > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> > > >
> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > > Dwight Engen wrote:
> > > > > >> Richard Davies wrote:
> > > > > >>> Vladimir Davydov wrote:
> > > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > > > >>>> Do not use it. We are working on making it usable though.
> > > > > > ...
> > > > > >>> What is the best mechanism available today, until kmem limits
> > > > > >>> mature?
> > > > > >>>
> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > > >>>
> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > > > >>
> > > > > >> I updated Frederic's task counter patches and included Max
> > > > > >> Kellermann's fork limiter here:
> > > > > >>
> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > > >>
> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > > >> would find it useful.
> > > > > >
> > > > > > Yes please, I would be interested in that. Ideally even against
> > > > > > 3.14.1 if you have that too.
> > > > >
> > > > > Dwight, do you have these patches in any public repo?
> > > > >
> > > > > I would like to test them also.
> > > >
> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > > >
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> > >
> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > > that a task limit would be a proper cgroup extension, and specifically
> > > that approximating that with a kmem limit is not a reasonable substitute.
> >
> > The current state of the kmem limit, which is improving a lot thanks to
> > Vladimir, is not a reason for a new extension/controller. We are just
> > not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
I was not the one who decided that the kmem extension of memory
controller should cover also the task number as a side effect but still
the decision sounds plausible to me because the kmem approach is more
generic.
Btw. if this is a problem them please go ahead and continue the original
discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the
other people involved.
I do not see any new arguments here, except that the kmem implementation
is not ready yet.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 15:43 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 15:43 UTC (permalink / raw
To: Serge Hallyn
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, Johannes Weiner,
Tejun Heo, David Rientjes
On Tue 29-04-14 13:03:53, Serge Hallyn wrote:
> Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
> > > Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
> > > > On Wed, 23 Apr 2014 09:07:28 +0300
> > > > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> > > >
> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
> > > > > > Dwight Engen wrote:
> > > > > >> Richard Davies wrote:
> > > > > >>> Vladimir Davydov wrote:
> > > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
> > > > > >>>> Do not use it. We are working on making it usable though.
> > > > > > ...
> > > > > >>> What is the best mechanism available today, until kmem limits
> > > > > >>> mature?
> > > > > >>>
> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
> > > > > >>>
> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
> > > > > >>
> > > > > >> I updated Frederic's task counter patches and included Max
> > > > > >> Kellermann's fork limiter here:
> > > > > >>
> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> > > > > >>
> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
> > > > > >> would find it useful.
> > > > > >
> > > > > > Yes please, I would be interested in that. Ideally even against
> > > > > > 3.14.1 if you have that too.
> > > > >
> > > > > Dwight, do you have these patches in any public repo?
> > > > >
> > > > > I would like to test them also.
> > > >
> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> > > >
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> > >
> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
> > > that a task limit would be a proper cgroup extension, and specifically
> > > that approximating that with a kmem limit is not a reasonable substitute.
> >
> > The current state of the kmem limit, which is improving a lot thanks to
> > Vladimir, is not a reason for a new extension/controller. We are just
> > not yet there.
>
> It has nothing to do with the state of the limit. I simply don't
> believe that emulating RLIMIT_NPROC by controlling stack size is a
> good idea.
I was not the one who decided that the kmem extension of memory
controller should cover also the task number as a side effect but still
the decision sounds plausible to me because the kmem approach is more
generic.
Btw. if this is a problem them please go ahead and continue the original
discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the
other people involved.
I do not see any new arguments here, except that the kmem implementation
is not ready yet.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 15:43 ` Michal Hocko
(?)
@ 2014-04-29 16:06 ` Tim Hockin
[not found] ` <CAO_RewYZDGLBAKit4CudTbqVk+zfDRX8kP0W6Zz90xJh7abM9Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-04-29 16:51 ` Frederic Weisbecker
-1 siblings, 2 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 16:06 UTC (permalink / raw
To: Michal Hocko
Cc: Serge Hallyn, Richard Davies, Vladimir Davydov, Marian Marinov,
Max Kellermann, Tim Hockin, Frederic Weisbecker, containers,
Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org,
William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes
Why the insistence that we manage something that REALLY IS a
first-class concept (hey, it has it's own RLIMIT) as a side effect of
something that doesn't quite capture what we want to achieve?
Is there some specific technical reason why you think this is a bad
idea? I would think, especially in a more unified hierarchy world,
that more cgroup controllers with smaller sets of responsibility would
make for more manageable code (within limits, obviously).
On Tue, Apr 29, 2014 at 8:43 AM, Michal Hocko <mhocko@suse.cz> wrote:
> On Tue 29-04-14 13:03:53, Serge Hallyn wrote:
>> Quoting Michal Hocko (mhocko@suse.cz):
>> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
>> > > Quoting Dwight Engen (dwight.engen@oracle.com):
>> > > > On Wed, 23 Apr 2014 09:07:28 +0300
>> > > > Marian Marinov <mm@yuhu.biz> wrote:
>> > > >
>> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
>> > > > > > Dwight Engen wrote:
>> > > > > >> Richard Davies wrote:
>> > > > > >>> Vladimir Davydov wrote:
>> > > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
>> > > > > >>>> Do not use it. We are working on making it usable though.
>> > > > > > ...
>> > > > > >>> What is the best mechanism available today, until kmem limits
>> > > > > >>> mature?
>> > > > > >>>
>> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
>> > > > > >>>
>> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
>> > > > > >>
>> > > > > >> I updated Frederic's task counter patches and included Max
>> > > > > >> Kellermann's fork limiter here:
>> > > > > >>
>> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>> > > > > >>
>> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
>> > > > > >> would find it useful.
>> > > > > >
>> > > > > > Yes please, I would be interested in that. Ideally even against
>> > > > > > 3.14.1 if you have that too.
>> > > > >
>> > > > > Dwight, do you have these patches in any public repo?
>> > > > >
>> > > > > I would like to test them also.
>> > > >
>> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>> > > >
>> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>> > >
>> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
>> > > that a task limit would be a proper cgroup extension, and specifically
>> > > that approximating that with a kmem limit is not a reasonable substitute.
>> >
>> > The current state of the kmem limit, which is improving a lot thanks to
>> > Vladimir, is not a reason for a new extension/controller. We are just
>> > not yet there.
>>
>> It has nothing to do with the state of the limit. I simply don't
>> believe that emulating RLIMIT_NPROC by controlling stack size is a
>> good idea.
>
> I was not the one who decided that the kmem extension of memory
> controller should cover also the task number as a side effect but still
> the decision sounds plausible to me because the kmem approach is more
> generic.
>
> Btw. if this is a problem them please go ahead and continue the original
> discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the
> other people involved.
>
> I do not see any new arguments here, except that the kmem implementation
> is not ready yet.
> --
> Michal Hocko
> SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429154345.GH15058-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
@ 2014-04-29 16:06 ` Tim Hockin
0 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 16:06 UTC (permalink / raw
To: Michal Hocko
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
Why the insistence that we manage something that REALLY IS a
first-class concept (hey, it has it's own RLIMIT) as a side effect of
something that doesn't quite capture what we want to achieve?
Is there some specific technical reason why you think this is a bad
idea? I would think, especially in a more unified hierarchy world,
that more cgroup controllers with smaller sets of responsibility would
make for more manageable code (within limits, obviously).
On Tue, Apr 29, 2014 at 8:43 AM, Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org> wrote:
> On Tue 29-04-14 13:03:53, Serge Hallyn wrote:
>> Quoting Michal Hocko (mhocko-AlSwsSmVLrQ@public.gmane.org):
>> > On Mon 28-04-14 18:00:25, Serge Hallyn wrote:
>> > > Quoting Dwight Engen (dwight.engen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org):
>> > > > On Wed, 23 Apr 2014 09:07:28 +0300
>> > > > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>> > > >
>> > > > > On 04/22/2014 11:05 PM, Richard Davies wrote:
>> > > > > > Dwight Engen wrote:
>> > > > > >> Richard Davies wrote:
>> > > > > >>> Vladimir Davydov wrote:
>> > > > > >>>> In short, kmem limiting for memory cgroups is currently broken.
>> > > > > >>>> Do not use it. We are working on making it usable though.
>> > > > > > ...
>> > > > > >>> What is the best mechanism available today, until kmem limits
>> > > > > >>> mature?
>> > > > > >>>
>> > > > > >>> RLIMIT_NPROC exists but is per-user, not per-container.
>> > > > > >>>
>> > > > > >>> Perhaps there is an up-to-date task counter patchset or similar?
>> > > > > >>
>> > > > > >> I updated Frederic's task counter patches and included Max
>> > > > > >> Kellermann's fork limiter here:
>> > > > > >>
>> > > > > >> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>> > > > > >>
>> > > > > >> I can send you a more recent patchset (against 3.13.10) if you
>> > > > > >> would find it useful.
>> > > > > >
>> > > > > > Yes please, I would be interested in that. Ideally even against
>> > > > > > 3.14.1 if you have that too.
>> > > > >
>> > > > > Dwight, do you have these patches in any public repo?
>> > > > >
>> > > > > I would like to test them also.
>> > > >
>> > > > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>> > > >
>> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>> > > > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>> > >
>> > > Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian
>> > > that a task limit would be a proper cgroup extension, and specifically
>> > > that approximating that with a kmem limit is not a reasonable substitute.
>> >
>> > The current state of the kmem limit, which is improving a lot thanks to
>> > Vladimir, is not a reason for a new extension/controller. We are just
>> > not yet there.
>>
>> It has nothing to do with the state of the limit. I simply don't
>> believe that emulating RLIMIT_NPROC by controlling stack size is a
>> good idea.
>
> I was not the one who decided that the kmem extension of memory
> controller should cover also the task number as a side effect but still
> the decision sounds plausible to me because the kmem approach is more
> generic.
>
> Btw. if this is a problem them please go ahead and continue the original
> discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the
> other people involved.
>
> I do not see any new arguments here, except that the kmem implementation
> is not ready yet.
> --
> Michal Hocko
> SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 16:06 ` Tim Hockin
[not found] ` <CAO_RewYZDGLBAKit4CudTbqVk+zfDRX8kP0W6Zz90xJh7abM9Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2014-04-29 16:51 ` Frederic Weisbecker
[not found] ` <20140429165114.GE6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2014-04-29 16:59 ` Tim Hockin
1 sibling, 2 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-29 16:51 UTC (permalink / raw
To: Tim Hockin
Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov,
Marian Marinov, Max Kellermann, Tim Hockin, containers,
Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org,
William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes
On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> Why the insistence that we manage something that REALLY IS a
> first-class concept (hey, it has it's own RLIMIT) as a side effect of
> something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly
motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve
a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does
but it brings other problems that aren't fixable in kmem itself, we can
certainly reconsider this cgroup subsystem. But I haven't yet seen
argument of this kind yet.
>
> Is there some specific technical reason why you think this is a bad
> idea?
> I would think, especially in a more unified hierarchy world,
> that more cgroup controllers with smaller sets of responsibility would
> make for more manageable code (within limits, obviously).
Because it's core code and it adds complications and overhead in the
fork/exit path. We just don't add new core code just for the sake of
slightly prettier interfaces.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <CAO_RewYZDGLBAKit4CudTbqVk+zfDRX8kP0W6Zz90xJh7abM9Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2014-04-29 16:51 ` Frederic Weisbecker
0 siblings, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-29 16:51 UTC (permalink / raw
To: Tim Hockin
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> Why the insistence that we manage something that REALLY IS a
> first-class concept (hey, it has it's own RLIMIT) as a side effect of
> something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly
motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve
a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does
but it brings other problems that aren't fixable in kmem itself, we can
certainly reconsider this cgroup subsystem. But I haven't yet seen
argument of this kind yet.
>
> Is there some specific technical reason why you think this is a bad
> idea?
> I would think, especially in a more unified hierarchy world,
> that more cgroup controllers with smaller sets of responsibility would
> make for more manageable code (within limits, obviously).
Because it's core code and it adds complications and overhead in the
fork/exit path. We just don't add new core code just for the sake of
slightly prettier interfaces.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429165114.GE6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
@ 2014-04-29 16:59 ` Tim Hockin
0 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 16:59 UTC (permalink / raw
To: Frederic Weisbecker
Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov,
Marian Marinov, Max Kellermann, Tim Hockin, containers,
Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org,
William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes
Here's the reason it doesn't work for us: It doesn't work. It was
something like 2 YEARS since we first wanted this, and it STILL does
not work. You're postponing a pretty simple request indefinitely in
favor of a much more complex feature, which still doesn't really give
me what I want. What I want is an API that works like rlimit but
per-cgroup, rather than per-UID.
On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec@gmail.com> wrote:
> On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
>> Why the insistence that we manage something that REALLY IS a
>> first-class concept (hey, it has it's own RLIMIT) as a side effect of
>> something that doesn't quite capture what we want to achieve?
>
> It's not a side effect, the kmem task stack control was partly
> motivated to solve forkbomb issues in containers.
>
> Also in general if we can reuse existing features and code to solve
> a problem without disturbing side issues, we just do it.
>
> Now if kmem doesn't solve the issue for you for any reason, or it does
> but it brings other problems that aren't fixable in kmem itself, we can
> certainly reconsider this cgroup subsystem. But I haven't yet seen
> argument of this kind yet.
>
>>
>> Is there some specific technical reason why you think this is a bad
>> idea?
>> I would think, especially in a more unified hierarchy world,
>> that more cgroup controllers with smaller sets of responsibility would
>> make for more manageable code (within limits, obviously).
>
> Because it's core code and it adds complications and overhead in the
> fork/exit path. We just don't add new core code just for the sake of
> slightly prettier interfaces.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429165114.GE6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
@ 2014-04-29 16:59 ` Tim Hockin
0 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 16:59 UTC (permalink / raw
To: Frederic Weisbecker
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
Here's the reason it doesn't work for us: It doesn't work. It was
something like 2 YEARS since we first wanted this, and it STILL does
not work. You're postponing a pretty simple request indefinitely in
favor of a much more complex feature, which still doesn't really give
me what I want. What I want is an API that works like rlimit but
per-cgroup, rather than per-UID.
On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
>> Why the insistence that we manage something that REALLY IS a
>> first-class concept (hey, it has it's own RLIMIT) as a side effect of
>> something that doesn't quite capture what we want to achieve?
>
> It's not a side effect, the kmem task stack control was partly
> motivated to solve forkbomb issues in containers.
>
> Also in general if we can reuse existing features and code to solve
> a problem without disturbing side issues, we just do it.
>
> Now if kmem doesn't solve the issue for you for any reason, or it does
> but it brings other problems that aren't fixable in kmem itself, we can
> certainly reconsider this cgroup subsystem. But I haven't yet seen
> argument of this kind yet.
>
>>
>> Is there some specific technical reason why you think this is a bad
>> idea?
>> I would think, especially in a more unified hierarchy world,
>> that more cgroup controllers with smaller sets of responsibility would
>> make for more manageable code (within limits, obviously).
>
> Because it's core code and it adds complications and overhead in the
> fork/exit path. We just don't add new core code just for the sake of
> slightly prettier interfaces.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 16:59 ` Tim Hockin
0 siblings, 0 replies; 84+ messages in thread
From: Tim Hockin @ 2014-04-29 16:59 UTC (permalink / raw
To: Frederic Weisbecker
Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov,
Marian Marinov, Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
Here's the reason it doesn't work for us: It doesn't work. It was
something like 2 YEARS since we first wanted this, and it STILL does
not work. You're postponing a pretty simple request indefinitely in
favor of a much more complex feature, which still doesn't really give
me what I want. What I want is an API that works like rlimit but
per-cgroup, rather than per-UID.
On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
>> Why the insistence that we manage something that REALLY IS a
>> first-class concept (hey, it has it's own RLIMIT) as a side effect of
>> something that doesn't quite capture what we want to achieve?
>
> It's not a side effect, the kmem task stack control was partly
> motivated to solve forkbomb issues in containers.
>
> Also in general if we can reuse existing features and code to solve
> a problem without disturbing side issues, we just do it.
>
> Now if kmem doesn't solve the issue for you for any reason, or it does
> but it brings other problems that aren't fixable in kmem itself, we can
> certainly reconsider this cgroup subsystem. But I haven't yet seen
> argument of this kind yet.
>
>>
>> Is there some specific technical reason why you think this is a bad
>> idea?
>> I would think, especially in a more unified hierarchy world,
>> that more cgroup controllers with smaller sets of responsibility would
>> make for more manageable code (within limits, obviously).
>
> Because it's core code and it adds complications and overhead in the
> fork/exit path. We just don't add new core code just for the sake of
> slightly prettier interfaces.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <CAO_Rewa20dneL8e3T4UPnu2Dkv28KTgFJR9_YSmRBKp-_yqewg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2014-04-29 17:06 ` Michal Hocko
2014-04-29 21:44 ` Frederic Weisbecker
1 sibling, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 17:06 UTC (permalink / raw
To: Tim Hockin
Cc: Frederic Weisbecker, Serge Hallyn, Richard Davies,
Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
containers, Daniel Walsh, cgroups, Glauber Costa,
linux-mm@kvack.org, William Dauchy, Johannes Weiner, Tejun Heo,
David Rientjes
On Tue 29-04-14 09:59:30, Tim Hockin wrote:
> Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
> It was something like 2 YEARS since we first wanted this, and it STILL
> does not work.
My recollection is that it was primarily Parallels and Google asking for
the kmem accounting. The reason why I didn't fight against inclusion
although the implementation at the time didn't have a proper slab
shrinking implemented was that that would happen later. Well, that later
hasn't happened yet and we are slowly getting there.
> You're postponing a pretty simple request indefinitely in
> favor of a much more complex feature, which still doesn't really give
> me what I want.
But we cannot simply add a new interface that will have to be maintained
for ever just because something else that is supposed to workaround bugs.
> What I want is an API that works like rlimit but per-cgroup, rather
> than per-UID.
You can use an out-of-tree patchset for the time being or help to get
kmem into shape. If there are principal reasons why kmem cannot be used
then you better articulate them.
> On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> >> Why the insistence that we manage something that REALLY IS a
> >> first-class concept (hey, it has it's own RLIMIT) as a side effect of
> >> something that doesn't quite capture what we want to achieve?
> >
> > It's not a side effect, the kmem task stack control was partly
> > motivated to solve forkbomb issues in containers.
> >
> > Also in general if we can reuse existing features and code to solve
> > a problem without disturbing side issues, we just do it.
> >
> > Now if kmem doesn't solve the issue for you for any reason, or it does
> > but it brings other problems that aren't fixable in kmem itself, we can
> > certainly reconsider this cgroup subsystem. But I haven't yet seen
> > argument of this kind yet.
> >
> >>
> >> Is there some specific technical reason why you think this is a bad
> >> idea?
> >> I would think, especially in a more unified hierarchy world,
> >> that more cgroup controllers with smaller sets of responsibility would
> >> make for more manageable code (within limits, obviously).
> >
> > Because it's core code and it adds complications and overhead in the
> > fork/exit path. We just don't add new core code just for the sake of
> > slightly prettier interfaces.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <CAO_Rewa20dneL8e3T4UPnu2Dkv28KTgFJR9_YSmRBKp-_yqewg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2014-04-29 17:06 ` Michal Hocko
2014-04-29 21:44 ` Frederic Weisbecker
1 sibling, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 17:06 UTC (permalink / raw
To: Tim Hockin
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
On Tue 29-04-14 09:59:30, Tim Hockin wrote:
> Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
> It was something like 2 YEARS since we first wanted this, and it STILL
> does not work.
My recollection is that it was primarily Parallels and Google asking for
the kmem accounting. The reason why I didn't fight against inclusion
although the implementation at the time didn't have a proper slab
shrinking implemented was that that would happen later. Well, that later
hasn't happened yet and we are slowly getting there.
> You're postponing a pretty simple request indefinitely in
> favor of a much more complex feature, which still doesn't really give
> me what I want.
But we cannot simply add a new interface that will have to be maintained
for ever just because something else that is supposed to workaround bugs.
> What I want is an API that works like rlimit but per-cgroup, rather
> than per-UID.
You can use an out-of-tree patchset for the time being or help to get
kmem into shape. If there are principal reasons why kmem cannot be used
then you better articulate them.
> On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> >> Why the insistence that we manage something that REALLY IS a
> >> first-class concept (hey, it has it's own RLIMIT) as a side effect of
> >> something that doesn't quite capture what we want to achieve?
> >
> > It's not a side effect, the kmem task stack control was partly
> > motivated to solve forkbomb issues in containers.
> >
> > Also in general if we can reuse existing features and code to solve
> > a problem without disturbing side issues, we just do it.
> >
> > Now if kmem doesn't solve the issue for you for any reason, or it does
> > but it brings other problems that aren't fixable in kmem itself, we can
> > certainly reconsider this cgroup subsystem. But I haven't yet seen
> > argument of this kind yet.
> >
> >>
> >> Is there some specific technical reason why you think this is a bad
> >> idea?
> >> I would think, especially in a more unified hierarchy world,
> >> that more cgroup controllers with smaller sets of responsibility would
> >> make for more manageable code (within limits, obviously).
> >
> > Because it's core code and it adds complications and overhead in the
> > fork/exit path. We just don't add new core code just for the sake of
> > slightly prettier interfaces.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 17:06 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 17:06 UTC (permalink / raw
To: Tim Hockin
Cc: Frederic Weisbecker, Serge Hallyn, Richard Davies,
Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
On Tue 29-04-14 09:59:30, Tim Hockin wrote:
> Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
> It was something like 2 YEARS since we first wanted this, and it STILL
> does not work.
My recollection is that it was primarily Parallels and Google asking for
the kmem accounting. The reason why I didn't fight against inclusion
although the implementation at the time didn't have a proper slab
shrinking implemented was that that would happen later. Well, that later
hasn't happened yet and we are slowly getting there.
> You're postponing a pretty simple request indefinitely in
> favor of a much more complex feature, which still doesn't really give
> me what I want.
But we cannot simply add a new interface that will have to be maintained
for ever just because something else that is supposed to workaround bugs.
> What I want is an API that works like rlimit but per-cgroup, rather
> than per-UID.
You can use an out-of-tree patchset for the time being or help to get
kmem into shape. If there are principal reasons why kmem cannot be used
then you better articulate them.
> On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker <fweisbec-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> >> Why the insistence that we manage something that REALLY IS a
> >> first-class concept (hey, it has it's own RLIMIT) as a side effect of
> >> something that doesn't quite capture what we want to achieve?
> >
> > It's not a side effect, the kmem task stack control was partly
> > motivated to solve forkbomb issues in containers.
> >
> > Also in general if we can reuse existing features and code to solve
> > a problem without disturbing side issues, we just do it.
> >
> > Now if kmem doesn't solve the issue for you for any reason, or it does
> > but it brings other problems that aren't fixable in kmem itself, we can
> > certainly reconsider this cgroup subsystem. But I haven't yet seen
> > argument of this kind yet.
> >
> >>
> >> Is there some specific technical reason why you think this is a bad
> >> idea?
> >> I would think, especially in a more unified hierarchy world,
> >> that more cgroup controllers with smaller sets of responsibility would
> >> make for more manageable code (within limits, obviously).
> >
> > Because it's core code and it adds complications and overhead in the
> > fork/exit path. We just don't add new core code just for the sake of
> > slightly prettier interfaces.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 17:06 ` Michal Hocko
(?)
@ 2014-04-29 17:30 ` Dwight Engen
[not found] ` <20140429133039.162d9dd7-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2014-04-29 18:09 ` Richard Davies
-1 siblings, 2 replies; 84+ messages in thread
From: Dwight Engen @ 2014-04-29 17:30 UTC (permalink / raw
To: Michal Hocko
Cc: Tim Hockin, Richard Davies, Vladimir Davydov, David Rientjes,
Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Serge Hallyn, Glauber Costa, linux-mm@kvack.org,
William Dauchy, Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
On Tue, 29 Apr 2014 19:06:39 +0200
Michal Hocko <mhocko@suse.cz> wrote:
> On Tue 29-04-14 09:59:30, Tim Hockin wrote:
> > Here's the reason it doesn't work for us: It doesn't work.
>
> There is a "simple" solution for that. Help us to fix it.
>
> > It was something like 2 YEARS since we first wanted this, and it
> > STILL does not work.
>
> My recollection is that it was primarily Parallels and Google asking
> for the kmem accounting. The reason why I didn't fight against
> inclusion although the implementation at the time didn't have a
> proper slab shrinking implemented was that that would happen later.
> Well, that later hasn't happened yet and we are slowly getting there.
>
> > You're postponing a pretty simple request indefinitely in
> > favor of a much more complex feature, which still doesn't really
> > give me what I want.
>
> But we cannot simply add a new interface that will have to be
> maintained for ever just because something else that is supposed to
> workaround bugs.
>
> > What I want is an API that works like rlimit but per-cgroup, rather
> > than per-UID.
>
> You can use an out-of-tree patchset for the time being or help to get
> kmem into shape. If there are principal reasons why kmem cannot be
> used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in
general? Richard would have to verify, but I suspect kmem is not currently
viable as a process limiter for him because icache/dcache/stack is all
accounted together.
> > On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker
> > <fweisbec@gmail.com> wrote:
> > > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> > >> Why the insistence that we manage something that REALLY IS a
> > >> first-class concept (hey, it has it's own RLIMIT) as a side
> > >> effect of something that doesn't quite capture what we want to
> > >> achieve?
> > >
> > > It's not a side effect, the kmem task stack control was partly
> > > motivated to solve forkbomb issues in containers.
> > >
> > > Also in general if we can reuse existing features and code to
> > > solve a problem without disturbing side issues, we just do it.
> > >
> > > Now if kmem doesn't solve the issue for you for any reason, or it
> > > does but it brings other problems that aren't fixable in kmem
> > > itself, we can certainly reconsider this cgroup subsystem. But I
> > > haven't yet seen argument of this kind yet.
> > >
> > >>
> > >> Is there some specific technical reason why you think this is a
> > >> bad idea?
> > >> I would think, especially in a more unified hierarchy world,
> > >> that more cgroup controllers with smaller sets of responsibility
> > >> would make for more manageable code (within limits, obviously).
> > >
> > > Because it's core code and it adds complications and overhead in
> > > the fork/exit path. We just don't add new core code just for the
> > > sake of slightly prettier interfaces.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429170639.GA25609-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
@ 2014-04-29 17:30 ` Dwight Engen
0 siblings, 0 replies; 84+ messages in thread
From: Dwight Engen @ 2014-04-29 17:30 UTC (permalink / raw
To: Michal Hocko
Cc: Richard Davies, Vladimir Davydov, Marian Marinov, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Johannes Weiner,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
David Rientjes, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
On Tue, 29 Apr 2014 19:06:39 +0200
Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org> wrote:
> On Tue 29-04-14 09:59:30, Tim Hockin wrote:
> > Here's the reason it doesn't work for us: It doesn't work.
>
> There is a "simple" solution for that. Help us to fix it.
>
> > It was something like 2 YEARS since we first wanted this, and it
> > STILL does not work.
>
> My recollection is that it was primarily Parallels and Google asking
> for the kmem accounting. The reason why I didn't fight against
> inclusion although the implementation at the time didn't have a
> proper slab shrinking implemented was that that would happen later.
> Well, that later hasn't happened yet and we are slowly getting there.
>
> > You're postponing a pretty simple request indefinitely in
> > favor of a much more complex feature, which still doesn't really
> > give me what I want.
>
> But we cannot simply add a new interface that will have to be
> maintained for ever just because something else that is supposed to
> workaround bugs.
>
> > What I want is an API that works like rlimit but per-cgroup, rather
> > than per-UID.
>
> You can use an out-of-tree patchset for the time being or help to get
> kmem into shape. If there are principal reasons why kmem cannot be
> used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in
general? Richard would have to verify, but I suspect kmem is not currently
viable as a process limiter for him because icache/dcache/stack is all
accounted together.
> > On Tue, Apr 29, 2014 at 9:51 AM, Frederic Weisbecker
> > <fweisbec-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > On Tue, Apr 29, 2014 at 09:06:22AM -0700, Tim Hockin wrote:
> > >> Why the insistence that we manage something that REALLY IS a
> > >> first-class concept (hey, it has it's own RLIMIT) as a side
> > >> effect of something that doesn't quite capture what we want to
> > >> achieve?
> > >
> > > It's not a side effect, the kmem task stack control was partly
> > > motivated to solve forkbomb issues in containers.
> > >
> > > Also in general if we can reuse existing features and code to
> > > solve a problem without disturbing side issues, we just do it.
> > >
> > > Now if kmem doesn't solve the issue for you for any reason, or it
> > > does but it brings other problems that aren't fixable in kmem
> > > itself, we can certainly reconsider this cgroup subsystem. But I
> > > haven't yet seen argument of this kind yet.
> > >
> > >>
> > >> Is there some specific technical reason why you think this is a
> > >> bad idea?
> > >> I would think, especially in a more unified hierarchy world,
> > >> that more cgroup controllers with smaller sets of responsibility
> > >> would make for more manageable code (within limits, obviously).
> > >
> > > Because it's core code and it adds complications and overhead in
> > > the fork/exit path. We just don't add new core code just for the
> > > sake of slightly prettier interfaces.
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 17:30 ` Dwight Engen
[not found] ` <20140429133039.162d9dd7-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
@ 2014-04-29 18:09 ` Richard Davies
[not found] ` <20140429180927.GB29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-29 18:27 ` Michal Hocko
1 sibling, 2 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-29 18:09 UTC (permalink / raw
To: Dwight Engen
Cc: Michal Hocko, Tim Hockin, Vladimir Davydov, David Rientjes,
Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
Dwight Engen wrote:
> Michal Hocko wrote:
> > Tim Hockin wrote:
> > > Here's the reason it doesn't work for us: It doesn't work.
> >
> > There is a "simple" solution for that. Help us to fix it.
> >
> > > It was something like 2 YEARS since we first wanted this, and it
> > > STILL does not work.
> >
> > My recollection is that it was primarily Parallels and Google asking
> > for the kmem accounting. The reason why I didn't fight against
> > inclusion although the implementation at the time didn't have a
> > proper slab shrinking implemented was that that would happen later.
> > Well, that later hasn't happened yet and we are slowly getting there.
> >
> > > You're postponing a pretty simple request indefinitely in
> > > favor of a much more complex feature, which still doesn't really
> > > give me what I want.
> >
> > But we cannot simply add a new interface that will have to be
> > maintained for ever just because something else that is supposed to
> > workaround bugs.
> >
> > > What I want is an API that works like rlimit but per-cgroup, rather
> > > than per-UID.
> >
> > You can use an out-of-tree patchset for the time being or help to get
> > kmem into shape. If there are principal reasons why kmem cannot be
> > used then you better articulate them.
>
> Is there a plan to separately account/limit stack pages vs kmem in
> general? Richard would have to verify, but I suspect kmem is not currently
> viable as a process limiter for him because icache/dcache/stack is all
> accounted together.
Certainly I would like to be able to limit container fork-bombs without
limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to
catch fork bombs early enough. I would definitely like more than 256MB of
disk caching.
So if we go the "working kmem" route, I would like to be able to specify a
limit excluding disk cache.
I am also somewhat worried that normal software use could legitimately go
above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
just by booting a distro with a few daemons in a container.
Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429133039.162d9dd7-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
@ 2014-04-29 18:09 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-29 18:09 UTC (permalink / raw
To: Dwight Engen
Cc: Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Johannes Weiner, Tim Hockin, Glauber Costa,
Michal Hocko, linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy,
David Rientjes, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
Dwight Engen wrote:
> Michal Hocko wrote:
> > Tim Hockin wrote:
> > > Here's the reason it doesn't work for us: It doesn't work.
> >
> > There is a "simple" solution for that. Help us to fix it.
> >
> > > It was something like 2 YEARS since we first wanted this, and it
> > > STILL does not work.
> >
> > My recollection is that it was primarily Parallels and Google asking
> > for the kmem accounting. The reason why I didn't fight against
> > inclusion although the implementation at the time didn't have a
> > proper slab shrinking implemented was that that would happen later.
> > Well, that later hasn't happened yet and we are slowly getting there.
> >
> > > You're postponing a pretty simple request indefinitely in
> > > favor of a much more complex feature, which still doesn't really
> > > give me what I want.
> >
> > But we cannot simply add a new interface that will have to be
> > maintained for ever just because something else that is supposed to
> > workaround bugs.
> >
> > > What I want is an API that works like rlimit but per-cgroup, rather
> > > than per-UID.
> >
> > You can use an out-of-tree patchset for the time being or help to get
> > kmem into shape. If there are principal reasons why kmem cannot be
> > used then you better articulate them.
>
> Is there a plan to separately account/limit stack pages vs kmem in
> general? Richard would have to verify, but I suspect kmem is not currently
> viable as a process limiter for him because icache/dcache/stack is all
> accounted together.
Certainly I would like to be able to limit container fork-bombs without
limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to
catch fork bombs early enough. I would definitely like more than 256MB of
disk caching.
So if we go the "working kmem" route, I would like to be able to specify a
limit excluding disk cache.
I am also somewhat worried that normal software use could legitimately go
above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
just by booting a distro with a few daemons in a container.
Richard.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429180927.GB29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-29 18:27 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 18:27 UTC (permalink / raw
To: Richard Davies
Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes,
Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
On Tue 29-04-14 19:09:27, Richard Davies wrote:
> Dwight Engen wrote:
> > Michal Hocko wrote:
> > > Tim Hockin wrote:
> > > > Here's the reason it doesn't work for us: It doesn't work.
> > >
> > > There is a "simple" solution for that. Help us to fix it.
> > >
> > > > It was something like 2 YEARS since we first wanted this, and it
> > > > STILL does not work.
> > >
> > > My recollection is that it was primarily Parallels and Google asking
> > > for the kmem accounting. The reason why I didn't fight against
> > > inclusion although the implementation at the time didn't have a
> > > proper slab shrinking implemented was that that would happen later.
> > > Well, that later hasn't happened yet and we are slowly getting there.
> > >
> > > > You're postponing a pretty simple request indefinitely in
> > > > favor of a much more complex feature, which still doesn't really
> > > > give me what I want.
> > >
> > > But we cannot simply add a new interface that will have to be
> > > maintained for ever just because something else that is supposed to
> > > workaround bugs.
> > >
> > > > What I want is an API that works like rlimit but per-cgroup, rather
> > > > than per-UID.
> > >
> > > You can use an out-of-tree patchset for the time being or help to get
> > > kmem into shape. If there are principal reasons why kmem cannot be
> > > used then you better articulate them.
> >
> > Is there a plan to separately account/limit stack pages vs kmem in
> > general? Richard would have to verify, but I suspect kmem is not currently
> > viable as a process limiter for him because icache/dcache/stack is all
> > accounted together.
>
> Certainly I would like to be able to limit container fork-bombs without
> limiting the amount of disk IO caching for processes in those containers.
>
> In my testing with of kmem limits, I needed a limit of 256MB or lower to
> catch fork bombs early enough. I would definitely like more than 256MB of
> disk caching.
>
> So if we go the "working kmem" route, I would like to be able to specify a
> limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a
userspace accounted memory with the memory cgroup controller. And you
do not have to limit that one. Kmem accounting refers to kernel internal
allocations - slab memory and per process kernel stack. You can see how
much memory is allocated per container by memory.kmem.usage_in_bytes or
have a look at /proc/slabinfo to see what kind of memory kernel
allocates globally and might be accounted for a container as well.
The primary problem with the kmem accounting right now is that such a
memory is not "reclaimed" and so if the kmem limit is reached all the
further kmem allocations fail. The biggest user of the kmem allocations
on many systems is dentry and inode chache which is reclaimable easily.
When this is implemented the kmem limit will be usable to both prevent
forkbombs but also other DOS scenarios when the kernel is pushed to
allocate a huge amount of memory.
HTH
> I am also somewhat worried that normal software use could legitimately go
> above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
> just by booting a distro with a few daemons in a container.
>
> Richard.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429180927.GB29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-29 18:27 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 18:27 UTC (permalink / raw
To: Richard Davies
Cc: Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Johannes Weiner,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA, Daniel Walsh
On Tue 29-04-14 19:09:27, Richard Davies wrote:
> Dwight Engen wrote:
> > Michal Hocko wrote:
> > > Tim Hockin wrote:
> > > > Here's the reason it doesn't work for us: It doesn't work.
> > >
> > > There is a "simple" solution for that. Help us to fix it.
> > >
> > > > It was something like 2 YEARS since we first wanted this, and it
> > > > STILL does not work.
> > >
> > > My recollection is that it was primarily Parallels and Google asking
> > > for the kmem accounting. The reason why I didn't fight against
> > > inclusion although the implementation at the time didn't have a
> > > proper slab shrinking implemented was that that would happen later.
> > > Well, that later hasn't happened yet and we are slowly getting there.
> > >
> > > > You're postponing a pretty simple request indefinitely in
> > > > favor of a much more complex feature, which still doesn't really
> > > > give me what I want.
> > >
> > > But we cannot simply add a new interface that will have to be
> > > maintained for ever just because something else that is supposed to
> > > workaround bugs.
> > >
> > > > What I want is an API that works like rlimit but per-cgroup, rather
> > > > than per-UID.
> > >
> > > You can use an out-of-tree patchset for the time being or help to get
> > > kmem into shape. If there are principal reasons why kmem cannot be
> > > used then you better articulate them.
> >
> > Is there a plan to separately account/limit stack pages vs kmem in
> > general? Richard would have to verify, but I suspect kmem is not currently
> > viable as a process limiter for him because icache/dcache/stack is all
> > accounted together.
>
> Certainly I would like to be able to limit container fork-bombs without
> limiting the amount of disk IO caching for processes in those containers.
>
> In my testing with of kmem limits, I needed a limit of 256MB or lower to
> catch fork bombs early enough. I would definitely like more than 256MB of
> disk caching.
>
> So if we go the "working kmem" route, I would like to be able to specify a
> limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a
userspace accounted memory with the memory cgroup controller. And you
do not have to limit that one. Kmem accounting refers to kernel internal
allocations - slab memory and per process kernel stack. You can see how
much memory is allocated per container by memory.kmem.usage_in_bytes or
have a look at /proc/slabinfo to see what kind of memory kernel
allocates globally and might be accounted for a container as well.
The primary problem with the kmem accounting right now is that such a
memory is not "reclaimed" and so if the kmem limit is reached all the
further kmem allocations fail. The biggest user of the kmem allocations
on many systems is dentry and inode chache which is reclaimable easily.
When this is implemented the kmem limit will be usable to both prevent
forkbombs but also other DOS scenarios when the kernel is pushed to
allocate a huge amount of memory.
HTH
> I am also somewhat worried that normal software use could legitimately go
> above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
> just by booting a distro with a few daemons in a container.
>
> Richard.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 18:27 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 18:27 UTC (permalink / raw
To: Richard Davies
Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes,
Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
William Dauchy, Johannes Weiner, Tejun Heo,
cgroups-u79uwXL29TY76Z2rM5mHXA, Daniel Walsh
On Tue 29-04-14 19:09:27, Richard Davies wrote:
> Dwight Engen wrote:
> > Michal Hocko wrote:
> > > Tim Hockin wrote:
> > > > Here's the reason it doesn't work for us: It doesn't work.
> > >
> > > There is a "simple" solution for that. Help us to fix it.
> > >
> > > > It was something like 2 YEARS since we first wanted this, and it
> > > > STILL does not work.
> > >
> > > My recollection is that it was primarily Parallels and Google asking
> > > for the kmem accounting. The reason why I didn't fight against
> > > inclusion although the implementation at the time didn't have a
> > > proper slab shrinking implemented was that that would happen later.
> > > Well, that later hasn't happened yet and we are slowly getting there.
> > >
> > > > You're postponing a pretty simple request indefinitely in
> > > > favor of a much more complex feature, which still doesn't really
> > > > give me what I want.
> > >
> > > But we cannot simply add a new interface that will have to be
> > > maintained for ever just because something else that is supposed to
> > > workaround bugs.
> > >
> > > > What I want is an API that works like rlimit but per-cgroup, rather
> > > > than per-UID.
> > >
> > > You can use an out-of-tree patchset for the time being or help to get
> > > kmem into shape. If there are principal reasons why kmem cannot be
> > > used then you better articulate them.
> >
> > Is there a plan to separately account/limit stack pages vs kmem in
> > general? Richard would have to verify, but I suspect kmem is not currently
> > viable as a process limiter for him because icache/dcache/stack is all
> > accounted together.
>
> Certainly I would like to be able to limit container fork-bombs without
> limiting the amount of disk IO caching for processes in those containers.
>
> In my testing with of kmem limits, I needed a limit of 256MB or lower to
> catch fork bombs early enough. I would definitely like more than 256MB of
> disk caching.
>
> So if we go the "working kmem" route, I would like to be able to specify a
> limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a
userspace accounted memory with the memory cgroup controller. And you
do not have to limit that one. Kmem accounting refers to kernel internal
allocations - slab memory and per process kernel stack. You can see how
much memory is allocated per container by memory.kmem.usage_in_bytes or
have a look at /proc/slabinfo to see what kind of memory kernel
allocates globally and might be accounted for a container as well.
The primary problem with the kmem accounting right now is that such a
memory is not "reclaimed" and so if the kmem limit is reached all the
further kmem allocations fail. The biggest user of the kmem allocations
on many systems is dentry and inode chache which is reclaimable easily.
When this is implemented the kmem limit will be usable to both prevent
forkbombs but also other DOS scenarios when the kernel is pushed to
allocate a huge amount of memory.
HTH
> I am also somewhat worried that normal software use could legitimately go
> above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
> just by booting a distro with a few daemons in a container.
>
> Richard.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 18:39 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-29 18:39 UTC (permalink / raw
To: Michal Hocko
Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes,
Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
Michal Hocko wrote:
> Richard Davies wrote:
> > Dwight Engen wrote:
> > > Is there a plan to separately account/limit stack pages vs kmem in
> > > general? Richard would have to verify, but I suspect kmem is not
> > > currently viable as a process limiter for him because
> > > icache/dcache/stack is all accounted together.
> >
> > Certainly I would like to be able to limit container fork-bombs without
> > limiting the amount of disk IO caching for processes in those containers.
> >
> > In my testing with of kmem limits, I needed a limit of 256MB or lower to
> > catch fork bombs early enough. I would definitely like more than 256MB of
> > disk caching.
> >
> > So if we go the "working kmem" route, I would like to be able to specify a
> > limit excluding disk cache.
>
> Page cache (which is what you mean by disk cache probably) is a
> userspace accounted memory with the memory cgroup controller. And you
> do not have to limit that one.
OK, that's helpful - thanks.
As an aside, with the normal (non-kmem) cgroup controller, is there a way
for me to exclude page cache and only limit the equivalent of the rss line
in memory.stat?
e.g. say I have a 256GB physical machine, running 200 containers, each with
1GB normal-mem limit (for running software) and 256MB kmem limit (to stop
fork-bombs).
The physical disk IO bandwidth is a shared resource between all the
containers, so ideally I would like the kernel to used the 56GB of RAM as
shared page cache however it best reduces physical IOPs, rather than having
a per-container limit.
Thanks,
Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 18:39 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-04-29 18:39 UTC (permalink / raw
To: Michal Hocko
Cc: Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Johannes Weiner,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA, Daniel Walsh
Michal Hocko wrote:
> Richard Davies wrote:
> > Dwight Engen wrote:
> > > Is there a plan to separately account/limit stack pages vs kmem in
> > > general? Richard would have to verify, but I suspect kmem is not
> > > currently viable as a process limiter for him because
> > > icache/dcache/stack is all accounted together.
> >
> > Certainly I would like to be able to limit container fork-bombs without
> > limiting the amount of disk IO caching for processes in those containers.
> >
> > In my testing with of kmem limits, I needed a limit of 256MB or lower to
> > catch fork bombs early enough. I would definitely like more than 256MB of
> > disk caching.
> >
> > So if we go the "working kmem" route, I would like to be able to specify a
> > limit excluding disk cache.
>
> Page cache (which is what you mean by disk cache probably) is a
> userspace accounted memory with the memory cgroup controller. And you
> do not have to limit that one.
OK, that's helpful - thanks.
As an aside, with the normal (non-kmem) cgroup controller, is there a way
for me to exclude page cache and only limit the equivalent of the rss line
in memory.stat?
e.g. say I have a 256GB physical machine, running 200 containers, each with
1GB normal-mem limit (for running software) and 256MB kmem limit (to stop
fork-bombs).
The physical disk IO bandwidth is a shared resource between all the
containers, so ideally I would like the kernel to used the 56GB of RAM as
shared page cache however it best reduces physical IOPs, rather than having
a per-container limit.
Thanks,
Richard.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 18:39 ` Richard Davies
(?)
@ 2014-04-29 19:03 ` Michal Hocko
-1 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 19:03 UTC (permalink / raw
To: Richard Davies
Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes,
Marian Marinov, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
On Tue 29-04-14 19:39:28, Richard Davies wrote:
> Michal Hocko wrote:
> > Richard Davies wrote:
> > > Dwight Engen wrote:
> > > > Is there a plan to separately account/limit stack pages vs kmem in
> > > > general? Richard would have to verify, but I suspect kmem is not
> > > > currently viable as a process limiter for him because
> > > > icache/dcache/stack is all accounted together.
> > >
> > > Certainly I would like to be able to limit container fork-bombs without
> > > limiting the amount of disk IO caching for processes in those containers.
> > >
> > > In my testing with of kmem limits, I needed a limit of 256MB or lower to
> > > catch fork bombs early enough. I would definitely like more than 256MB of
> > > disk caching.
> > >
> > > So if we go the "working kmem" route, I would like to be able to specify a
> > > limit excluding disk cache.
> >
> > Page cache (which is what you mean by disk cache probably) is a
> > userspace accounted memory with the memory cgroup controller. And you
> > do not have to limit that one.
>
> OK, that's helpful - thanks.
>
> As an aside, with the normal (non-kmem) cgroup controller, is there a way
> for me to exclude page cache and only limit the equivalent of the rss line
> in memory.stat?
No
> e.g. say I have a 256GB physical machine, running 200 containers, each with
> 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop
> fork-bombs).
>
> The physical disk IO bandwidth is a shared resource between all the
> containers, so ideally I would like the kernel to used the 56GB of RAM as
> shared page cache however it best reduces physical IOPs, rather than having
> a per-container limit.
Then do not use any memory.limit_in_bytes and if there is a memory
pressure then the global reclaim will shrink all the containers
proportionally and the page cache will be the #1 target for the
reclaim (but we are getting off-topic here I am afraid).
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429183928.GF29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-04-29 19:03 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-29 19:03 UTC (permalink / raw
To: Richard Davies
Cc: Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Johannes Weiner,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA, Daniel Walsh
On Tue 29-04-14 19:39:28, Richard Davies wrote:
> Michal Hocko wrote:
> > Richard Davies wrote:
> > > Dwight Engen wrote:
> > > > Is there a plan to separately account/limit stack pages vs kmem in
> > > > general? Richard would have to verify, but I suspect kmem is not
> > > > currently viable as a process limiter for him because
> > > > icache/dcache/stack is all accounted together.
> > >
> > > Certainly I would like to be able to limit container fork-bombs without
> > > limiting the amount of disk IO caching for processes in those containers.
> > >
> > > In my testing with of kmem limits, I needed a limit of 256MB or lower to
> > > catch fork bombs early enough. I would definitely like more than 256MB of
> > > disk caching.
> > >
> > > So if we go the "working kmem" route, I would like to be able to specify a
> > > limit excluding disk cache.
> >
> > Page cache (which is what you mean by disk cache probably) is a
> > userspace accounted memory with the memory cgroup controller. And you
> > do not have to limit that one.
>
> OK, that's helpful - thanks.
>
> As an aside, with the normal (non-kmem) cgroup controller, is there a way
> for me to exclude page cache and only limit the equivalent of the rss line
> in memory.stat?
No
> e.g. say I have a 256GB physical machine, running 200 containers, each with
> 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop
> fork-bombs).
>
> The physical disk IO bandwidth is a shared resource between all the
> containers, so ideally I would like the kernel to used the 56GB of RAM as
> shared page cache however it best reduces physical IOPs, rather than having
> a per-container limit.
Then do not use any memory.limit_in_bytes and if there is a memory
pressure then the global reclaim will shrink all the containers
proportionally and the page cache will be the #1 target for the
reclaim (but we are getting off-topic here I am afraid).
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 21:36 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-29 21:36 UTC (permalink / raw
To: Michal Hocko, Richard Davies
Cc: Dwight Engen, Tim Hockin, Vladimir Davydov, David Rientjes,
Max Kellermann, Tim Hockin, Frederic Weisbecker, containers,
Serge Hallyn, Glauber Costa, linux-mm, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
On 04/29/2014 09:27 PM, Michal Hocko wrote:
> On Tue 29-04-14 19:09:27, Richard Davies wrote:
>> Dwight Engen wrote:
>>> Michal Hocko wrote:
>>>> Tim Hockin wrote:
>>>>> Here's the reason it doesn't work for us: It doesn't work.
>>>>
>>>> There is a "simple" solution for that. Help us to fix it.
>>>>
>>>>> It was something like 2 YEARS since we first wanted this, and it
>>>>> STILL does not work.
>>>>
>>>> My recollection is that it was primarily Parallels and Google asking
>>>> for the kmem accounting. The reason why I didn't fight against
>>>> inclusion although the implementation at the time didn't have a
>>>> proper slab shrinking implemented was that that would happen later.
>>>> Well, that later hasn't happened yet and we are slowly getting there.
>>>>
>>>>> You're postponing a pretty simple request indefinitely in
>>>>> favor of a much more complex feature, which still doesn't really
>>>>> give me what I want.
>>>>
>>>> But we cannot simply add a new interface that will have to be
>>>> maintained for ever just because something else that is supposed to
>>>> workaround bugs.
>>>>
>>>>> What I want is an API that works like rlimit but per-cgroup, rather
>>>>> than per-UID.
>>>>
>>>> You can use an out-of-tree patchset for the time being or help to get
>>>> kmem into shape. If there are principal reasons why kmem cannot be
>>>> used then you better articulate them.
>>>
>>> Is there a plan to separately account/limit stack pages vs kmem in
>>> general? Richard would have to verify, but I suspect kmem is not currently
>>> viable as a process limiter for him because icache/dcache/stack is all
>>> accounted together.
>>
>> Certainly I would like to be able to limit container fork-bombs without
>> limiting the amount of disk IO caching for processes in those containers.
>>
>> In my testing with of kmem limits, I needed a limit of 256MB or lower to
>> catch fork bombs early enough. I would definitely like more than 256MB of
>> disk caching.
>>
>> So if we go the "working kmem" route, I would like to be able to specify a
>> limit excluding disk cache.
>
> Page cache (which is what you mean by disk cache probably) is a
> userspace accounted memory with the memory cgroup controller. And you
> do not have to limit that one. Kmem accounting refers to kernel internal
> allocations - slab memory and per process kernel stack. You can see how
> much memory is allocated per container by memory.kmem.usage_in_bytes or
> have a look at /proc/slabinfo to see what kind of memory kernel
> allocates globally and might be accounted for a container as well.
>
> The primary problem with the kmem accounting right now is that such a
> memory is not "reclaimed" and so if the kmem limit is reached all the
> further kmem allocations fail. The biggest user of the kmem allocations
> on many systems is dentry and inode chache which is reclaimable easily.
> When this is implemented the kmem limit will be usable to both prevent
> forkbombs but also other DOS scenarios when the kernel is pushed to
> allocate a huge amount of memory.
I would have to disagree here.
If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem.
The simple scheduling of so many processes generates have load on the machine.
Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure on the kmem, but will would not produce many
processes.
On the other hand, forking thousands of processes with low memory footprint will hit the scheduler a lot faster then
hitting the kmem limit.
Kmem limit is something that we need! But firmly believe that we need a simple NPROC limit for cgroups.
-hackman
>
> HTH
>
>> I am also somewhat worried that normal software use could legitimately go
>> above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
>> just by booting a distro with a few daemons in a container.
>>
>> Richard.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 21:36 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-04-29 21:36 UTC (permalink / raw
To: Michal Hocko, Richard Davies
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Johannes Weiner,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On 04/29/2014 09:27 PM, Michal Hocko wrote:
> On Tue 29-04-14 19:09:27, Richard Davies wrote:
>> Dwight Engen wrote:
>>> Michal Hocko wrote:
>>>> Tim Hockin wrote:
>>>>> Here's the reason it doesn't work for us: It doesn't work.
>>>>
>>>> There is a "simple" solution for that. Help us to fix it.
>>>>
>>>>> It was something like 2 YEARS since we first wanted this, and it
>>>>> STILL does not work.
>>>>
>>>> My recollection is that it was primarily Parallels and Google asking
>>>> for the kmem accounting. The reason why I didn't fight against
>>>> inclusion although the implementation at the time didn't have a
>>>> proper slab shrinking implemented was that that would happen later.
>>>> Well, that later hasn't happened yet and we are slowly getting there.
>>>>
>>>>> You're postponing a pretty simple request indefinitely in
>>>>> favor of a much more complex feature, which still doesn't really
>>>>> give me what I want.
>>>>
>>>> But we cannot simply add a new interface that will have to be
>>>> maintained for ever just because something else that is supposed to
>>>> workaround bugs.
>>>>
>>>>> What I want is an API that works like rlimit but per-cgroup, rather
>>>>> than per-UID.
>>>>
>>>> You can use an out-of-tree patchset for the time being or help to get
>>>> kmem into shape. If there are principal reasons why kmem cannot be
>>>> used then you better articulate them.
>>>
>>> Is there a plan to separately account/limit stack pages vs kmem in
>>> general? Richard would have to verify, but I suspect kmem is not currently
>>> viable as a process limiter for him because icache/dcache/stack is all
>>> accounted together.
>>
>> Certainly I would like to be able to limit container fork-bombs without
>> limiting the amount of disk IO caching for processes in those containers.
>>
>> In my testing with of kmem limits, I needed a limit of 256MB or lower to
>> catch fork bombs early enough. I would definitely like more than 256MB of
>> disk caching.
>>
>> So if we go the "working kmem" route, I would like to be able to specify a
>> limit excluding disk cache.
>
> Page cache (which is what you mean by disk cache probably) is a
> userspace accounted memory with the memory cgroup controller. And you
> do not have to limit that one. Kmem accounting refers to kernel internal
> allocations - slab memory and per process kernel stack. You can see how
> much memory is allocated per container by memory.kmem.usage_in_bytes or
> have a look at /proc/slabinfo to see what kind of memory kernel
> allocates globally and might be accounted for a container as well.
>
> The primary problem with the kmem accounting right now is that such a
> memory is not "reclaimed" and so if the kmem limit is reached all the
> further kmem allocations fail. The biggest user of the kmem allocations
> on many systems is dentry and inode chache which is reclaimable easily.
> When this is implemented the kmem limit will be usable to both prevent
> forkbombs but also other DOS scenarios when the kernel is pushed to
> allocate a huge amount of memory.
I would have to disagree here.
If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem.
The simple scheduling of so many processes generates have load on the machine.
Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure on the kmem, but will would not produce many
processes.
On the other hand, forking thousands of processes with low memory footprint will hit the scheduler a lot faster then
hitting the kmem limit.
Kmem limit is something that we need! But firmly believe that we need a simple NPROC limit for cgroups.
-hackman
>
> HTH
>
>> I am also somewhat worried that normal software use could legitimately go
>> above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
>> just by booting a distro with a few daemons in a container.
>>
>> Richard.
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <CAO_Rewa20dneL8e3T4UPnu2Dkv28KTgFJR9_YSmRBKp-_yqewg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2014-04-29 21:44 ` Frederic Weisbecker
2014-04-29 21:44 ` Frederic Weisbecker
1 sibling, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-29 21:44 UTC (permalink / raw
To: Tim Hockin
Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov,
Marian Marinov, Max Kellermann, Tim Hockin, containers,
Daniel Walsh, cgroups, Glauber Costa, linux-mm@kvack.org,
William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes
On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
> Here's the reason it doesn't work for us: It doesn't work. It was
> something like 2 YEARS since we first wanted this, and it STILL does
> not work.
When I was working on the task counter cgroup subsystem 2 years
ago, the patches were actually pushed back by google people, in favour
of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of
number of tasks as a resource is awkward and that the real resource
in the game comes from kernel memory exhaustion due to task stack being
allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially
since that could solve others people issues at the same time. kmem
cgroup has a quite large domain of application.
> You're postponing a pretty simple request indefinitely in
> favor of a much more complex feature, which still doesn't really give
> me what I want. What I want is an API that works like rlimit but
> per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter
cgroup subsystem is simpler than extending the kmem code to apply limits
to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already
familiar with this.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <CAO_Rewa20dneL8e3T4UPnu2Dkv28KTgFJR9_YSmRBKp-_yqewg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-04-29 17:06 ` Michal Hocko
@ 2014-04-29 21:44 ` Frederic Weisbecker
1 sibling, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-29 21:44 UTC (permalink / raw
To: Tim Hockin
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA,
Daniel Walsh
On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
> Here's the reason it doesn't work for us: It doesn't work. It was
> something like 2 YEARS since we first wanted this, and it STILL does
> not work.
When I was working on the task counter cgroup subsystem 2 years
ago, the patches were actually pushed back by google people, in favour
of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of
number of tasks as a resource is awkward and that the real resource
in the game comes from kernel memory exhaustion due to task stack being
allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially
since that could solve others people issues at the same time. kmem
cgroup has a quite large domain of application.
> You're postponing a pretty simple request indefinitely in
> favor of a much more complex feature, which still doesn't really give
> me what I want. What I want is an API that works like rlimit but
> per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter
cgroup subsystem is simpler than extending the kmem code to apply limits
to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already
familiar with this.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-29 21:44 ` Frederic Weisbecker
0 siblings, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-29 21:44 UTC (permalink / raw
To: Tim Hockin
Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov,
Marian Marinov, Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
> Here's the reason it doesn't work for us: It doesn't work. It was
> something like 2 YEARS since we first wanted this, and it STILL does
> not work.
When I was working on the task counter cgroup subsystem 2 years
ago, the patches were actually pushed back by google people, in favour
of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of
number of tasks as a resource is awkward and that the real resource
in the game comes from kernel memory exhaustion due to task stack being
allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially
since that could solve others people issues at the same time. kmem
cgroup has a quite large domain of application.
> You're postponing a pretty simple request indefinitely in
> favor of a much more complex feature, which still doesn't really give
> me what I want. What I want is an API that works like rlimit but
> per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter
cgroup subsystem is simpler than extending the kmem code to apply limits
to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already
familiar with this.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-29 21:44 ` Frederic Weisbecker
(?)
(?)
@ 2014-04-30 13:12 ` Daniel J Walsh
[not found] ` <5360F6B4.9010308-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2014-04-30 13:28 ` Frederic Weisbecker
-1 siblings, 2 replies; 84+ messages in thread
From: Daniel J Walsh @ 2014-04-30 13:12 UTC (permalink / raw
To: Frederic Weisbecker, Tim Hockin
Cc: Michal Hocko, Serge Hallyn, Richard Davies, Vladimir Davydov,
Marian Marinov, Max Kellermann, Tim Hockin, containers, cgroups,
Glauber Costa, linux-mm@kvack.org, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
On 04/29/2014 05:44 PM, Frederic Weisbecker wrote:
> On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
>> Here's the reason it doesn't work for us: It doesn't work. It was
>> something like 2 YEARS since we first wanted this, and it STILL does
>> not work.
> When I was working on the task counter cgroup subsystem 2 years
> ago, the patches were actually pushed back by google people, in favour
> of task stack kmem cgroup subsystem.
>
> The reason was that expressing the forkbomb issue in terms of
> number of tasks as a resource is awkward and that the real resource
> in the game comes from kernel memory exhaustion due to task stack being
> allocated over and over, swap ping-pong and stuffs...
>
> And that was a pretty good argument. I still agree with that. Especially
> since that could solve others people issues at the same time. kmem
> cgroup has a quite large domain of application.
>
>> You're postponing a pretty simple request indefinitely in
>> favor of a much more complex feature, which still doesn't really give
>> me what I want. What I want is an API that works like rlimit but
>> per-cgroup, rather than per-UID.
> The request is simple but I don't think that adding the task counter
> cgroup subsystem is simpler than extending the kmem code to apply limits
> to only task stack. Especially in terms of maintainance.
>
> Also you guys have very good mm kernel developers who are already
> familiar with this.
I would look at this from a Usability point of view. It is a lot easier
to understand number of processes then the mount of KMEM those processes
will need. Setting something like
ProcessLimit=1000 in a systemd unit file is easy to explain. Now if
systemd has the ability to translate this into something that makes
sense in terms of kmem cgroup, then my argument goes away.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140429214454.GF6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
@ 2014-04-30 13:12 ` Daniel J Walsh
0 siblings, 0 replies; 84+ messages in thread
From: Daniel J Walsh @ 2014-04-30 13:12 UTC (permalink / raw
To: Frederic Weisbecker, Tim Hockin
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On 04/29/2014 05:44 PM, Frederic Weisbecker wrote:
> On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
>> Here's the reason it doesn't work for us: It doesn't work. It was
>> something like 2 YEARS since we first wanted this, and it STILL does
>> not work.
> When I was working on the task counter cgroup subsystem 2 years
> ago, the patches were actually pushed back by google people, in favour
> of task stack kmem cgroup subsystem.
>
> The reason was that expressing the forkbomb issue in terms of
> number of tasks as a resource is awkward and that the real resource
> in the game comes from kernel memory exhaustion due to task stack being
> allocated over and over, swap ping-pong and stuffs...
>
> And that was a pretty good argument. I still agree with that. Especially
> since that could solve others people issues at the same time. kmem
> cgroup has a quite large domain of application.
>
>> You're postponing a pretty simple request indefinitely in
>> favor of a much more complex feature, which still doesn't really give
>> me what I want. What I want is an API that works like rlimit but
>> per-cgroup, rather than per-UID.
> The request is simple but I don't think that adding the task counter
> cgroup subsystem is simpler than extending the kmem code to apply limits
> to only task stack. Especially in terms of maintainance.
>
> Also you guys have very good mm kernel developers who are already
> familiar with this.
I would look at this from a Usability point of view. It is a lot easier
to understand number of processes then the mount of KMEM those processes
will need. Setting something like
ProcessLimit=1000 in a systemd unit file is easy to explain. Now if
systemd has the ability to translate this into something that makes
sense in terms of kmem cgroup, then my argument goes away.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <5360F6B4.9010308-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2014-04-30 13:28 ` Frederic Weisbecker
0 siblings, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-30 13:28 UTC (permalink / raw
To: Daniel J Walsh
Cc: Tim Hockin, Michal Hocko, Serge Hallyn, Richard Davies,
Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
containers, cgroups, Glauber Costa, linux-mm@kvack.org,
William Dauchy, Johannes Weiner, Tejun Heo, David Rientjes
On Wed, Apr 30, 2014 at 09:12:20AM -0400, Daniel J Walsh wrote:
>
> On 04/29/2014 05:44 PM, Frederic Weisbecker wrote:
> > On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
> >> Here's the reason it doesn't work for us: It doesn't work. It was
> >> something like 2 YEARS since we first wanted this, and it STILL does
> >> not work.
> > When I was working on the task counter cgroup subsystem 2 years
> > ago, the patches were actually pushed back by google people, in favour
> > of task stack kmem cgroup subsystem.
> >
> > The reason was that expressing the forkbomb issue in terms of
> > number of tasks as a resource is awkward and that the real resource
> > in the game comes from kernel memory exhaustion due to task stack being
> > allocated over and over, swap ping-pong and stuffs...
> >
> > And that was a pretty good argument. I still agree with that. Especially
> > since that could solve others people issues at the same time. kmem
> > cgroup has a quite large domain of application.
> >
> >> You're postponing a pretty simple request indefinitely in
> >> favor of a much more complex feature, which still doesn't really give
> >> me what I want. What I want is an API that works like rlimit but
> >> per-cgroup, rather than per-UID.
> > The request is simple but I don't think that adding the task counter
> > cgroup subsystem is simpler than extending the kmem code to apply limits
> > to only task stack. Especially in terms of maintainance.
> >
> > Also you guys have very good mm kernel developers who are already
> > familiar with this.
> I would look at this from a Usability point of view. It is a lot easier
> to understand number of processes then the mount of KMEM those processes
> will need. Setting something like
> ProcessLimit=1000 in a systemd unit file is easy to explain.
Yeah that's a fair point.
> Now if systemd has the ability to translate this into something that makes
> sense in terms of kmem cgroup, then my argument goes away.
Yeah if we keep the kmem direction, this can be a place where we do the mapping.
Now I just hope the amount of stack memory allocated doesn't differ too much per arch.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <5360F6B4.9010308-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2014-04-30 13:28 ` Frederic Weisbecker
0 siblings, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-30 13:28 UTC (permalink / raw
To: Daniel J Walsh
Cc: Richard Davies, Vladimir Davydov, David Rientjes, Marian Marinov,
Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On Wed, Apr 30, 2014 at 09:12:20AM -0400, Daniel J Walsh wrote:
>
> On 04/29/2014 05:44 PM, Frederic Weisbecker wrote:
> > On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
> >> Here's the reason it doesn't work for us: It doesn't work. It was
> >> something like 2 YEARS since we first wanted this, and it STILL does
> >> not work.
> > When I was working on the task counter cgroup subsystem 2 years
> > ago, the patches were actually pushed back by google people, in favour
> > of task stack kmem cgroup subsystem.
> >
> > The reason was that expressing the forkbomb issue in terms of
> > number of tasks as a resource is awkward and that the real resource
> > in the game comes from kernel memory exhaustion due to task stack being
> > allocated over and over, swap ping-pong and stuffs...
> >
> > And that was a pretty good argument. I still agree with that. Especially
> > since that could solve others people issues at the same time. kmem
> > cgroup has a quite large domain of application.
> >
> >> You're postponing a pretty simple request indefinitely in
> >> favor of a much more complex feature, which still doesn't really give
> >> me what I want. What I want is an API that works like rlimit but
> >> per-cgroup, rather than per-UID.
> > The request is simple but I don't think that adding the task counter
> > cgroup subsystem is simpler than extending the kmem code to apply limits
> > to only task stack. Especially in terms of maintainance.
> >
> > Also you guys have very good mm kernel developers who are already
> > familiar with this.
> I would look at this from a Usability point of view. It is a lot easier
> to understand number of processes then the mount of KMEM those processes
> will need. Setting something like
> ProcessLimit=1000 in a systemd unit file is easy to explain.
Yeah that's a fair point.
> Now if systemd has the ability to translate this into something that makes
> sense in terms of kmem cgroup, then my argument goes away.
Yeah if we keep the kmem direction, this can be a place where we do the mapping.
Now I just hope the amount of stack memory allocated doesn't differ too much per arch.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-30 13:28 ` Frederic Weisbecker
0 siblings, 0 replies; 84+ messages in thread
From: Frederic Weisbecker @ 2014-04-30 13:28 UTC (permalink / raw
To: Daniel J Walsh
Cc: Tim Hockin, Michal Hocko, Serge Hallyn, Richard Davies,
Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
On Wed, Apr 30, 2014 at 09:12:20AM -0400, Daniel J Walsh wrote:
>
> On 04/29/2014 05:44 PM, Frederic Weisbecker wrote:
> > On Tue, Apr 29, 2014 at 09:59:30AM -0700, Tim Hockin wrote:
> >> Here's the reason it doesn't work for us: It doesn't work. It was
> >> something like 2 YEARS since we first wanted this, and it STILL does
> >> not work.
> > When I was working on the task counter cgroup subsystem 2 years
> > ago, the patches were actually pushed back by google people, in favour
> > of task stack kmem cgroup subsystem.
> >
> > The reason was that expressing the forkbomb issue in terms of
> > number of tasks as a resource is awkward and that the real resource
> > in the game comes from kernel memory exhaustion due to task stack being
> > allocated over and over, swap ping-pong and stuffs...
> >
> > And that was a pretty good argument. I still agree with that. Especially
> > since that could solve others people issues at the same time. kmem
> > cgroup has a quite large domain of application.
> >
> >> You're postponing a pretty simple request indefinitely in
> >> favor of a much more complex feature, which still doesn't really give
> >> me what I want. What I want is an API that works like rlimit but
> >> per-cgroup, rather than per-UID.
> > The request is simple but I don't think that adding the task counter
> > cgroup subsystem is simpler than extending the kmem code to apply limits
> > to only task stack. Especially in terms of maintainance.
> >
> > Also you guys have very good mm kernel developers who are already
> > familiar with this.
> I would look at this from a Usability point of view. It is a lot easier
> to understand number of processes then the mount of KMEM those processes
> will need. Setting something like
> ProcessLimit=1000 in a systemd unit file is easy to explain.
Yeah that's a fair point.
> Now if systemd has the ability to translate this into something that makes
> sense in terms of kmem cgroup, then my argument goes away.
Yeah if we keep the kmem direction, this can be a place where we do the mapping.
Now I just hope the amount of stack memory allocated doesn't differ too much per arch.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <53601B68.60906-NV7Lj0SOnH0@public.gmane.org>
@ 2014-04-30 13:31 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-30 13:31 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Dwight Engen, Tim Hockin, Vladimir Davydov,
David Rientjes, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers, Serge Hallyn, Glauber Costa, linux-mm, William Dauchy,
Johannes Weiner, Tejun Heo, cgroups, Daniel Walsh
On Wed 30-04-14 00:36:40, Marian Marinov wrote:
> On 04/29/2014 09:27 PM, Michal Hocko wrote:
> >On Tue 29-04-14 19:09:27, Richard Davies wrote:
> >>Dwight Engen wrote:
> >>>Michal Hocko wrote:
> >>>>Tim Hockin wrote:
> >>>>>Here's the reason it doesn't work for us: It doesn't work.
> >>>>
> >>>>There is a "simple" solution for that. Help us to fix it.
> >>>>
> >>>>>It was something like 2 YEARS since we first wanted this, and it
> >>>>>STILL does not work.
> >>>>
> >>>>My recollection is that it was primarily Parallels and Google asking
> >>>>for the kmem accounting. The reason why I didn't fight against
> >>>>inclusion although the implementation at the time didn't have a
> >>>>proper slab shrinking implemented was that that would happen later.
> >>>>Well, that later hasn't happened yet and we are slowly getting there.
> >>>>
> >>>>>You're postponing a pretty simple request indefinitely in
> >>>>>favor of a much more complex feature, which still doesn't really
> >>>>>give me what I want.
> >>>>
> >>>>But we cannot simply add a new interface that will have to be
> >>>>maintained for ever just because something else that is supposed to
> >>>>workaround bugs.
> >>>>
> >>>>>What I want is an API that works like rlimit but per-cgroup, rather
> >>>>>than per-UID.
> >>>>
> >>>>You can use an out-of-tree patchset for the time being or help to get
> >>>>kmem into shape. If there are principal reasons why kmem cannot be
> >>>>used then you better articulate them.
> >>>
> >>>Is there a plan to separately account/limit stack pages vs kmem in
> >>>general? Richard would have to verify, but I suspect kmem is not currently
> >>>viable as a process limiter for him because icache/dcache/stack is all
> >>>accounted together.
> >>
> >>Certainly I would like to be able to limit container fork-bombs without
> >>limiting the amount of disk IO caching for processes in those containers.
> >>
> >>In my testing with of kmem limits, I needed a limit of 256MB or lower to
> >>catch fork bombs early enough. I would definitely like more than 256MB of
> >>disk caching.
> >>
> >>So if we go the "working kmem" route, I would like to be able to specify a
> >>limit excluding disk cache.
> >
> >Page cache (which is what you mean by disk cache probably) is a
> >userspace accounted memory with the memory cgroup controller. And you
> >do not have to limit that one. Kmem accounting refers to kernel internal
> >allocations - slab memory and per process kernel stack. You can see how
> >much memory is allocated per container by memory.kmem.usage_in_bytes or
> >have a look at /proc/slabinfo to see what kind of memory kernel
> >allocates globally and might be accounted for a container as well.
> >
> >The primary problem with the kmem accounting right now is that such a
> >memory is not "reclaimed" and so if the kmem limit is reached all the
> >further kmem allocations fail. The biggest user of the kmem allocations
> >on many systems is dentry and inode chache which is reclaimable easily.
> >When this is implemented the kmem limit will be usable to both prevent
> >forkbombs but also other DOS scenarios when the kernel is pushed to
> >allocate a huge amount of memory.
>
> I would have to disagree here.
> If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem.
> The simple scheduling of so many processes generates have load on the machine.
> Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
What prevents you from setting the kmem limit to NR_PROC * 8K + slab_pillow?
> Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure
> on the kmem, but will would not produce many processes.
Once we have a proper slab reclaim implementation this shouldn't be a
problem.
> On the other hand, forking thousands of processes with low memory footprint
> will hit the scheduler a lot faster then hitting the kmem limit.
>
> Kmem limit is something that we need! But firmly believe that we need
> a simple NPROC limit for cgroups.
Once again. If you feel that your usecase is not covered by the kmem
limit follow up on the original email thread I have referenced earlier
in the thread. Splitting up the discussion doesn't help at all.
> -hackman
>
> >
> >HTH
> >
> >>I am also somewhat worried that normal software use could legitimately go
> >>above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
> >>just by booting a distro with a few daemons in a container.
> >>
> >>Richard.
> >
>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <53601B68.60906-NV7Lj0SOnH0@public.gmane.org>
@ 2014-04-30 13:31 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-30 13:31 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Tim Hockin, Glauber Costa, Johannes Weiner,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On Wed 30-04-14 00:36:40, Marian Marinov wrote:
> On 04/29/2014 09:27 PM, Michal Hocko wrote:
> >On Tue 29-04-14 19:09:27, Richard Davies wrote:
> >>Dwight Engen wrote:
> >>>Michal Hocko wrote:
> >>>>Tim Hockin wrote:
> >>>>>Here's the reason it doesn't work for us: It doesn't work.
> >>>>
> >>>>There is a "simple" solution for that. Help us to fix it.
> >>>>
> >>>>>It was something like 2 YEARS since we first wanted this, and it
> >>>>>STILL does not work.
> >>>>
> >>>>My recollection is that it was primarily Parallels and Google asking
> >>>>for the kmem accounting. The reason why I didn't fight against
> >>>>inclusion although the implementation at the time didn't have a
> >>>>proper slab shrinking implemented was that that would happen later.
> >>>>Well, that later hasn't happened yet and we are slowly getting there.
> >>>>
> >>>>>You're postponing a pretty simple request indefinitely in
> >>>>>favor of a much more complex feature, which still doesn't really
> >>>>>give me what I want.
> >>>>
> >>>>But we cannot simply add a new interface that will have to be
> >>>>maintained for ever just because something else that is supposed to
> >>>>workaround bugs.
> >>>>
> >>>>>What I want is an API that works like rlimit but per-cgroup, rather
> >>>>>than per-UID.
> >>>>
> >>>>You can use an out-of-tree patchset for the time being or help to get
> >>>>kmem into shape. If there are principal reasons why kmem cannot be
> >>>>used then you better articulate them.
> >>>
> >>>Is there a plan to separately account/limit stack pages vs kmem in
> >>>general? Richard would have to verify, but I suspect kmem is not currently
> >>>viable as a process limiter for him because icache/dcache/stack is all
> >>>accounted together.
> >>
> >>Certainly I would like to be able to limit container fork-bombs without
> >>limiting the amount of disk IO caching for processes in those containers.
> >>
> >>In my testing with of kmem limits, I needed a limit of 256MB or lower to
> >>catch fork bombs early enough. I would definitely like more than 256MB of
> >>disk caching.
> >>
> >>So if we go the "working kmem" route, I would like to be able to specify a
> >>limit excluding disk cache.
> >
> >Page cache (which is what you mean by disk cache probably) is a
> >userspace accounted memory with the memory cgroup controller. And you
> >do not have to limit that one. Kmem accounting refers to kernel internal
> >allocations - slab memory and per process kernel stack. You can see how
> >much memory is allocated per container by memory.kmem.usage_in_bytes or
> >have a look at /proc/slabinfo to see what kind of memory kernel
> >allocates globally and might be accounted for a container as well.
> >
> >The primary problem with the kmem accounting right now is that such a
> >memory is not "reclaimed" and so if the kmem limit is reached all the
> >further kmem allocations fail. The biggest user of the kmem allocations
> >on many systems is dentry and inode chache which is reclaimable easily.
> >When this is implemented the kmem limit will be usable to both prevent
> >forkbombs but also other DOS scenarios when the kernel is pushed to
> >allocate a huge amount of memory.
>
> I would have to disagree here.
> If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem.
> The simple scheduling of so many processes generates have load on the machine.
> Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
What prevents you from setting the kmem limit to NR_PROC * 8K + slab_pillow?
> Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure
> on the kmem, but will would not produce many processes.
Once we have a proper slab reclaim implementation this shouldn't be a
problem.
> On the other hand, forking thousands of processes with low memory footprint
> will hit the scheduler a lot faster then hitting the kmem limit.
>
> Kmem limit is something that we need! But firmly believe that we need
> a simple NPROC limit for cgroups.
Once again. If you feel that your usecase is not covered by the kmem
limit follow up on the original email thread I have referenced earlier
in the thread. Splitting up the discussion doesn't help at all.
> -hackman
>
> >
> >HTH
> >
> >>I am also somewhat worried that normal software use could legitimately go
> >>above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
> >>just by booting a distro with a few daemons in a container.
> >>
> >>Richard.
> >
>
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-04-30 13:31 ` Michal Hocko
0 siblings, 0 replies; 84+ messages in thread
From: Michal Hocko @ 2014-04-30 13:31 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Dwight Engen, Tim Hockin, Vladimir Davydov,
David Rientjes, Max Kellermann, Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Serge Hallyn, Glauber Costa, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
William Dauchy, Johannes Weiner, Tejun Heo,
cgroups-u79uwXL29TY76Z2rM5mHXA, Daniel Walsh
On Wed 30-04-14 00:36:40, Marian Marinov wrote:
> On 04/29/2014 09:27 PM, Michal Hocko wrote:
> >On Tue 29-04-14 19:09:27, Richard Davies wrote:
> >>Dwight Engen wrote:
> >>>Michal Hocko wrote:
> >>>>Tim Hockin wrote:
> >>>>>Here's the reason it doesn't work for us: It doesn't work.
> >>>>
> >>>>There is a "simple" solution for that. Help us to fix it.
> >>>>
> >>>>>It was something like 2 YEARS since we first wanted this, and it
> >>>>>STILL does not work.
> >>>>
> >>>>My recollection is that it was primarily Parallels and Google asking
> >>>>for the kmem accounting. The reason why I didn't fight against
> >>>>inclusion although the implementation at the time didn't have a
> >>>>proper slab shrinking implemented was that that would happen later.
> >>>>Well, that later hasn't happened yet and we are slowly getting there.
> >>>>
> >>>>>You're postponing a pretty simple request indefinitely in
> >>>>>favor of a much more complex feature, which still doesn't really
> >>>>>give me what I want.
> >>>>
> >>>>But we cannot simply add a new interface that will have to be
> >>>>maintained for ever just because something else that is supposed to
> >>>>workaround bugs.
> >>>>
> >>>>>What I want is an API that works like rlimit but per-cgroup, rather
> >>>>>than per-UID.
> >>>>
> >>>>You can use an out-of-tree patchset for the time being or help to get
> >>>>kmem into shape. If there are principal reasons why kmem cannot be
> >>>>used then you better articulate them.
> >>>
> >>>Is there a plan to separately account/limit stack pages vs kmem in
> >>>general? Richard would have to verify, but I suspect kmem is not currently
> >>>viable as a process limiter for him because icache/dcache/stack is all
> >>>accounted together.
> >>
> >>Certainly I would like to be able to limit container fork-bombs without
> >>limiting the amount of disk IO caching for processes in those containers.
> >>
> >>In my testing with of kmem limits, I needed a limit of 256MB or lower to
> >>catch fork bombs early enough. I would definitely like more than 256MB of
> >>disk caching.
> >>
> >>So if we go the "working kmem" route, I would like to be able to specify a
> >>limit excluding disk cache.
> >
> >Page cache (which is what you mean by disk cache probably) is a
> >userspace accounted memory with the memory cgroup controller. And you
> >do not have to limit that one. Kmem accounting refers to kernel internal
> >allocations - slab memory and per process kernel stack. You can see how
> >much memory is allocated per container by memory.kmem.usage_in_bytes or
> >have a look at /proc/slabinfo to see what kind of memory kernel
> >allocates globally and might be accounted for a container as well.
> >
> >The primary problem with the kmem accounting right now is that such a
> >memory is not "reclaimed" and so if the kmem limit is reached all the
> >further kmem allocations fail. The biggest user of the kmem allocations
> >on many systems is dentry and inode chache which is reclaimable easily.
> >When this is implemented the kmem limit will be usable to both prevent
> >forkbombs but also other DOS scenarios when the kernel is pushed to
> >allocate a huge amount of memory.
>
> I would have to disagree here.
> If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem.
> The simple scheduling of so many processes generates have load on the machine.
> Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
What prevents you from setting the kmem limit to NR_PROC * 8K + slab_pillow?
> Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure
> on the kmem, but will would not produce many processes.
Once we have a proper slab reclaim implementation this shouldn't be a
problem.
> On the other hand, forking thousands of processes with low memory footprint
> will hit the scheduler a lot faster then hitting the kmem limit.
>
> Kmem limit is something that we need! But firmly believe that we need
> a simple NPROC limit for cgroups.
Once again. If you feel that your usecase is not covered by the kmem
limit follow up on the original email thread I have referenced earlier
in the thread. Splitting up the discussion doesn't help at all.
> -hackman
>
> >
> >HTH
> >
> >>I am also somewhat worried that normal software use could legitimately go
> >>above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing
> >>just by booting a distro with a few daemons in a container.
> >>
> >>Richard.
> >
>
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-04-23 12:49 ` Dwight Engen
2014-04-28 18:00 ` Serge Hallyn
@ 2014-05-06 11:40 ` Marian Marinov
[not found] ` <5368CA47.7030007-NV7Lj0SOnH0@public.gmane.org>
2014-05-07 17:15 ` Dwight Engen
2014-06-10 14:50 ` Marian Marinov
[not found] ` <20140423084942.560ae837-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
3 siblings, 2 replies; 84+ messages in thread
From: Marian Marinov @ 2014-05-06 11:40 UTC (permalink / raw
To: Dwight Engen
Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker,
David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann,
Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko,
Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm
On 04/23/2014 03:49 PM, Dwight Engen wrote:
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm@yuhu.biz> wrote:
>
>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>> Dwight Engen wrote:
>>>> Richard Davies wrote:
>>>>> Vladimir Davydov wrote:
>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>> Do not use it. We are working on making it usable though.
>>> ...
>>>>> What is the best mechanism available today, until kmem limits
>>>>> mature?
>>>>>
>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>
>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>
>>>> I updated Frederic's task counter patches and included Max
>>>> Kellermann's fork limiter here:
>>>>
>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>
>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>> would find it useful.
>>>
>>> Yes please, I would be interested in that. Ideally even against
>>> 3.14.1 if you have that too.
>>
>> Dwight, do you have these patches in any public repo?
>>
>> I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>
Guys I tested the patches with 3.12.16. However I see a problem with them.
Trying to set the limit to a cgroup which already have processes in it does not work:
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#
I have even tried to remove this check:
+ if (cgroup_task_count(cgrp) || !list_empty(&cgrp->children))
+ return -EBUSY;
But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
>> Marian
>>
>>>
>>> Thanks,
>>>
>>> Richard.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140423084942.560ae837-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
@ 2014-05-06 11:40 ` Marian Marinov
2014-06-10 14:50 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-05-06 11:40 UTC (permalink / raw
To: Dwight Engen
Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On 04/23/2014 03:49 PM, Dwight Engen wrote:
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>
>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>> Dwight Engen wrote:
>>>> Richard Davies wrote:
>>>>> Vladimir Davydov wrote:
>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>> Do not use it. We are working on making it usable though.
>>> ...
>>>>> What is the best mechanism available today, until kmem limits
>>>>> mature?
>>>>>
>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>
>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>
>>>> I updated Frederic's task counter patches and included Max
>>>> Kellermann's fork limiter here:
>>>>
>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>
>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>> would find it useful.
>>>
>>> Yes please, I would be interested in that. Ideally even against
>>> 3.14.1 if you have that too.
>>
>> Dwight, do you have these patches in any public repo?
>>
>> I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>
Guys I tested the patches with 3.12.16. However I see a problem with them.
Trying to set the limit to a cgroup which already have processes in it does not work:
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#
I have even tried to remove this check:
+ if (cgroup_task_count(cgrp) || !list_empty(&cgrp->children))
+ return -EBUSY;
But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
>> Marian
>>
>>>
>>> Thanks,
>>>
>>> Richard.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
2014-05-06 11:40 ` Marian Marinov
[not found] ` <5368CA47.7030007-NV7Lj0SOnH0@public.gmane.org>
@ 2014-05-07 17:15 ` Dwight Engen
2014-05-07 22:39 ` Marian Marinov
1 sibling, 1 reply; 84+ messages in thread
From: Dwight Engen @ 2014-05-07 17:15 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker,
David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann,
Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko,
Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm
On Tue, 06 May 2014 14:40:55 +0300
Marian Marinov <mm@yuhu.biz> wrote:
> On 04/23/2014 03:49 PM, Dwight Engen wrote:
> > On Wed, 23 Apr 2014 09:07:28 +0300
> > Marian Marinov <mm@yuhu.biz> wrote:
> >
> >> On 04/22/2014 11:05 PM, Richard Davies wrote:
> >>> Dwight Engen wrote:
> >>>> Richard Davies wrote:
> >>>>> Vladimir Davydov wrote:
> >>>>>> In short, kmem limiting for memory cgroups is currently broken.
> >>>>>> Do not use it. We are working on making it usable though.
> >>> ...
> >>>>> What is the best mechanism available today, until kmem limits
> >>>>> mature?
> >>>>>
> >>>>> RLIMIT_NPROC exists but is per-user, not per-container.
> >>>>>
> >>>>> Perhaps there is an up-to-date task counter patchset or similar?
> >>>>
> >>>> I updated Frederic's task counter patches and included Max
> >>>> Kellermann's fork limiter here:
> >>>>
> >>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> >>>>
> >>>> I can send you a more recent patchset (against 3.13.10) if you
> >>>> would find it useful.
> >>>
> >>> Yes please, I would be interested in that. Ideally even against
> >>> 3.14.1 if you have that too.
> >>
> >> Dwight, do you have these patches in any public repo?
> >>
> >> I would like to test them also.
> >
> > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> >
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >
> Guys I tested the patches with 3.12.16. However I see a problem with
> them.
>
> Trying to set the limit to a cgroup which already have processes in
> it does not work:
This is a similar check/limitation to the one for kmem in memcg, and is
done here to keep the res_counters consistent and from going negative.
It could probably be relaxed slightly by using res_counter_set_limit()
instead, but you would still need to initially set a limit before
adding tasks to the group.
> [root@sp2 lxc]# echo 50 > cpuacct.task_limit
> -bash: echo: write error: Device or resource busy
> [root@sp2 lxc]# echo 0 > cpuacct.task_limit
> -bash: echo: write error: Device or resource busy
> [root@sp2 lxc]#
>
> I have even tried to remove this check:
> + if (cgroup_task_count(cgrp)
> || !list_empty(&cgrp->children))
> + return -EBUSY;
> But still give me 'Device or resource busy'.
>
> Any pointers of why is this happening ?
>
> Marian
>
> >> Marian
> >>
> >>>
> >>> Thanks,
> >>>
> >>> Richard.
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe cgroups"
> >>> in the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>
> >>>
> >>
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe cgroups"
> > in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <5368CA47.7030007-NV7Lj0SOnH0@public.gmane.org>
@ 2014-05-07 17:15 ` Dwight Engen
0 siblings, 0 replies; 84+ messages in thread
From: Dwight Engen @ 2014-05-07 17:15 UTC (permalink / raw
To: Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On Tue, 06 May 2014 14:40:55 +0300
Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> On 04/23/2014 03:49 PM, Dwight Engen wrote:
> > On Wed, 23 Apr 2014 09:07:28 +0300
> > Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> >
> >> On 04/22/2014 11:05 PM, Richard Davies wrote:
> >>> Dwight Engen wrote:
> >>>> Richard Davies wrote:
> >>>>> Vladimir Davydov wrote:
> >>>>>> In short, kmem limiting for memory cgroups is currently broken.
> >>>>>> Do not use it. We are working on making it usable though.
> >>> ...
> >>>>> What is the best mechanism available today, until kmem limits
> >>>>> mature?
> >>>>>
> >>>>> RLIMIT_NPROC exists but is per-user, not per-container.
> >>>>>
> >>>>> Perhaps there is an up-to-date task counter patchset or similar?
> >>>>
> >>>> I updated Frederic's task counter patches and included Max
> >>>> Kellermann's fork limiter here:
> >>>>
> >>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
> >>>>
> >>>> I can send you a more recent patchset (against 3.13.10) if you
> >>>> would find it useful.
> >>>
> >>> Yes please, I would be interested in that. Ideally even against
> >>> 3.14.1 if you have that too.
> >>
> >> Dwight, do you have these patches in any public repo?
> >>
> >> I would like to test them also.
> >
> > Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> >
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> > git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >
> Guys I tested the patches with 3.12.16. However I see a problem with
> them.
>
> Trying to set the limit to a cgroup which already have processes in
> it does not work:
This is a similar check/limitation to the one for kmem in memcg, and is
done here to keep the res_counters consistent and from going negative.
It could probably be relaxed slightly by using res_counter_set_limit()
instead, but you would still need to initially set a limit before
adding tasks to the group.
> [root@sp2 lxc]# echo 50 > cpuacct.task_limit
> -bash: echo: write error: Device or resource busy
> [root@sp2 lxc]# echo 0 > cpuacct.task_limit
> -bash: echo: write error: Device or resource busy
> [root@sp2 lxc]#
>
> I have even tried to remove this check:
> + if (cgroup_task_count(cgrp)
> || !list_empty(&cgrp->children))
> + return -EBUSY;
> But still give me 'Device or resource busy'.
>
> Any pointers of why is this happening ?
>
> Marian
>
> >> Marian
> >>
> >>>
> >>> Thanks,
> >>>
> >>> Richard.
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe cgroups"
> >>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>
> >>>
> >>
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe cgroups"
> > in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-05-07 22:39 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-05-07 22:39 UTC (permalink / raw
To: Dwight Engen, Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann,
Tim Hockin, Frederic Weisbecker, containers, Johannes Weiner,
Glauber Costa, Michal Hocko, linux-mm, William Dauchy,
David Rientjes, Tejun Heo, cgroups
On 05/07/2014 08:15 PM, Dwight Engen wrote:
> On Tue, 06 May 2014 14:40:55 +0300
> Marian Marinov <mm@yuhu.biz> wrote:
>
>> On 04/23/2014 03:49 PM, Dwight Engen wrote:
>>> On Wed, 23 Apr 2014 09:07:28 +0300
>>> Marian Marinov <mm@yuhu.biz> wrote:
>>>
>>>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>>>> Dwight Engen wrote:
>>>>>> Richard Davies wrote:
>>>>>>> Vladimir Davydov wrote:
>>>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>>>> Do not use it. We are working on making it usable though.
>>>>> ...
>>>>>>> What is the best mechanism available today, until kmem limits
>>>>>>> mature?
>>>>>>>
>>>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>>>
>>>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>>>
>>>>>> I updated Frederic's task counter patches and included Max
>>>>>> Kellermann's fork limiter here:
>>>>>>
>>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>>>
>>>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>>>> would find it useful.
>>>>>
>>>>> Yes please, I would be interested in that. Ideally even against
>>>>> 3.14.1 if you have that too.
>>>>
>>>> Dwight, do you have these patches in any public repo?
>>>>
>>>> I would like to test them also.
>>>
>>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>>>
>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>>>
>> Guys I tested the patches with 3.12.16. However I see a problem with
>> them.
>>
>> Trying to set the limit to a cgroup which already have processes in
>> it does not work:
>
> This is a similar check/limitation to the one for kmem in memcg, and is
> done here to keep the res_counters consistent and from going negative.
> It could probably be relaxed slightly by using res_counter_set_limit()
> instead, but you would still need to initially set a limit before
> adding tasks to the group.
I have removed the check entirely and still receive the EBUSY... I just don't understand what is returning it. If you
have any pointers, I would be happy to take a look.
I'll look at set_limit(), thanks for pointing that one.
What I'm proposing is the following checks:
if (val > RES_COUNTER_MAX || val < 0)
return -EBUSY;
if (val != 0 && val <= cgroup_task_count(cgrp))
return -EBUSY;
res_counter_write_u64(&ca->task_limit, type, val);
This way we ensure that val is within the limits > 0 and < RES_COUNTER_MAX. And also allow only values of 0 or greater
then the current task count.
Marian
>
>> [root@sp2 lxc]# echo 50 > cpuacct.task_limit
>> -bash: echo: write error: Device or resource busy
>> [root@sp2 lxc]# echo 0 > cpuacct.task_limit
>> -bash: echo: write error: Device or resource busy
>> [root@sp2 lxc]#
>>
>> I have even tried to remove this check:
>> + if (cgroup_task_count(cgrp)
>> || !list_empty(&cgrp->children))
>> + return -EBUSY;
>> But still give me 'Device or resource busy'.
>>
>> Any pointers of why is this happening ?
>>
>> Marian
>>
>>>> Marian
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Richard.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>>>> in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> _______________________________________________
> Containers mailing list
> Containers@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers
>
--
Marian Marinov
Founder & CEO of 1H Ltd.
Jabber/GTalk: hackman@jabber.org
ICQ: 7556201
Mobile: +359 886 660 270
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-05-07 22:39 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-05-07 22:39 UTC (permalink / raw
To: Dwight Engen, Marian Marinov
Cc: Richard Davies, Vladimir Davydov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
Michal Hocko, linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
On 05/07/2014 08:15 PM, Dwight Engen wrote:
> On Tue, 06 May 2014 14:40:55 +0300
> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>
>> On 04/23/2014 03:49 PM, Dwight Engen wrote:
>>> On Wed, 23 Apr 2014 09:07:28 +0300
>>> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>>>
>>>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>>>> Dwight Engen wrote:
>>>>>> Richard Davies wrote:
>>>>>>> Vladimir Davydov wrote:
>>>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>>>> Do not use it. We are working on making it usable though.
>>>>> ...
>>>>>>> What is the best mechanism available today, until kmem limits
>>>>>>> mature?
>>>>>>>
>>>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>>>
>>>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>>>
>>>>>> I updated Frederic's task counter patches and included Max
>>>>>> Kellermann's fork limiter here:
>>>>>>
>>>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>>>
>>>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>>>> would find it useful.
>>>>>
>>>>> Yes please, I would be interested in that. Ideally even against
>>>>> 3.14.1 if you have that too.
>>>>
>>>> Dwight, do you have these patches in any public repo?
>>>>
>>>> I would like to test them also.
>>>
>>> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>>>
>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
>>> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
>>>
>> Guys I tested the patches with 3.12.16. However I see a problem with
>> them.
>>
>> Trying to set the limit to a cgroup which already have processes in
>> it does not work:
>
> This is a similar check/limitation to the one for kmem in memcg, and is
> done here to keep the res_counters consistent and from going negative.
> It could probably be relaxed slightly by using res_counter_set_limit()
> instead, but you would still need to initially set a limit before
> adding tasks to the group.
I have removed the check entirely and still receive the EBUSY... I just don't understand what is returning it. If you
have any pointers, I would be happy to take a look.
I'll look at set_limit(), thanks for pointing that one.
What I'm proposing is the following checks:
if (val > RES_COUNTER_MAX || val < 0)
return -EBUSY;
if (val != 0 && val <= cgroup_task_count(cgrp))
return -EBUSY;
res_counter_write_u64(&ca->task_limit, type, val);
This way we ensure that val is within the limits > 0 and < RES_COUNTER_MAX. And also allow only values of 0 or greater
then the current task count.
Marian
>
>> [root@sp2 lxc]# echo 50 > cpuacct.task_limit
>> -bash: echo: write error: Device or resource busy
>> [root@sp2 lxc]# echo 0 > cpuacct.task_limit
>> -bash: echo: write error: Device or resource busy
>> [root@sp2 lxc]#
>>
>> I have even tried to remove this check:
>> + if (cgroup_task_count(cgrp)
>> || !list_empty(&cgrp->children))
>> + return -EBUSY;
>> But still give me 'Device or resource busy'.
>>
>> Any pointers of why is this happening ?
>>
>> Marian
>>
>>>> Marian
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Richard.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>>>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers
>
--
Marian Marinov
Founder & CEO of 1H Ltd.
Jabber/GTalk: hackman-/eSpBmjxGS4dnm+yROfE0A@public.gmane.org
ICQ: 7556201
Mobile: +359 886 660 270
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-05-08 15:25 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-05-08 15:25 UTC (permalink / raw
To: Marian Marinov
Cc: Dwight Engen, Marian Marinov, Vladimir Davydov, Daniel Walsh,
Max Kellermann, Tim Hockin, Frederic Weisbecker, containers,
Johannes Weiner, Glauber Costa, Michal Hocko, linux-mm,
William Dauchy, David Rientjes, Tejun Heo, cgroups
Marian Marinov wrote:
> On 05/07/2014 08:15 PM, Dwight Engen wrote:
> >On Tue, 06 May 2014 14:40:55 +0300
> >Marian Marinov <mm@yuhu.biz> wrote:
> >
> >>On 04/23/2014 03:49 PM, Dwight Engen wrote:
> >>>On Wed, 23 Apr 2014 09:07:28 +0300
> >>>Marian Marinov <mm@yuhu.biz> wrote:
> >>>
> >>>>On 04/22/2014 11:05 PM, Richard Davies wrote:
> >>>>>Dwight Engen wrote:
> >>>>>>Richard Davies wrote:
> >>>>>>>Vladimir Davydov wrote:
> >>>>>>>>In short, kmem limiting for memory cgroups is currently broken.
> >>>>>>>>Do not use it. We are working on making it usable though.
> >>>>>...
> >>>>>>>What is the best mechanism available today, until kmem limits
> >>>>>>>mature?
> >>>>>>>
> >>>>>>>RLIMIT_NPROC exists but is per-user, not per-container.
> >>>>>>>
> >>>>>>>Perhaps there is an up-to-date task counter patchset or similar?
> >>>>>>
> >>>>>>I updated Frederic's task counter patches and included Max
> >>>>>>Kellermann's fork limiter here:
> >>>>>>
> >>>>>>http://thread.gmane.org/gmane.linux.kernel.containers/27212
> >>>>>>
> >>>>>>I can send you a more recent patchset (against 3.13.10) if you
> >>>>>>would find it useful.
> >>>>>
> >>>>>Yes please, I would be interested in that. Ideally even against
> >>>>>3.14.1 if you have that too.
> >>>>
> >>>>Dwight, do you have these patches in any public repo?
> >>>>
> >>>>I would like to test them also.
> >>>
> >>>Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> >>>
> >>>git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> >>>git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >>>
> >>Guys I tested the patches with 3.12.16. However I see a problem with
> >>them.
> >>
> >>Trying to set the limit to a cgroup which already have processes in
> >>it does not work:
> >
> >This is a similar check/limitation to the one for kmem in memcg, and is
> >done here to keep the res_counters consistent and from going negative.
> >It could probably be relaxed slightly by using res_counter_set_limit()
> >instead, but you would still need to initially set a limit before
> >adding tasks to the group.
>
> I have removed the check entirely and still receive the EBUSY... I
> just don't understand what is returning it. If you have any
> pointers, I would be happy to take a look.
>
> I'll look at set_limit(), thanks for pointing that one.
>
> What I'm proposing is the following checks:
>
> if (val > RES_COUNTER_MAX || val < 0)
> return -EBUSY;
> if (val != 0 && val <= cgroup_task_count(cgrp))
> return -EBUSY;
>
> res_counter_write_u64(&ca->task_limit, type, val);
>
> This way we ensure that val is within the limits > 0 and <
> RES_COUNTER_MAX. And also allow only values of 0 or greater then the
> current task count.
I have also noticed that I can't change many different cgroup limits while
there are tasks running in the cgroup - not just cpuacct.task_limit, but
also kmem and even normal memory.limit_in_bytes
I would like to be able to change all of these limits, as long as the new
limit is greater than the actual current use.
Could a method like this be used for all of the others too?
Richard.
> >>[root@sp2 lxc]# echo 50 > cpuacct.task_limit
> >>-bash: echo: write error: Device or resource busy
> >>[root@sp2 lxc]# echo 0 > cpuacct.task_limit
> >>-bash: echo: write error: Device or resource busy
> >>[root@sp2 lxc]#
> >>
> >>I have even tried to remove this check:
> >>+ if (cgroup_task_count(cgrp)
> >>|| !list_empty(&cgrp->children))
> >>+ return -EBUSY;
> >>But still give me 'Device or resource busy'.
> >>
> >>Any pointers of why is this happening ?
> >>
> >>Marian
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-05-08 15:25 ` Richard Davies
0 siblings, 0 replies; 84+ messages in thread
From: Richard Davies @ 2014-05-08 15:25 UTC (permalink / raw
To: Marian Marinov
Cc: Vladimir Davydov, Marian Marinov, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Daniel Walsh, cgroups-u79uwXL29TY76Z2rM5mHXA, Glauber Costa,
Michal Hocko, linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy,
Johannes Weiner, Tejun Heo, David Rientjes
Marian Marinov wrote:
> On 05/07/2014 08:15 PM, Dwight Engen wrote:
> >On Tue, 06 May 2014 14:40:55 +0300
> >Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> >
> >>On 04/23/2014 03:49 PM, Dwight Engen wrote:
> >>>On Wed, 23 Apr 2014 09:07:28 +0300
> >>>Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
> >>>
> >>>>On 04/22/2014 11:05 PM, Richard Davies wrote:
> >>>>>Dwight Engen wrote:
> >>>>>>Richard Davies wrote:
> >>>>>>>Vladimir Davydov wrote:
> >>>>>>>>In short, kmem limiting for memory cgroups is currently broken.
> >>>>>>>>Do not use it. We are working on making it usable though.
> >>>>>...
> >>>>>>>What is the best mechanism available today, until kmem limits
> >>>>>>>mature?
> >>>>>>>
> >>>>>>>RLIMIT_NPROC exists but is per-user, not per-container.
> >>>>>>>
> >>>>>>>Perhaps there is an up-to-date task counter patchset or similar?
> >>>>>>
> >>>>>>I updated Frederic's task counter patches and included Max
> >>>>>>Kellermann's fork limiter here:
> >>>>>>
> >>>>>>http://thread.gmane.org/gmane.linux.kernel.containers/27212
> >>>>>>
> >>>>>>I can send you a more recent patchset (against 3.13.10) if you
> >>>>>>would find it useful.
> >>>>>
> >>>>>Yes please, I would be interested in that. Ideally even against
> >>>>>3.14.1 if you have that too.
> >>>>
> >>>>Dwight, do you have these patches in any public repo?
> >>>>
> >>>>I would like to test them also.
> >>>
> >>>Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
> >>>
> >>>git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> >>>git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
> >>>
> >>Guys I tested the patches with 3.12.16. However I see a problem with
> >>them.
> >>
> >>Trying to set the limit to a cgroup which already have processes in
> >>it does not work:
> >
> >This is a similar check/limitation to the one for kmem in memcg, and is
> >done here to keep the res_counters consistent and from going negative.
> >It could probably be relaxed slightly by using res_counter_set_limit()
> >instead, but you would still need to initially set a limit before
> >adding tasks to the group.
>
> I have removed the check entirely and still receive the EBUSY... I
> just don't understand what is returning it. If you have any
> pointers, I would be happy to take a look.
>
> I'll look at set_limit(), thanks for pointing that one.
>
> What I'm proposing is the following checks:
>
> if (val > RES_COUNTER_MAX || val < 0)
> return -EBUSY;
> if (val != 0 && val <= cgroup_task_count(cgrp))
> return -EBUSY;
>
> res_counter_write_u64(&ca->task_limit, type, val);
>
> This way we ensure that val is within the limits > 0 and <
> RES_COUNTER_MAX. And also allow only values of 0 or greater then the
> current task count.
I have also noticed that I can't change many different cgroup limits while
there are tasks running in the cgroup - not just cpuacct.task_limit, but
also kmem and even normal memory.limit_in_bytes
I would like to be able to change all of these limits, as long as the new
limit is greater than the actual current use.
Could a method like this be used for all of the others too?
Richard.
> >>[root@sp2 lxc]# echo 50 > cpuacct.task_limit
> >>-bash: echo: write error: Device or resource busy
> >>[root@sp2 lxc]# echo 0 > cpuacct.task_limit
> >>-bash: echo: write error: Device or resource busy
> >>[root@sp2 lxc]#
> >>
> >>I have even tried to remove this check:
> >>+ if (cgroup_task_count(cgrp)
> >>|| !list_empty(&cgrp->children))
> >>+ return -EBUSY;
> >>But still give me 'Device or resource busy'.
> >>
> >>Any pointers of why is this happening ?
> >>
> >>Marian
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140422200531.GA19334-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
@ 2014-06-10 12:18 ` Alin Dobre
2014-04-23 6:07 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Alin Dobre @ 2014-06-10 12:18 UTC (permalink / raw
To: Richard Davies, Dwight Engen
Cc: Vladimir Davydov, Frederic Weisbecker, David Rientjes,
Glauber Costa, Tejun Heo, Max Kellermann, Johannes Weiner,
William Dauchy, Tim Hockin, Michal Hocko, Daniel Walsh,
Daniel Berrange, cgroups, containers, linux-mm
On 22/04/14 21:05, Richard Davies wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>>> Vladimir Davydov wrote:
>>>> In short, kmem limiting for memory cgroups is currently broken. Do
>>>> not use it. We are working on making it usable though.
> ...
>>> What is the best mechanism available today, until kmem limits mature?
>>>
>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>
>>> Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
Any chance for a 3.15 rebase, since the changes from cgroup_fork() makes
the operation no longer trivial.
Cheers,
Alin.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-06-10 12:18 ` Alin Dobre
0 siblings, 0 replies; 84+ messages in thread
From: Alin Dobre @ 2014-06-10 12:18 UTC (permalink / raw
To: Richard Davies, Dwight Engen
Cc: Vladimir Davydov, Daniel Walsh, Max Kellermann, Tim Hockin,
Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
On 22/04/14 21:05, Richard Davies wrote:
> Dwight Engen wrote:
>> Richard Davies wrote:
>>> Vladimir Davydov wrote:
>>>> In short, kmem limiting for memory cgroups is currently broken. Do
>>>> not use it. We are working on making it usable though.
> ...
>>> What is the best mechanism available today, until kmem limits mature?
>>>
>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>
>>> Perhaps there is an up-to-date task counter patchset or similar?
>>
>> I updated Frederic's task counter patches and included Max Kellermann's
>> fork limiter here:
>>
>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>
>> I can send you a more recent patchset (against 3.13.10) if you would
>> find it useful.
>
> Yes please, I would be interested in that. Ideally even against 3.14.1 if
> you have that too.
Any chance for a 3.15 rebase, since the changes from cgroup_fork() makes
the operation no longer trivial.
Cheers,
Alin.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140423084942.560ae837-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
@ 2014-06-10 14:50 ` Marian Marinov
2014-06-10 14:50 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-06-10 14:50 UTC (permalink / raw
To: Dwight Engen
Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker,
David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann,
Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko,
Daniel Walsh, Daniel Berrange, cgroups, containers, linux-mm
[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]
On 04/23/2014 03:49 PM, Dwight Engen wrote:
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm@yuhu.biz> wrote:
>
>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>> Dwight Engen wrote:
>>>> Richard Davies wrote:
>>>>> Vladimir Davydov wrote:
>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>> Do not use it. We are working on making it usable though.
>>> ...
>>>>> What is the best mechanism available today, until kmem limits
>>>>> mature?
>>>>>
>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>
>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>
>>>> I updated Frederic's task counter patches and included Max
>>>> Kellermann's fork limiter here:
>>>>
>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>
>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>> would find it useful.
>>>
>>> Yes please, I would be interested in that. Ideally even against
>>> 3.14.1 if you have that too.
>>
>> Dwight, do you have these patches in any public repo?
>>
>> I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
I did a backport of the patches to 3.12.16 and forward ported them to 3.12.20.
I'm very happy with how they work.
I used the patches on machines with 10-20k processes and it worked perfectly when some of the containers spawned 100s of
processes. It really saved us when one of the containers was attacked :)
The only thing that I'm going to add is on the fly change of the limit.
Marian
>
>> Marian
>>
>>>
>>> Thanks,
>>>
>>> Richard.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
[not found] ` <20140423084942.560ae837-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2014-05-06 11:40 ` Marian Marinov
@ 2014-06-10 14:50 ` Marian Marinov
1 sibling, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-06-10 14:50 UTC (permalink / raw
To: Dwight Engen
Cc: Richard Davies, Vladimir Davydov, Daniel Walsh, Max Kellermann,
Tim Hockin, Frederic Weisbecker,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
Johannes Weiner, Glauber Costa, Michal Hocko,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg, William Dauchy, David Rientjes,
Tejun Heo, cgroups-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1.1: Type: text/plain, Size: 2306 bytes --]
On 04/23/2014 03:49 PM, Dwight Engen wrote:
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>
>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>> Dwight Engen wrote:
>>>> Richard Davies wrote:
>>>>> Vladimir Davydov wrote:
>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>> Do not use it. We are working on making it usable though.
>>> ...
>>>>> What is the best mechanism available today, until kmem limits
>>>>> mature?
>>>>>
>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>
>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>
>>>> I updated Frederic's task counter patches and included Max
>>>> Kellermann's fork limiter here:
>>>>
>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>
>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>> would find it useful.
>>>
>>> Yes please, I would be interested in that. Ideally even against
>>> 3.14.1 if you have that too.
>>
>> Dwight, do you have these patches in any public repo?
>>
>> I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
I did a backport of the patches to 3.12.16 and forward ported them to 3.12.20.
I'm very happy with how they work.
I used the patches on machines with 10-20k processes and it worked perfectly when some of the containers spawned 100s of
processes. It really saved us when one of the containers was attacked :)
The only thing that I'm going to add is on the fly change of the limit.
Marian
>
>> Marian
>>
>>>
>>> Thanks,
>>>
>>> Richard.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
[-- Attachment #2: Type: text/plain, Size: 205 bytes --]
_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
@ 2014-06-10 14:50 ` Marian Marinov
0 siblings, 0 replies; 84+ messages in thread
From: Marian Marinov @ 2014-06-10 14:50 UTC (permalink / raw
To: Dwight Engen
Cc: Richard Davies, Vladimir Davydov, Frederic Weisbecker,
David Rientjes, Glauber Costa, Tejun Heo, Max Kellermann,
Johannes Weiner, William Dauchy, Tim Hockin, Michal Hocko,
Daniel Walsh, Daniel Berrange, cgroups-u79uwXL29TY76Z2rM5mHXA,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg
[-- Attachment #1: Type: text/plain, Size: 2306 bytes --]
On 04/23/2014 03:49 PM, Dwight Engen wrote:
> On Wed, 23 Apr 2014 09:07:28 +0300
> Marian Marinov <mm-NV7Lj0SOnH0@public.gmane.org> wrote:
>
>> On 04/22/2014 11:05 PM, Richard Davies wrote:
>>> Dwight Engen wrote:
>>>> Richard Davies wrote:
>>>>> Vladimir Davydov wrote:
>>>>>> In short, kmem limiting for memory cgroups is currently broken.
>>>>>> Do not use it. We are working on making it usable though.
>>> ...
>>>>> What is the best mechanism available today, until kmem limits
>>>>> mature?
>>>>>
>>>>> RLIMIT_NPROC exists but is per-user, not per-container.
>>>>>
>>>>> Perhaps there is an up-to-date task counter patchset or similar?
>>>>
>>>> I updated Frederic's task counter patches and included Max
>>>> Kellermann's fork limiter here:
>>>>
>>>> http://thread.gmane.org/gmane.linux.kernel.containers/27212
>>>>
>>>> I can send you a more recent patchset (against 3.13.10) if you
>>>> would find it useful.
>>>
>>> Yes please, I would be interested in that. Ideally even against
>>> 3.14.1 if you have that too.
>>
>> Dwight, do you have these patches in any public repo?
>>
>> I would like to test them also.
>
> Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
>
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.13
> git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
I did a backport of the patches to 3.12.16 and forward ported them to 3.12.20.
I'm very happy with how they work.
I used the patches on machines with 10-20k processes and it worked perfectly when some of the containers spawned 100s of
processes. It really saved us when one of the containers was attacked :)
The only thing that I'm going to add is on the fly change of the limit.
Marian
>
>> Marian
>>
>>>
>>> Thanks,
>>>
>>> Richard.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe cgroups"
>>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 84+ messages in thread
end of thread, other threads:[~2014-06-10 14:50 UTC | newest]
Thread overview: 84+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-16 15:46 memcg with kmem limit doesn't recover after disk i/o causes limit to be hit Richard Davies
2014-04-18 15:59 ` Michal Hocko
2014-04-18 17:57 ` Vladimir Davydov
2014-04-18 17:57 ` Vladimir Davydov
2014-04-18 18:20 ` Michal Hocko
2014-04-18 18:20 ` Michal Hocko
2014-04-18 18:37 ` Vladimir Davydov
2014-04-20 14:28 ` Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit] Richard Davies
2014-04-20 18:35 ` Tim Hockin
[not found] ` <20140420142830.GC22077-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-20 18:35 ` Tim Hockin
2014-04-22 18:39 ` Dwight Engen
2014-04-22 18:39 ` Dwight Engen
2014-04-22 20:05 ` Richard Davies
2014-04-22 20:13 ` Tim Hockin
2014-04-23 6:07 ` Marian Marinov
2014-04-23 6:07 ` Marian Marinov
[not found] ` <535758A0.5000500-NV7Lj0SOnH0@public.gmane.org>
2014-04-23 12:49 ` Dwight Engen
2014-04-23 12:49 ` Dwight Engen
2014-04-28 18:00 ` Serge Hallyn
2014-04-28 18:00 ` Serge Hallyn
2014-04-29 7:25 ` Michal Hocko
2014-04-29 7:25 ` Michal Hocko
[not found] ` <20140429072515.GB15058-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
2014-04-29 13:03 ` Serge Hallyn
2014-04-29 13:03 ` Serge Hallyn
2014-04-29 13:03 ` Serge Hallyn
2014-04-29 13:57 ` Marian Marinov
2014-04-29 13:57 ` Marian Marinov
2014-04-29 13:57 ` Marian Marinov
2014-04-29 14:04 ` Tim Hockin
2014-04-29 14:04 ` Tim Hockin
2014-04-29 15:43 ` Michal Hocko
2014-04-29 15:43 ` Michal Hocko
2014-04-29 15:43 ` Michal Hocko
2014-04-29 16:06 ` Tim Hockin
[not found] ` <CAO_RewYZDGLBAKit4CudTbqVk+zfDRX8kP0W6Zz90xJh7abM9Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-04-29 16:51 ` Frederic Weisbecker
2014-04-29 16:51 ` Frederic Weisbecker
[not found] ` <20140429165114.GE6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2014-04-29 16:59 ` Tim Hockin
2014-04-29 16:59 ` Tim Hockin
2014-04-29 16:59 ` Tim Hockin
2014-04-29 17:06 ` Michal Hocko
2014-04-29 17:06 ` Michal Hocko
2014-04-29 17:30 ` Dwight Engen
[not found] ` <20140429133039.162d9dd7-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2014-04-29 18:09 ` Richard Davies
2014-04-29 18:09 ` Richard Davies
[not found] ` <20140429180927.GB29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-29 18:27 ` Michal Hocko
2014-04-29 18:27 ` Michal Hocko
2014-04-29 18:27 ` Michal Hocko
2014-04-29 18:39 ` Richard Davies
2014-04-29 18:39 ` Richard Davies
2014-04-29 19:03 ` Michal Hocko
[not found] ` <20140429183928.GF29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-29 19:03 ` Michal Hocko
2014-04-29 21:36 ` Marian Marinov
2014-04-29 21:36 ` Marian Marinov
2014-04-30 13:31 ` Michal Hocko
2014-04-30 13:31 ` Michal Hocko
[not found] ` <53601B68.60906-NV7Lj0SOnH0@public.gmane.org>
2014-04-30 13:31 ` Michal Hocko
[not found] ` <20140429170639.GA25609-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
2014-04-29 17:30 ` Dwight Engen
[not found] ` <CAO_Rewa20dneL8e3T4UPnu2Dkv28KTgFJR9_YSmRBKp-_yqewg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-04-29 17:06 ` Michal Hocko
2014-04-29 21:44 ` Frederic Weisbecker
2014-04-29 21:44 ` Frederic Weisbecker
2014-04-29 21:44 ` Frederic Weisbecker
[not found] ` <20140429214454.GF6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2014-04-30 13:12 ` Daniel J Walsh
2014-04-30 13:12 ` Daniel J Walsh
[not found] ` <5360F6B4.9010308-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2014-04-30 13:28 ` Frederic Weisbecker
2014-04-30 13:28 ` Frederic Weisbecker
2014-04-30 13:28 ` Frederic Weisbecker
[not found] ` <20140429154345.GH15058-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
2014-04-29 16:06 ` Tim Hockin
2014-05-06 11:40 ` Marian Marinov
[not found] ` <5368CA47.7030007-NV7Lj0SOnH0@public.gmane.org>
2014-05-07 17:15 ` Dwight Engen
2014-05-07 17:15 ` Dwight Engen
2014-05-07 22:39 ` Marian Marinov
2014-05-07 22:39 ` Marian Marinov
2014-05-08 15:25 ` Richard Davies
2014-05-08 15:25 ` Richard Davies
2014-06-10 14:50 ` Marian Marinov
2014-06-10 14:50 ` Marian Marinov
[not found] ` <20140423084942.560ae837-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2014-05-06 11:40 ` Marian Marinov
2014-06-10 14:50 ` Marian Marinov
[not found] ` <20140422200531.GA19334-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org>
2014-04-22 20:13 ` Tim Hockin
2014-04-23 6:07 ` Marian Marinov
2014-06-10 12:18 ` Alin Dobre
2014-06-10 12:18 ` Alin Dobre
[not found] ` <20140422143943.20609800-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2014-04-22 20:05 ` Richard Davies
[not found] ` <5351679F.5040908-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
2014-04-20 14:28 ` Richard Davies
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.