From: Vitaly Wool <vitalywool@gmail.com>
To: Minchan Kim <minchan@kernel.org>
Cc: "Sergey Senozhatsky" <sergey.senozhatsky@gmail.com>,
"Dan Streetman" <ddstreet@ieee.org>,
LKML <linux-kernel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>, 김준수 <iamjoonsoo.kim@lge.com>,
"Gioh Kim" <gioh.kim@lge.com>
Subject: Re: [PATCH 0/3] allow zram to use zbud as underlying allocator
Date: Fri, 25 Sep 2015 11:54:57 +0200 [thread overview]
Message-ID: <CAMJBoFM_bMvQthAJPK+w4uQznqp7eFLdk=c7ZtT-R1aoF-1SeA@mail.gmail.com> (raw)
In-Reply-To: <20150915061349.GA16485@bbox>
Hello Minchan,
the main use case where I see unacceptably long stalls in UI with
zsmalloc is switching between users in Android.
There is a way to automate user creation and switching between them so
the test I run both to get vmstat statistics and to profile stalls is
to create a user, switch to it and switch back. Each test cycle does
that 10 times, and all the results presented below are averages for 20
runs.
Kernel configurations used for testing:
(1): vanilla
(2): (1) plus "make SLUB atomic" patch [1]
(3): (1) with zbud instead of zsmalloc
(4): (2) with compaction defer logic mostly disabled
> KSM? Is there any reason you mentioned *KSM* in this context?
> IOW, if you don't use KSM, you couldn't see a problem?
If I don't use KSM, latenices get smaller in both cases. Worst case
wise, zbud still gives more deterministic behavior.
>> I ran into several occasions when moving pages from compressed swap back
>> into the "normal" part of RAM caused significant latencies in system operation.
>
> What kernel version did you use? Did you enable CMA? ION?
> What was major factor for the latencies?
CMA and ION are both enabled. The working kernel is 3.18 based with
most of the mm/ stuff backported from 4.2.
The major factors for the latencies was a) fragmentation and b)
compaction deferral. See also below.
> Decompress? zsmalloc-compaction overhead? rmap overheads?
> compaction overheads?
> There are several potential culprits.
> It would be very helpful if you provide some numbers(perf will help you).
The UI is blocked after user switching for, average:
(1) 1.84 seconds
(2) 0.89 seconds
(3) 1.32 seconds
(4) 0.87 seconds
The UI us blocked after user switching for, worst-case:
(1) 2.91
(2) 1.12
(3) 1.79
(4) 1.34
Selected vmstat results, average:
I. allocstall
(1) 7814
(2) 4615
(3) 2004
(4) 2611
II. compact_stall
(1) 1869
(2) 1135
(3) 727
(4) 638
III. compact_fail
(1) 914
(2) 520
(3) 230
(4) 218
IV. compact_success
(1) 876
(2) 535
(3) 419
(4) 443
More data available on request.
>> By using zbud I lose in compression ratio but gain in determinism,
>> lower latencies and lower fragmentation, so in the coming patches
>> I tried to generalize what I've done to enable zbud for zram so far.
>
> Before that, I'd like to know what is root cause.
> From my side, I had an similar experience.
> At that time, problem was that *compaction* which triggered to reclaim
> lots of page cache pages. The reason compaction triggered a lot was
> fragmentation caused by zsmalloc, GPU and high-order allocation
> request by SLUB and somethings(ex, ION, fork).
>
> Recently, Joonsoo fixed SLUB side.
> http://marc.info/?l=linux-kernel&m=143891343223853&w=2
Yes, it makes things better, see above. However, worst case is still
looking not so nice.
> And we added zram-auto-compaction recently so zram try to compact
> objects to reduce memory usage. It might be helpful for fragment
> problem as side effect but please keep it mind that it would be opposite.
> Currently, zram-auto-compaction is not aware of virtual memory compaction
> so as worst case, zsmalloc can spread out pinned object into movable
> pageblocks via zsmalloc-compaction.
> Gioh and I try to solve the issue with below patches but is pending now
> by other urgent works.
> https://lwn.net/Articles/650917/
> https://lkml.org/lkml/2015/8/10/90
>
> In summary, we need to clarify what's the root cause before diving into
> code and hiding it.
I'm not "hiding" anything. This statement is utterly bogus.
Summarizing my test results, I would like to stress that:
* zbud gives better worst-times
* the system's behavior with zbud is way more stable and predictable
* zsmalloc-based zram operation depends very much on kernel memory
management subsystem changes
* zsmalloc operates significantly worse with compaction deferral logic
introduced after ca. 3.18
As a bottom line, zsmalloc operation is substantially more fragile and
far less predictable than zbud's. If that is not a good reason to _at
least_ have *an option* to use zram with the latter, then I don't know
what is.
~vitaly
WARNING: multiple messages have this Message-ID (diff)
From: Vitaly Wool <vitalywool@gmail.com>
To: Minchan Kim <minchan@kernel.org>
Cc: "Sergey Senozhatsky" <sergey.senozhatsky@gmail.com>,
"Dan Streetman" <ddstreet@ieee.org>,
LKML <linux-kernel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>, 김준수 <iamjoonsoo.kim@lge.com>,
"Gioh Kim" <gioh.kim@lge.com>
Subject: Re: [PATCH 0/3] allow zram to use zbud as underlying allocator
Date: Fri, 25 Sep 2015 11:54:57 +0200 [thread overview]
Message-ID: <CAMJBoFM_bMvQthAJPK+w4uQznqp7eFLdk=c7ZtT-R1aoF-1SeA@mail.gmail.com> (raw)
In-Reply-To: <20150915061349.GA16485@bbox>
Hello Minchan,
the main use case where I see unacceptably long stalls in UI with
zsmalloc is switching between users in Android.
There is a way to automate user creation and switching between them so
the test I run both to get vmstat statistics and to profile stalls is
to create a user, switch to it and switch back. Each test cycle does
that 10 times, and all the results presented below are averages for 20
runs.
Kernel configurations used for testing:
(1): vanilla
(2): (1) plus "make SLUB atomic" patch [1]
(3): (1) with zbud instead of zsmalloc
(4): (2) with compaction defer logic mostly disabled
> KSM? Is there any reason you mentioned *KSM* in this context?
> IOW, if you don't use KSM, you couldn't see a problem?
If I don't use KSM, latenices get smaller in both cases. Worst case
wise, zbud still gives more deterministic behavior.
>> I ran into several occasions when moving pages from compressed swap back
>> into the "normal" part of RAM caused significant latencies in system operation.
>
> What kernel version did you use? Did you enable CMA? ION?
> What was major factor for the latencies?
CMA and ION are both enabled. The working kernel is 3.18 based with
most of the mm/ stuff backported from 4.2.
The major factors for the latencies was a) fragmentation and b)
compaction deferral. See also below.
> Decompress? zsmalloc-compaction overhead? rmap overheads?
> compaction overheads?
> There are several potential culprits.
> It would be very helpful if you provide some numbers(perf will help you).
The UI is blocked after user switching for, average:
(1) 1.84 seconds
(2) 0.89 seconds
(3) 1.32 seconds
(4) 0.87 seconds
The UI us blocked after user switching for, worst-case:
(1) 2.91
(2) 1.12
(3) 1.79
(4) 1.34
Selected vmstat results, average:
I. allocstall
(1) 7814
(2) 4615
(3) 2004
(4) 2611
II. compact_stall
(1) 1869
(2) 1135
(3) 727
(4) 638
III. compact_fail
(1) 914
(2) 520
(3) 230
(4) 218
IV. compact_success
(1) 876
(2) 535
(3) 419
(4) 443
More data available on request.
>> By using zbud I lose in compression ratio but gain in determinism,
>> lower latencies and lower fragmentation, so in the coming patches
>> I tried to generalize what I've done to enable zbud for zram so far.
>
> Before that, I'd like to know what is root cause.
> From my side, I had an similar experience.
> At that time, problem was that *compaction* which triggered to reclaim
> lots of page cache pages. The reason compaction triggered a lot was
> fragmentation caused by zsmalloc, GPU and high-order allocation
> request by SLUB and somethings(ex, ION, fork).
>
> Recently, Joonsoo fixed SLUB side.
> http://marc.info/?l=linux-kernel&m=143891343223853&w=2
Yes, it makes things better, see above. However, worst case is still
looking not so nice.
> And we added zram-auto-compaction recently so zram try to compact
> objects to reduce memory usage. It might be helpful for fragment
> problem as side effect but please keep it mind that it would be opposite.
> Currently, zram-auto-compaction is not aware of virtual memory compaction
> so as worst case, zsmalloc can spread out pinned object into movable
> pageblocks via zsmalloc-compaction.
> Gioh and I try to solve the issue with below patches but is pending now
> by other urgent works.
> https://lwn.net/Articles/650917/
> https://lkml.org/lkml/2015/8/10/90
>
> In summary, we need to clarify what's the root cause before diving into
> code and hiding it.
I'm not "hiding" anything. This statement is utterly bogus.
Summarizing my test results, I would like to stress that:
* zbud gives better worst-times
* the system's behavior with zbud is way more stable and predictable
* zsmalloc-based zram operation depends very much on kernel memory
management subsystem changes
* zsmalloc operates significantly worse with compaction deferral logic
introduced after ca. 3.18
As a bottom line, zsmalloc operation is substantially more fragile and
far less predictable than zbud's. If that is not a good reason to _at
least_ have *an option* to use zram with the latter, then I don't know
what is.
~vitaly
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2015-09-25 9:55 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-09-14 13:49 [PATCH 0/3] allow zram to use zbud as underlying allocator Vitaly Wool
2015-09-14 13:49 ` Vitaly Wool
2015-09-14 13:50 ` [PATCH 1/3] zram: make max_zpage_size configurable Vitaly Wool
2015-09-14 13:50 ` Vitaly Wool
2015-09-15 1:00 ` Sergey Senozhatsky
2015-09-15 1:00 ` Sergey Senozhatsky
2015-09-15 7:18 ` Vitaly Wool
2015-09-15 7:38 ` Sergey Senozhatsky
2015-09-15 7:38 ` Sergey Senozhatsky
2015-09-15 5:42 ` Dan Streetman
2015-09-15 5:42 ` Dan Streetman
2015-09-15 6:08 ` Sergey Senozhatsky
2015-09-15 6:08 ` Sergey Senozhatsky
2015-09-14 13:51 ` [PATCH 2/3] zpool/zsmalloc/zbud: align on interfaces Vitaly Wool
2015-09-14 13:51 ` Vitaly Wool
2015-09-15 1:06 ` Sergey Senozhatsky
2015-09-15 1:06 ` Sergey Senozhatsky
2015-09-15 5:09 ` Dan Streetman
2015-09-15 5:09 ` Dan Streetman
2015-09-14 13:55 ` [PATCH 3/3] zram: use common zpool interface Vitaly Wool
2015-09-14 13:55 ` Vitaly Wool
2015-09-15 1:12 ` Sergey Senozhatsky
2015-09-15 1:12 ` Sergey Senozhatsky
2015-09-15 6:03 ` Dan Streetman
2015-09-15 6:03 ` Dan Streetman
2015-09-14 14:01 ` [PATCH 0/3] allow zram to use zbud as underlying allocator Vlastimil Babka
2015-09-14 14:01 ` Vlastimil Babka
2015-09-14 14:12 ` Vitaly Wool
2015-09-14 14:12 ` Vitaly Wool
2015-09-14 14:14 ` Vlastimil Babka
2015-09-14 14:14 ` Vlastimil Babka
2015-09-15 4:08 ` Dan Streetman
2015-09-15 4:08 ` Dan Streetman
2015-09-15 4:22 ` Sergey Senozhatsky
2015-09-15 4:22 ` Sergey Senozhatsky
2015-09-17 6:21 ` Vlastimil Babka
2015-09-17 6:21 ` Vlastimil Babka
2015-09-17 9:19 ` Sergey Senozhatsky
2015-09-17 9:19 ` Sergey Senozhatsky
2015-09-15 0:49 ` Sergey Senozhatsky
2015-09-15 0:49 ` Sergey Senozhatsky
2015-09-15 6:13 ` Minchan Kim
2015-09-15 6:13 ` Minchan Kim
2015-09-25 9:54 ` Vitaly Wool [this message]
2015-09-25 9:54 ` Vitaly Wool
2015-09-30 7:52 ` Minchan Kim
2015-09-30 7:52 ` Minchan Kim
2015-09-30 8:01 ` Vitaly Wool
2015-09-30 8:01 ` Vitaly Wool
2015-09-30 8:13 ` Minchan Kim
2015-09-30 8:13 ` Minchan Kim
2015-09-30 8:18 ` Vitaly Wool
2015-09-30 8:18 ` Vitaly Wool
2015-09-30 15:37 ` Vlastimil Babka
2015-09-30 15:37 ` Vlastimil Babka
2015-09-30 15:46 ` Vitaly Wool
2015-09-30 15:46 ` Vitaly Wool
2015-10-01 7:52 ` Vlastimil Babka
2015-10-01 7:52 ` Vlastimil Babka
2015-10-10 9:33 ` Vitaly Wool
2015-10-10 9:33 ` Vitaly Wool
2015-10-14 13:28 ` Vlastimil Babka
2015-10-14 13:28 ` Vlastimil Babka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAMJBoFM_bMvQthAJPK+w4uQznqp7eFLdk=c7ZtT-R1aoF-1SeA@mail.gmail.com' \
--to=vitalywool@gmail.com \
--cc=ddstreet@ieee.org \
--cc=gioh.kim@lge.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=minchan@kernel.org \
--cc=sergey.senozhatsky@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.