From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755225AbbFLKLo (ORCPT ); Fri, 12 Jun 2015 06:11:44 -0400 Received: from mail-wi0-f181.google.com ([209.85.212.181]:33445 "EHLO mail-wi0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751161AbbFLKLk convert rfc822-to-8bit (ORCPT ); Fri, 12 Jun 2015 06:11:40 -0400 From: Michal Nazarewicz To: Vlastimil Babka , Andrew Morton , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Vlastimil Babka , Minchan Kim , Mel Gorman , Joonsoo Kim , Naoya Horiguchi , Christoph Lameter , Rik van Riel , David Rientjes Subject: Re: [PATCH 4/6] mm, compaction: always skip compound pages by order in migrate scanner In-Reply-To: <1433928754-966-5-git-send-email-vbabka@suse.cz> Organization: http://mina86.com/ References: <1433928754-966-1-git-send-email-vbabka@suse.cz> <1433928754-966-5-git-send-email-vbabka@suse.cz> User-Agent: Notmuch/0.19+53~g2e63a09 (http://notmuchmail.org) Emacs/25.0.50.1 (x86_64-unknown-linux-gnu) X-Face: PbkBB1w#)bOqd`iCe"Ds{e+!C7`pkC9a|f)Qo^BMQvy\q5x3?vDQJeN(DS?|-^$uMti[3D*#^_Ts"pU$jBQLq~Ud6iNwAw_r_o_4]|JO?]}P_}Nc&"p#D(ZgUb4uCNPe7~a[DbPG0T~!&c.y$Ur,=N4RT>]dNpd;KFrfMCylc}gc??'U2j,!8%xdD Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWbfGlUPDDHgE57V0jUupKjgIObY0PLrom9mH4dFRK4gmjPs41MxjOgAAACQElEQVQ4jW3TMWvbQBQHcBk1xE6WyALX1069oZBMlq+ouUwpEQQ6uRjttkWP4CmBgGM0BQLBdPFZYPsyFUo6uEtKDQ7oy/U96XR2Ux8ehH/89Z6enqxBcS7Lg81jmSuujrfCZcLI/TYYvbGj+jbgFpHJ/bqQAUISj8iLyu4LuFHJTosxsucO4jSDNE0Hq3hwK/ceQ5sx97b8LcUDsILfk+ovHkOIsMbBfg43VuQ5Ln9YAGCkUdKJoXR9EclFBhixy3EGVz1K6eEkhxCAkeMMnqoAhAKwhoUJkDrCqvbecaYINlFKSRS1i12VKH1XpUd4qxL876EkMcDvHj3s5RBajHHMlA5iK32e0C7VgG0RlzFPvoYHZLRmAC0BmNcBruhkE0KsMsbEc62ZwUJDxWUdMsMhVqovoT96i/DnX/ASvz/6hbCabELLk/6FF/8PNpPCGqcZTGFcBhhAaZZDbQPaAB3+KrWWy2XgbYDNIinkdWAFcCpraDE/knwe5DBqGmgzESl1p2E4MWAz0VUPgYYzmfWb9yS4vCvgsxJriNTHoIBz5YteBvg+VGISQWUqhMiByPIPpygeDBE6elD973xWwKkEiHZAHKjhuPsFnBuArrzxtakRcISv+XMIPl4aGBUJm8Emk7qBYU8IlgNEIpiJhk/No24jHwkKTFHDWfPniR4iw5vJaw2nzSjfq2zffcE/GDjRC2dn0J0XwPAbDL84TvaFCJEU4Oml9pRyEUhR3Cl2t01AoEjRbs0sYugp14/4X5n4pU4EHHnMAAAAAElFTkSuQmCC X-PGP: 50751FF4 X-PGP-FP: AC1F 5F5C D418 88F8 CC84 5858 2060 4012 5075 1FF4 X-Hashcash: 1:20:150612:minchan@kernel.org::GYer8i+7HmINMuVG:0000000000000000000000000000000000000000000074N X-Hashcash: 1:20:150612:vbabka@suse.cz::yw74Fwdg8H2vpKL1:0000LD2 X-Hashcash: 1:20:150612:rientjes@google.com::rQAnsp3nNcLzkFDd:0000000000000000000000000000000000000000000GHt X-Hashcash: 1:20:150612:n-horiguchi@ah.jp.nec.com::XCqqKoD7Ppxw4EuE:0000000000000000000000000000000000000mUY X-Hashcash: 1:20:150612:cl@linux.com::DKdB456NbEN0L7jH:000001XRq X-Hashcash: 1:20:150612:vbabka@suse.cz::5pBHhHXWzn1OQpBD:0001ukg X-Hashcash: 1:20:150612:linux-mm@kvack.org::SPs8idlKY2vRxcDs:00000000000000000000000000000000000000000001zQk X-Hashcash: 1:20:150612:linux-kernel@vger.kernel.org::NEEAK7EIpnIcsCXC:0000000000000000000000000000000002TDx X-Hashcash: 1:20:150612:riel@redhat.com::VyrB6kVKm5wLex73:002K9k X-Hashcash: 1:20:150612:mgorman@suse.de::g8dj5Qikdgfsnkc9:002z3M X-Hashcash: 1:20:150612:iamjoonsoo.kim@lge.com::7bD4U+zgIKTc53uO:0000000000000000000000000000000000000004wHh X-Hashcash: 1:20:150612:akpm@linux-foundation.org::DYuo2/WjKEvHPGwu:00000000000000000000000000000000000097y5 Date: Fri, 12 Jun 2015 12:11:35 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 10 2015, Vlastimil Babka wrote: > The compaction migrate scanner tries to skip compound pages by their order, to > reduce number of iterations for pages it cannot isolate. The check is only done > if PageLRU() is true, which means it applies to THP pages, but not e.g. > hugetlbfs pages or any other non-LRU compound pages, which we have to iterate > by base pages. > > This limitation comes from the assumption that it's only safe to read > compound_order() when we have the zone's lru_lock and THP cannot be split under > us. But the only danger (after filtering out order values that are not below > MAX_ORDER, to prevent overflows) is that we skip too much or too little after > reading a bogus compound_order() due to a rare race. This is the same reasoning > as patch 99c0fd5e51c4 ("mm, compaction: skip buddy pages by their order in the > migrate scanner") introduced for unsafely reading PageBuddy() order. > > After this patch, all pages are tested for PageCompound() and we skip them by > compound_order(). The test is done after the test for balloon_page_movable() > as we don't want to assume if balloon pages (or other pages with own isolation > and migration implementation if a generic API gets implemented) are compound > or not. > > When tested with stress-highalloc from mmtests on 4GB system with 1GB hugetlbfs > pages, the vmstat compact_migrate_scanned count decreased by 15%. > > Signed-off-by: Vlastimil Babka > Cc: Minchan Kim > Cc: Mel Gorman > Cc: Joonsoo Kim > Cc: Michal Nazarewicz Acked-by: Michal Nazarewicz > Cc: Naoya Horiguchi > Cc: Christoph Lameter > Cc: Rik van Riel > Cc: David Rientjes > --- > mm/compaction.c | 36 +++++++++++++++++------------------- > 1 file changed, 17 insertions(+), 19 deletions(-) > > diff --git a/mm/compaction.c b/mm/compaction.c > index d334bb3..e37d361 100644 > --- a/mm/compaction.c > +++ b/mm/compaction.c > @@ -680,6 +680,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, > > /* Time to isolate some pages for migration */ > for (; low_pfn < end_pfn; low_pfn++) { > + bool is_lru; > + > /* > * Periodically drop the lock (if held) regardless of its > * contention, to give chance to IRQs. Abort async compaction > @@ -723,39 +725,35 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, > * It's possible to migrate LRU pages and balloon pages > * Skip any other type of page > */ > - if (!PageLRU(page)) { > + is_lru = PageLRU(page); > + if (!is_lru) { > if (unlikely(balloon_page_movable(page))) { > if (balloon_page_isolate(page)) { > /* Successfully isolated */ > goto isolate_success; > } > } > - continue; > } > > /* > - * PageLRU is set. lru_lock normally excludes isolation > - * splitting and collapsing (collapsing has already happened > - * if PageLRU is set) but the lock is not necessarily taken > - * here and it is wasteful to take it just to check transhuge. > - * Check PageCompound without lock and skip the whole pageblock > - * if it's a transhuge page, as calling compound_order() > - * without preventing THP from splitting the page underneath us > - * may return surprising results. > - * If we happen to check a THP tail page, compound_order() > - * returns 0. It should be rare enough to not bother with > - * using compound_head() in that case. > + * Regardless of being on LRU, compound pages such as THP and > + * hugetlbfs are not to be compacted. We can potentially save > + * a lot of iterations if we skip them at once. The check is > + * racy, but we can consider only valid values and the only > + * danger is skipping too much. > */ > if (PageCompound(page)) { > - int nr; > - if (locked) > - nr = 1 << compound_order(page); > - else > - nr = pageblock_nr_pages; > - low_pfn += nr - 1; > + unsigned int comp_order = compound_order(page); > + > + if (comp_order > 0 && comp_order < MAX_ORDER) > + low_pfn += (1UL << comp_order) - 1; > + > continue; > } > > + if (!is_lru) > + continue; > + > /* > * Migration will fail if an anonymous page is pinned in memory, > * so avoid taking lru_lock and isolating it unnecessarily in an > -- > 2.1.4 > -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michał “mina86” Nazarewicz (o o) ooo +------ooO--(_)--Ooo-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-f53.google.com (mail-wg0-f53.google.com [74.125.82.53]) by kanga.kvack.org (Postfix) with ESMTP id 9CDF96B0083 for ; Fri, 12 Jun 2015 06:11:41 -0400 (EDT) Received: by wgez8 with SMTP id z8so21280028wge.0 for ; Fri, 12 Jun 2015 03:11:41 -0700 (PDT) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com. [2a00:1450:400c:c05::22d]) by mx.google.com with ESMTPS id ly8si6302673wjb.40.2015.06.12.03.11.39 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Jun 2015 03:11:40 -0700 (PDT) Received: by wibut5 with SMTP id ut5so13775710wib.1 for ; Fri, 12 Jun 2015 03:11:39 -0700 (PDT) From: Michal Nazarewicz Subject: Re: [PATCH 4/6] mm, compaction: always skip compound pages by order in migrate scanner In-Reply-To: <1433928754-966-5-git-send-email-vbabka@suse.cz> References: <1433928754-966-1-git-send-email-vbabka@suse.cz> <1433928754-966-5-git-send-email-vbabka@suse.cz> Date: Fri, 12 Jun 2015 12:11:35 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka , Andrew Morton , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Minchan Kim , Mel Gorman , Joonsoo Kim , Naoya Horiguchi , Christoph Lameter , Rik van Riel , David Rientjes On Wed, Jun 10 2015, Vlastimil Babka wrote: > The compaction migrate scanner tries to skip compound pages by their orde= r, to > reduce number of iterations for pages it cannot isolate. The check is onl= y done > if PageLRU() is true, which means it applies to THP pages, but not e.g. > hugetlbfs pages or any other non-LRU compound pages, which we have to ite= rate > by base pages. > > This limitation comes from the assumption that it's only safe to read > compound_order() when we have the zone's lru_lock and THP cannot be split= under > us. But the only danger (after filtering out order values that are not be= low > MAX_ORDER, to prevent overflows) is that we skip too much or too little a= fter > reading a bogus compound_order() due to a rare race. This is the same rea= soning > as patch 99c0fd5e51c4 ("mm, compaction: skip buddy pages by their order i= n the > migrate scanner") introduced for unsafely reading PageBuddy() order. > > After this patch, all pages are tested for PageCompound() and we skip the= m by > compound_order(). The test is done after the test for balloon_page_movab= le() > as we don't want to assume if balloon pages (or other pages with own isol= ation > and migration implementation if a generic API gets implemented) are compo= und > or not. > > When tested with stress-highalloc from mmtests on 4GB system with 1GB hug= etlbfs > pages, the vmstat compact_migrate_scanned count decreased by 15%. > > Signed-off-by: Vlastimil Babka > Cc: Minchan Kim > Cc: Mel Gorman > Cc: Joonsoo Kim > Cc: Michal Nazarewicz Acked-by: Michal Nazarewicz > Cc: Naoya Horiguchi > Cc: Christoph Lameter > Cc: Rik van Riel > Cc: David Rientjes > --- > mm/compaction.c | 36 +++++++++++++++++------------------- > 1 file changed, 17 insertions(+), 19 deletions(-) > > diff --git a/mm/compaction.c b/mm/compaction.c > index d334bb3..e37d361 100644 > --- a/mm/compaction.c > +++ b/mm/compaction.c > @@ -680,6 +680,8 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, >=20=20 > /* Time to isolate some pages for migration */ > for (; low_pfn < end_pfn; low_pfn++) { > + bool is_lru; > + > /* > * Periodically drop the lock (if held) regardless of its > * contention, to give chance to IRQs. Abort async compaction > @@ -723,39 +725,35 @@ isolate_migratepages_block(struct compact_control *= cc, unsigned long low_pfn, > * It's possible to migrate LRU pages and balloon pages > * Skip any other type of page > */ > - if (!PageLRU(page)) { > + is_lru =3D PageLRU(page); > + if (!is_lru) { > if (unlikely(balloon_page_movable(page))) { > if (balloon_page_isolate(page)) { > /* Successfully isolated */ > goto isolate_success; > } > } > - continue; > } >=20=20 > /* > - * PageLRU is set. lru_lock normally excludes isolation > - * splitting and collapsing (collapsing has already happened > - * if PageLRU is set) but the lock is not necessarily taken > - * here and it is wasteful to take it just to check transhuge. > - * Check PageCompound without lock and skip the whole pageblock > - * if it's a transhuge page, as calling compound_order() > - * without preventing THP from splitting the page underneath us > - * may return surprising results. > - * If we happen to check a THP tail page, compound_order() > - * returns 0. It should be rare enough to not bother with > - * using compound_head() in that case. > + * Regardless of being on LRU, compound pages such as THP and > + * hugetlbfs are not to be compacted. We can potentially save > + * a lot of iterations if we skip them at once. The check is > + * racy, but we can consider only valid values and the only > + * danger is skipping too much. > */ > if (PageCompound(page)) { > - int nr; > - if (locked) > - nr =3D 1 << compound_order(page); > - else > - nr =3D pageblock_nr_pages; > - low_pfn +=3D nr - 1; > + unsigned int comp_order =3D compound_order(page); > + > + if (comp_order > 0 && comp_order < MAX_ORDER) > + low_pfn +=3D (1UL << comp_order) - 1; > + > continue; > } >=20=20 > + if (!is_lru) > + continue; > + > /* > * Migration will fail if an anonymous page is pinned in memory, > * so avoid taking lru_lock and isolating it unnecessarily in an > --=20 > 2.1.4 > --=20 Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=3D./ `o ..o | Computer Science, Micha=C5=82 =E2=80=9Cmina86=E2=80=9D Nazarewicz = (o o) ooo +------ooO--(_)--Ooo-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org