[PATCH 5/6] mm, compaction: skip compound pages by order in free scanner

All the mail mirrored from lore.kernel.org
 help / color / mirror / Atom feed

From: Vlastimil Babka <vbabka@suse.cz>
To: Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, Vlastimil Babka <vbabka@suse.cz>,
	Minchan Kim <minchan@kernel.org>, Mel Gorman <mgorman@suse.de>,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>,
	Michal Nazarewicz <mina86@mina86.com>,
	Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>,
	Christoph Lameter <cl@linux.com>, Rik van Riel <riel@redhat.com>,
	David Rientjes <rientjes@google.com>
Subject: [PATCH 5/6] mm, compaction: skip compound pages by order in free scanner
Date: Wed, 10 Jun 2015 11:32:33 +0200	[thread overview]
Message-ID: <1433928754-966-6-git-send-email-vbabka@suse.cz> (raw)
In-Reply-To: <1433928754-966-1-git-send-email-vbabka@suse.cz>

The compaction free scanner is looking for PageBuddy() pages and skipping all
others.  For large compound pages such as THP or hugetlbfs, we can save a lot
of iterations if we skip them at once using their compound_order(). This is
generally unsafe and we can read a bogus value of order due to a race, but if
we are careful, the only danger is skipping too much.

When tested with stress-highalloc from mmtests on 4GB system with 1GB hugetlbfs
pages, the vmstat compact_free_scanned count decreased by at least 15%.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: David Rientjes <rientjes@google.com>
---
 mm/compaction.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/mm/compaction.c b/mm/compaction.c
index e37d361..4a14084 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -437,6 +437,24 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
 
 		if (!valid_page)
 			valid_page = page;
+
+		/*
+		 * For compound pages such as THP and hugetlbfs, we can save
+		 * potentially a lot of iterations if we skip them at once.
+		 * The check is racy, but we can consider only valid values
+		 * and the only danger is skipping too much.
+		 */
+		if (PageCompound(page)) {
+			unsigned int comp_order = compound_order(page);
+
+			if (comp_order > 0 && comp_order < MAX_ORDER) {
+				blockpfn += (1UL << comp_order) - 1;
+				cursor += (1UL << comp_order) - 1;
+			}
+
+			goto isolate_fail;
+		}
+
 		if (!PageBuddy(page))
 			goto isolate_fail;
 
@@ -496,6 +514,13 @@ isolate_fail:
 
 	}
 
+	/*
+	 * There is a tiny chance that we have read bogus compound_order(),
+	 * so be careful to not go outside of the pageblock.
+	 */
+	if (unlikely(blockpfn > end_pfn))
+		blockpfn = end_pfn;
+
 	trace_mm_compaction_isolate_freepages(*start_pfn, blockpfn,
 					nr_scanned, total_isolated);
 
-- 
2.1.4

WARNING: multiple messages have this Message-ID (diff)

From: Vlastimil Babka <vbabka@suse.cz>
To: Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, Vlastimil Babka <vbabka@suse.cz>,
	Minchan Kim <minchan@kernel.org>, Mel Gorman <mgorman@suse.de>,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>,
	Michal Nazarewicz <mina86@mina86.com>,
	Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>,
	Christoph Lameter <cl@linux.com>, Rik van Riel <riel@redhat.com>,
	David Rientjes <rientjes@google.com>
Subject: [PATCH 5/6] mm, compaction: skip compound pages by order in free scanner
Date: Wed, 10 Jun 2015 11:32:33 +0200	[thread overview]
Message-ID: <1433928754-966-6-git-send-email-vbabka@suse.cz> (raw)
In-Reply-To: <1433928754-966-1-git-send-email-vbabka@suse.cz>

The compaction free scanner is looking for PageBuddy() pages and skipping all
others.  For large compound pages such as THP or hugetlbfs, we can save a lot
of iterations if we skip them at once using their compound_order(). This is
generally unsafe and we can read a bogus value of order due to a race, but if
we are careful, the only danger is skipping too much.

When tested with stress-highalloc from mmtests on 4GB system with 1GB hugetlbfs
pages, the vmstat compact_free_scanned count decreased by at least 15%.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: David Rientjes <rientjes@google.com>
---
 mm/compaction.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/mm/compaction.c b/mm/compaction.c
index e37d361..4a14084 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -437,6 +437,24 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
 
 		if (!valid_page)
 			valid_page = page;
+
+		/*
+		 * For compound pages such as THP and hugetlbfs, we can save
+		 * potentially a lot of iterations if we skip them at once.
+		 * The check is racy, but we can consider only valid values
+		 * and the only danger is skipping too much.
+		 */
+		if (PageCompound(page)) {
+			unsigned int comp_order = compound_order(page);
+
+			if (comp_order > 0 && comp_order < MAX_ORDER) {
+				blockpfn += (1UL << comp_order) - 1;
+				cursor += (1UL << comp_order) - 1;
+			}
+
+			goto isolate_fail;
+		}
+
 		if (!PageBuddy(page))
 			goto isolate_fail;
 
@@ -496,6 +514,13 @@ isolate_fail:
 
 	}
 
+	/*
+	 * There is a tiny chance that we have read bogus compound_order(),
+	 * so be careful to not go outside of the pageblock.
+	 */
+	if (unlikely(blockpfn > end_pfn))
+		blockpfn = end_pfn;
+
 	trace_mm_compaction_isolate_freepages(*start_pfn, blockpfn,
 					nr_scanned, total_isolated);
 
-- 
2.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

next prev parent reply	other threads:[~2015-06-10  9:35 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-06-10  9:32 [PATCH 0/6] Assorted compaction cleanups and optimizations Vlastimil Babka
2015-06-10  9:32 ` Vlastimil Babka
2015-06-10  9:32 ` [PATCH 1/6] mm, compaction: more robust check for scanners meeting Vlastimil Babka
2015-06-10  9:32   ` Vlastimil Babka
2015-06-10 18:02   ` Rik van Riel
2015-06-10 18:02     ` Rik van Riel
2015-06-12  9:55   ` Michal Nazarewicz
2015-06-12  9:55     ` Michal Nazarewicz
2015-06-16  5:37   ` Joonsoo Kim
2015-06-16  5:37     ` Joonsoo Kim
2015-06-19 13:41   ` Mel Gorman
2015-06-19 13:41     ` Mel Gorman
2015-06-10  9:32 ` [PATCH 2/6] mm, compaction: simplify handling restart position in free pages scanner Vlastimil Babka
2015-06-10  9:32   ` Vlastimil Babka
2015-06-16  5:38   ` Joonsoo Kim
2015-06-16  5:38     ` Joonsoo Kim
2015-06-19 13:48   ` Mel Gorman
2015-06-19 13:48     ` Mel Gorman
2015-06-10  9:32 ` [PATCH 3/6] mm, compaction: encapsulate resetting cached scanner positions Vlastimil Babka
2015-06-10  9:32   ` Vlastimil Babka
2015-06-12 10:07   ` Michal Nazarewicz
2015-06-12 10:07     ` Michal Nazarewicz
2015-06-16  5:41   ` Joonsoo Kim
2015-06-16  5:41     ` Joonsoo Kim
2015-06-16 12:13     ` Vlastimil Babka
2015-06-16 12:13       ` Vlastimil Babka
2015-06-10  9:32 ` [PATCH 4/6] mm, compaction: always skip compound pages by order in migrate scanner Vlastimil Babka
2015-06-10  9:32   ` Vlastimil Babka
2015-06-12 10:11   ` Michal Nazarewicz
2015-06-12 10:11     ` Michal Nazarewicz
2015-06-16  5:44   ` Joonsoo Kim
2015-06-16  5:44     ` Joonsoo Kim
2015-06-16 12:16     ` Vlastimil Babka
2015-06-16 12:16       ` Vlastimil Babka
2015-06-19 13:58   ` Mel Gorman
2015-06-19 13:58     ` Mel Gorman
2015-06-10  9:32 ` Vlastimil Babka [this message]
2015-06-10  9:32   ` [PATCH 5/6] mm, compaction: skip compound pages by order in free scanner Vlastimil Babka
2015-06-12 10:18   ` Michal Nazarewicz
2015-06-12 10:18     ` Michal Nazarewicz
2015-06-16  5:45   ` Joonsoo Kim
2015-06-16  5:45     ` Joonsoo Kim
2015-06-10  9:32 ` [PATCH 6/6] mm, compaction: decouple updating pageblock_skip and cached pfn Vlastimil Babka
2015-06-10  9:32   ` Vlastimil Babka
2015-06-16  6:10   ` Joonsoo Kim
2015-06-16  6:10     ` Joonsoo Kim
2015-06-16 12:33     ` Vlastimil Babka
2015-06-16 12:33       ` Vlastimil Babka
2015-06-16 13:03       ` Joonsoo Kim
2015-06-16 13:03         ` Joonsoo Kim

find likely ancestor, descendant, or conflicting patches for this message:
dfblob:e37d361 dfblob:e37d361 dfblob:4a14084 dfblob:4a14084
	(help)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1433928754-966-6-git-send-email-vbabka@suse.cz \
    --to=vbabka@suse.cz \
    --cc=akpm@linux-foundation.org \
    --cc=cl@linux.com \
    --cc=iamjoonsoo.kim@lge.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@suse.de \
    --cc=mina86@mina86.com \
    --cc=minchan@kernel.org \
    --cc=n-horiguchi@ah.jp.nec.com \
    --cc=riel@redhat.com \
    --cc=rientjes@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Be sure your reply has a Subject: header at the top and a blank line before the message body.

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.