From: Kefeng Wang <wangkefeng.wang@huawei.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>,
<linux-mm@kvack.org>, <linux-fsdevel@vger.kernel.org>,
Kefeng Wang <wangkefeng.wang@huawei.com>
Subject: [PATCH rfc 4/4] mm: filemap: try to batch lruvec stat updating
Date: Mon, 29 Apr 2024 15:24:17 +0800 [thread overview]
Message-ID: <20240429072417.2146732-5-wangkefeng.wang@huawei.com> (raw)
In-Reply-To: <20240429072417.2146732-1-wangkefeng.wang@huawei.com>
The filemap_map_pages() tries to map few pages(eg, 16 pages), but the
lruvec stat updating is called on each mapping, since the updating is
time-consuming, especially with memcg, so try to batch it when the memcg
and pgdat are same during the mapping, if luckily, we could save most of
time of lruvec stat updating, the lat_pagefault shows 3~4% improvement.
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
mm/filemap.c | 33 ++++++++++++++++++++++++++++++---
1 file changed, 30 insertions(+), 3 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index 3966b6616d02..b27281707098 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3615,6 +3615,20 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,
return ret;
}
+static void filemap_lruvec_stat_update(struct mem_cgroup *memcg,
+ pg_data_t *pgdat, int nr)
+{
+ struct lruvec *lruvec;
+
+ if (!memcg) {
+ __mod_node_page_state(pgdat, NR_FILE_MAPPED, nr);
+ return;
+ }
+
+ lruvec = mem_cgroup_lruvec(memcg, pgdat);
+ __mod_lruvec_state(lruvec, NR_FILE_MAPPED, nr);
+}
+
vm_fault_t filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
@@ -3628,6 +3642,9 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
vm_fault_t ret = 0;
unsigned long rss = 0;
unsigned int nr_pages = 0, mmap_miss = 0, mmap_miss_saved, folio_type;
+ struct mem_cgroup *memcg, *memcg_cur;
+ pg_data_t *pgdat, *pgdat_cur;
+ int nr_mapped = 0;
rcu_read_lock();
folio = next_uptodate_folio(&xas, mapping, end_pgoff);
@@ -3648,9 +3665,20 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
}
folio_type = mm_counter_file(folio);
+ memcg = folio_memcg(folio);
+ pgdat = folio_pgdat(folio);
do {
unsigned long end;
- int nr_mapped = 0;
+
+ memcg_cur = folio_memcg(folio);
+ pgdat_cur = folio_pgdat(folio);
+
+ if (unlikely(memcg != memcg_cur || pgdat != pgdat_cur)) {
+ filemap_lruvec_stat_update(memcg, pgdat, nr_mapped);
+ nr_mapped = 0;
+ memcg = memcg_cur;
+ pgdat = pgdat_cur;
+ }
addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
vmf->pte += xas.xa_index - last_pgoff;
@@ -3668,11 +3696,10 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
nr_pages, &rss, &nr_mapped,
&mmap_miss);
- __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr_mapped);
-
folio_unlock(folio);
folio_put(folio);
} while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL);
+ filemap_lruvec_stat_update(memcg, pgdat, nr_mapped);
add_mm_counter(vma->vm_mm, folio_type, rss);
pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
--
2.27.0
next prev parent reply other threads:[~2024-04-29 7:24 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-29 7:24 [PATCH rfc 0/4] mm: filemap: try to batch lruvec stat updating Kefeng Wang
2024-04-29 7:24 ` [PATCH rfc 1/4] mm: memory: add prepare_range_pte_entry() Kefeng Wang
2024-04-29 7:24 ` [PATCH rfc 2/4] mm: filemap: add filemap_set_pte_range() Kefeng Wang
2024-04-29 7:24 ` [PATCH rfc 3/4] mm: filemap: move __lruvec_stat_mod_folio() out of filemap_set_pte_range() Kefeng Wang
2024-05-07 11:11 ` David Hildenbrand
2024-05-07 13:12 ` Kefeng Wang
2024-05-08 9:33 ` David Hildenbrand
2024-05-08 11:15 ` Kefeng Wang
2024-05-08 11:27 ` David Hildenbrand
2024-05-08 13:56 ` Kefeng Wang
2024-04-29 7:24 ` Kefeng Wang [this message]
2024-05-07 9:06 ` [PATCH rfc 4/4] mm: filemap: try to batch lruvec stat updating Kefeng Wang
2024-05-09 14:01 ` Johannes Weiner
2024-05-10 1:55 ` Kefeng Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240429072417.2146732-5-wangkefeng.wang@huawei.com \
--to=wangkefeng.wang@huawei.com \
--cc=akpm@linux-foundation.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).