From: Suren Baghdasaryan <surenb@google.com>
To: akpm@linux-foundation.org
Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com,
josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com,
laurent.dufour@fr.ibm.com, michel@lespinasse.org,
liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz,
minchan@google.com, dave@stgolabs.net,
punit.agrawal@bytedance.com, lstoakes@gmail.com,
hdanton@sina.com, apopple@nvidia.com, peterx@redhat.com,
ying.huang@intel.com, david@redhat.com, yuzhao@google.com,
dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk,
brauner@kernel.org, pasha.tatashin@soleen.com, surenb@google.com,
linux-mm@kvack.org, linux-fsdevel@vger.kernel.org,
linux-kernel@vger.kernel.org, kernel-team@android.com
Subject: [PATCH v2 2/6] mm: handle swap page faults under VMA lock if page is uncontended
Date: Thu, 8 Jun 2023 17:51:54 -0700 [thread overview]
Message-ID: <20230609005158.2421285-3-surenb@google.com> (raw)
In-Reply-To: <20230609005158.2421285-1-surenb@google.com>
When page fault is handled under VMA lock protection, all swap page
faults are retried with mmap_lock because folio_lock_or_retry
implementation has to drop and reacquire mmap_lock if folio could
not be immediately locked.
Instead of retrying all swapped page faults, retry only when folio
locking fails.
Note that the only time do_swap_page calls synchronous swap_readpage
is when SWP_SYNCHRONOUS_IO is set, which is only set for
QUEUE_FLAG_SYNCHRONOUS devices: brd, zram and nvdimms (both btt and
pmem). Therefore we don't sleep in this path, and there's no need to
drop the mmap or per-vma lock.
Drivers implementing ops->migrate_to_ram might still rely on mmap_lock,
therefore fall back to mmap_lock in this case.
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
---
mm/filemap.c | 6 ++++++
mm/memory.c | 14 +++++++++-----
2 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index b4c9bd368b7e..7cb0a3776a07 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1706,6 +1706,8 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait)
* mmap_lock has been released (mmap_read_unlock(), unless flags had both
* FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in
* which case mmap_lock is still held.
+ * If flags had FAULT_FLAG_VMA_LOCK set, meaning the operation is performed
+ * with VMA lock only, the VMA lock is still held.
*
* If neither ALLOW_RETRY nor KILLABLE are set, will always return true
* with the folio locked and the mmap_lock unperturbed.
@@ -1713,6 +1715,10 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait)
bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm,
unsigned int flags)
{
+ /* Can't do this if not holding mmap_lock */
+ if (flags & FAULT_FLAG_VMA_LOCK)
+ return false;
+
if (fault_flag_allow_retry_first(flags)) {
/*
* CAUTION! In this case, mmap_lock is not released
diff --git a/mm/memory.c b/mm/memory.c
index f69fbc251198..41f45819a923 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3711,11 +3711,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
if (!pte_unmap_same(vmf))
goto out;
- if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
- ret = VM_FAULT_RETRY;
- goto out;
- }
-
entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
@@ -3725,6 +3720,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
vmf->page = pfn_swap_entry_to_page(entry);
ret = remove_device_exclusive_entry(vmf);
} else if (is_device_private_entry(entry)) {
+ if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
+ /*
+ * migrate_to_ram is not yet ready to operate
+ * under VMA lock.
+ */
+ ret |= VM_FAULT_RETRY;
+ goto out;
+ }
+
vmf->page = pfn_swap_entry_to_page(entry);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
--
2.41.0.162.gfafddb0af9-goog
next prev parent reply other threads:[~2023-06-09 0:52 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-09 0:51 [PATCH v2 0/6] Per-vma lock support for swap and userfaults Suren Baghdasaryan
2023-06-09 0:51 ` [PATCH v2 1/6] swap: remove remnants of polling from read_swap_cache_async Suren Baghdasaryan
2023-06-09 1:57 ` Huang, Ying
2023-06-09 3:13 ` Ming Lei
2023-06-09 18:50 ` Suren Baghdasaryan
2023-06-12 4:53 ` Christoph Hellwig
2023-06-09 0:51 ` Suren Baghdasaryan [this message]
2023-06-09 20:25 ` [PATCH v2 2/6] mm: handle swap page faults under VMA lock if page is uncontended Peter Xu
2023-06-09 20:35 ` Matthew Wilcox
2023-06-09 20:45 ` Peter Xu
2023-06-09 22:34 ` Suren Baghdasaryan
2023-06-12 13:59 ` Peter Xu
2023-06-12 16:09 ` Suren Baghdasaryan
2023-06-09 0:51 ` [PATCH v2 3/6] mm: add missing VM_FAULT_RESULT_TRACE name for VM_FAULT_COMPLETED Suren Baghdasaryan
2023-06-09 20:34 ` Peter Xu
2023-06-09 0:51 ` [PATCH v2 4/6] mm: drop VMA lock before waiting for migration Suren Baghdasaryan
2023-06-09 20:42 ` Peter Xu
2023-06-09 22:30 ` Suren Baghdasaryan
2023-06-10 1:29 ` Suren Baghdasaryan
2023-06-12 13:36 ` Peter Xu
2023-06-12 16:07 ` Suren Baghdasaryan
2023-06-12 18:34 ` Peter Xu
2023-06-12 18:44 ` Suren Baghdasaryan
2023-06-12 18:57 ` Peter Xu
2023-06-12 13:28 ` Peter Xu
2023-06-12 15:41 ` Suren Baghdasaryan
2023-06-09 0:51 ` [PATCH v2 5/6] mm: implement folio wait under VMA lock Suren Baghdasaryan
2023-06-09 15:03 ` Matthew Wilcox
2023-06-09 18:49 ` Suren Baghdasaryan
2023-06-09 18:55 ` Suren Baghdasaryan
2023-06-09 19:36 ` Matthew Wilcox
2023-06-09 22:49 ` Suren Baghdasaryan
2023-06-09 20:54 ` Peter Xu
2023-06-09 22:48 ` Suren Baghdasaryan
2023-06-12 13:56 ` Peter Xu
2023-06-09 0:51 ` [PATCH v2 6/6] mm: handle userfaults " Suren Baghdasaryan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230609005158.2421285-3-surenb@google.com \
--to=surenb@google.com \
--cc=akpm@linux-foundation.org \
--cc=apopple@nvidia.com \
--cc=brauner@kernel.org \
--cc=dave@stgolabs.net \
--cc=david@redhat.com \
--cc=dhowells@redhat.com \
--cc=hannes@cmpxchg.org \
--cc=hdanton@sina.com \
--cc=hughd@google.com \
--cc=jack@suse.cz \
--cc=jglisse@google.com \
--cc=josef@toxicpanda.com \
--cc=kernel-team@android.com \
--cc=laurent.dufour@fr.ibm.com \
--cc=ldufour@linux.ibm.com \
--cc=liam.howlett@oracle.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lstoakes@gmail.com \
--cc=mhocko@suse.com \
--cc=michel@lespinasse.org \
--cc=minchan@google.com \
--cc=pasha.tatashin@soleen.com \
--cc=peterx@redhat.com \
--cc=punit.agrawal@bytedance.com \
--cc=vbabka@suse.cz \
--cc=viro@zeniv.linux.org.uk \
--cc=willy@infradead.org \
--cc=ying.huang@intel.com \
--cc=yuzhao@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).