From: Andrew Morton <akpm@linux-foundation.org>
To: aarcange@redhat.com, adobriyan@gmail.com, airlied@linux.ie,
akpm@linux-foundation.org, daniel@ffwll.ch,
david@gibson.dropbear.id.au, galpress@amazon.com, hch@lst.de,
jack@suse.cz, jannh@google.com, jgg@ziepe.ca,
kirill@shutemov.name, ktkhai@virtuozzo.com, linmiaohe@huawei.com,
linux-graphics-maintainer@vmware.com, linux-mm@kvack.org,
mike.kravetz@oracle.com, mm-commits@vger.kernel.org,
peterx@redhat.com, rppt@linux.vnet.ibm.com, sroland@vmware.com,
torvalds@linux-foundation.org, willy@infradead.org,
wzam@amazon.com
Subject: [patch 08/29] mm: introduce page_needs_cow_for_dma() for deciding whether cow
Date: Fri, 12 Mar 2021 21:07:26 -0800 [thread overview]
Message-ID: <20210313050726.2Cr09FU7B%akpm@linux-foundation.org> (raw)
In-Reply-To: <20210312210632.9b7d62973d72a56fb13c7a03@linux-foundation.org>
From: Peter Xu <peterx@redhat.com>
Subject: mm: introduce page_needs_cow_for_dma() for deciding whether cow
We've got quite a few places (pte, pmd, pud) that explicitly checked
against whether we should break the cow right now during fork(). It's
easier to provide a helper, especially before we work the same thing on
hugetlbfs.
Since we'll reference is_cow_mapping() in mm.h, move it there too.
Actually it suites mm.h more since internal.h is mm/ only, but mm.h is
exported to the whole kernel. With that we should expect another patch to
use is_cow_mapping() whenever we can across the kernel since we do use it
quite a lot but it's always done with raw code against VM_* flags.
Link: https://lkml.kernel.org/r/20210217233547.93892-4-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: David Airlie <airlied@linux.ie>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Gal Pressman <galpress@amazon.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Kirill Shutemov <kirill@shutemov.name>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
Cc: Roland Scheidegger <sroland@vmware.com>
Cc: VMware Graphics <linux-graphics-maintainer@vmware.com>
Cc: Wei Zhang <wzam@amazon.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/mm.h | 21 +++++++++++++++++++++
mm/huge_memory.c | 8 ++------
mm/internal.h | 5 -----
mm/memory.c | 8 +-------
4 files changed, 24 insertions(+), 18 deletions(-)
--- a/include/linux/mm.h~mm-introduce-page_needs_cow_for_dma-for-deciding-whether-cow
+++ a/include/linux/mm.h
@@ -1300,6 +1300,27 @@ static inline bool page_maybe_dma_pinned
GUP_PIN_COUNTING_BIAS;
}
+static inline bool is_cow_mapping(vm_flags_t flags)
+{
+ return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
+}
+
+/*
+ * This should most likely only be called during fork() to see whether we
+ * should break the cow immediately for a page on the src mm.
+ */
+static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma,
+ struct page *page)
+{
+ if (!is_cow_mapping(vma->vm_flags))
+ return false;
+
+ if (!atomic_read(&vma->vm_mm->has_pinned))
+ return false;
+
+ return page_maybe_dma_pinned(page);
+}
+
#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
#define SECTION_IN_PAGE_FLAGS
#endif
--- a/mm/huge_memory.c~mm-introduce-page_needs_cow_for_dma-for-deciding-whether-cow
+++ a/mm/huge_memory.c
@@ -1100,9 +1100,7 @@ int copy_huge_pmd(struct mm_struct *dst_
* best effort that the pinned pages won't be replaced by another
* random page during the coming copy-on-write.
*/
- if (unlikely(is_cow_mapping(vma->vm_flags) &&
- atomic_read(&src_mm->has_pinned) &&
- page_maybe_dma_pinned(src_page))) {
+ if (unlikely(page_needs_cow_for_dma(vma, src_page))) {
pte_free(dst_mm, pgtable);
spin_unlock(src_ptl);
spin_unlock(dst_ptl);
@@ -1214,9 +1212,7 @@ int copy_huge_pud(struct mm_struct *dst_
}
/* Please refer to comments in copy_huge_pmd() */
- if (unlikely(is_cow_mapping(vma->vm_flags) &&
- atomic_read(&src_mm->has_pinned) &&
- page_maybe_dma_pinned(pud_page(pud)))) {
+ if (unlikely(page_needs_cow_for_dma(vma, pud_page(pud)))) {
spin_unlock(src_ptl);
spin_unlock(dst_ptl);
__split_huge_pud(vma, src_pud, addr);
--- a/mm/internal.h~mm-introduce-page_needs_cow_for_dma-for-deciding-whether-cow
+++ a/mm/internal.h
@@ -296,11 +296,6 @@ static inline unsigned int buddy_order(s
*/
#define buddy_order_unsafe(page) READ_ONCE(page_private(page))
-static inline bool is_cow_mapping(vm_flags_t flags)
-{
- return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
-}
-
/*
* These three helpers classifies VMAs for virtual memory accounting.
*/
--- a/mm/memory.c~mm-introduce-page_needs_cow_for_dma-for-deciding-whether-cow
+++ a/mm/memory.c
@@ -809,12 +809,8 @@ copy_present_page(struct vm_area_struct
pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss,
struct page **prealloc, pte_t pte, struct page *page)
{
- struct mm_struct *src_mm = src_vma->vm_mm;
struct page *new_page;
- if (!is_cow_mapping(src_vma->vm_flags))
- return 1;
-
/*
* What we want to do is to check whether this page may
* have been pinned by the parent process. If so,
@@ -828,9 +824,7 @@ copy_present_page(struct vm_area_struct
* the page count. That might give false positives for
* for pinning, but it will work correctly.
*/
- if (likely(!atomic_read(&src_mm->has_pinned)))
- return 1;
- if (likely(!page_maybe_dma_pinned(page)))
+ if (likely(!page_needs_cow_for_dma(src_vma, page)))
return 1;
new_page = *prealloc;
_
next prev parent reply other threads:[~2021-03-13 5:07 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-13 5:06 incoming Andrew Morton
2021-03-13 5:07 ` [patch 01/29] memblock: fix section mismatch warning Andrew Morton
2021-03-13 5:07 ` [patch 02/29] stop_machine: mark helpers __always_inline Andrew Morton
2021-03-13 5:07 ` [patch 03/29] init/Kconfig: make COMPILE_TEST depend on HAS_IOMEM Andrew Morton
2021-03-13 5:07 ` [patch 04/29] mm/page_alloc.c: refactor initialization of struct page for holes in memory layout Andrew Morton
2021-03-13 5:07 ` [patch 05/29] mm/fork: clear PASID for new mm Andrew Morton
2021-03-13 5:07 ` [patch 06/29] hugetlb: dedup the code to add a new file_region Andrew Morton
2021-03-13 5:07 ` [patch 07/29] hugetlb: break earlier in add_reservation_in_range() when we can Andrew Morton
2021-03-13 5:07 ` Andrew Morton [this message]
2021-03-13 5:07 ` [patch 09/29] mm: use is_cow_mapping() across tree where proper Andrew Morton
2021-03-13 5:07 ` [patch 10/29] hugetlb: do early cow when page pinned on src mm Andrew Morton
2021-03-13 5:07 ` [patch 11/29] mm/highmem.c: fix zero_user_segments() with start > end Andrew Morton
2021-03-13 5:07 ` [patch 12/29] binfmt_misc: fix possible deadlock in bm_register_write Andrew Morton
2021-03-13 5:07 ` [patch 13/29] MAINTAINERS: exclude uapi directories in API/ABI section Andrew Morton
2021-03-13 5:07 ` [patch 14/29] linux/compiler-clang.h: define HAVE_BUILTIN_BSWAP* Andrew Morton
2021-03-13 5:07 ` [patch 15/29] kfence: fix printk format for ptrdiff_t Andrew Morton
2021-03-13 5:07 ` [patch 16/29] kfence, slab: fix cache_alloc_debugcheck_after() for bulk allocations Andrew Morton
2021-03-13 5:08 ` [patch 17/29] kfence: fix reports if constant function prefixes exist Andrew Morton
2021-03-13 5:08 ` [patch 18/29] include/linux/sched/mm.h: use rcu_dereference in in_vfork() Andrew Morton
2021-03-13 5:08 ` [patch 19/29] mm/madvise: replace ptrace attach requirement for process_madvise Andrew Morton
2021-03-13 5:08 ` [patch 20/29] kasan, mm: fix crash with HW_TAGS and DEBUG_PAGEALLOC Andrew Morton
2021-03-13 5:08 ` [patch 21/29] kasan: fix KASAN_STACK dependency for HW_TAGS Andrew Morton
2021-03-13 5:08 ` [patch 22/29] mm/userfaultfd: fix memory corruption due to writeprotect Andrew Morton
2021-03-13 5:08 ` [patch 23/29] mm, hwpoison: do not lock page again when me_huge_page() successfully recovers Andrew Morton
2021-03-13 19:23 ` Linus Torvalds
2021-03-14 6:36 ` HORIGUCHI NAOYA(堀口 直也)
2021-03-13 5:08 ` [patch 24/29] ia64: fix ia64_syscall_get_set_arguments() for break-based syscalls Andrew Morton
2021-03-13 5:08 ` [patch 25/29] ia64: fix ptrace(PTRACE_SYSCALL_INFO_EXIT) sign Andrew Morton
2021-03-13 5:08 ` [patch 26/29] mm/memcg: rename mem_cgroup_split_huge_fixup to split_page_memcg and add nr_pages argument Andrew Morton
2021-03-13 5:08 ` [patch 27/29] mm/memcg: set memcg when splitting page Andrew Morton
2021-03-13 5:08 ` [patch 28/29] zram: fix return value on writeback_store Andrew Morton
2021-03-13 5:08 ` [patch 29/29] zram: fix broken page writeback Andrew Morton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210313050726.2Cr09FU7B%akpm@linux-foundation.org \
--to=akpm@linux-foundation.org \
--cc=aarcange@redhat.com \
--cc=adobriyan@gmail.com \
--cc=airlied@linux.ie \
--cc=daniel@ffwll.ch \
--cc=david@gibson.dropbear.id.au \
--cc=galpress@amazon.com \
--cc=hch@lst.de \
--cc=jack@suse.cz \
--cc=jannh@google.com \
--cc=jgg@ziepe.ca \
--cc=kirill@shutemov.name \
--cc=ktkhai@virtuozzo.com \
--cc=linmiaohe@huawei.com \
--cc=linux-graphics-maintainer@vmware.com \
--cc=linux-mm@kvack.org \
--cc=mike.kravetz@oracle.com \
--cc=mm-commits@vger.kernel.org \
--cc=peterx@redhat.com \
--cc=rppt@linux.vnet.ibm.com \
--cc=sroland@vmware.com \
--cc=torvalds@linux-foundation.org \
--cc=willy@infradead.org \
--cc=wzam@amazon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.