From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 308DBC4345F for ; Fri, 26 Apr 2024 17:15:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7932A6B0085; Fri, 26 Apr 2024 13:15:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 742E06B0087; Fri, 26 Apr 2024 13:15:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60AE36B0088; Fri, 26 Apr 2024 13:15:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 414F66B0085 for ; Fri, 26 Apr 2024 13:15:15 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B7F88161461 for ; Fri, 26 Apr 2024 17:15:14 +0000 (UTC) X-FDA: 82052333748.30.A08E026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id B77B740017 for ; Fri, 26 Apr 2024 17:15:11 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dfqvuQcE; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714151712; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=m4G8nHzQIXCHO1y8p4LPoi+zqr2MAifzLs+fYjo9zpE=; b=7pCUkALED3qhzH8HIYssdKBYfsBXurDkEy1jXyZucSIbRQmO3gFwUKyecE4O20jKhpa2oq NGyA91+jMvb4mO3jj82V26lv1a2cs6OSwYWl/VwAuZKbnF/uc8S2kXyjW88wuo9F34geez JX+YmEkDYOftbwdQkgBsA2yEcvytpXk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714151712; a=rsa-sha256; cv=none; b=uWQoLITbZMgzRFqZcDj4bKV1dHWfUtDSMNE6B2IX9lgrdajNtN2ln1T6rLMUbz3IQwIpCA dRfnOSIXrbX4LHZTXTJig3JhukuhcA/fEPHpxwuGDE/BZL73ok8V4gNIkBelfsybII6YcU BRjbIoOeWzRTSUP5+rq8h11WU+fq8eM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dfqvuQcE; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:Message-ID: Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:In-Reply-To:References; bh=m4G8nHzQIXCHO1y8p4LPoi+zqr2MAifzLs+fYjo9zpE=; b=dfqvuQcEnWC+1Ka5P7W34aMx1I SgfipMEBewsRWdhlWJ/Uu0YeHnwY+gt+obbQXckdtKxh2ex8+UyUgCNS1nnIei7KNFwZeUmZQ96nt f51+pwdKwN1wk9UQd6Z95sty6wW+xr7Yq4VtFpuwXaiBCfSF2L7VGHU6pSJHudcgMUuLvuTog34nG tL9WE/U0odLpIThOIY+7lJ7kUUNKj8s+UlSPd1StTMgcxe/BVtFSpXrfYid8dFb1439HQ+NABernk 0ly6VPW2uQJfhO0MC/3oIqOfWLivf1GyfyFfczD76D1xS9HeR8zFm90tyOz7gw09J0RGDVCBppp7K jSlJ2h5Q==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0PAB-00000005ezT-1Zdf; Fri, 26 Apr 2024 17:15:07 +0000 Date: Fri, 26 Apr 2024 18:15:07 +0100 From: Matthew Wilcox To: linux-mm@kvack.org, netdev@vger.kernel.org Cc: Suren Baghdasaryan , "Liam R. Howlett" , Arjun Roy Subject: [RFC] Make find_tcp_vma() more efficient Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspamd-Queue-Id: B77B740017 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 5x9ifya6z8be1towzkh8szi4hs8jbks8 X-HE-Tag: 1714151711-639721 X-HE-Meta: U2FsdGVkX197t4089mIVL41MYRXOGBfNOVRUunQRb7f7aH9lg9H0hS03+0LrnQNLCbYTmh9HAIugC51qM71Lx1OquTspEaAUTS1nViILmUDu4nW4g1sjkmZFoOOQR6dFjiqVh3tUQtqzjplZ2kR9q3SsK6s5WZbbrrJBMlcUoCDPJWCURcJho6y2nx6NDB46BTfdGY7xDZi8EMAoKawcsdqt6/U+6g5Z5q+T9V7uqD43l6QZRoSDnSR+LdYZXfXKbLi/BJOiGMnIgHqmtrUlY9BGP9XSA8V0fx2dtHBV9nroNlIFE7WwA9IIe4LpHnUDb3UuALn15DaokP/BhEuoXYQN6ip5Huw3+N+gchNOGkWKFXdBNP0oBpFKTp2Dn4OcTjNxoPEpy4ZsOw7RpkxErJ1U+8BkWXW6f/n1ce0GZFxBR8/As+XCLkMJJy6Dh/OdVrEZW2WdO1hrew909eQdS3k2k4ehZJul+TexPug85f4fG0QGJmDoTu8ON0TwC5AsPrQz07+Dq9j+lF55UN7a6ReXB4Eazfw5PkEiU6LBWUdnJdEbibw7oiIpzTFvKgGyVzlZHNCdqng6Gx25872PX0BFCx06VJoCIIoW252r80CxYDHdH+BMxMbSRM/d4i3PEwW4yLUtAj1FeVNUcKF2mdUttKG3eu9bVLAEpOKXMEBCdXfc4zNc7YPU4+Z/wMvW/lMITrtbHZ5oyPnJSDxSjFmOzgrTtUxfSHNTPJ2H2gdHyNohAHV7awqm9ZOMJF3xMAB58yhR7zkGljuyhxCiWvMfz0wtq8yof63i68jaOkEZrQtytKBGvYAXqnJ9k/TqgQZdL1syWsIVUAhhglBXS0Q8vU3/iTkdjs8FG1NHYj0qblRn3ke3m3APEqL/QG+RVU+oYhMTLHrrv1Xq1DSZiHcJ6EEBzVhSTBUNA/IN6GQ8WGBgyXqa9gVDBcHXfZsGeoX7ajNYonoZdZnVFdi G051tNE9 kE2JaXCTOA2NrBSCiEmCBbbuKl4+4dflf3EdQt1dt8xB9KXWCLRERkSkijzJh3EP6wsFnloc7KtW6zyzsiUeHPf7WMmp/d7k+51JL6UZwS1NaUDyMbBoJGOwI76s2lwzkcXxYhq1OKkpnd1A4yvXTdvRHNOejOfvvrT/SrGoDn4p142moa9G2bvuxUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Liam asked me if we could do away with the "bool *mmap_locked" parameter, and the problem is that some architctures don't support CONFIG_PER_VMA_LOCK yet. But we can abstract it ... something like this maybe? (not particularly proposing this for inclusion; just wrote it and want to get it out of my tree so I can get back to other projects. If anyone wants it, they can test it and submit it for inclusion and stick my S-o-B on it) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9849dfda44d4..570763351508 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -779,11 +779,22 @@ static inline void assert_fault_locked(struct vm_fault *vmf) struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address); +static inline void lock_vma_under_mmap_lock(struct vm_area_struct *vma) +{ + down_read(&vma->vm_lock->lock); + mmap_read_unlock(vma->vm_mm); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } -static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_end_read(struct vm_area_struct *vma) +{ + mmap_read_unlock(vma->vm_mm); +} + +static inline void lock_vma_under_mmap_lock(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index f23b97777ea5..e763916e5185 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2051,27 +2051,25 @@ static void tcp_zc_finalize_rx_tstamp(struct sock *sk, } static struct vm_area_struct *find_tcp_vma(struct mm_struct *mm, - unsigned long address, - bool *mmap_locked) + unsigned long address) { struct vm_area_struct *vma = lock_vma_under_rcu(mm, address); - if (vma) { - if (vma->vm_ops != &tcp_vm_ops) { - vma_end_read(vma); + if (!vma) { + mmap_read_lock(mm); + vma = vma_lookup(mm, address); + if (vma) { + lock_vma_under_mmap_lock(vma); + } else { + mmap_read_unlock(mm); return NULL; } - *mmap_locked = false; - return vma; } - - mmap_read_lock(mm); - vma = vma_lookup(mm, address); - if (!vma || vma->vm_ops != &tcp_vm_ops) { - mmap_read_unlock(mm); + if (vma->vm_ops != &tcp_vm_ops) { + vma_end_read(vma); return NULL; } - *mmap_locked = true; + return vma; } @@ -2092,7 +2090,6 @@ static int tcp_zerocopy_receive(struct sock *sk, u32 seq = tp->copied_seq; u32 total_bytes_to_map; int inq = tcp_inq(sk); - bool mmap_locked; int ret; zc->copybuf_len = 0; @@ -2117,7 +2114,7 @@ static int tcp_zerocopy_receive(struct sock *sk, return 0; } - vma = find_tcp_vma(current->mm, address, &mmap_locked); + vma = find_tcp_vma(current->mm, address); if (!vma) return -EINVAL; @@ -2194,10 +2191,7 @@ static int tcp_zerocopy_receive(struct sock *sk, zc, total_bytes_to_map); } out: - if (mmap_locked) - mmap_read_unlock(current->mm); - else - vma_end_read(vma); + vma_end_read(vma); /* Try to copy straggler data. */ if (!ret) copylen = tcp_zc_handle_leftover(zc, sk, skb, &seq, copybuf_len, tss);