From: "Chuck Lever" <cel@kernel.org>
To: "Mike Snitzer" <snitzer@kernel.org>
Cc: NeilBrown <neilb@ownmail.net>, "Jeff Layton" <jlayton@kernel.org>,
"Olga Kornievskaia" <okorniev@redhat.com>,
"Dai Ngo" <dai.ngo@oracle.com>, "Tom Talpey" <tom@talpey.com>,
"Daire Byrne" <daire@dneg.com>,
linux-nfs@vger.kernel.org, "Chuck Lever" <chuck.lever@oracle.com>
Subject: Re: [RFC PATCH 0/7] sunrpc: Reduce lock contention for NFSD TCP sockets
Date: Mon, 30 Mar 2026 15:04:30 -0400 [thread overview]
Message-ID: <f3e70f87-df09-4c3d-8995-ced68a028ed8@app.fastmail.com> (raw)
In-Reply-To: <acrHmvgWjJijCo1U@kernel.org>
On Mon, Mar 30, 2026, at 2:57 PM, Mike Snitzer wrote:
> Hi Chuck,
>
> On Thu, Feb 05, 2026 at 10:57:22AM -0500, Chuck Lever wrote:
>> From: Chuck Lever <chuck.lever@oracle.com>
>>
>> High-throughput NFSD workloads exhibit significant lock contention on
>> TCP connections. Worker threads compete for the socket lock during
>> receives and serialize on xpt_mutex during sends, limiting scalability.
>>
>> This series addresses both paths:
>>
>> - Receive: A dedicated kernel thread per TCP connection owns all
>> sock_recvmsg() calls and queues complete RPC messages for workers
>> via lock-free llist. This eliminates socket lock contention among
>> workers.
>>
>> - Transmit: Flat combining allows one thread to send on behalf of
>> multiple waiters. Threads enqueue requests; the mutex holder
>> ("combiner") processes the batch, amortizing lock acquisition and
>> enabling TCP segment coalescing via MSG_MORE.
>>
>> Supporting changes include a workqueue affinity fix for single-LLC
>> systems, a page recycling pool for receive buffers, and explicit TCP
>> buffer sizing for high bandwidth-delay product networks.
>>
>> Base commit: v6.19-rc8
>>
>> ---
>>
>> Chuck Lever (7):
>> workqueue: Automatic affinity scope fallback for single-pod topologies
>> sunrpc: split svc_data_ready into protocol-specific callbacks
>> sunrpc: add per-transport page recycling pool
>> sunrpc: add dedicated TCP receiver thread
>> sunrpc: implement flat combining for TCP socket sends
>> sunrpc: unify fore and backchannel server TCP send paths
>> SUNRPC: Set explicit TCP socket buffer sizes for NFSD
>
> Curious to know your thinking on this patchset? Can you post v2 given
> your note about having adapted based on Tejun's feedback here?
>
> https://lore.kernel.org/linux-nfs/af3a034c-4829-469d-b55d-9414409ee425@app.fastmail.com/
The workqueue-related patch is not important for the server,
it turns out, but I've adapted it for use on the client.
Regarding the series you listed above:
One reviewer balked at the idea of starting a kthread per
connection on the server. I've reworked the receive side
to use ->read_sock instead of ->recvmsg.
The flat combining changes on the send side are still good.
One further optimization was to replace xpt_mutex with an
llist queuing mechanism.
A number of other improvements are in the pipeline ahead
of or behind these two.
--
Chuck Lever
prev parent reply other threads:[~2026-03-30 19:04 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-02-05 15:57 [RFC PATCH 0/7] sunrpc: Reduce lock contention for NFSD TCP sockets Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 1/7] workqueue: Automatic affinity scope fallback for single-pod topologies Chuck Lever
2026-02-06 14:57 ` Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 2/7] sunrpc: split svc_data_ready into protocol-specific callbacks Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 3/7] sunrpc: add per-transport page recycling pool Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 4/7] sunrpc: add dedicated TCP receiver thread Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 5/7] sunrpc: implement flat combining for TCP socket sends Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 6/7] sunrpc: unify fore and backchannel server TCP send paths Chuck Lever
2026-02-05 15:57 ` [RFC PATCH 7/7] SUNRPC: Set explicit TCP socket buffer sizes for NFSD Chuck Lever
2026-03-30 18:57 ` [RFC PATCH 0/7] sunrpc: Reduce lock contention for NFSD TCP sockets Mike Snitzer
2026-03-30 19:04 ` Chuck Lever [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f3e70f87-df09-4c3d-8995-ced68a028ed8@app.fastmail.com \
--to=cel@kernel.org \
--cc=chuck.lever@oracle.com \
--cc=dai.ngo@oracle.com \
--cc=daire@dneg.com \
--cc=jlayton@kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=neilb@ownmail.net \
--cc=okorniev@redhat.com \
--cc=snitzer@kernel.org \
--cc=tom@talpey.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).