From: Olga Kornievskaia <aglo@umich.edu>
To: Rik Theys <Rik.Theys@esat.kuleuven.be>
Cc: Linux Nfs <linux-nfs@vger.kernel.org>
Subject: Re: NFS4ERR_SEQ_MISORDERED errors and NFS client very slow
Date: Thu, 30 Apr 2026 13:29:56 -0400 [thread overview]
Message-ID: <CAN-5tyGhC8q=iB_H6JaFZpwpWAqEz5NObVrzZ8m=3OzgLgJnpw@mail.gmail.com> (raw)
In-Reply-To: <2cb85a89-f896-4504-b1cf-e4494d344ffe@esat.kuleuven.be>
On Thu, Apr 30, 2026 at 2:54 AM Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:
>
> Hi,
>
> We have a Rocky 8 client running Linux 7.0.2 (kernel-ml from elrepo)
> that is an NFS client to a RHEL10 server.
>
> Lately we've noticed that NFS performance is very poor for certain
> workloads (We saw the same issue on the stock EL8 kernel, 6.18.20 and
> now 7.0.2). For example cloning git repositories is extremely slow.
>
> Looking at the server side there don't seem to be any saturations of the
> disk or network subsystems.
>
> I've taken a network dump between the client and server. In that dump I
> see that the server frequently responds to requests from the client with
> NFS4ERR_SEQ_MISORDERED (10063). What could be the cause of these
> mismatches? Is this always a client issue, or can this be caused by the
> server?
This might have been fixed by mentioned patch below. This patch will
be included in RHEL10.2 release.
If you have the ability to change the kernel on your NFS server I
would suggest trying some upstream version that has this patch
included to see if the problem goes away or wait until when RHEL10.2
comes out and test it.
commit 1cff14b7fc7f31363c39d0269563ce75c714f7ae
Author: NeilBrown <neil@brown.name>
Date: Thu Oct 16 09:49:57 2025 -0400
nfsd: ensure SEQUENCE replay sends a valid reply.
nfsd4_enc_sequence_replay() uses nfsd4_encode_operation() to encode a
new SEQUENCE reply when replaying a request from the slot cache - only
ops after the SEQUENCE are replayed from the cache in ->sl_data.
However it does this in nfsd4_replay_cache_entry() which is called
*before* nfsd4_sequence() has filled in reply fields.
This means that in the replayed SEQUENCE reply:
maxslots will be whatever the client sent
target_maxslots will be -1 (assuming init to zero, and
nfsd4_encode_sequence() subtracts 1)
status_flags will be zero
The incorrect maxslots value, in particular, can cause the client to
think the slot table has been reduced in size so it can discard its
knowledge of current sequence number of the later slots, though the
server has not discarded those slots. When the client later wants to
use a later slot, it can get NFS4ERR_SEQ_MISORDERED from the server.
This patch moves the setup of the reply into a new helper function and
call it *before* nfsd4_replay_cache_entry() is called. Only one of the
updated fields was used after this point - maxslots. So the
nfsd4_sequence struct has been extended to have separate maxslots for
the request and the response.
Reported-by: Olga Kornievskaia <okorniev@redhat.com>
Closes: https://lore.kernel.org/linux-nfs/20251010194449.10281-1-okorniev@redhat.com/
Tested-by: Olga Kornievskaia <okorniev@redhat.com>
Signed-off-by: NeilBrown <neil@brown.name>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>
> Should the client not recover?
>
> Regards,
>
> Rik
>
> --
> Rik Theys
> System Engineer
> KU Leuven - Dept. Elektrotechniek (ESAT)
> Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
> +32(0)16/32.11.07
> ----------------------------------------------------------------
> <<Any errors in spelling, tact or fact are transmission errors>>
>
>
next prev parent reply other threads:[~2026-04-30 17:30 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-30 6:53 NFS4ERR_SEQ_MISORDERED errors and NFS client very slow Rik Theys
2026-04-30 13:27 ` Benjamin Coddington
2026-04-30 14:26 ` Rik Theys
2026-04-30 14:34 ` Rick Macklem
2026-04-30 14:39 ` Rick Macklem
2026-04-30 15:03 ` Rik Theys
2026-04-30 15:02 ` Rick Macklem
2026-04-30 15:27 ` Rik Theys
2026-04-30 15:35 ` Rick Macklem
2026-04-30 16:24 ` Rick Macklem
2026-04-30 17:04 ` Rik Theys
2026-04-30 17:29 ` Olga Kornievskaia [this message]
2026-04-30 18:08 ` Rik Theys
2026-04-30 18:28 ` Olga Kornievskaia
2026-05-04 7:30 ` Rik Theys
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAN-5tyGhC8q=iB_H6JaFZpwpWAqEz5NObVrzZ8m=3OzgLgJnpw@mail.gmail.com' \
--to=aglo@umich.edu \
--cc=Rik.Theys@esat.kuleuven.be \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).