From: Olga Kornievskaia <aglo@umich.edu>
To: Rik Theys <Rik.Theys@esat.kuleuven.be>
Cc: Linux Nfs <linux-nfs@vger.kernel.org>
Subject: Re: NFS4ERR_SEQ_MISORDERED errors and NFS client very slow
Date: Thu, 30 Apr 2026 14:28:53 -0400 [thread overview]
Message-ID: <CAN-5tyEec5qG7uC0UOHKY0WSsRRE30VtOjDoOX87z89X7UAiWw@mail.gmail.com> (raw)
In-Reply-To: <baa0afef-f004-4ee4-945c-8992c05e7d2a@esat.kuleuven.be>
On Thu, Apr 30, 2026 at 2:08 PM Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:
>
> Hi,
>
> On 4/30/26 7:29 PM, Olga Kornievskaia wrote:
> > On Thu, Apr 30, 2026 at 2:54 AM Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:
> >> Hi,
> >>
> >> We have a Rocky 8 client running Linux 7.0.2 (kernel-ml from elrepo)
> >> that is an NFS client to a RHEL10 server.
> >>
> >> Lately we've noticed that NFS performance is very poor for certain
> >> workloads (We saw the same issue on the stock EL8 kernel, 6.18.20 and
> >> now 7.0.2). For example cloning git repositories is extremely slow.
> >>
> >> Looking at the server side there don't seem to be any saturations of the
> >> disk or network subsystems.
> >>
> >> I've taken a network dump between the client and server. In that dump I
> >> see that the server frequently responds to requests from the client with
> >> NFS4ERR_SEQ_MISORDERED (10063). What could be the cause of these
> >> mismatches? Is this always a client issue, or can this be caused by the
> >> server?
> > This might have been fixed by mentioned patch below. This patch will
> > be included in RHEL10.2 release.
> >
> > If you have the ability to change the kernel on your NFS server I
> > would suggest trying some upstream version that has this patch
> > included to see if the problem goes away or wait until when RHEL10.2
> > comes out and test it.
>
> Thanks for the hint! Note that the client can still access (some) files
> on the server, but it can get really slow.
>
> If I read
> https://lore.kernel.org/linux-nfs/20251010194449.10281-1-okorniev@redhat.com/
> correctly, I would expect the server to send this response for all requests?
If I recall correctly, slotid=0 was not affected but all other slots
were (the ones that were previously used but then went idle, then the
server (accidently) resets the client's session slot table by setting
the highest_slot_used so client resets) Then at some point when client
starts using slots other then slot 0 it ends in this SEQ_MISORDERED
state. Again, I'm not saying this is your problem but I'm aware of a
problem in RHEL10 which looks like what your network trace shows. If
you are able to get a larger network trace that captures the use of
non-zero slots that ended up getting SEQ_MISORDERED we might be able
to confirm or discover something else. But I understand that capturing
such trace isn't easy.
Fix is also in CentOs kernel anything at/after kernel-6.12.0-201. In
case that helps.
> Regards,
>
> Rik
>
> >
> > commit 1cff14b7fc7f31363c39d0269563ce75c714f7ae
> > Author: NeilBrown <neil@brown.name>
> > Date: Thu Oct 16 09:49:57 2025 -0400
> >
> > nfsd: ensure SEQUENCE replay sends a valid reply.
> >
> > nfsd4_enc_sequence_replay() uses nfsd4_encode_operation() to encode a
> > new SEQUENCE reply when replaying a request from the slot cache - only
> > ops after the SEQUENCE are replayed from the cache in ->sl_data.
> >
> > However it does this in nfsd4_replay_cache_entry() which is called
> > *before* nfsd4_sequence() has filled in reply fields.
> >
> > This means that in the replayed SEQUENCE reply:
> > maxslots will be whatever the client sent
> > target_maxslots will be -1 (assuming init to zero, and
> > nfsd4_encode_sequence() subtracts 1)
> > status_flags will be zero
> >
> > The incorrect maxslots value, in particular, can cause the client to
> > think the slot table has been reduced in size so it can discard its
> > knowledge of current sequence number of the later slots, though the
> > server has not discarded those slots. When the client later wants to
> > use a later slot, it can get NFS4ERR_SEQ_MISORDERED from the server.
> >
> > This patch moves the setup of the reply into a new helper function and
> > call it *before* nfsd4_replay_cache_entry() is called. Only one of the
> > updated fields was used after this point - maxslots. So the
> > nfsd4_sequence struct has been extended to have separate maxslots for
> > the request and the response.
> >
> > Reported-by: Olga Kornievskaia <okorniev@redhat.com>
> > Closes: https://lore.kernel.org/linux-nfs/20251010194449.10281-1-okorniev@redhat.com/
> > Tested-by: Olga Kornievskaia <okorniev@redhat.com>
> > Signed-off-by: NeilBrown <neil@brown.name>
> > Reviewed-by: Jeff Layton <jlayton@kernel.org>
> > Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> >
> >> Should the client not recover?
> >>
> >> Regards,
> >>
> >> Rik
> >>
> >> --
> >> Rik Theys
> >> System Engineer
> >> KU Leuven - Dept. Elektrotechniek (ESAT)
> >> Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
> >> +32(0)16/32.11.07
> >> ----------------------------------------------------------------
> >> <<Any errors in spelling, tact or fact are transmission errors>>
> >>
> >>
> --
> Rik Theys
> System Engineer
> KU Leuven - Dept. Elektrotechniek (ESAT)
> Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
> +32(0)16/32.11.07
> ----------------------------------------------------------------
> <<Any errors in spelling, tact or fact are transmission errors>>
>
next prev parent reply other threads:[~2026-04-30 18:29 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-30 6:53 NFS4ERR_SEQ_MISORDERED errors and NFS client very slow Rik Theys
2026-04-30 13:27 ` Benjamin Coddington
2026-04-30 14:26 ` Rik Theys
2026-04-30 14:34 ` Rick Macklem
2026-04-30 14:39 ` Rick Macklem
2026-04-30 15:03 ` Rik Theys
2026-04-30 15:02 ` Rick Macklem
2026-04-30 15:27 ` Rik Theys
2026-04-30 15:35 ` Rick Macklem
2026-04-30 16:24 ` Rick Macklem
2026-04-30 17:04 ` Rik Theys
2026-04-30 17:29 ` Olga Kornievskaia
2026-04-30 18:08 ` Rik Theys
2026-04-30 18:28 ` Olga Kornievskaia [this message]
2026-05-04 7:30 ` Rik Theys
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAN-5tyEec5qG7uC0UOHKY0WSsRRE30VtOjDoOX87z89X7UAiWw@mail.gmail.com \
--to=aglo@umich.edu \
--cc=Rik.Theys@esat.kuleuven.be \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).