On Fri, 2026-03-27 at 12:57 -0400, Mike Snitzer wrote: > On Fri, Mar 27, 2026 at 09:19:07AM -0400, Chuck Lever wrote: > > On 3/27/26 7:32 AM, Jeff Layton wrote: > > > On Thu, 2026-03-26 at 16:48 -0400, Mike Snitzer wrote: > > > > > > > Your bandwidth for 1MB sequential IO of 793 MB/s for O_DIRECT and > > > > 4,952 MB/s for buffered and dontcache is considerably less than the 72 > > > > GB/s offered in Jon's testbed. Your testing isn't exposing the > > > > bottlenecks (contention) of the MM subsystem for buffered IO... not > > > > yet put my finger on _why_ that is. > > > > > > That may very well be, but not everyone has a box as large as the one > > > you and Jon were working with. > > > > Right, and this is kind of a blocker for us. Practically speaking, Jon's > > results are not reproducible. > > > > It would be immensely helpful if the MM-tipover behavior could be > > reproduced on smaller systems. Reduced physical memory size, lower > > network and storage speed, and so on, so that Jeff and I can study the > > issue on our own systems. > > Hammerspace still has the same performance lab setup, so we can > certainly reproduce if needed (and try to scale it down, etc) but > unfortunately its currently tied up with other important work. Jon > Flynn's testing was "only" with 2 systems (the NFS server has 16 fast > NVMe, client system connecting over 400 GbE with RDMA). But I'll see > about shaking a couple systems loose... > > Might be useful for us to document the setup of the NFS server side > and give pointers for how to mount from the client system and then run > fio commandline. > > I think Jens has a couple beefy servers with lots of fast NVMe. Maybe > he'd be open to testing NFS server scalability when he isn't touring > around the country watching his kid win motorsports events.. JENS!? ;) > > I'm not sure we really even need NFS in the mix to show this. Here's the results from testing them both on local XFS (same host in this case, just no NFS in the mix): https://text.is/VN0K3 tl;dr: DONTCACHE_LAZY seems to be better than DONTCACHE across the board, and it even seems to slightly edge out O_DIRECT for writes. YMMV on your hw, of course. The attached patch is what I'm planning to propose, if you want to give it a spin. I'm testing one more optimization now, to see if I can get the flush contention down even more. Personally, I wonder if we ought to consider a proportional flush like this in the normal buffered write codepath too. I've long thought that we wait too long to kick off writeback: the flusher threads have a 5s duty cycle, and all of our dirty writeback percentages were last set when RAM sizes were much smaller. Doing something like this in the normal buffered case might allow us to get ahead of heavy writers easier. -- Jeff Layton