* [PATCH V2 0/1] pmem: set QUEUE_FLAG_NOWAIT
@ 2023-07-31 22:46 Chaitanya Kulkarni
2023-07-31 22:46 ` [PATCH V2 1/1] " Chaitanya Kulkarni
0 siblings, 1 reply; 14+ messages in thread
From: Chaitanya Kulkarni @ 2023-07-31 22:46 UTC (permalink / raw
To: dan.j.williams, vishal.l.verma, dave.jiang, ira.weiny
Cc: hch, nvdimm, Chaitanya Kulkarni
Hi,
Set the QUEUE_FLAG_NOWAIT. Following are the performance numbers with
io_uring fio engine for random read, note that device has been populated
fully with randwrite workload before taking these numbers :-
linux-block (pmem-nowait-on) # grep IOPS pmem*fio | column -t
pmem-nowait-off-1.fio: read: IOPS=3683k, BW=14.0GiB/s
pmem-nowait-off-2.fio: read: IOPS=3819k, BW=14.6GiB/s
pmem-nowait-off-3.fio: read: IOPS=3999k, BW=15.3GiB/s
pmem-nowait-on-1.fio: read: IOPS=5837k, BW=22.3GiB/s
pmem-nowait-on-2.fio: read: IOPS=5936k, BW=22.6GiB/s
pmem-nowait-on-3.fio: read: IOPS=5945k, BW=22.7GiB/s
linux-block (pmem-nowait-on) # grep cpu pmem*fio | column -t
pmem-nowait-off-1.fio: cpu : usr=7.09%, sys=29.65%, ctx=198742065
pmem-nowait-off-2.fio: cpu : usr=6.89%, sys=30.56%, ctx=205817652
pmem-nowait-off-3.fio: cpu : usr=6.86%, sys=30.94%, ctx=222627094
pmem-nowait-on-1.fio: cpu : usr=10.58%, sys=88.44%, ctx=27181
pmem-nowait-on-2.fio: cpu : usr=10.50%, sys=87.75%, ctx=25746
pmem-nowait-on-3.fio: cpu : usr=10.67%, sys=88.60%, ctx=28261
linux-block (pmem-nowait-on) # grep slat pmem*fio | column -t
pmem-nowait-off-1.fio: slat (nsec): min=432, max=50847k, avg=9324.69
pmem-nowait-off-2.fio: slat (nsec): min=441, max=52557k, avg=9132.45
pmem-nowait-off-3.fio: slat (nsec): min=430, max=36113k, avg=9132.63
pmem-nowait-on-1.fio: slat (nsec): min=1393, max=68090k, avg=7615.31
pmem-nowait-on-2.fio: slat (nsec): min=1222, max=44137k, avg=7493.77
pmem-nowait-on-3.fio: slat (nsec): min=1493, max=40100k, avg=7486.36
Please let me know if further testing is needed I've ran fio verification
job in order to make verify these changes.
-ck
V2:-
Unconditionally set the QUEUE_FLAG_NOWAIT in pmem_attach_disk() along
with the other queue flags.
Chaitanya Kulkarni (1):
pmem: set QUEUE_FLAG_NOWAIT
drivers/nvdimm/pmem.c | 1 +
1 file changed, 1 insertion(+)
linux-block (pmem-nowait-on) # ./test-pmem.sh
++ unload_mod
++ rmmod nd_pmem
++ rmmod nd_btt
++ git checkout for-next
Switched to branch 'for-next'
Your branch is ahead of 'origin/for-next' by 155 commits.
(use "git push" to publish your local commits)
++ git log -1
commit e50c5e801b5a9e1797eb5a157eac1b5e50084486 (HEAD -> for-next)
Merge: e6dfe861227b e98acd815ebf
Author: Chaitanya Kulkarni <kch@nvidia.com>
Date: Mon Jul 31 14:48:39 2023 -0700
Merge branch 'for-next' of git://git.kernel.dk/linux-block into for-next
++ makej M=drivers/nvdimm
CC [M] drivers/nvdimm/pmem.o
LD [M] drivers/nvdimm/nd_pmem.o
MODPOST drivers/nvdimm/Module.symvers
LD [M] drivers/nvdimm/nd_pmem.ko
++ load_mod
++ insmod drivers/nvdimm/nd_btt.ko
++ insmod drivers/nvdimm/nd_pmem.ko
++ sleep 1
++ test_pmem nowait-off
++ sleep 1
++ fio fio/verify.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=16
fio-3.34
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=4662: Mon Jul 31 15:24:07 2023
read: IOPS=265k, BW=1036MiB/s (1087MB/s)(566MiB/546msec)
slat (nsec): min=501, max=31820, avg=2733.20, stdev=1179.52
clat (nsec): min=17022, max=96063, avg=56544.09, stdev=5848.22
lat (usec): min=20, max=101, avg=59.28, stdev= 6.06
clat percentiles (nsec):
| 1.00th=[44288], 5.00th=[46848], 10.00th=[48896], 20.00th=[51456],
| 30.00th=[53504], 40.00th=[55552], 50.00th=[56576], 60.00th=[58112],
| 70.00th=[59648], 80.00th=[61184], 90.00th=[63744], 95.00th=[66048],
| 99.00th=[72192], 99.50th=[74240], 99.90th=[80384], 99.95th=[82432],
| 99.99th=[88576]
write: IOPS=209k, BW=818MiB/s (857MB/s)(896MiB/1096msec); 0 zone resets
slat (nsec): min=1352, max=113484, avg=4293.77, stdev=1425.01
clat (usec): min=25, max=285, avg=71.81, stdev= 9.33
lat (usec): min=31, max=288, avg=76.10, stdev= 9.56
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 68],
| 30.00th=[ 70], 40.00th=[ 71], 50.00th=[ 72], 60.00th=[ 73],
| 70.00th=[ 75], 80.00th=[ 77], 90.00th=[ 80], 95.00th=[ 84],
| 99.00th=[ 102], 99.50th=[ 113], 99.90th=[ 169], 99.95th=[ 180],
| 99.99th=[ 219]
bw ( KiB/s): min=152408, max=857568, per=73.07%, avg=611669.33, stdev=398064.54, samples=3
iops : min=38102, max=214392, avg=152917.33, stdev=99516.13, samples=3
lat (usec) : 20=0.01%, 50=6.40%, 100=92.93%, 250=0.67%, 500=0.01%
cpu : usr=35.49%, sys=63.60%, ctx=2561, majf=0, minf=3973
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=144875,229376,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1036MiB/s (1087MB/s), 1036MiB/s-1036MiB/s (1087MB/s-1087MB/s), io=566MiB (593MB), run=546-546msec
WRITE: bw=818MiB/s (857MB/s), 818MiB/s-818MiB/s (857MB/s-857MB/s), io=896MiB (940MB), run=1096-1096msec
Disk stats (read/write):
pmem0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
++ fio fio/randwrite.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0
RANDWRITE: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
...
fio-3.34
Starting 48 processes
Jobs: 48 (f=48): [w(48)][100.0%][w=20.6GiB/s][w=5389k IOPS][eta 00m:00s]
RANDWRITE: (groupid=0, jobs=48): err= 0: pid=4681: Mon Jul 31 15:25:07 2023
write: IOPS=5147k, BW=19.6GiB/s (21.1GB/s)(1178GiB/60002msec); 0 zone resets
slat (nsec): min=380, max=57147k, avg=6895.36, stdev=28357.20
clat (nsec): min=130, max=57173k, avg=11233.63, stdev=44260.60
lat (nsec): min=1944, max=57174k, avg=18128.99, stdev=53168.60
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6],
| 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9],
| 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 21], 95.00th=[ 32],
| 99.00th=[ 58], 99.50th=[ 74], 99.90th=[ 135], 99.95th=[ 186],
| 99.99th=[ 742]
bw ( MiB/s): min= 5793, max=30898, per=100.00%, avg=20121.35, stdev=146.64, samples=5712
iops : min=1483165, max=7909996, avg=5151064.04, stdev=37540.43, samples=5712
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.33%, 4=2.22%, 10=71.71%, 20=15.22%, 50=9.07%
lat (usec) : 100=1.23%, 250=0.18%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=9.36%, sys=31.21%, ctx=286817971, majf=0, minf=607
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,308833813,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
WRITE: bw=19.6GiB/s (21.1GB/s), 19.6GiB/s-19.6GiB/s (21.1GB/s-21.1GB/s), io=1178GiB (1265GB), run=60002-60002msec
Disk stats (read/write):
pmem0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
++ for i in 1 2 3
++ fio fio/randread.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0 --output=pmem-nowait-off-1.fio
++ for i in 1 2 3[r(48)][100.0%][r=14.7GiB/s][r=3864k IOPS][eta 00m:00s]
++ fio fio/randread.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0 --output=pmem-nowait-off-2.fio
++ for i in 1 2 3[r(48)][100.0%][r=15.7GiB/s][r=4116k IOPS][eta 00m:00s]
++ fio fio/randread.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0 --output=pmem-nowait-off-3.fio
++ unload_mod8): [r(48)][100.0%][r=15.1GiB/s][r=3966k IOPS][eta 00m:00s]
++ rmmod nd_pmem
++ rmmod nd_btt
++ git checkout pmem-nowait-on
Switched to branch 'pmem-nowait-on'
++ git log -1
commit 24573494ec05e8d7bb7acb82e4a0e400297272aa (HEAD -> pmem-nowait-on)
Author: Chaitanya Kulkarni <kch@nvidia.com>
Date: Fri May 12 03:24:54 2023 -0700
pmem: set QUEUE_FLAG_NOWAIT
Set the QUEUE_FLAG_NOWAIT. Following are the performance numbers with
io_uring fio engine for random read, note that device has been populated
fully with randwrite workload before taking these numbers :-
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
++ rmmod nd_pmem
rmmod: ERROR: Module nd_pmem is not currently loaded
++ makej M=drivers/nvdimm
CC [M] drivers/nvdimm/pmem.o
LD [M] drivers/nvdimm/nd_pmem.o
MODPOST drivers/nvdimm/Module.symvers
LD [M] drivers/nvdimm/nd_pmem.ko
++ load_mod
++ insmod drivers/nvdimm/nd_btt.ko
++ insmod drivers/nvdimm/nd_pmem.ko
++ sleep 1
++ test_pmem nowait-on
++ sleep 1
++ fio fio/verify.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=16
fio-3.34
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=5662: Mon Jul 31 15:28:13 2023
read: IOPS=448k, BW=1750MiB/s (1835MB/s)(567MiB/324msec)
slat (nsec): min=1092, max=56347, avg=1349.16, stdev=838.70
clat (nsec): min=741, max=152570, avg=33482.76, stdev=9895.86
lat (usec): min=2, max=154, avg=34.83, stdev=10.20
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 35], 95.00th=[ 49],
| 99.00th=[ 87], 99.50th=[ 96], 99.90th=[ 113], 99.95th=[ 122],
| 99.99th=[ 139]
write: IOPS=207k, BW=810MiB/s (849MB/s)(896MiB/1106msec); 0 zone resets
slat (nsec): min=2135, max=81445, avg=4433.09, stdev=1568.20
clat (nsec): min=441, max=210380, avg=72445.60, stdev=10663.97
lat (usec): min=3, max=254, avg=76.88, stdev=11.22
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 62], 20.00th=[ 65],
| 30.00th=[ 68], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 75],
| 70.00th=[ 77], 80.00th=[ 79], 90.00th=[ 83], 95.00th=[ 86],
| 99.00th=[ 120], 99.50th=[ 135], 99.90th=[ 161], 99.95th=[ 169],
| 99.99th=[ 192]
bw ( KiB/s): min=764336, max=876176, per=98.88%, avg=820256.00, stdev=79082.82, samples=2
iops : min=191084, max=219044, avg=205064.00, stdev=19770.71, samples=2
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=36.90%, 100=62.08%
lat (usec) : 250=1.01%
cpu : usr=37.58%, sys=62.28%, ctx=3, majf=0, minf=3983
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=145180,229376,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1750MiB/s (1835MB/s), 1750MiB/s-1750MiB/s (1835MB/s-1835MB/s), io=567MiB (595MB), run=324-324msec
WRITE: bw=810MiB/s (849MB/s), 810MiB/s-810MiB/s (849MB/s-849MB/s), io=896MiB (940MB), run=1106-1106msec
Disk stats (read/write):
pmem0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
++ fio fio/randwrite.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0
RANDWRITE: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
...
fio-3.34
Starting 48 processes
Jobs: 48 (f=48): [w(48)][100.0%][w=39.7GiB/s][w=10.4M IOPS][eta 00m:00s]
RANDWRITE: (groupid=0, jobs=48): err= 0: pid=5665: Mon Jul 31 15:29:13 2023
write: IOPS=9926k, BW=37.9GiB/s (40.7GB/s)(2272GiB/60001msec); 0 zone resets
slat (nsec): min=1192, max=32084k, avg=4269.53, stdev=21566.61
clat (nsec): min=501, max=32087k, avg=5022.97, stdev=23279.45
lat (nsec): min=1854, max=32098k, avg=9292.50, stdev=31761.04
clat percentiles (nsec):
| 1.00th=[ 2544], 5.00th=[ 2992], 10.00th=[ 3280], 20.00th=[ 3600],
| 30.00th=[ 3920], 40.00th=[ 4256], 50.00th=[ 4576], 60.00th=[ 4960],
| 70.00th=[ 5408], 80.00th=[ 5984], 90.00th=[ 7008], 95.00th=[ 7968],
| 99.00th=[10944], 99.50th=[14528], 99.90th=[24192], 99.95th=[33536],
| 99.99th=[63744]
bw ( MiB/s): min=26941, max=42586, per=100.00%, avg=38794.41, stdev=73.04, samples=5712
iops : min=6897082, max=10902168, avg=9931366.71, stdev=18699.20, samples=5712
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=32.93%, 10=65.63%, 20=1.26%, 50=0.16%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=18.08%, sys=80.64%, ctx=30470, majf=0, minf=587
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,595577284,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
WRITE: bw=37.9GiB/s (40.7GB/s), 37.9GiB/s-37.9GiB/s (40.7GB/s-40.7GB/s), io=2272GiB (2439GB), run=60001-60001msec
Disk stats (read/write):
pmem0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
++ for i in 1 2 3
++ fio fio/randread.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0 --output=pmem-nowait-on-1.fio
++ for i in 1 2 3[r(48)][100.0%][r=22.7GiB/s][r=5960k IOPS][eta 00m:00s]
++ fio fio/randread.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0 --output=pmem-nowait-on-2.fio
++ for i in 1 2 3[r(48)][100.0%][r=22.7GiB/s][r=5953k IOPS][eta 00m:00s]
++ fio fio/randread.fio --ioengine=io_uring --size=896M --filename=/dev/pmem0 --output=pmem-nowait-on-3.fio
linux-block (pmem-nowait-on) # ][r=22.7GiB/s][r=5946k IOPS][eta 00m:00s]
linux-block (pmem-nowait-on) #
linux-block (pmem-nowait-on) # grep IOPS pmem*fio | column -t
pmem-nowait-off-1.fio: read: IOPS=3683k, BW=14.0GiB/s (15.1GB/s)(843GiB/60002msec)
pmem-nowait-off-2.fio: read: IOPS=3819k, BW=14.6GiB/s (15.6GB/s)(874GiB/60002msec)
pmem-nowait-off-3.fio: read: IOPS=3999k, BW=15.3GiB/s (16.4GB/s)(915GiB/60002msec)
pmem-nowait-on-1.fio: read: IOPS=5837k, BW=22.3GiB/s (23.9GB/s)(1336GiB/60002msec)
pmem-nowait-on-2.fio: read: IOPS=5936k, BW=22.6GiB/s (24.3GB/s)(1359GiB/60002msec)
pmem-nowait-on-3.fio: read: IOPS=5945k, BW=22.7GiB/s (24.3GB/s)(1361GiB/60002msec)
linux-block (pmem-nowait-on) # grep cpu pmem*fio | column -t
pmem-nowait-off-1.fio: cpu : usr=7.09%, sys=29.65%, ctx=198742065
pmem-nowait-off-2.fio: cpu : usr=6.89%, sys=30.56%, ctx=205817652
pmem-nowait-off-3.fio: cpu : usr=6.86%, sys=30.94%, ctx=222627094
pmem-nowait-on-1.fio: cpu : usr=10.58%, sys=88.44%, ctx=27181,
pmem-nowait-on-2.fio: cpu : usr=10.50%, sys=87.75%, ctx=25746,
pmem-nowait-on-3.fio: cpu : usr=10.67%, sys=88.60%, ctx=28261,
linux-block (pmem-nowait-on) # grep slat pmem*fio | column -t
pmem-nowait-off-1.fio: slat (nsec): min=432, max=50847k, avg=9324.69
pmem-nowait-off-2.fio: slat (nsec): min=441, max=52557k, avg=9132.45
pmem-nowait-off-3.fio: slat (nsec): min=430, max=36113k, avg=9132.63
pmem-nowait-on-1.fio: slat (nsec): min=1393, max=68090k, avg=7615.31
pmem-nowait-on-2.fio: slat (nsec): min=1222, max=44137k, avg=7493.77
pmem-nowait-on-3.fio: slat (nsec): min=1493, max=40100k, avg=7486.36
linux-block (pmem-nowait-on) #
--
2.40.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-07-31 22:46 [PATCH V2 0/1] pmem: set QUEUE_FLAG_NOWAIT Chaitanya Kulkarni
@ 2023-07-31 22:46 ` Chaitanya Kulkarni
2023-08-01 15:23 ` Jeff Moyer
0 siblings, 1 reply; 14+ messages in thread
From: Chaitanya Kulkarni @ 2023-07-31 22:46 UTC (permalink / raw
To: dan.j.williams, vishal.l.verma, dave.jiang, ira.weiny
Cc: hch, nvdimm, Chaitanya Kulkarni
Set the QUEUE_FLAG_NOWAIT. Following are the performance numbers with
io_uring fio engine for random read, note that device has been populated
fully with randwrite workload before taking these numbers :-
linux-block (pmem-nowait-on) # grep IOPS pmem*fio | column -t
pmem-nowait-off-1.fio: read: IOPS=3683k, BW=14.0GiB/s
pmem-nowait-off-2.fio: read: IOPS=3819k, BW=14.6GiB/s
pmem-nowait-off-3.fio: read: IOPS=3999k, BW=15.3GiB/s
pmem-nowait-on-1.fio: read: IOPS=5837k, BW=22.3GiB/s
pmem-nowait-on-2.fio: read: IOPS=5936k, BW=22.6GiB/s
pmem-nowait-on-3.fio: read: IOPS=5945k, BW=22.7GiB/s
linux-block (pmem-nowait-on) # grep cpu pmem*fio | column -t
pmem-nowait-off-1.fio: cpu : usr=7.09%, sys=29.65%, ctx=198742065
pmem-nowait-off-2.fio: cpu : usr=6.89%, sys=30.56%, ctx=205817652
pmem-nowait-off-3.fio: cpu : usr=6.86%, sys=30.94%, ctx=222627094
pmem-nowait-on-1.fio: cpu : usr=10.58%, sys=88.44%, ctx=27181
pmem-nowait-on-2.fio: cpu : usr=10.50%, sys=87.75%, ctx=25746
pmem-nowait-on-3.fio: cpu : usr=10.67%, sys=88.60%, ctx=28261
linux-block (pmem-nowait-on) # grep slat pmem*fio | column -t
pmem-nowait-off-1.fio: slat (nsec): min=432, max=50847k, avg=9324.69
pmem-nowait-off-2.fio: slat (nsec): min=441, max=52557k, avg=9132.45
pmem-nowait-off-3.fio: slat (nsec): min=430, max=36113k, avg=9132.63
pmem-nowait-on-1.fio: slat (nsec): min=1393, max=68090k, avg=7615.31
pmem-nowait-on-2.fio: slat (nsec): min=1222, max=44137k, avg=7493.77
pmem-nowait-on-3.fio: slat (nsec): min=1493, max=40100k, avg=7486.36
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvdimm/pmem.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 46e094e56159..ddd485c377eb 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -543,6 +543,7 @@ static int pmem_attach_disk(struct device *dev,
blk_queue_max_hw_sectors(q, UINT_MAX);
blk_queue_flag_set(QUEUE_FLAG_NONROT, q);
blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, q);
+ blk_queue_flag_set(QUEUE_FLAG_NOWAIT, q);
if (pmem->pfn_flags & PFN_MAP)
blk_queue_flag_set(QUEUE_FLAG_DAX, q);
--
2.40.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-07-31 22:46 ` [PATCH V2 1/1] " Chaitanya Kulkarni
@ 2023-08-01 15:23 ` Jeff Moyer
2023-08-01 15:59 ` Christoph Hellwig
0 siblings, 1 reply; 14+ messages in thread
From: Jeff Moyer @ 2023-08-01 15:23 UTC (permalink / raw
To: Chaitanya Kulkarni
Cc: dan.j.williams, vishal.l.verma, dave.jiang, ira.weiny, hch,
nvdimm
Chaitanya Kulkarni <kch@nvidia.com> writes:
> Set the QUEUE_FLAG_NOWAIT. Following are the performance numbers with
> io_uring fio engine for random read, note that device has been populated
> fully with randwrite workload before taking these numbers :-
I am slightly embarrassed to have to ask this question, but what are the
implications of setting this queue flag? Is the submit_bio routine
expected to never block? Is the I/O expected to be performed
asynchronously? If either of those is the case, then pmem does not
qualify.
-Jeff
>
> linux-block (pmem-nowait-on) # grep IOPS pmem*fio | column -t
> pmem-nowait-off-1.fio: read: IOPS=3683k, BW=14.0GiB/s
> pmem-nowait-off-2.fio: read: IOPS=3819k, BW=14.6GiB/s
> pmem-nowait-off-3.fio: read: IOPS=3999k, BW=15.3GiB/s
>
> pmem-nowait-on-1.fio: read: IOPS=5837k, BW=22.3GiB/s
> pmem-nowait-on-2.fio: read: IOPS=5936k, BW=22.6GiB/s
> pmem-nowait-on-3.fio: read: IOPS=5945k, BW=22.7GiB/s
>
> linux-block (pmem-nowait-on) # grep cpu pmem*fio | column -t
> pmem-nowait-off-1.fio: cpu : usr=7.09%, sys=29.65%, ctx=198742065
> pmem-nowait-off-2.fio: cpu : usr=6.89%, sys=30.56%, ctx=205817652
> pmem-nowait-off-3.fio: cpu : usr=6.86%, sys=30.94%, ctx=222627094
>
> pmem-nowait-on-1.fio: cpu : usr=10.58%, sys=88.44%, ctx=27181
> pmem-nowait-on-2.fio: cpu : usr=10.50%, sys=87.75%, ctx=25746
> pmem-nowait-on-3.fio: cpu : usr=10.67%, sys=88.60%, ctx=28261
>
> linux-block (pmem-nowait-on) # grep slat pmem*fio | column -t
> pmem-nowait-off-1.fio: slat (nsec): min=432, max=50847k, avg=9324.69
> pmem-nowait-off-2.fio: slat (nsec): min=441, max=52557k, avg=9132.45
> pmem-nowait-off-3.fio: slat (nsec): min=430, max=36113k, avg=9132.63
>
> pmem-nowait-on-1.fio: slat (nsec): min=1393, max=68090k, avg=7615.31
> pmem-nowait-on-2.fio: slat (nsec): min=1222, max=44137k, avg=7493.77
> pmem-nowait-on-3.fio: slat (nsec): min=1493, max=40100k, avg=7486.36
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> ---
> drivers/nvdimm/pmem.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
> index 46e094e56159..ddd485c377eb 100644
> --- a/drivers/nvdimm/pmem.c
> +++ b/drivers/nvdimm/pmem.c
> @@ -543,6 +543,7 @@ static int pmem_attach_disk(struct device *dev,
> blk_queue_max_hw_sectors(q, UINT_MAX);
> blk_queue_flag_set(QUEUE_FLAG_NONROT, q);
> blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, q);
> + blk_queue_flag_set(QUEUE_FLAG_NOWAIT, q);
> if (pmem->pfn_flags & PFN_MAP)
> blk_queue_flag_set(QUEUE_FLAG_DAX, q);
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-01 15:23 ` Jeff Moyer
@ 2023-08-01 15:59 ` Christoph Hellwig
2023-08-01 17:49 ` Jeff Moyer
0 siblings, 1 reply; 14+ messages in thread
From: Christoph Hellwig @ 2023-08-01 15:59 UTC (permalink / raw
To: Jeff Moyer
Cc: Chaitanya Kulkarni, dan.j.williams, vishal.l.verma, dave.jiang,
ira.weiny, hch, nvdimm
On Tue, Aug 01, 2023 at 11:23:36AM -0400, Jeff Moyer wrote:
> I am slightly embarrassed to have to ask this question, but what are the
> implications of setting this queue flag? Is the submit_bio routine
> expected to never block?
Yes, at least not significantly.
> Is the I/O expected to be performed
> asynchronously?
Not nessecarily if it is fast enough..
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-01 15:59 ` Christoph Hellwig
@ 2023-08-01 17:49 ` Jeff Moyer
2023-08-02 1:25 ` Chaitanya Kulkarni
0 siblings, 1 reply; 14+ messages in thread
From: Jeff Moyer @ 2023-08-01 17:49 UTC (permalink / raw
To: Christoph Hellwig
Cc: Chaitanya Kulkarni, dan.j.williams, vishal.l.verma, dave.jiang,
ira.weiny, nvdimm
Hi, Christoph,
Christoph Hellwig <hch@lst.de> writes:
> On Tue, Aug 01, 2023 at 11:23:36AM -0400, Jeff Moyer wrote:
>> I am slightly embarrassed to have to ask this question, but what are the
>> implications of setting this queue flag? Is the submit_bio routine
>> expected to never block?
>
> Yes, at least not significantly.
If there happens to be poisoned memory, the write path can make an acpi
device specific method call. That involves taking a mutex (see the call
chain starting at acpi_ex_enter_interpreter()). I'm not sure how long a
DSM takes, but I doubt it's fast.
>> Is the I/O expected to be performed
>> asynchronously?
>
> Not nessecarily if it is fast enough..
Clear as mud. :) It's a memcpy on potentially "slow" memory. So, the
amount of time spent copying depends on the speed of the cpu, the media
and the size of the I/O. I don't know off-hand what the upper bound
would be on today's pmem.
-Jeff
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-01 17:49 ` Jeff Moyer
@ 2023-08-02 1:25 ` Chaitanya Kulkarni
2023-08-02 12:30 ` Christoph Hellwig
0 siblings, 1 reply; 14+ messages in thread
From: Chaitanya Kulkarni @ 2023-08-02 1:25 UTC (permalink / raw
To: Jeff Moyer, Christoph Hellwig
Cc: Chaitanya Kulkarni, dan.j.williams@intel.com,
vishal.l.verma@intel.com, dave.jiang@intel.com,
ira.weiny@intel.com, nvdimm@lists.linux.dev
On 8/1/23 10:49, Jeff Moyer wrote:
> Hi, Christoph,
>
> Christoph Hellwig <hch@lst.de> writes:
>
>> On Tue, Aug 01, 2023 at 11:23:36AM -0400, Jeff Moyer wrote:
>>> I am slightly embarrassed to have to ask this question, but what are the
>>> implications of setting this queue flag? Is the submit_bio routine
>>> expected to never block?
>> Yes, at least not significantly.
> If there happens to be poisoned memory, the write path can make an acpi
> device specific method call. That involves taking a mutex (see the call
> chain starting at acpi_ex_enter_interpreter()). I'm not sure how long a
> DSM takes, but I doubt it's fast.
for this case I can add bio->bio_opf & REQ_NOWAIT check and return
with bio_wouldblock_error() that will take care of blocking case e.g.
something like this untested as a prep patch for write path :-
linux-block (pmem-nowait-on) # git diff
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index ddd485c377eb..eff100f74357 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -179,12 +179,16 @@ static blk_status_t pmem_do_read(struct
pmem_device *pmem,
static blk_status_t pmem_do_write(struct pmem_device *pmem,
struct page *page, unsigned int page_off,
- sector_t sector, unsigned int len)
+ sector_t sector, unsigned int len, struct bio *bio)
{
phys_addr_t pmem_off = to_offset(pmem, sector);
void *pmem_addr = pmem->virt_addr + pmem_off;
if (unlikely(is_bad_pmem(&pmem->bb, sector, len))) {
+ if (bio && bio->bi_opf & REQ_NOWAIT) {
+ bio_wouldblock_error(bio);
+ return BLK_STS_AGAIN;
+ }
blk_status_t rc = pmem_clear_poison(pmem, pmem_off, len);
if (rc != BLK_STS_OK)
@@ -217,7 +221,7 @@ static void pmem_submit_bio(struct bio *bio)
bio_for_each_segment(bvec, bio, iter) {
if (op_is_write(bio_op(bio)))
rc = pmem_do_write(pmem, bvec.bv_page,
bvec.bv_offset,
- iter.bi_sector, bvec.bv_len);
+ iter.bi_sector, bvec.bv_len, bio);
else
rc = pmem_do_read(pmem, bvec.bv_page,
bvec.bv_offset,
iter.bi_sector, bvec.bv_len);
@@ -297,7 +301,7 @@ static int pmem_dax_zero_page_range(struct
dax_device *dax_dev, pgoff_t pgoff,
return blk_status_to_errno(pmem_do_write(pmem, ZERO_PAGE(0), 0,
PFN_PHYS(pgoff) >> SECTOR_SHIFT,
- PAGE_SIZE));
+ PAGE_SIZE, NULL));
}
static long pmem_dax_direct_access(struct dax_device *dax_dev,
>>> Is the I/O expected to be performed
>>> asynchronously?
>> Not nessecarily if it is fast enough..
> Clear as mud. :) It's a memcpy on potentially "slow" memory. So, the
> amount of time spent copying depends on the speed of the cpu, the media
> and the size of the I/O. I don't know off-hand what the upper bound
> would be on today's pmem.
Above scenario depends on the system and I'm not sure if we can
generalize this
for all the deployments. In case we end up generalizing above scenario
then we
can always add a modparam to disable nowait so user has total control
whether
to enable or disable this an incremental patch ..
For the deployments those are not suffering from the "mempcy on potentially
slow memory" this patch shows clear performance win with enabling NOWAIT
for io_uring ..
-ck
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 1:25 ` Chaitanya Kulkarni
@ 2023-08-02 12:30 ` Christoph Hellwig
2023-08-02 15:27 ` Jeff Moyer
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Christoph Hellwig @ 2023-08-02 12:30 UTC (permalink / raw
To: Chaitanya Kulkarni
Cc: Jeff Moyer, Christoph Hellwig, dan.j.williams@intel.com,
vishal.l.verma@intel.com, dave.jiang@intel.com,
ira.weiny@intel.com, nvdimm@lists.linux.dev, axboe, linux-block
Given that pmem simply loops over an arbitrarily large bio I think
we also need a threshold for which to allow nowait I/O. While it
won't block for giant I/Os, doing all of them in the submitter
context isn't exactly the idea behind the nowait I/O.
I'm not really sure what a good theshold would be, though.
Btw, please also always add linux-block to the Cc list for block
driver patches that are even the slightest bit about the block
layer interface.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 12:30 ` Christoph Hellwig
@ 2023-08-02 15:27 ` Jeff Moyer
2023-08-03 3:24 ` Chaitanya Kulkarni
2023-08-02 15:36 ` Jens Axboe
2023-08-03 3:17 ` Chaitanya Kulkarni
2 siblings, 1 reply; 14+ messages in thread
From: Jeff Moyer @ 2023-08-02 15:27 UTC (permalink / raw
To: Christoph Hellwig
Cc: Chaitanya Kulkarni, dan.j.williams@intel.com,
vishal.l.verma@intel.com, dave.jiang@intel.com,
ira.weiny@intel.com, nvdimm@lists.linux.dev, axboe, linux-block
Christoph Hellwig <hch@lst.de> writes:
> Given that pmem simply loops over an arbitrarily large bio I think
> we also need a threshold for which to allow nowait I/O. While it
> won't block for giant I/Os, doing all of them in the submitter
> context isn't exactly the idea behind the nowait I/O.
>
> I'm not really sure what a good theshold would be, though.
There's no mention of the latency of the submission side in the
documentation for RWF_NOWAIT. The man page says "Do not wait for data
which is not immediately available." Data in pmem *is* immediately
available. If we wanted to enforce a latency threshold for submission,
it would have to be configurable and, ideally, a part of the API. I
don't think it's something we should even try to guarantee, though,
unless application writers are asking for it.
So, I think with the change to return -EAGAIN for writes to poisoned
memory, this patch is probably ok.
Chaitanya, could you send a v2?
Thanks,
Jeff
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 12:30 ` Christoph Hellwig
2023-08-02 15:27 ` Jeff Moyer
@ 2023-08-02 15:36 ` Jens Axboe
2023-08-02 16:25 ` Christoph Hellwig
2023-08-03 3:21 ` Chaitanya Kulkarni
2023-08-03 3:17 ` Chaitanya Kulkarni
2 siblings, 2 replies; 14+ messages in thread
From: Jens Axboe @ 2023-08-02 15:36 UTC (permalink / raw
To: Christoph Hellwig, Chaitanya Kulkarni
Cc: Jeff Moyer, dan.j.williams@intel.com, vishal.l.verma@intel.com,
dave.jiang@intel.com, ira.weiny@intel.com, nvdimm@lists.linux.dev,
linux-block
On 8/2/23 6:30?AM, Christoph Hellwig wrote:
> Given that pmem simply loops over an arbitrarily large bio I think
> we also need a threshold for which to allow nowait I/O. While it
> won't block for giant I/Os, doing all of them in the submitter
> context isn't exactly the idea behind the nowait I/O.
You can do a LOT of looping over a giant bio and still come out way
ahead compared to needing to punt to a different thread. So I do think
it's the right choice. But I'm making assumptions here on what it looks
like, as I haven't seen the patch...
> Btw, please also always add linux-block to the Cc list for block
> driver patches that are even the slightest bit about the block
> layer interface.
Indeed. Particularly for these nowait changes, as some of them have been
pretty broken in the past.
--
Jens Axboe
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 15:36 ` Jens Axboe
@ 2023-08-02 16:25 ` Christoph Hellwig
2023-08-03 3:21 ` Chaitanya Kulkarni
1 sibling, 0 replies; 14+ messages in thread
From: Christoph Hellwig @ 2023-08-02 16:25 UTC (permalink / raw
To: Jens Axboe
Cc: Christoph Hellwig, Chaitanya Kulkarni, Jeff Moyer,
dan.j.williams@intel.com, vishal.l.verma@intel.com,
dave.jiang@intel.com, ira.weiny@intel.com, nvdimm@lists.linux.dev,
linux-block
On Wed, Aug 02, 2023 at 09:36:32AM -0600, Jens Axboe wrote:
> You can do a LOT of looping over a giant bio and still come out way
> ahead compared to needing to punt to a different thread. So I do think
> it's the right choice. But I'm making assumptions here on what it looks
> like, as I haven't seen the patch...
"a LOT" is relative. A GB or two will block the submission thread for
quite a while.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 12:30 ` Christoph Hellwig
2023-08-02 15:27 ` Jeff Moyer
2023-08-02 15:36 ` Jens Axboe
@ 2023-08-03 3:17 ` Chaitanya Kulkarni
2 siblings, 0 replies; 14+ messages in thread
From: Chaitanya Kulkarni @ 2023-08-03 3:17 UTC (permalink / raw
To: Christoph Hellwig
Cc: Jeff Moyer, dan.j.williams@intel.com, vishal.l.verma@intel.com,
dave.jiang@intel.com, ira.weiny@intel.com, nvdimm@lists.linux.dev,
axboe@kernel.dk, linux-block@vger.kernel.org
On 8/2/23 05:30, Christoph Hellwig wrote:
> Given that pmem simply loops over an arbitrarily large bio I think
> we also need a threshold for which to allow nowait I/O. While it
> won't block for giant I/Os, doing all of them in the submitter
> context isn't exactly the idea behind the nowait I/O.
>
> I'm not really sure what a good theshold would be, though.
>
> Btw, please also always add linux-block to the Cc list for block
> driver patches that are even the slightest bit about the block
> layer interface.
will do ..
-ck
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 15:36 ` Jens Axboe
2023-08-02 16:25 ` Christoph Hellwig
@ 2023-08-03 3:21 ` Chaitanya Kulkarni
1 sibling, 0 replies; 14+ messages in thread
From: Chaitanya Kulkarni @ 2023-08-03 3:21 UTC (permalink / raw
To: Jens Axboe
Cc: Jeff Moyer, dan.j.williams@intel.com, vishal.l.verma@intel.com,
dave.jiang@intel.com, ira.weiny@intel.com, nvdimm@lists.linux.dev,
linux-block@vger.kernel.org, Christoph Hellwig
On 8/2/23 08:36, Jens Axboe wrote:
> On 8/2/23 6:30?AM, Christoph Hellwig wrote:
>> Given that pmem simply loops over an arbitrarily large bio I think
>> we also need a threshold for which to allow nowait I/O. While it
>> won't block for giant I/Os, doing all of them in the submitter
>> context isn't exactly the idea behind the nowait I/O.
> You can do a LOT of looping over a giant bio and still come out way
> ahead compared to needing to punt to a different thread. So I do think
> it's the right choice. But I'm making assumptions here on what it looks
> like, as I haven't seen the patch...
>
will send out the V3 soon and CC you and linux-block ...
>> Btw, please also always add linux-block to the Cc list for block
>> driver patches that are even the slightest bit about the block
>> layer interface.
> Indeed. Particularly for these nowait changes, as some of them have been
> pretty broken in the past.
>
yes, didn't forgot, lately dealing with bunch of stupidity, I'll send
zram and null_blk nowait changes soon with your comments fixed ..
-ck
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-02 15:27 ` Jeff Moyer
@ 2023-08-03 3:24 ` Chaitanya Kulkarni
2023-08-03 13:11 ` Jeff Moyer
0 siblings, 1 reply; 14+ messages in thread
From: Chaitanya Kulkarni @ 2023-08-03 3:24 UTC (permalink / raw
To: Jeff Moyer, Christoph Hellwig
Cc: dan.j.williams@intel.com, vishal.l.verma@intel.com,
dave.jiang@intel.com, ira.weiny@intel.com, nvdimm@lists.linux.dev,
axboe@kernel.dk, linux-block@vger.kernel.org
On 8/2/23 08:27, Jeff Moyer wrote:
> Christoph Hellwig <hch@lst.de> writes:
>
>> Given that pmem simply loops over an arbitrarily large bio I think
>> we also need a threshold for which to allow nowait I/O. While it
>> won't block for giant I/Os, doing all of them in the submitter
>> context isn't exactly the idea behind the nowait I/O.
>>
>> I'm not really sure what a good theshold would be, though.
> There's no mention of the latency of the submission side in the
> documentation for RWF_NOWAIT. The man page says "Do not wait for data
> which is not immediately available." Data in pmem *is* immediately
> available. If we wanted to enforce a latency threshold for submission,
> it would have to be configurable and, ideally, a part of the API. I
> don't think it's something we should even try to guarantee, though,
> unless application writers are asking for it.
I need to see the usecase from application writter who cannot come
with a solution so kernel have to provide this interface, I'll be happy
to do that ..
> So, I think with the change to return -EAGAIN for writes to poisoned
> memory, this patch is probably ok.
I believe you mean the same one I've provided earlier incremental ..
> Chaitanya, could you send a v2?
sure ...
-ck
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH V2 1/1] pmem: set QUEUE_FLAG_NOWAIT
2023-08-03 3:24 ` Chaitanya Kulkarni
@ 2023-08-03 13:11 ` Jeff Moyer
0 siblings, 0 replies; 14+ messages in thread
From: Jeff Moyer @ 2023-08-03 13:11 UTC (permalink / raw
To: Chaitanya Kulkarni
Cc: Christoph Hellwig, dan.j.williams@intel.com,
vishal.l.verma@intel.com, dave.jiang@intel.com,
ira.weiny@intel.com, nvdimm@lists.linux.dev, axboe@kernel.dk,
linux-block@vger.kernel.org
Chaitanya Kulkarni <chaitanyak@nvidia.com> writes:
> On 8/2/23 08:27, Jeff Moyer wrote:
>> So, I think with the change to return -EAGAIN for writes to poisoned
>> memory, this patch is probably ok.
>
> I believe you mean the same one I've provided earlier incremental ..
Yes, sorry if that wasn't clear.
>> Chaitanya, could you send a v2?
>
> sure ...
and I guess I should have said v3. ;-)
Cheers,
Jeff
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-08-03 13:05 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-07-31 22:46 [PATCH V2 0/1] pmem: set QUEUE_FLAG_NOWAIT Chaitanya Kulkarni
2023-07-31 22:46 ` [PATCH V2 1/1] " Chaitanya Kulkarni
2023-08-01 15:23 ` Jeff Moyer
2023-08-01 15:59 ` Christoph Hellwig
2023-08-01 17:49 ` Jeff Moyer
2023-08-02 1:25 ` Chaitanya Kulkarni
2023-08-02 12:30 ` Christoph Hellwig
2023-08-02 15:27 ` Jeff Moyer
2023-08-03 3:24 ` Chaitanya Kulkarni
2023-08-03 13:11 ` Jeff Moyer
2023-08-02 15:36 ` Jens Axboe
2023-08-02 16:25 ` Christoph Hellwig
2023-08-03 3:21 ` Chaitanya Kulkarni
2023-08-03 3:17 ` Chaitanya Kulkarni
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).