* testing thin provisioned block device
@ 2016-05-23 14:24 Tim Walker
2016-05-23 14:40 ` Alireza Haghdoost
2016-05-23 15:02 ` Andrey Kuzmin
0 siblings, 2 replies; 8+ messages in thread
From: Tim Walker @ 2016-05-23 14:24 UTC (permalink / raw
To: fio
Hello-
I need to do random writes/reads to some thin-provisioned block
devices. Writes are no problem, but I have to make sure the reads
come from blocks that have already been written (or else the device
synthesizes fill data). Pre-filling the device is not desirable since
they are 8-12 TiB, plus that isn't the way they are really used.
IO at the block level is our customer's requirement, but I'd also be
interested in the same concept at the file level, where fio randomly
reads from files that it has randomly written.
I'm sure I'm not the first person who has come up against this, but
I've searched/Googled the best I can and have come up empty. Can
somebody point me to the correct switches to force reads to be
randomly selected from the blocks that have been written by that test
sequence?
Thanks in advance.
Best regards,
-Tim
--
Tim Walker
Product Design Systems Engineering, Seagate Technology
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-23 14:24 testing thin provisioned block device Tim Walker
@ 2016-05-23 14:40 ` Alireza Haghdoost
2016-05-23 15:01 ` Alireza Haghdoost
2016-05-23 15:02 ` Andrey Kuzmin
1 sibling, 1 reply; 8+ messages in thread
From: Alireza Haghdoost @ 2016-05-23 14:40 UTC (permalink / raw
To: Tim Walker; +Cc: fio@vger.kernel.org
On Mon, May 23, 2016 at 9:24 AM, Tim Walker <tim.t.walker@seagate.com> wrote:
>
> Hello-
>
> I need to do random writes/reads to some thin-provisioned block
> devices. Writes are no problem, but I have to make sure the reads
> come from blocks that have already been written (or else the device
> synthesizes fill data). Pre-filling the device is not desirable since
> they are 8-12 TiB, plus that isn't the way they are really used.
>
> IO at the block level is our customer's requirement, but I'd also be
> interested in the same concept at the file level, where fio randomly
> reads from files that it has randomly written.
>
> I'm sure I'm not the first person who has come up against this, but
> I've searched/Googled the best I can and have come up empty. Can
> somebody point me to the correct switches to force reads to be
> randomly selected from the blocks that have been written by that test
> sequence?
>
> Thanks in advance.
>
> Best regards,
> -Tim
Tim,
If you don't want to mix read and write workload, could you just write
randomly at your block device with one job first, then use the same
random seed in the next job to read those random blocks ?
--Alireza
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-23 14:40 ` Alireza Haghdoost
@ 2016-05-23 15:01 ` Alireza Haghdoost
0 siblings, 0 replies; 8+ messages in thread
From: Alireza Haghdoost @ 2016-05-23 15:01 UTC (permalink / raw
To: Tim Walker; +Cc: fio@vger.kernel.org
On Mon, May 23, 2016 at 9:40 AM, Alireza Haghdoost <alireza@cs.umn.edu> wrote:
> On Mon, May 23, 2016 at 9:24 AM, Tim Walker <tim.t.walker@seagate.com> wrote:
>>
>> Hello-
>>
>> I need to do random writes/reads to some thin-provisioned block
>> devices. Writes are no problem, but I have to make sure the reads
>> come from blocks that have already been written (or else the device
>> synthesizes fill data). Pre-filling the device is not desirable since
>> they are 8-12 TiB, plus that isn't the way they are really used.
>>
>> IO at the block level is our customer's requirement, but I'd also be
>> interested in the same concept at the file level, where fio randomly
>> reads from files that it has randomly written.
>>
>> I'm sure I'm not the first person who has come up against this, but
>> I've searched/Googled the best I can and have come up empty. Can
>> somebody point me to the correct switches to force reads to be
>> randomly selected from the blocks that have been written by that test
>> sequence?
>>
>> Thanks in advance.
>>
>> Best regards,
>> -Tim
>
> Tim,
>
> If you don't want to mix read and write workload, could you just write
> randomly at your block device with one job first, then use the same
> random seed in the next job to read those random blocks ?
>
> --Alireza
Or you could fire two Jobs simultaneously with the same random seed.
First job only writes with large queue depth and the second job only
reads with small queue depth. That would result in a mixed workload
where the writer writes on random blocks with a faster speed than the
reader job.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-23 14:24 testing thin provisioned block device Tim Walker
2016-05-23 14:40 ` Alireza Haghdoost
@ 2016-05-23 15:02 ` Andrey Kuzmin
2016-05-23 17:41 ` Sitsofe Wheeler
1 sibling, 1 reply; 8+ messages in thread
From: Andrey Kuzmin @ 2016-05-23 15:02 UTC (permalink / raw
To: Tim Walker; +Cc: fio@vger.kernel.org
On Mon, May 23, 2016 at 5:24 PM, Tim Walker <tim.t.walker@seagate.com> wrote:
> Hello-
>
> I need to do random writes/reads to some thin-provisioned block
> devices. Writes are no problem, but I have to make sure the reads
> come from blocks that have already been written (or else the device
> synthesizes fill data). Pre-filling the device is not desirable since
> they are 8-12 TiB, plus that isn't the way they are really used.
>
> IO at the block level is our customer's requirement, but I'd also be
> interested in the same concept at the file level, where fio randomly
> reads from files that it has randomly written.
>
> I'm sure I'm not the first person who has come up against this, but
> I've searched/Googled the best I can and have come up empty. Can
> somebody point me to the correct switches to force reads to be
> randomly selected from the blocks that have been written by that test
> sequence?
To my knowledge, there's no prebuilt configuration to read only those blocks
that have been written before.
At the same time, when randommap is on, FIO keeps track of the blocks written,
so it may be possible to limit the read generation to those by adding
a bit of code.
Regards,
Andrey
>
> Thanks in advance.
>
> Best regards,
> -Tim
>
> --
> Tim Walker
> Product Design Systems Engineering, Seagate Technology
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-23 15:02 ` Andrey Kuzmin
@ 2016-05-23 17:41 ` Sitsofe Wheeler
2016-05-24 3:52 ` Tim Walker
2016-05-24 14:58 ` Jens Axboe
0 siblings, 2 replies; 8+ messages in thread
From: Sitsofe Wheeler @ 2016-05-23 17:41 UTC (permalink / raw
To: Andrey Kuzmin; +Cc: Tim Walker, fio@vger.kernel.org
On 23 May 2016 at 16:02, Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
>
> To my knowledge, there's no prebuilt configuration to read only those blocks
> that have been written before.
>
> At the same time, when randommap is on, FIO keeps track of the blocks written,
> so it may be possible to limit the read generation to those by adding
> a bit of code.
I thought this type of facility had to exist so that verification
triggers (https://github.com/axboe/fio/blob/fio-2.10/HOWTO#L2214 ) via
verify_state_load could work?
--
Sitsofe | http://sucs.org/~sits/
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-23 17:41 ` Sitsofe Wheeler
@ 2016-05-24 3:52 ` Tim Walker
2016-05-24 4:41 ` Sitsofe Wheeler
2016-05-24 14:58 ` Jens Axboe
1 sibling, 1 reply; 8+ messages in thread
From: Tim Walker @ 2016-05-24 3:52 UTC (permalink / raw
To: Sitsofe Wheeler; +Cc: Andrey Kuzmin, fio@vger.kernel.org
This looks interesting. Thanks!
I will report back...
-Tim
On Mon, May 23, 2016 at 11:41 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 23 May 2016 at 16:02, Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
>>
>> To my knowledge, there's no prebuilt configuration to read only those blocks
>> that have been written before.
>>
>> At the same time, when randommap is on, FIO keeps track of the blocks written,
>> so it may be possible to limit the read generation to those by adding
>> a bit of code.
>
> I thought this type of facility had to exist so that verification
> triggers (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_axboe_fio_blob_fio-2D2.10_HOWTO-23L2214&d=CwIBaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=NW1X0yRHNNEluZ8sOGXBxCbQJZPWcIkPT0Uy3ynVsFU&m=kMV_vJkDG44PlTPqwWzkRqZGd9WboeQTwskJijXbUB4&s=hocA_d46stzA2Fk0_dkI2uDHlaEs81U8qVRHKQhSfo0&e= ) via
> verify_state_load could work?
>
> --
> Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=CwIBaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=NW1X0yRHNNEluZ8sOGXBxCbQJZPWcIkPT0Uy3ynVsFU&m=kMV_vJkDG44PlTPqwWzkRqZGd9WboeQTwskJijXbUB4&s=j0EyneqqMvyjeXgg8I4VEb8qly5XZH14rVVpAtmzXK4&e=
--
Tim Walker
Product Design Systems Engineering, Seagate Technology
(303) 775-3770
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-24 3:52 ` Tim Walker
@ 2016-05-24 4:41 ` Sitsofe Wheeler
0 siblings, 0 replies; 8+ messages in thread
From: Sitsofe Wheeler @ 2016-05-24 4:41 UTC (permalink / raw
To: Tim Walker; +Cc: Andrey Kuzmin, fio@vger.kernel.org
On 24 May 2016 at 04:52, Tim Walker <tim.t.walker@seagate.com> wrote:
> This looks interesting. Thanks!
> I will report back...
Other possibilities would be to use verify_backlog
(https://github.com/axboe/fio/blob/fio-2.10/HOWTO#L1467 ) so you're
re-reading periodically as you're going along or to use
io_limit/io_size
(https://github.com/axboe/fio/blob/fio-2.10/HOWTO#L495 ) with two
stonewalled jobs using the same seeds but where the first job was a
write and the second a read.
--
Sitsofe | http://sucs.org/~sits/
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: testing thin provisioned block device
2016-05-23 17:41 ` Sitsofe Wheeler
2016-05-24 3:52 ` Tim Walker
@ 2016-05-24 14:58 ` Jens Axboe
1 sibling, 0 replies; 8+ messages in thread
From: Jens Axboe @ 2016-05-24 14:58 UTC (permalink / raw
To: Sitsofe Wheeler, Andrey Kuzmin; +Cc: Tim Walker, fio@vger.kernel.org
On 05/23/2016 11:41 AM, Sitsofe Wheeler wrote:
> On 23 May 2016 at 16:02, Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
>>
>> To my knowledge, there's no prebuilt configuration to read only those blocks
>> that have been written before.
>>
>> At the same time, when randommap is on, FIO keeps track of the blocks written,
>> so it may be possible to limit the read generation to those by adding
>> a bit of code.
>
> I thought this type of facility had to exist so that verification
> triggers (https://github.com/axboe/fio/blob/fio-2.10/HOWTO#L2214 ) via
> verify_state_load could work?
Definitely. Similarly to what Alireza suggested, using the same random
seed for a reader would read the exact same blocks that fio previously
wrote. You don't strictly need the random map for that. If you seed
things the same way, then the offsets/sizes/etc will be generated
identically. The random map can help with verifies if you overwrite
blocks or need to trim, but for a pure write-then-read setup, it's not
needed.
For instance, if you do verifies, you can have a write job file that you
run to completion. Then you take that exact same job file, and change
the write (random or sequential) to the equivalent read, and run fio
again. The same thing is of course possible without doing the
verification, but just using that same mechanism to read only the parts
that were written.
--
Jens Axboe
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-05-24 14:58 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-05-23 14:24 testing thin provisioned block device Tim Walker
2016-05-23 14:40 ` Alireza Haghdoost
2016-05-23 15:01 ` Alireza Haghdoost
2016-05-23 15:02 ` Andrey Kuzmin
2016-05-23 17:41 ` Sitsofe Wheeler
2016-05-24 3:52 ` Tim Walker
2016-05-24 4:41 ` Sitsofe Wheeler
2016-05-24 14:58 ` Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.