From: Xiao Ni <xni@redhat.com>
To: tada keisuke <keisuke1.tada@kioxia.com>
Cc: "song@kernel.org" <song@kernel.org>,
"yukuai3@huawei.com" <yukuai3@huawei.com>,
"linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 00/11] md: avoid counter operation conflicts
Date: Thu, 18 Apr 2024 17:58:11 +0800 [thread overview]
Message-ID: <CALTww2_zw8=_wizsA=msnYsx-cKAHzHBB4maYp7T+GZmkAQvqg@mail.gmail.com> (raw)
In-Reply-To: <3c0ab8eb94ad4b6fb4cb2159f0638650@kioxia.com>
On Thu, Apr 18, 2024 at 2:21 PM tada keisuke <keisuke1.tada@kioxia.com> wrote:
>
> Changes in v2:
> - Add message of performance in cover.
> - Fix a problem of error code initialization in patch 6.
> - Fix a problem of sleeping during rcu_read_lock() in patch 9.
> - Change base-commit from md-6.9 to md-6.10
>
> Currently, active_aligned_reads and nr_pending used as counters are atomic types.
> Therefore, when inc/dec in a multi-core results in conflicts and READ I/O becomes slow.
> To improve performance, use "percpu_ref" counters that can avoid conflicts and maintain consistency.
>
> Switch modes of percpu_ref to achieve both consistency and conflict avoidance.
> During normal operations such as inc/dec, it operates as percpu mode.
> When consistency is required, it operates as atomic mode.
> The operations that require consistency are as follows:
> - Zero check for the counter
> - All operations in RAID 1/10
>
> Patches 1, 3, 6 change active_aligned_reads, and patches 2, 4, 5, 7 to 11 change nr_pending.
> nr_pending temporarily switch from percpu mode to atomic mode in patch 7.
> This is to reduce the amount of changes from patches 8 to 10.
> Finally, nr_pending switch from atomic mode to percpu mode in patch 11.
>
> We applied the patch to base-commit and used fio to compare IOPS.
> CPU: AMD EPYC 7313P (3.0GHz, 16cores)
> DISK: ramdisk x 3 (modprobe brd rd_nr=3)
> RAID: level 5
> fio config: bs=4k, rw=randread, iodepth=128, numjobs=16
>
> without patch: 3.64 MIOPS
> with patch : 3.84 MIOPS
Hi Tada
Thanks for the patch set. Have you done tests with nvme/ssd or hdd?
It's better to see the results with real disks.
Best Regards
Xiao
>
> Keisuke TADA (11):
> add infra for active_aligned_reads changes
> add infra for nr_pending changes
> workaround for inconsistency of config state in takeover
> minimize execution of zero check for nr_pending
> match the type of variables to percpu_ref
> avoid conflicts in active_aligned_reads operations
> change the type of nr_pending from atomic_t to percpu_ref
> add atomic mode switching in RAID 1/10
> add atomic mode switching when removing disk
> add atomic mode switching when I/O completion
> avoid conflicts in nr_pending operations
>
> drivers/md/md-bitmap.c | 2 +-
> drivers/md/md.c | 48 ++++++++++++++++++---
> drivers/md/md.h | 62 +++++++++++++++++++++++----
> drivers/md/raid1.c | 37 +++++++++++------
> drivers/md/raid10.c | 60 ++++++++++++++++-----------
> drivers/md/raid5-cache.c | 4 +-
> drivers/md/raid5.c | 90 +++++++++++++++++++++++++++-------------
> drivers/md/raid5.h | 17 +++++++-
> 8 files changed, 238 insertions(+), 82 deletions(-)
>
>
> base-commit: 9d1110f99c253ccef82e480bfe9f38a12eb797a7
> --
> 2.34.1
>
>
>
prev parent reply other threads:[~2024-04-18 9:58 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-18 5:43 [PATCH v2 00/11] md: avoid counter operation conflicts tada keisuke
2024-04-18 9:58 ` Xiao Ni [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALTww2_zw8=_wizsA=msnYsx-cKAHzHBB4maYp7T+GZmkAQvqg@mail.gmail.com' \
--to=xni@redhat.com \
--cc=keisuke1.tada@kioxia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=song@kernel.org \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).