From: Patrick Plenefisch <simonpatp@gmail.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: Mike Snitzer <snitzer@kernel.org>,
Goffredo Baroncelli <kreijack@inwind.it>,
linux-kernel@vger.kernel.org, Alasdair Kergon <agk@redhat.com>,
Mikulas Patocka <mpatocka@redhat.com>, Chris Mason <clm@fb.com>,
Josef Bacik <josef@toxicpanda.com>,
David Sterba <dsterba@suse.com>,
regressions@lists.linux.dev, dm-devel@lists.linux.dev,
linux-btrfs@vger.kernel.org
Subject: Re: LVM-on-LVM: error while submitting device barriers
Date: Tue, 12 Mar 2024 18:54:59 -0400 [thread overview]
Message-ID: <CAOCpoWd5VWZnAaYvkFDYo736ZXDK0bExC9NkwVGfLv_CATj9Rw@mail.gmail.com> (raw)
In-Reply-To: <Ze8DZLBHhCxgzc+r@fedora>
On Mon, Mar 11, 2024 at 9:13 AM Ming Lei <ming.lei@redhat.com> wrote:
>
> On Sun, Mar 10, 2024 at 02:11:11PM -0400, Patrick Plenefisch wrote:
> > On Sun, Mar 10, 2024 at 11:27 AM Mike Snitzer <snitzer@kernel.org> wrote:
> > >
> > > On Sun, Mar 10 2024 at 7:34P -0400,
> > > Ming Lei <ming.lei@redhat.com> wrote:
> > >
> > > > On Sat, Mar 09, 2024 at 03:39:02PM -0500, Patrick Plenefisch wrote:
> > > > > On Wed, Mar 6, 2024 at 11:00 AM Ming Lei <ming.lei@redhat.com> wrote:
> > > > > >
> > > > > > #!/usr/bin/bpftrace
> > > > > >
> > > > > > #ifndef BPFTRACE_HAVE_BTF
> > > > > > #include <linux/blkdev.h>
> > > > > > #endif
> > > > > >
> > > > > > kprobe:submit_bio_noacct,
> > > > > > kprobe:submit_bio
> > > > > > / (((struct bio *)arg0)->bi_opf & (1 << __REQ_PREFLUSH)) != 0 /
> > > > > > {
> > > > > > $bio = (struct bio *)arg0;
> > > > > > @submit_stack[arg0] = kstack;
> > > > > > @tracked[arg0] = 1;
> > > > > > }
> > > > > >
> > > > > > kprobe:bio_endio
> > > > > > /@tracked[arg0] != 0/
> > > > > > {
> > > > > > $bio = (struct bio *)arg0;
> > > > > >
> > > > > > if (($bio->bi_flags & (1 << BIO_CHAIN)) && $bio->__bi_remaining.counter > 1) {
> > > > > > return;
> > > > > > }
> > > > > >
> > > > > > if ($bio->bi_status != 0) {
> > > > > > printf("dev %s bio failed %d, submitter %s completion %s\n",
> > > > > > $bio->bi_bdev->bd_disk->disk_name,
> > > > > > $bio->bi_status, @submit_stack[arg0], kstack);
> > > > > > }
> > > > > > delete(@submit_stack[arg0]);
> > > > > > delete(@tracked[arg0]);
> > > > > > }
> > > > > >
> > > > > > END {
> > > > > > clear(@submit_stack);
> > > > > > clear(@tracked);
> > > > > > }
> > > > > >
> > > > >
> > > > > Attaching 4 probes...
> > > > > dev dm-77 bio failed 10, submitter
> > > > > submit_bio_noacct+5
> > > > > __send_duplicate_bios+358
> > > > > __send_empty_flush+179
> > > > > dm_submit_bio+857
> > > > > __submit_bio+132
> > > > > submit_bio_noacct_nocheck+345
> > > > > write_all_supers+1718
> > > > > btrfs_commit_transaction+2342
> > > > > transaction_kthread+345
> > > > > kthread+229
> > > > > ret_from_fork+49
> > > > > ret_from_fork_asm+27
> > > > > completion
> > > > > bio_endio+5
> > > > > dm_submit_bio+955
> > > > > __submit_bio+132
> > > > > submit_bio_noacct_nocheck+345
> > > > > write_all_supers+1718
> > > > > btrfs_commit_transaction+2342
> > > > > transaction_kthread+345
> > > > > kthread+229
> > > > > ret_from_fork+49
> > > > > ret_from_fork_asm+27
> > > > >
> > > > > dev dm-86 bio failed 10, submitter
> > > > > submit_bio_noacct+5
> > > > > write_all_supers+1718
> > > > > btrfs_commit_transaction+2342
> > > > > transaction_kthread+345
> > > > > kthread+229
> > > > > ret_from_fork+49
> > > > > ret_from_fork_asm+27
> > > > > completion
> > > > > bio_endio+5
> > > > > clone_endio+295
> > > > > clone_endio+295
> > > > > process_one_work+369
> > > > > worker_thread+635
> > > > > kthread+229
> > > > > ret_from_fork+49
> > > > > ret_from_fork_asm+27
> > > > >
> > > > >
> > > > > For context, dm-86 is /dev/lvm/brokenDisk and dm-77 is /dev/lowerVG/lvmPool
> > > >
> > > > io_status is 10(BLK_STS_IOERR), which is produced in submission code path on
> > > > /dev/dm-77(/dev/lowerVG/lvmPool) first, so looks it is one device mapper issue.
> > > >
> > > > The error should be from the following code only:
> > > >
> > > > static void __map_bio(struct bio *clone)
> > > >
> > > > ...
> > > > if (r == DM_MAPIO_KILL)
> > > > dm_io_dec_pending(io, BLK_STS_IOERR);
> > > > else
> > > > dm_io_dec_pending(io, BLK_STS_DM_REQUEUE);
> > > > break;
> > >
> > > I agree that the above bpf stack traces for dm-77 indicate that
> > > dm_submit_bio failed, which would end up in the above branch if the
> > > target's ->map() returned DM_MAPIO_KILL or DM_MAPIO_REQUEUE.
> > >
> > > But such an early failure speaks to the flush bio never being
> > > submitted to the underlying storage. No?
> > >
> > > dm-raid.c:raid_map does return DM_MAPIO_REQUEUE with:
> > >
> > > /*
> > > * If we're reshaping to add disk(s)), ti->len and
> > > * mddev->array_sectors will differ during the process
> > > * (ti->len > mddev->array_sectors), so we have to requeue
> > > * bios with addresses > mddev->array_sectors here or
> > > * there will occur accesses past EOD of the component
> > > * data images thus erroring the raid set.
> > > */
> > > if (unlikely(bio_end_sector(bio) > mddev->array_sectors))
> > > return DM_MAPIO_REQUEUE;
> > >
> > > But a flush doesn't have an end_sector (it'd be 0 afaik).. so it seems
> > > weird relative to a flush.
> > >
> > > > Patrick, you mentioned lvmPool is raid1, can you explain how lvmPool is
> > > > built? It is dm-raid1 target or over plain raid1 device which is
> > > > build over /dev/lowerVG?
> >
> > LVM raid1:
> > lvcreate --type raid1 -m 1 ...
>
> OK, that is the reason, as Mike mentioned.
>
> dm-raid.c:raid_map returns DM_MAPIO_REQUEUE, which is translated into
> BLK_STS_IOERR in dm_io_complete().
>
> Empty flush bio is sent from btrfs, both .bi_size and .bi_sector are set
> as zero, but the top dm is linear, which(linear_map()) maps new
> bio->bi_iter.bi_sector, and the mapped bio is sent to dm-raid(raid_map()),
> then DM_MAPIO_REQUEUE is returned.
>
> The one-line patch I sent in last email should solve this issue.
>
> https://lore.kernel.org/dm-devel/a783e5ed-db56-4100-956a-353170b1b7ed@inwind.it/T/#m8fce3ecb2f98370b7d7ce8db6714bbf644af5459
With this patch on a 6.6.13 base, I can modify files and the BTRFS
volume stays RW, while no errors are logged in dmesg!
>
> But DM_MAPIO_REQUEUE misuse needs close look, and I believe Mike is working
> on that bigger problem.
>
> I guess most of dm targets don't deal with empty bio well, at least
> linear & dm-raid, not look into others yet, :-(
>
>
> Thanks,
> Ming
>
prev parent reply other threads:[~2024-03-12 22:55 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAOCpoWc_HQy4UJzTi9pqtJdO740Wx5Yd702O-mwXBE6RVBX1Eg@mail.gmail.com>
[not found] ` <CAOCpoWf3TSQkUUo-qsj0LVEOm-kY0hXdmttLE82Ytc0hjpTSPw@mail.gmail.com>
2024-02-28 17:25 ` [REGRESSION] LVM-on-LVM: error while submitting device barriers Patrick Plenefisch
2024-02-28 19:19 ` Goffredo Baroncelli
2024-02-28 19:37 ` Patrick Plenefisch
2024-02-29 19:56 ` Goffredo Baroncelli
2024-02-29 20:22 ` Patrick Plenefisch
2024-02-29 22:05 ` Goffredo Baroncelli
2024-03-05 17:45 ` Mike Snitzer
2024-03-06 15:59 ` Ming Lei
2024-03-09 20:39 ` Patrick Plenefisch
2024-03-10 11:34 ` Ming Lei
2024-03-10 15:27 ` Mike Snitzer
2024-03-10 15:47 ` Ming Lei
2024-03-10 18:11 ` Patrick Plenefisch
2024-03-11 13:13 ` Ming Lei
2024-03-12 22:54 ` Patrick Plenefisch [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOCpoWd5VWZnAaYvkFDYo736ZXDK0bExC9NkwVGfLv_CATj9Rw@mail.gmail.com \
--to=simonpatp@gmail.com \
--cc=agk@redhat.com \
--cc=clm@fb.com \
--cc=dm-devel@lists.linux.dev \
--cc=dsterba@suse.com \
--cc=josef@toxicpanda.com \
--cc=kreijack@inwind.it \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=mpatocka@redhat.com \
--cc=regressions@lists.linux.dev \
--cc=snitzer@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).