From: Chris Murphy <lists@colorremedies.com>
To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Cc: Sebastian Roller <sebastian.roller@gmail.com>,
Qu Wenruo <quwenruo.btrfs@gmx.com>,
David Sterba <dsterba@suse.com>
Subject: Re: All files are damaged after btrfs restore
Date: Tue, 16 Mar 2021 13:34:35 -0600 [thread overview]
Message-ID: <CAJCQCtT47jRBkYn0D2CXtyeuc1_XFKAJ4oj1pnfOjmxY+NdxgQ@mail.gmail.com> (raw)
In-Reply-To: <CALS+qHNfxPZqmYsWWqZZqPsyGLfWV5+-vrrqUMD84nRU6V_3VQ@mail.gmail.com>
Hi,
The problem exceeds my knowledge of both Btrfs and bcache/ssd failure
modes. I'm not sure what professional data recovery can really do,
other than throw a bunch of people at stitching things back together
again without any help from the file system. I know that the state of
the repair tools is not great, and it is confusing what to use in what
order. I don't know if a support contract from one of the distros
supporting Btrfs (most likely SUSE) is a better way to get assistance
with this kind of recovery while also supporting development. But
that's a question for SUSE sales :)
Most of the emphasis of upstream development has been on preventing
problems from happening to critical Btrfs metadata in the first place.
Its ability to self-heal really depends on it having independent block
devices to write to, e.g. metadata raid 1. Metadata DUP might normally
help with only spinning drives, but with a cache device, it's going to
cache all of these concurrent metadata writes.
If critical metadata is seriously damaged or missing, it's probably
impossible to fix or even skip over with the current state of the
tools. Current code needs an entry point into the chunk tree in order
to make the logical to physical mapping; and then needs an entry point
to the root tree to get to the proper snapshot file tree. If all the
recent and critical metadata is lost on the failed bcache caching
device, then a totally different strategy is needed.
The file btree for the snapshot you want should be on the backing
device, as well as its data chunks, and the mapping in the ~94% of the
chunk tree that's on disk. I won't be surprised if the file system is
broken beyond repair, but I'd be a little surprised if someone more
knowledgeable can't figure out a way to get the data out of a week old
snapshot. But that's speculation on my part. I really have no idea how
long it could take for bcache in writeback mode to flush to the
backing device.
--
Chris Murphy
On Tue, Mar 16, 2021 at 3:35 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:
>
> Hi again.
>
> > Looks like the answer is no. The chunk tree really has to be correct
> > first before anything else because it's central to doing all the
> > logical to physical address translation. And if it's busted and can't
> > be repaired then nothing else is likely to work or be repairable. It's
> > that critical.
> >
> > > I already ran chunk-recover. It needs two days to finish. But I used
> > > btrfs-tools version 4.14 and it failed.
> >
> > I'd have to go dig in git history to even know if there's been
> > improvements in chunk recover since then. But I pretty much consider
> > any file system's tool obsolete within a year. I think it's total
> > nonsense that distributions are intentionally using old tools.
> > >
> > > root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
> > > Scanning: DONE in dev0
> > > checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > bytenr mismatch, want=124762809384960, have=0
> > > open with broken chunk error
> > > Chunk tree recovery failed
> > >
> > > I could try again with a newer version. (?) Because with version 4.14
> > > also btrfs restore failed.
> >
> > It is entirely possible that 5.11 fails exactly the same way because
> > it's just too badly damaged for the current state of the recovery
> > tools to deal with damage of this kind. But it's also possible it'll
> > work. It's a coin toss unless someone else a lot more familiar with
> > the restore code speaks up. But looking at just the summary change
> > log, it looks like no work has happened in chunk recover for a while.
> >
> > https://btrfs.wiki.kernel.org/index.php/Changelog
>
> So I ran another chunk-recover with btrfs-progs version 5.11. This is
> part of the output. (The list doesn't allow me attach the whole output
> to this mail (5 mb zipped). But if you let me know what's important I
> can send that.)
>
> root@hikitty:~$ nohup /root/install/btrfs-progs-5.11/btrfs -v rescue
> chunk-recover /dev/sdi1 >
> /transfer/sebroll/btrfs-rescue-chunk-recover.out.txt 2>&1 &
> nohup: ignoring input
> All Devices:
> Device: id = 2, name = /dev/sdi1
> Scanning: DONE in dev0
>
> DEVICE SCAN RESULT:
> Filesystem Information:
> sectorsize: 4096
> nodesize: 16384
> tree root generation: 825256
> chunk root generation: 825256
>
> All Devices:
> Device: id = 2, name = /dev/sdi1
>
> All Block Groups:
> Block Group: start = 49477515739136, len = 1073741824, flag = 1
> Block Group: start = 49478589480960, len = 1073741824, flag = 1
> (…)
> Block Group: start = 141942960685056, len = 1073741824, flag = 1
> Block Group: start = 141944034426880, len = 33554432, flag = 22
>
> All Chunks:
> Chunk: start = 49477515739136, len = 1073741824, type = 1,
> num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 1048576
> Chunk: start = 49478589480960, len = 1073741824, type = 1,
> num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 1074790400
> (…)
> Chunk: start = 141942960685056, len = 1073741824, type = 1,
> num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 75414325166080
> Chunk: start = 141944034426880, len = 33554432, type = 22,
> num_stripes = 2
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 2034741805056
> [ 1] Stripe: devid = 2, offset = 2034775359488
>
> All Device Extents:
> Device extent: devid = 2, start = 1048576, len = 1073741824,
> chunk offset = 49477515739136
> Device extent: devid = 2, start = 1074790400, len =
> 1073741824, chunk offset = 49478589480960
> Device extent: devid = 2, start = 2148532224, len =
> 1073741824, chunk offset = 49479663222784
> (…)
> Device extent: devid = 2, start = 75412177682432, len =
> 1073741824, chunk offset = 141940813201408
> Device extent: devid = 2, start = 75413251424256, len =
> 1073741824, chunk offset = 141941886943232
> Device extent: devid = 2, start = 75414325166080, len =
> 1073741824, chunk offset = 141942960685056
>
> CHECK RESULT:
> Recoverable Chunks:
> Chunk: start = 49477515739136, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 1048576
> Block Group: start = 49477515739136, len = 1073741824, flag = 1
> Device extent list:
> [ 0]Device extent: devid = 2, start = 1048576, len =
> 1073741824, chunk offset = 49477515739136
> Chunk: start = 49478589480960, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 1074790400
> Block Group: start = 49478589480960, len = 1073741824, flag = 1
> Device extent list:
> [ 0]Device extent: devid = 2, start = 1074790400, len =
> 1073741824, chunk offset = 49478589480960
> Chunk: start = 49479663222784, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 2148532224
> Block Group: start = 49479663222784, len = 1073741824, flag = 1
> Device extent list:
> [ 0]Device extent: devid = 2, start = 2148532224, len =
> 1073741824, chunk offset = 49479663222784
> (…)
> Chunk: start = 141085812719616, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 74690623176704
> No block group.
> Device extent list:
> [ 0]Device extent: devid = 2, start = 74690623176704, len =
> 1073741824, chunk offset = 141085812719616
> Chunk: start = 141403740962816, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 73223891845120
> No block group.
> Device extent list:
> [ 0]Device extent: devid = 2, start = 73223891845120, len =
> 1073741824, chunk offset = 141403740962816
> Unrecoverable Chunks:
> Chunk: start = 73953460617216, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 23810225995776
> No block group.
> No device extent.
> Chunk: start = 75698291081216, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 25555056459776
> No block group.
> No device extent.
> (…)
> Chunk: start = 139435974852608, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 49594055524352
> Block Group: start = 139435974852608, len = 1073741824, flag = 1
> No device extent.
> Chunk: start = 140101996773376checksum verify failed on
> 99593231630336 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
> ERROR: failed to read block groups: Input/output error
> open with broken chunk error
> , len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 58501817696256
> Block Group: start = 140101996773376, len = 1073741824, flag = 1
> No device extent.
> Chunk: start = 140221215670272, len = 1073741824, type = 1, num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 58416992092160
> Block Group: start = 140221215670272, len = 1073741824, flag = 1
> No device extent.
> (…)
> Chunk: start = 141836593135616, len = 33554432, type = 22, num_stripes = 2
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 2034741805056
> [ 1] Stripe: devid = 2, offset = 2034775359488
> No block group.
> No device extent.
> Chunk: start = 141269456125952, len = 33554432, type = 22, num_stripes = 0
> Stripes list:
> Block Group: start = 141269456125952, len = 33554432, flag = 22
> No device extent.
>
> Total Chunks: 72859
> Recoverable: 68784
> Unrecoverable: 4075
>
> Orphan Block Groups:
>
> Orphan Device Extents:
>
> Chunk tree recovery failed
>
>
>
> So. Is this good or bad? First, there are a lot of unrecoverable
> chunks. And some of them are system chunks.
> But my biggest issue seems to be that "bad tree block" error that is
> showing up and which I'm getting any time I try to access the
> file-system. This error also seems to prevent mounting and btrfs
> check. Can I use the position given there to look up the corresponding
> chunk? If so:
>
> Block Group and Chunks around bad tree block 124762809384960:
>
> Block Group: start = 124760809799680, len = 1073741824, flag = 1
> Block Group: start = 124761883541504, len = 1073741824, flag = 24
> Block Group: start = 124798390763520, len = 1073741824, flag =
> 1
>
> Chunk: start = 124760809799680, len = 1073741824, type = 1,
> num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 67164766732288
> Chunk: start = 124761883541504, len = 1073741824, type = 24,
> num_stripes = 2
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 67165840474112
> [ 1] Stripe: devid = 2, offset = 67166914215936
> Chunk: start = 124777989668864, len = 1073741824, type = 1,
> num_stripes = 1
> Stripes list:
> [ 0] Stripe: devid = 2, offset = 67183020343296
>
> The bad tree block seems to contain meta data. Can I somehow recover
> this? It seems to be DUP (2 Stripes)…
>
> > > > And you'll need to look for a snapshot name in there, find its bytenr,
> > > > and let's first see if just using that works. If it doesn't then maybe
> > > > combining it with the next most recent root tree will work.
> > >
> > > I am working backwards right now using btrfs restore -f in combination
> > > with -t. So far no success.
> >
> > Yep. I think it comes down to the chunk tree needing to be reasonable
> > first, before anything else is possible.
>
> Now I tried every possible combination with btrfs restore.
> Unfortunately I found no working solution to restore the data
> undamaged. As mentioned before the files contain "holes" of 0x00.
> These holes are exactly 4 MiB in size. Does this tell anything?
> Any ideas how to proceed from here on? Accept the data loss and try to
> repair the files as good as possible -- putting them together from
> different saves (if available) is a real grind. Or professional data
> recovery -- but wouldn't they face the same issue regarding the
> chunk-tree?
>
> Kind regards,
> Sebastian
--
Chris Murphy
next prev parent reply other threads:[~2021-03-16 19:35 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-23 15:45 All files are damaged after btrfs restore Sebastian Roller
2021-02-25 5:40 ` Chris Murphy
2021-02-25 5:52 ` Chris Murphy
2021-02-26 16:01 ` Sebastian Roller
2021-02-27 1:04 ` Chris Murphy
2021-03-04 15:34 ` Sebastian Roller
2021-03-05 3:01 ` Chris Murphy
2021-03-07 13:58 ` Sebastian Roller
2021-03-08 0:56 ` Chris Murphy
2021-03-09 17:02 ` Sebastian Roller
2021-03-09 20:34 ` Chris Murphy
2021-03-16 9:35 ` Sebastian Roller
2021-03-16 19:34 ` Chris Murphy [this message]
2021-03-17 1:38 ` Qu Wenruo
2021-03-17 2:59 ` Chris Murphy
2021-03-17 9:01 ` Sebastian Roller
2021-03-17 1:54 ` Dāvis Mosāns
2021-03-17 10:50 ` Sebastian Roller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJCQCtT47jRBkYn0D2CXtyeuc1_XFKAJ4oj1pnfOjmxY+NdxgQ@mail.gmail.com \
--to=lists@colorremedies.com \
--cc=dsterba@suse.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=quwenruo.btrfs@gmx.com \
--cc=sebastian.roller@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.