| Message ID | cover.1668529099.git.fdmanana@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | btrfs-progs: receive: fix a silent data loss bug with encoded writes | expand |
Hey. I recently sent | received a lot of previous data, thus asking: What exactly does encoded write mean? Is one *only* affected when ones used compression - respectively if one DID NOT do any filesystem compression (i.e. compress mount option)... can one be sure to be safe? Thanks, Chris. [0] https://lore.kernel.org/linux-btrfs/cover.1660690698.git.osandov@fb.com/
On Tue, Nov 15, 2022 at 4:37 PM Christoph Anton Mitterer <calestyo@scientia.org> wrote: > > Hey. > > I recently sent | received a lot of previous data, thus asking: > > What exactly does encoded write mean? Right now, it means a write operation where the data is compressed. I.e. there was no decompression at the sender side. > Is one *only* affected when ones > used compression - respectively if one DID NOT do any filesystem > compression (i.e. compress mount option)... can one be sure to be safe? If you haven't used 'btrfs send' with the --compressed-data option or you are sure you don't have any compressed files, then you're fine. Otherwise check your files by comparing them between the sending side and the receiving side. > > Thanks, > Chris. > > > [0] https://lore.kernel.org/linux-btrfs/cover.1660690698.git.osandov@fb.com/
Hey again. Something more general: Just as with the most recent (at least as far as I've noticed) possible silent data corruption[0], it's IMO really most unfortunate and a structural problem, that there is no useful way for (interested) end users to take notice them. If I weren't following the list a bit - and even there only by chance and depending on an alerting commit summary - I'd probably never ever hear about it... yet I might still suffer from it without realising immediately. Right now people may still have backups respectively the sources they btrfs-sent from... but perhaps not so in a year when they possibly notice the corruption (if at all). This is really no criticism that such bugs do happen - btrfs is quite actively developed, so I fully understand that such bugs show up. But for end users its really awful, if especially for silent corruption issues there is no alerting mechanism. And yes I know, other filesystems don't have one either - doesn't make it better though. I'd say a simple corruptions announcement mailing list would do: No other announcements like "new btrfs progs version" (unless that would fix a data corruption issue). Also usually no announcement for issues that were 100% user-visible (like general crash when btrfs tries to mount a fs) OR which have no impact on data consistency (e.g. if a bug would prevent the "compress" mount option to be considered at all). Only really serious things that could cause data loss (perhaps including confidentiality issues, like with fscrypt). Per issue, one announcement mail with description on what it is, how likely it happened, if/how one can find out whether one was affected, how one can fix (or if checks against / recovery from backups are needed). Perhaps other filesystems would even want to join in, then mails should be prefixed with "btrfs: ", "xfs:", "ext4", etc.: Regards, Chris.
On Tue, 2022-11-15 at 16:47 +0000, Filipe Manana wrote: > > Is one *only* affected when ones > > used compression - respectively if one DID NOT do any filesystem > > compression (i.e. compress mount option)... can one be sure to be > > safe? > > If you haven't used 'btrfs send' with the --compressed-data option or > you are sure you don't have any compressed files, then you're fine. Thanks a lot... so all good for me. :-) But still, as I wrote in the other mail,... other people might be affected... and it would be reeeeally nice if there was some good way for them to get alerted about such cases. Thanks, Chris.
On Tue, Nov 15, 2022 at 4:53 PM Christoph Anton Mitterer <calestyo@scientia.org> wrote: > > On Tue, 2022-11-15 at 16:47 +0000, Filipe Manana wrote: > > > Is one *only* affected when ones > > > used compression - respectively if one DID NOT do any filesystem > > > compression (i.e. compress mount option)... can one be sure to be > > > safe? > > > > If you haven't used 'btrfs send' with the --compressed-data option or > > you are sure you don't have any compressed files, then you're fine. > > Thanks a lot... so all good for me. :-) > > But still, as I wrote in the other mail,... other people might be > affected... and it would be reeeeally nice if there was some good way > for them to get alerted about such cases. There will always be users who'll miss such alerts, many don't read all the emails in this list, many are not even subscribed to the mailing list, etc. Do you have examples of other projects that have an effective alert system? > > > Thanks, > Chris.
On Wed, 2022-11-16 at 10:50 +0000, Filipe Manana wrote: > There will always be users who'll miss such alerts, many don't read > all the emails in this list, many are not even subscribed to the > mailing list, etc. Sure... and it will never be possible to have a system which really reaches 100% of the desired users. But this isn't only the case for a something that notices about data corruption issues, it's also the case for any other warning system: - civil defense via sirens (people may life to far away, wear headphones with loud music, be deaf, sleep with earmuffs) via apps (they often simply fail or are overloaded, people may not have a smartphone at all, it maybe powered off) - software security advisories (again, people may not be subscribed to such mailing lists, may not run upgrades regularly respectively check for security updates in some automated fashion, or attackers may try to specifically block such information from certain people) - etc. pp. - and even *if* the information reaches someone, it's still not guaranteed that he cares about or understands it well enough I don't think the goal is ever about reaching everyone - the goal is having a simple way of reaching those who care. Because whether it's storage farm admins who keep important scientific data on btrfs or some end user who values his precious collection of pictures, etc. ... it would be quite good for them to be able to react in cases of (especially silent) data corruption. And I should emphasis, that this is in no way targeted for lazy people who don't make backups. I for example do have some rather sophisticated backup strategy, but if there's silent data corruption it's often quite hard to tell whether the data is still valid, even if one does things like storing hash sums of them (and verifying these of some 10TB of data already takes several days). > Do you have examples of other projects that have an effective alert > system? Well, as I've said, not at the filesystem level. But it's e.g. common practise for security incidents. Upstream developers put quite some effort into finding and fixing such bugs, which is appreciated, but IMO that is a bit ridiculed by the fact that people then may still loose their data, just because they never take notice about such issues, when they'd still have valid backups. Cheers, Chris.
On Tue, Nov 15, 2022 at 04:25:23PM +0000, fdmanana@kernel.org wrote: > From: Filipe Manana <fdmanana@suse.com> > > When using a v2 stream, with encoded writes, if the receiver fails to > perform an encoded write, it fallbacks to decompressing the data and then > write it using regular buffered IO. However that logic is currenty buggy, > and it can result in writing less data than expected or no data at all. > This results in a silent data loss. > > Patch 3/3 fixes that bug and has all the details about how/why it happens, > while previous patches just added debug messages to the callbacks for > encoded writes and fallocate, which are currently missing and are very > useful for debugging. > > Filipe Manana (3): > btrfs-progs: receive: add debug messages when processing encoded writes > btrfs-progs: receive: add debug messages when processing fallocate > btrfs-progs: receive: fix silent data loss after fall back from encoded write Added to devel, thanks.
From: Filipe Manana <fdmanana@suse.com> When using a v2 stream, with encoded writes, if the receiver fails to perform an encoded write, it fallbacks to decompressing the data and then write it using regular buffered IO. However that logic is currenty buggy, and it can result in writing less data than expected or no data at all. This results in a silent data loss. Patch 3/3 fixes that bug and has all the details about how/why it happens, while previous patches just added debug messages to the callbacks for encoded writes and fallocate, which are currently missing and are very useful for debugging. Filipe Manana (3): btrfs-progs: receive: add debug messages when processing encoded writes btrfs-progs: receive: add debug messages when processing fallocate btrfs-progs: receive: fix silent data loss after fall back from encoded write cmds/receive.c | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-)