improve documentation of snapshot unaware defrag
diff mbox

Message ID 1451185407-11422-1-git-send-email-mail@christoph.anton.mitterer.name
State New
Headers show

Commit Message

Christoph Anton Mitterer Dec. 27, 2015, 3:03 a.m. UTC
In btrfs-filesystem(8), improved the documentation of snapshot unaware
defragmentation and included the exact kernel version numbers being affected as
well as the possible effects.
No longer use th word "unlink" which is easily understood as "deleting a file".
Moved the warning more to the beginning of "defragment" subcommand's
documentation where it's more visible to readers.

Added the same warning to the "autodefrag" option of btrfs-mount(5).

Signed-off-by: Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>
---
 Documentation/btrfs-filesystem.asciidoc | 11 +++++++----
 Documentation/btrfs-mount.asciidoc      |  7 +++++++
 2 files changed, 14 insertions(+), 4 deletions(-)

Comments

Christoph Anton Mitterer Dec. 27, 2015, 3:10 a.m. UTC | #1
On Sun, 2015-12-27 at 04:03 +0100, Christoph Anton Mitterer wrote:
> -WARNING: defragmenting with kernels up to 2.6.37 will unlink COW-ed
Perhaps someone can also check the above.

I was looking through the git history, but, couldn't find anything wrt
2.6.37...
The commit's I've basically searched for in the non-stable repo were
38c227d87c49ad5d173cb5d4374d49acec6a495d (adding the ref-link aware defrag)
8101c8dbf6243ba517aab58d69bf1bc37d8b7b9c (removing it)

But maybe I've missed something.


Also, the wiki mentioned it for 3.13, I changed that:
https://btrfs.wiki.kernel.org/index.php?title=Changelog&action=historysubmit&diff=29765&oldid=29697

Please correct if wrong.

HTH,
Chris
Duncan Dec. 27, 2015, 7:09 a.m. UTC | #2
Christoph Anton Mitterer posted on Sun, 27 Dec 2015 04:03:27 +0100 as
excerpted:

[Rewrapped here but all added lines.]

> +WARNING: Defragmenting with Linux kernel versions < 3.9 or ? 3.14-rc2
> as well as 
> +with Linux stable kernel versions ? 3.10.31, ? 3.12.12 or ?
> 3.13.4 will break up 
> +the ref-links of CoW data (for example files copied with 
> `cp --reflink`,
> +snapshots or de-duplicated data).
> +This may cause considerable increase of space usage depending on the
> broken up +ref-links.

Thanks.  I had looked at that a few times and thought it needed updated, 
but I think it hadn't reached my pain threshold yet[1], so I hadn't yet 
posted about it.

Glad it reached someone's pain threshold. =:^)

---
[1] Pain threshold:  Or more like, I was always doing something else at 
the time, which is probably everybody else's excuse too.  But by contrast 
it can be noted that I posted right away when I noticed the mkfs.btrfs 
manpage totally lost raid1 mode with one update, because I use it, 
regardless of what else I was doing.  I guess that must have hit my pain 
threshold...
Christoph Anton Mitterer Dec. 28, 2015, 12:50 a.m. UTC | #3
On Sun, 2015-12-27 at 07:09 +0000, Duncan wrote:
> raid1 mode
I wonder when that reaches my pain threshold... and I submit a patch
that renames it "notreallyraid1" in all places ;-)

Cheers,
Chris.
Hugo Mills Dec. 28, 2015, 1:58 a.m. UTC | #4
On Mon, Dec 28, 2015 at 01:50:09AM +0100, Christoph Anton Mitterer wrote:
> On Sun, 2015-12-27 at 07:09 +0000, Duncan wrote:
> > raid1 mode
> I wonder when that reaches my pain threshold... and I submit a patch
> that renames it "notreallyraid1" in all places ;-)

   Isn't this an FAQ already? There is already a patch to rename the
RAID modes. It's been sitting in the progs patch queue for about 2
years, because none of the senior devs has acked it yet (since it's a
big user-visible change).

   Hugo.
Christoph Anton Mitterer Dec. 28, 2015, 2:07 a.m. UTC | #5
On Mon, 2015-12-28 at 01:58 +0000, Hugo Mills wrote:
>    Isn't this an FAQ already? There is already a patch to rename the
> RAID modes. It's been sitting in the progs patch queue for about 2
> years, because none of the senior devs has acked it yet (since it's a
> big user-visible change).
Uhm... yeah, it's a bit invasive... but that happens when such improper
naming is done in the first place :-/

It's similar to when tools wrongly or ambiguously use SI prefixes,
instead of proper kB, MB, GB, etc. (for base 1000) respectively KiB,
MiB, GiB, etc. (for base 1024).
Especially just using K,M,G, is simply evil and should lead to public
punishment ;-)
I'm also not really fond of what btrfs took over from LVM,
namely kKmMgGtTpPeE,... it's ambiguous or at least unclean as well...


Best would be probably if we don't use "raid" as names at all (or just
as aliases for the actual canonical names), but rather describe what's
actually...
E.g.
classic RAID1 = mirror
btrfs RAID1   = dup2 or something similar like clone2, replica2
classic RAID0 = striped
RAID5/6       = parityN



Cheers,
Chris.
Duncan Dec. 28, 2015, 2:51 a.m. UTC | #6
Christoph Anton Mitterer posted on Mon, 28 Dec 2015 01:50:09 +0100 as
excerpted:

> On Sun, 2015-12-27 at 07:09 +0000, Duncan wrote:

>> raid1 mode

> I wonder when that reaches my pain threshold... and I submit a patch
> that renames it "notreallyraid1" in all places ;-)

I've seen two responses to that, both correct, AFAIK.

1) Btrfs very specifically and deliberately uses *lowercase* raidN in 
part to make that distinction, as the btrfs variants are chunk-level (and 
designed so that at some point in the future they can be subvolume and/or 
file level), not device-level (and at that future point, not necessarily 
filesystem level either).

As we've seen in discussion in other threads, for raid10 in particular, 
that makes a profound difference in robustness in the multi-device 
failure case.

2) Regarding btrfs raid1 and raid10's current very specific two-way-
mirroring in particular, limiting to two-way-mirroring in the 3+ devices 
case is well within established definitions and historic usage.  
Apparently, the N-devices = N-way-mirroring usage is relatively new, 
arguably first popularized by Linux mdraid, after which various hardware 
raid suppliers also implemented it due to competitive pressure.  But only 
two-way-mirroring is required by the RAID-1 definition.

Even were that not the case, point #1, btrfs' very specific use of 
*lowercase* raid1, still covers the two-way-limitation case just as well 
as it covers the chunk-level case.


That said, that the limited pair-mirroring btrfs implements even in the 
3+ device case still meets formal RAID-1 definitions was originally news 
to me as well, however well I might now accept the fact.  But once my 
earlier naive assumptions were corrected, the remaining clarification 
issues fell below my pain threshold.


But for those for whom it's still very close to their pain threshold, due 
to the above a patch effectively doing s/raid1/notreallyraid1/g is 
unlikely to be accepted.  Much more likely to be accepted would be a 
patch to the btrfs-balance and mkfs.btrfs manpages adding note, 
preferably accounting for the raid10 situation as well, explaining that 
btrfs raid (lowercase) isn't RAID (uppercase) in the the traditional 
sense, that it's chunk-scope not device-scope and that this has 
implications in for instance robustness in the raid10 multi-device 
failure case, and that both raid1 and raid10 are (currently) limited to 
two-way-mirroring.

Meanwhile, for anyone considering writing that patch, I'd also strongly 
recommend that the two-way-mirroring wording is separated out, at least 
onto its own lines if not a separate paragraph, so it can be cleanly 
deleted and/or modified once N-way-mirroring is introduced as a feature, 
without having to rewrite the chunk-level and raid10 bit as well.
Christoph Anton Mitterer Dec. 28, 2015, 3:03 a.m. UTC | #7
On Mon, 2015-12-28 at 02:51 +0000, Duncan wrote:
> 1) Btrfs very specifically and deliberately uses *lowercase* raidN
> in part to make that distinction, as the btrfs variants are chunk-
> level (and designed so that at some point in the future they can be
> subvolume and/or file level), not device-level (and at that future
> point, not necessarily filesystem level either).
I guess no "normal" user would expect or understand that lower/upper
case would imply any distinction.


> 2) Regarding btrfs raid1 and raid10's current very specific two-way-
> mirroring in particular, limiting to two-way-mirroring in the 3+
> devices 
> case is well within established definitions and historic usage.  
> Apparently, the N-devices = N-way-mirroring usage is relatively new, 
> arguably first popularized by Linux mdraid, after which various
> hardware 
> raid suppliers also implemented it due to competitive pressure.  But
> only 
> two-way-mirroring is required by the RAID-1 definition.
No, this is not true.

This http://www.eecs.berkeley.edu/Pubs/TechRpts/1987/CSD-87-391.pdf is
the original paper on RAID.
Chapter 7 describes RAID1 and the clearly says "all disks are
duplicated" as well as "Level 1 RAID has only one data disk".

I wouldn't know any single case of a HW RAID controller (and we've had
quite a few of them here at the Tier2) or other software implementation
where RAID1 had another meaning than "N disks, N mirrors".



> Even were that not the case, point #1, btrfs' very specific use of 
> *lowercase* raid1, still covers the two-way-limitation case just as
> well 
> as it covers the chunk-level case.
Hmm wouldn't still change anything, IMHO,... saying "lower case RAID is
something different than upper case RAID" would be just a bit ...
uhm... weird.

Actually, because btrfs doing it at the chunk level (while RAID being
at the device level), proves while my point that "raid" or "RAID" or
any other lower/upper case combination shouldn't be used at all.


Cheers,
Chris.
Duncan Dec. 28, 2015, 6:12 a.m. UTC | #8
Christoph Anton Mitterer posted on Mon, 28 Dec 2015 04:03:05 +0100 as
excerpted:

> On Mon, 2015-12-28 at 02:51 +0000, Duncan wrote:

>> 1) Btrfs very specifically and deliberately uses *lowercase* raidN in
>> part to make that distinction, as the btrfs variants are chunk- level
>> (and designed so that at some point in the future they can be subvolume
>> and/or file level), not device-level (and at that future point, not
>> necessarily filesystem level either).

> I guess no "normal" user would expect or understand that lower/upper
> case would imply any distinction.

I /could/ argue the case based on definition of the "normal" in "normal 
user", but I won't, as in any case I agree with you at least to the 
extent that a better explanation of the details should eventually be 
found both on the wiki (where it is arguably already covered in the 
sysadmin's and multiple devices pages) and in the btrfs-balance and 
mkfs.btrfs manpages (where it remains uncovered).

>> 2) Regarding btrfs raid1 and raid10's current very specific two-way-
>> mirroring in particular, limiting to two-way-mirroring in the 3+
>> devices case is well within established definitions and historic usage.
>> Apparently, the N-devices = N-way-mirroring usage is relatively new,
>> arguably first popularized by Linux mdraid, after which various
>> hardware raid suppliers also implemented it due to competitive
>> pressure.  But only two-way-mirroring is required by the RAID-1
>> definition.

> No, this is not true.
> 
> This http://www.eecs.berkeley.edu/Pubs/TechRpts/1987/CSD-87-391.pdf is
> the original paper on RAID.
> Chapter 7 describes RAID1 and the clearly says "all disks are
> duplicated" as well as "Level 1 RAID has only one data disk".

Kudos for digging up the reference. =:^)

Never-the-less, I (and others from which I got the position), believe 
your interpretation is arguably in error.  More precisely...

1) In the context of the Level 1 RAID discussed in chapter 7, from 
earlier in the paper, in chapter 6, introducing RAID, on page six of the 
paper which is page 8 of the PDF (quotes here between the >>>>> and <<<<< 
demarcs, [...] indicating elision, as traditional):

>>>>>

Reliability: Our basic approach will be to break the arrays into 
reliability groups, with each group having extra "check" disks containing 
redundant information. [...]

Here are some other terms that we use:

D = total number of disks with data (not including the extra check disks);

G = number of data disks in a group (not including the extra check disks);

[...]

C = number of check disks in a group;

<<<<<

That's the context, disks grouped for reliability, with data and check 
disks in a group, but multiple such groups.

Then later in the paper, in the First Level RAID discussion in chapter 7, 
starting on page 9 of the paper, page 11 of the pdf:

>>>>>

Mirrored disks are a traditional approach for improving reliability of 
magnetic disks.  This is the most expensive option since all disks are 
duplicated (G=1 and C=1), and every write to a data disk is also a write 
to a check disk.

<<<<<

With the definitions and context above, we see that the "(G=1 and C=1)" 
defines First Level RAID as exactly one data disk and one check disk in a 
reliability group, with multiple such groups.  So yes, it has "only one 
data disk"... in a defined context where that's per group, with exactly 
one check disk as well, with multiple groups, such that each write to a 
group writes to exactly one data disk and one check disk, but a full 
write may be to many groups.

This can be further seen by examining Table II on page 10 of the paper 
(12 of the pdf), where total number of disks is declared to be 2D (twice 
the number of data disks, based on the above definition of D), and usable 
storage capacity to be 50%.

Further, in the commentary on the same page, "Since a Level 1 RAID has 
only one data disk in its group, we assume that the large transfer 
requires the same number of disk acting in concert as found in groups of 
the higher level RAIDs: 10 to 25 disks."  Again, that emphasizes the per-
group aspect of the G=1, C=1 definition, and the fact that there's many 
such groups in the deployment.

Finally, "Duplicating all disks can mean doubling the cost of the 
database system or using only 50% of the disk storage capacity."  Again, 
very clearly pair-mirroring, with many such pair-mirrors in the array.

Which, other than the per-chunk rather than per-disk granularity, is 
_exactly_ what btrfs does.

It would actually seem that the N-way-mirroring, where N=number-of-
devices, usage of so-called raid1 is out of kilter with the original 
definition, not btrfs' very specific two-way-mirroring, regardless of the 
number of devices, which is actually very close to the original 
definition of two devices per groups, many such groups in an array.

Tho I'll certainly agree that in today's usage, RAID-1 certainly 
/incorporates/ the N-way-mirroring usage, and would even agree that, 
within my rather limited exposure at least, it's the more common usage.  
But that doesn't make it the original usage, nor does it mean that 
there's no room in today's broader definition for the original usage, 
which then must remain as valid as the broader usage, today.

So other than the per-chunk scope, btrfs raid1 would indeed seem to be 
real RAID-1.

Never-the-less, given the broader usage today, there's definitely a need 
for some word of explanation in the mkfs.btrfs and btrfs-balance 
manpages.  I'll agree there, but then I never disagreed with that in the 
first place, and indeed, that was my opinion from when I myself thought 
pair-mirroring wasn't proper raid1 -- that much hasn't changed.


Meanwhile, I've actually quoted about 50% of the original paper's raid1 
discussion in the above.  The Level 1 RAID discussion is actually quite 
short, under a double-spaced page in the original paper, which itself is 
only 26 pdf pages long, including two pages of title and blank page at 
the beginning (thus the pdf page numbering being two pages higher than 
the paper's page numbering), and two plus pages of acknowledgments, 
references and appendix at the end, so only 22 pages of well spaced 
actual content.  Those who haven't clicked thru to actually read it may 
be interested in doing so.  Here it is again for convenience. =:^)

http://www.eecs.berkeley.edu/Pubs/TechRpts/1987/CSD-87-391.pdf


> I wouldn't know any single case of a HW RAID controller (and we've had
> quite a few of them here at the Tier2) or other software implementation
> where RAID1 had another meaning than "N disks, N mirrors".

That may be.  I'm sure you have more experience with it than I do.  But 
that doesn't change the original definition, or mean that usage 
consistent with that original definition is incorrect, even if uncommon 
today.

>> Even were that not the case, point #1, btrfs' very specific use of
>> *lowercase* raid1, still covers the two-way-limitation case just as
>> well as it covers the chunk-level case.

> Hmm wouldn't still change anything, IMHO,... saying "lower case RAID is
> something different than upper case RAID" would be just a bit ... uhm...
> weird.
> 
> Actually, because btrfs doing it at the chunk level (while RAID being at
> the device level), proves while my point that "raid" or "RAID" or any
> other lower/upper case combination shouldn't be used at all.

I don't actually disagree with you there.  Weird it is, agreed.  But it's 
also the case, at least currently, and based on what Hugo said about a 
patch to change the terminology being in limbo for two years, during 
which the currently used terminology has become even more entrenched as 
btrfs is widely deployed in distro installations now (even if it isn't 
entirely stable yet), that it's unlikely to change.  The best that could 
be done at this point is make raid1 an alias for something else, but even 
then, I'd guess the raid1 terminology would continue pretty much 
unabated, since it's already widely used and well entrenched in the 
various google engines as well as the archives for this list.
Duncan Dec. 28, 2015, 9:12 a.m. UTC | #9
Hugo Mills posted on Mon, 28 Dec 2015 01:58:07 +0000 as excerpted:

> On Mon, Dec 28, 2015 at 01:50:09AM +0100, Christoph Anton Mitterer
> wrote:
>> On Sun, 2015-12-27 at 07:09 +0000, Duncan wrote:
>> > raid1 mode
>> I wonder when that reaches my pain threshold... and I submit a patch
>> that renames it "notreallyraid1" in all places ;-)
> 
> Isn't this an FAQ already? There is already a patch to rename the
> RAID modes. It's been sitting in the progs patch queue for about 2
> years, because none of the senior devs has acked it yet (since it's a
> big user-visible change).

I don't see it in the FAQ, but I see hints on both the sysadmin's guide 
and the usecases pages.

(Either the wiki or firefox seems to be having certificate problems ATM 
and all I'm getting is an OCSP response has an invalid sig error.  But 
the resurrect this page extension to the rescue, click the resurrect via 
google and I get it.  Links has no problem loading the page, but lynx 
does, so it's not just firefox.)

UseCases:

First section is RAID, first question there is on creating a raid1 mirror 
in btrfs.  It has this to say at the end of the answer:

>>>>>

NOTE This does not do the 'usual thing' for 3 or more drives. Until "N-
Way" (traditional) RAID-1 is implemented: Loss of more than one drive 
might crash the array. For now, RAID-1 means 'one copy of what's 
important exists on two of the drives in the array no matter how many 
drives there may be in it'.

<<<<<

SysadminGuide:

Second section is data usage and allocation.  First subsection there is 
RAID and data replication.  The first paragraph there is:

>>>>>

Btrfs's "RAID" implementation bears only passing resemblance to 
traditional RAID implementations. Instead, btrfs replicates data on a per-
chunk basis. If the filesystem is configured to use "RAID-1", for 
example, chunks are allocated in pairs, with each chunk of the pair being 
taken from a different block device. Data written to such a chunk pair 
will be duplicated across both chunks.

<<<<<

The multi-device page has little theoretical discussion, only discussing 
current status and having a bunch of specific commandline examples.

Patch
diff mbox

diff --git a/Documentation/btrfs-filesystem.asciidoc b/Documentation/btrfs-filesystem.asciidoc
index 31cd51b..600bbac 100644
--- a/Documentation/btrfs-filesystem.asciidoc
+++ b/Documentation/btrfs-filesystem.asciidoc
@@ -55,6 +55,13 @@  if the free space is too fragmented.
 Use 0 to take the kernel default, which is 256kB but may change in the future.
 You can also turn on compression in defragment operations.
 +
+WARNING: Defragmenting with Linux kernel versions < 3.9 or ? 3.14-rc2 as well as
+with Linux stable kernel versions ? 3.10.31, ? 3.12.12 or ? 3.13.4 will break up
+the ref-links of CoW data (for example files copied with `cp --reflink`,
+snapshots or de-duplicated data).
+This may cause considerable increase of space usage depending on the broken up
+ref-links.
++
 `Options`
 +
 -v::::
@@ -79,10 +86,6 @@  target extent size, do not touch extents bigger than <size>
 For <start>, <len>, <size> it is possible to append
 units designator: \'K', \'M', \'G', \'T', \'P', or \'E', which represent
 KiB, MiB, GiB, TiB, PiB, or EiB, respectively. Case does not matter.
-+
-WARNING: defragmenting with kernels up to 2.6.37 will unlink COW-ed copies of data,
-don't use it if you use snapshots, have de-duplicated your data or made
-copies with `cp --reflink`.
 
 *label* [<dev>|<mountpoint>] [<newlabel>]::
 Show or update the label of a filesystem.
diff --git a/Documentation/btrfs-mount.asciidoc b/Documentation/btrfs-mount.asciidoc
index 39215a8..d364594 100644
--- a/Documentation/btrfs-mount.asciidoc
+++ b/Documentation/btrfs-mount.asciidoc
@@ -26,6 +26,13 @@  MOUNT OPTIONS
 	Auto defragmentation detects small random writes into files and queue
 	them up for the defrag process.  Works best for small files;
 	Not well suited for large database workloads.
+	+
+	WARNING: Defragmenting with Linux kernel versions < 3.9 or ? 3.14-rc2 as
+	well as with Linux stable kernel versions ? 3.10.31, ? 3.12.12 or
+	? 3.13.4 will break up the ref-links of CoW data (for example files
+	copied with `cp --reflink`, snapshots or de-duplicated data).
+	This may cause considerable increase of space usage depending on the
+	broken up ref-links.
 
 *check_int*::
 *check_int_data*::