[v2,7/7] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option
diff mbox

Message ID 20170911063612.32114-8-quwenruo.btrfs@gmx.com
State New
Headers show

Commit Message

Qu Wenruo Sept. 11, 2017, 6:36 a.m. UTC
Add extra limitation explained for --rootdir option, including:
1) Size limitation
   Now I decide to follow "mkfs.ext4 -d" behavior, so we user is
   responsible to make sure the block device/file is large enough.

2) Read permission
   If user can't read the content, mkfs will just fail.
   So user is also responsible to make sure to have enough privilege.

3) Extra warning about the behavior change
   Since we we don't shrink fs the create file image, add such warning
   in documentation.

Signed-off-by: Qu Wenruo <quwenruo.btrfs@gmx.com>
---
 Documentation/mkfs.btrfs.asciidoc | 11 +++++++++++
 1 file changed, 11 insertions(+)

Comments

David Sterba Sept. 12, 2017, 5:03 p.m. UTC | #1
On Mon, Sep 11, 2017 at 03:36:12PM +0900, Qu Wenruo wrote:
> Add extra limitation explained for --rootdir option, including:
> 1) Size limitation
>    Now I decide to follow "mkfs.ext4 -d" behavior, so we user is
>    responsible to make sure the block device/file is large enough.

That's fine and I'm ok for changing that to be the default now. But the
shrinking could be still useful, if I read the usecase that Austin
mentioned in the previous thread correctly.

Say I want to prepare a minimal image but will provide a large file at
the beginning because I don't know what's the resulting size going to
be. In this case, something like

$ mkfs.btrfs --rootdir dir/ --minimize image

would make it possible. I'm not sure if somebody hasn't proposed that
already, but this sounds good to me and I think it's a valid usecase.
This means adding back the patch from V1 and then making it optional.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Goffredo Baroncelli Sept. 12, 2017, 5:50 p.m. UTC | #2
On 09/12/2017 07:03 PM, David Sterba wrote:
> Say I want to prepare a minimal image but will provide a large file 
                                                ^^^^^^^^^^^^^^^^^^^^^^

I think that if the target is a file AND --minimize is passed, it is a reasonable expectation that the file is created "on the fly" and grown up to what needed.  

What I mean is that "--minimize" is passed (and a file is passed), mkfs.btrfs should 
a) create the file if it doesn't exist, and avoid any check about its length
b) truncate the file at the end

unfortunately the checks are in more places, so removing these checks is a quite intrusive change...

> at
> the beginning because I don't know what's the resulting size going to
> be. In this case, something like
> 
> $ mkfs.btrfs --rootdir dir/ --minimize image

BR
G.Baroncelli
David Sterba Sept. 12, 2017, 6:07 p.m. UTC | #3
On Tue, Sep 12, 2017 at 07:50:19PM +0200, Goffredo Baroncelli wrote:
> On 09/12/2017 07:03 PM, David Sterba wrote:
> > Say I want to prepare a minimal image but will provide a large file 
>                                                 ^^^^^^^^^^^^^^^^^^^^^^
> 
> I think that if the target is a file AND --minimize is passed, it is a
> reasonable expectation that the file is created "on the fly" and grown
> up to what needed.  
> 
> What I mean is that "--minimize" is passed (and a file is passed), mkfs.btrfs should 
> a) create the file if it doesn't exist, and avoid any check about its length
> b) truncate the file at the end

I think the farthest-offset write will extend the file size, so we
should not truncate it, in the sense 'make it smaller than it is', but
rather 'align it to the size where the filesystem expects live data".

Eg. if we write a blockgroup header to the beginning of 256MB, but fill
only a few kilobytes, effectively writing to some range, say, [0..32MB].
Then reads past the 32MB (eg. through loop device) would fail.

As long as the file is zero, minimizing will be implicit. My example
with the provided large file was wrong in retrospect. I'll think about
that more.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Qu Wenruo Sept. 13, 2017, 12:32 a.m. UTC | #4
On 2017年09月13日 02:07, David Sterba wrote:
> On Tue, Sep 12, 2017 at 07:50:19PM +0200, Goffredo Baroncelli wrote:
>> On 09/12/2017 07:03 PM, David Sterba wrote:
>>> Say I want to prepare a minimal image but will provide a large file
>>                                                  ^^^^^^^^^^^^^^^^^^^^^^
>>
>> I think that if the target is a file AND --minimize is passed, it is a
>> reasonable expectation that the file is created "on the fly" and grown
>> up to what needed.
>>
>> What I mean is that "--minimize" is passed (and a file is passed), mkfs.btrfs should
>> a) create the file if it doesn't exist, and avoid any check about its length
>> b) truncate the file at the end
> 
> I think the farthest-offset write will extend the file size, so we
> should not truncate it, in the sense 'make it smaller than it is', but
> rather 'align it to the size where the filesystem expects live data".

Unfortunately, that's not the case.

Current implementation must determine its size at the very beginning.

That's to say, if mkfs determines that it's going to use 1G size, then 
current btrfs-progs will be guaranteed that no write over 1G boundary.
(Unless there is some hidden bug)

So if we are realling going to implement "--minimize" option, we are 
doing asymmetric behavior:

1) For fs smaller than file
    Good, we truncate it

2) For fs larger than file
    Bad, we will get ENOSPC error.

One workaround I mentioned in V1 is to make a large enough sparse file 
first and then truncate it.

But such workaround has some problem, especially related to the chunk 
allocator.

For a 1G sparse file as example, we will have about 100M data chunk.
While for 128M space file, we will have about 10M data chunk.
That will cause a lot of space wasted if using large spare file and then 
truncating it than using a small existing file.

Considering the all the variables involved, I choose the KISS method.
Let user to handle it, and mkfs.btrfs --rootdir will just do the copy work.

If user really wants to do a minimal image, the best (AFAIK the only 
correct) method is to implement btrfs-progs balance (offline balanace) 
to compact all these meta and data, then truncate it (offline resize).

The initial --rootdir is doing too many things at the same time in mkfs.
And that's why I want to make it simple.

BTW, just as I stated in cover letter, "mkfs.ext4" -d is doing it easy 
as what I did here, and the complex work is handled by out of e2fsprogs 
utility, genext2fs.



And there may be some use case relying on this truncation, but hey, if 
the need is really high, there should already be a lot of complains 
about --rootdir shrinking fs, and more tested/documented --rootdir 
implementation.

So it's not really worth it to implement a complex code base to handle 
something while there is not much user base.

Thanks,
Qu

> 
> Eg. if we write a blockgroup header to the beginning of 256MB, but fill
> only a few kilobytes, effectively writing to some range, say, [0..32MB].
> Then reads past the 32MB (eg. through loop device) would fail.
> 
> As long as the file is zero, minimizing will be implicit. My example
> with the provided large file was wrong in retrospect. I'll think about
> that more.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Qu Wenruo Sept. 16, 2017, 1:15 a.m. UTC | #5
On 2017年09月15日 23:48, David Sterba wrote:
> On Fri, Sep 15, 2017 at 09:24:19PM +0800, Qu Wenruo wrote:
>>> I'm going to review & merge this series to devel. Tests and
>>> documentation should be updated to make the usecase clear.
>>
>> I'm happy to address any comment, both code and doc/test.
> 
> For the tests I'd like to see:
> 
> * with file target, try non-existent file, zero-length file, too-small
>    file and large-enough

No problem.

But it would be better to define the expected behavior first.
I'll show the behavior difference below to see if you're OK with it.

For non-existent file:
New: Fail, no such file at the very beginning
Old: Create new one

Zero-length:
New: Fail, too small file at the very beginning
Old: Expand file and continue mkfs

Too-small:
New: Fail, showing -28 errno half way
Old: Same as zero length.

Large-enough:
New: Success, without shrink
Old: Success, shrink

> 
> * for block device target, I think loop device would do, too-small and
>    large-enough

Same as above too-small and large-enough case.

> 
> * after the test, verify that the files have same size and contents

I'll reuse convert facilities to do that.

And I'll add one extra test case:
No enough privilege to read some file in the directory:
New: Fail gracefully
Old: Fail with BUG_ON.

If you're OK with the behavior difference, I'll create new test cases 
for mkfs --rootdir.

Thanks,
Qu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch
diff mbox

diff --git a/Documentation/mkfs.btrfs.asciidoc b/Documentation/mkfs.btrfs.asciidoc
index d53d9e26..843e25cd 100644
--- a/Documentation/mkfs.btrfs.asciidoc
+++ b/Documentation/mkfs.btrfs.asciidoc
@@ -106,6 +106,17 @@  Please see the mount option 'discard' for that in `btrfs`(5).
 *-r|--rootdir <rootdir>*::
 Populate the toplevel subvolume with files from 'rootdir'.  This does not
 require root permissions and does not mount the filesystem.
++
+With this option, only one device can be specified.
++
+NOTE: User should make sure the block device/file has large enough space to
+contain the source directory and has enough previllege to read source directory.
+Or mkfs will just fail.
++
+WARNING: Before v4.14 btrfs-progs, *--rootdir* will shrink the filesystem,
+prevent user to make use of the remaining space.
+In v4.14 btrfs-progs, this behavior is changed, and will not shrink the fs.
+The result should be the same as `mkfs`, `mount` and then `cp -r`.
 
 *-O|--features <feature1>[,<feature2>...]*::
 A list of filesystem features turned on at mkfs time. Not all features are