| Message ID | 20170911063612.32114-8-quwenruo.btrfs@gmx.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, Sep 11, 2017 at 03:36:12PM +0900, Qu Wenruo wrote: > Add extra limitation explained for --rootdir option, including: > 1) Size limitation > Now I decide to follow "mkfs.ext4 -d" behavior, so we user is > responsible to make sure the block device/file is large enough. That's fine and I'm ok for changing that to be the default now. But the shrinking could be still useful, if I read the usecase that Austin mentioned in the previous thread correctly. Say I want to prepare a minimal image but will provide a large file at the beginning because I don't know what's the resulting size going to be. In this case, something like $ mkfs.btrfs --rootdir dir/ --minimize image would make it possible. I'm not sure if somebody hasn't proposed that already, but this sounds good to me and I think it's a valid usecase. This means adding back the patch from V1 and then making it optional. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 09/12/2017 07:03 PM, David Sterba wrote: > Say I want to prepare a minimal image but will provide a large file ^^^^^^^^^^^^^^^^^^^^^^ I think that if the target is a file AND --minimize is passed, it is a reasonable expectation that the file is created "on the fly" and grown up to what needed. What I mean is that "--minimize" is passed (and a file is passed), mkfs.btrfs should a) create the file if it doesn't exist, and avoid any check about its length b) truncate the file at the end unfortunately the checks are in more places, so removing these checks is a quite intrusive change... > at > the beginning because I don't know what's the resulting size going to > be. In this case, something like > > $ mkfs.btrfs --rootdir dir/ --minimize image BR G.Baroncelli
On Tue, Sep 12, 2017 at 07:50:19PM +0200, Goffredo Baroncelli wrote: > On 09/12/2017 07:03 PM, David Sterba wrote: > > Say I want to prepare a minimal image but will provide a large file > ^^^^^^^^^^^^^^^^^^^^^^ > > I think that if the target is a file AND --minimize is passed, it is a > reasonable expectation that the file is created "on the fly" and grown > up to what needed. > > What I mean is that "--minimize" is passed (and a file is passed), mkfs.btrfs should > a) create the file if it doesn't exist, and avoid any check about its length > b) truncate the file at the end I think the farthest-offset write will extend the file size, so we should not truncate it, in the sense 'make it smaller than it is', but rather 'align it to the size where the filesystem expects live data". Eg. if we write a blockgroup header to the beginning of 256MB, but fill only a few kilobytes, effectively writing to some range, say, [0..32MB]. Then reads past the 32MB (eg. through loop device) would fail. As long as the file is zero, minimizing will be implicit. My example with the provided large file was wrong in retrospect. I'll think about that more. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2017年09月13日 02:07, David Sterba wrote: > On Tue, Sep 12, 2017 at 07:50:19PM +0200, Goffredo Baroncelli wrote: >> On 09/12/2017 07:03 PM, David Sterba wrote: >>> Say I want to prepare a minimal image but will provide a large file >> ^^^^^^^^^^^^^^^^^^^^^^ >> >> I think that if the target is a file AND --minimize is passed, it is a >> reasonable expectation that the file is created "on the fly" and grown >> up to what needed. >> >> What I mean is that "--minimize" is passed (and a file is passed), mkfs.btrfs should >> a) create the file if it doesn't exist, and avoid any check about its length >> b) truncate the file at the end > > I think the farthest-offset write will extend the file size, so we > should not truncate it, in the sense 'make it smaller than it is', but > rather 'align it to the size where the filesystem expects live data". Unfortunately, that's not the case. Current implementation must determine its size at the very beginning. That's to say, if mkfs determines that it's going to use 1G size, then current btrfs-progs will be guaranteed that no write over 1G boundary. (Unless there is some hidden bug) So if we are realling going to implement "--minimize" option, we are doing asymmetric behavior: 1) For fs smaller than file Good, we truncate it 2) For fs larger than file Bad, we will get ENOSPC error. One workaround I mentioned in V1 is to make a large enough sparse file first and then truncate it. But such workaround has some problem, especially related to the chunk allocator. For a 1G sparse file as example, we will have about 100M data chunk. While for 128M space file, we will have about 10M data chunk. That will cause a lot of space wasted if using large spare file and then truncating it than using a small existing file. Considering the all the variables involved, I choose the KISS method. Let user to handle it, and mkfs.btrfs --rootdir will just do the copy work. If user really wants to do a minimal image, the best (AFAIK the only correct) method is to implement btrfs-progs balance (offline balanace) to compact all these meta and data, then truncate it (offline resize). The initial --rootdir is doing too many things at the same time in mkfs. And that's why I want to make it simple. BTW, just as I stated in cover letter, "mkfs.ext4" -d is doing it easy as what I did here, and the complex work is handled by out of e2fsprogs utility, genext2fs. And there may be some use case relying on this truncation, but hey, if the need is really high, there should already be a lot of complains about --rootdir shrinking fs, and more tested/documented --rootdir implementation. So it's not really worth it to implement a complex code base to handle something while there is not much user base. Thanks, Qu > > Eg. if we write a blockgroup header to the beginning of 256MB, but fill > only a few kilobytes, effectively writing to some range, say, [0..32MB]. > Then reads past the 32MB (eg. through loop device) would fail. > > As long as the file is zero, minimizing will be implicit. My example > with the provided large file was wrong in retrospect. I'll think about > that more. > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2017年09月15日 23:48, David Sterba wrote: > On Fri, Sep 15, 2017 at 09:24:19PM +0800, Qu Wenruo wrote: >>> I'm going to review & merge this series to devel. Tests and >>> documentation should be updated to make the usecase clear. >> >> I'm happy to address any comment, both code and doc/test. > > For the tests I'd like to see: > > * with file target, try non-existent file, zero-length file, too-small > file and large-enough No problem. But it would be better to define the expected behavior first. I'll show the behavior difference below to see if you're OK with it. For non-existent file: New: Fail, no such file at the very beginning Old: Create new one Zero-length: New: Fail, too small file at the very beginning Old: Expand file and continue mkfs Too-small: New: Fail, showing -28 errno half way Old: Same as zero length. Large-enough: New: Success, without shrink Old: Success, shrink > > * for block device target, I think loop device would do, too-small and > large-enough Same as above too-small and large-enough case. > > * after the test, verify that the files have same size and contents I'll reuse convert facilities to do that. And I'll add one extra test case: No enough privilege to read some file in the directory: New: Fail gracefully Old: Fail with BUG_ON. If you're OK with the behavior difference, I'll create new test cases for mkfs --rootdir. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/Documentation/mkfs.btrfs.asciidoc b/Documentation/mkfs.btrfs.asciidoc index d53d9e26..843e25cd 100644 --- a/Documentation/mkfs.btrfs.asciidoc +++ b/Documentation/mkfs.btrfs.asciidoc @@ -106,6 +106,17 @@ Please see the mount option 'discard' for that in `btrfs`(5). *-r|--rootdir <rootdir>*:: Populate the toplevel subvolume with files from 'rootdir'. This does not require root permissions and does not mount the filesystem. ++ +With this option, only one device can be specified. ++ +NOTE: User should make sure the block device/file has large enough space to +contain the source directory and has enough previllege to read source directory. +Or mkfs will just fail. ++ +WARNING: Before v4.14 btrfs-progs, *--rootdir* will shrink the filesystem, +prevent user to make use of the remaining space. +In v4.14 btrfs-progs, this behavior is changed, and will not shrink the fs. +The result should be the same as `mkfs`, `mount` and then `cp -r`. *-O|--features <feature1>[,<feature2>...]*:: A list of filesystem features turned on at mkfs time. Not all features are
Add extra limitation explained for --rootdir option, including: 1) Size limitation Now I decide to follow "mkfs.ext4 -d" behavior, so we user is responsible to make sure the block device/file is large enough. 2) Read permission If user can't read the content, mkfs will just fail. So user is also responsible to make sure to have enough privilege. 3) Extra warning about the behavior change Since we we don't shrink fs the create file image, add such warning in documentation. Signed-off-by: Qu Wenruo <quwenruo.btrfs@gmx.com> --- Documentation/mkfs.btrfs.asciidoc | 11 +++++++++++ 1 file changed, 11 insertions(+)