diff mbox series

[v3] overlay/066: copy-up test for variant sparse files

Message ID 20191024122923.24689-1-cgxu519@mykernel.net (mailing list archive)
State New, archived
Headers show
Series [v3] overlay/066: copy-up test for variant sparse files | expand

Commit Message

Chengguang Xu Oct. 24, 2019, 12:29 p.m. UTC
This is intensive copy-up test for sparse files,
these cases will be mainly used for regression test
of copy-up improvement for sparse files.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>

---
v1->v2:
- Call _get_block_size to get fs block size.
- Add comment for test space requirement.
- Print meaningful error message when copy-up fail.
- Adjust random hole range to 1M~5M.
- Fix typo.

v2->v3:
- Fix space requiremnt for test.
- Add more descriptions for test files and hole patterns.
- Define well named variables to replace unexplained numbers.
- Fix random hole algorithm to what Amir suggested.
- Adjust iosize to start from 1K.
- Remove from quick test group.

 tests/overlay/066     | 130 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/066.out |   2 +
 tests/overlay/group   |   1 +
 3 files changed, 133 insertions(+)
 create mode 100755 tests/overlay/066
 create mode 100644 tests/overlay/066.out

Comments

Amir Goldstein Oct. 24, 2019, 9:02 p.m. UTC | #1
On Thu, Oct 24, 2019 at 3:29 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
>
> This is intensive copy-up test for sparse files,
> these cases will be mainly used for regression test
> of copy-up improvement for sparse files.
>
> Reviewed-by: Amir Goldstein <amir73il@gmail.com>
> Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>
>
> ---
> v1->v2:
> - Call _get_block_size to get fs block size.
> - Add comment for test space requirement.
> - Print meaningful error message when copy-up fail.
> - Adjust random hole range to 1M~5M.
> - Fix typo.
>
> v2->v3:
> - Fix space requiremnt for test.
> - Add more descriptions for test files and hole patterns.
> - Define well named variables to replace unexplained numbers.
> - Fix random hole algorithm to what Amir suggested.
> - Adjust iosize to start from 1K.
> - Remove from quick test group.

Why? you said it takes 7s without the kernel patch.
The test overlay/001 is in quick group and it copies up 2*4GB
sparse files.

Tests that are not in quick group are far less likely to be run
regularly by developers.

Otherwise looks good.

Thanks,
Amir.
Chengguang Xu Oct. 25, 2019, 1:19 a.m. UTC | #2
---- 在 星期五, 2019-10-25 05:02:07 Amir Goldstein <amir73il@gmail.com> 撰写 ----
 > On Thu, Oct 24, 2019 at 3:29 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
 > >
 > > This is intensive copy-up test for sparse files,
 > > these cases will be mainly used for regression test
 > > of copy-up improvement for sparse files.
 > >
 > > Reviewed-by: Amir Goldstein <amir73il@gmail.com>
 > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>
 > >
 > > ---
 > > v1->v2:
 > > - Call _get_block_size to get fs block size.
 > > - Add comment for test space requirement.
 > > - Print meaningful error message when copy-up fail.
 > > - Adjust random hole range to 1M~5M.
 > > - Fix typo.
 > >
 > > v2->v3:
 > > - Fix space requiremnt for test.
 > > - Add more descriptions for test files and hole patterns.
 > > - Define well named variables to replace unexplained numbers.
 > > - Fix random hole algorithm to what Amir suggested.
 > > - Adjust iosize to start from 1K.
 > > - Remove from quick test group.
 > 
 > Why? you said it takes 7s without the kernel patch.
 > The test overlay/001 is in quick group and it copies up 2*4GB
 > sparse files.

I noticed that after changed to start from 1K iosize the test took about 23s.
I'm afraid maybe it will take more time on low performance VM env.

The test overlay/001 took 8s/1s with/without kernel patch, so mainly test time
wasted on creating test files on test overlay/066.

 > 
 > Tests that are not in quick group are far less likely to be run
 > regularly by developers.

hmm...well, lets add 'quick' group again and remove it  if anyone complains later. 

Thanks,
Chengguang
Amir Goldstein Oct. 27, 2019, 1:59 p.m. UTC | #3
On Fri, Oct 25, 2019 at 4:19 AM Chengguang Xu <cgxu519@mykernel.net> wrote:
>
>  ---- 在 星期五, 2019-10-25 05:02:07 Amir Goldstein <amir73il@gmail.com> 撰写 ----
>  > On Thu, Oct 24, 2019 at 3:29 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
>  > >
>  > > This is intensive copy-up test for sparse files,
>  > > these cases will be mainly used for regression test
>  > > of copy-up improvement for sparse files.
>  > >
>  > > Reviewed-by: Amir Goldstein <amir73il@gmail.com>
>  > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>
>  > >
>  > > ---
>  > > v1->v2:
>  > > - Call _get_block_size to get fs block size.
>  > > - Add comment for test space requirement.
>  > > - Print meaningful error message when copy-up fail.
>  > > - Adjust random hole range to 1M~5M.
>  > > - Fix typo.
>  > >
>  > > v2->v3:
>  > > - Fix space requiremnt for test.
>  > > - Add more descriptions for test files and hole patterns.
>  > > - Define well named variables to replace unexplained numbers.
>  > > - Fix random hole algorithm to what Amir suggested.
>  > > - Adjust iosize to start from 1K.

Chengguang,

Sorry, I did't notice that you did that. Why?
As you can see below, this change has a very bad impact on test run time.
Any reason not to use _get_block_size?


>  > > - Remove from quick test group.
>  >
>  > Why? you said it takes 7s without the kernel patch.
>  > The test overlay/001 is in quick group and it copies up 2*4GB
>  > sparse files.
>
> I noticed that after changed to start from 1K iosize the test took about 23s.
> I'm afraid maybe it will take more time on low performance VM env.
>
> The test overlay/001 took 8s/1s with/without kernel patch, so mainly test time
> wasted on creating test files on test overlay/066.

You are correct about the time spent on creating the files, but...

On my low perf VM, the test runs 95s with overlay over xfs+reflink

But if I set start iosize=4 (which what my fs block size is) the test
runs only 30s.

IOW, most of the test time is spent on creating the files with small iosize
below fs block size, which doesn't test copy up of holes at all.

If I further change file size to be a multiply of iosize (x10),
test run time drops to 6s!
I don't think we loose too much test coverage if we do that?
If anything we gain testing different file sizes.

The disk space requirement formula for ${iosize}K_holefiles becomes:
10*(2^0 + 2^11)K*12/2 =~ 10 * 1024 * 12
same as before, just needs explaining.
(the formula assumes the worst case of min_iosize=1)

-------------
 #
 # |-- hole --|-- data --| ... |-- data --|-- hole --|

-iosize=1
+min_iosize=$(($(_get_block_size "${lowerdir}") / 1024 ))
+iosize=$min_iosize
 max_iosize=2048
-file_size=10240
-max_pos=`expr $file_size - $max_iosize`

 while [ $iosize -le $max_iosize ]; do
+       file_size=$((10*$iosize))
+       max_pos=`expr $file_size - $iosize`
+       date >>$seqres.full
+       echo "Creating ${testfile}_iosize${iosize}K_holefile..." >>$seqres.full
        pos=$iosize
        $XFS_IO_PROG -fc "truncate ${file_size}K" \
                "${lowerdir}/${testfile}_iosize${iosize}K_holefile"
>>$seqres.full
-----------


>
>  >
>  > Tests that are not in quick group are far less likely to be run
>  > regularly by developers.
>
> hmm...well, lets add 'quick' group again and remove it  if anyone complains later.
>

I am now complaining ;-), but after fixes above, test is really quick

Please send a fix patch (to already merged test) to fir test runtime
and possibly use _get_block_size.

Thanks,
Amir.
Chengguang Xu Oct. 28, 2019, 12:08 p.m. UTC | #4
---- 在 星期日, 2019-10-27 21:59:36 Amir Goldstein <amir73il@gmail.com> 撰写 ----
 > On Fri, Oct 25, 2019 at 4:19 AM Chengguang Xu <cgxu519@mykernel.net> wrote:
 > >
 > >  ---- 在 星期五, 2019-10-25 05:02:07 Amir Goldstein <amir73il@gmail.com> 撰写 ----
 > >  > On Thu, Oct 24, 2019 at 3:29 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
 > >  > >
 > >  > > This is intensive copy-up test for sparse files,
 > >  > > these cases will be mainly used for regression test
 > >  > > of copy-up improvement for sparse files.
 > >  > >
 > >  > > Reviewed-by: Amir Goldstein <amir73il@gmail.com>
 > >  > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>
 > >  > >
 > >  > > ---
 > >  > > v1->v2:
 > >  > > - Call _get_block_size to get fs block size.
 > >  > > - Add comment for test space requirement.
 > >  > > - Print meaningful error message when copy-up fail.
 > >  > > - Adjust random hole range to 1M~5M.
 > >  > > - Fix typo.
 > >  > >
 > >  > > v2->v3:
 > >  > > - Fix space requiremnt for test.
 > >  > > - Add more descriptions for test files and hole patterns.
 > >  > > - Define well named variables to replace unexplained numbers.
 > >  > > - Fix random hole algorithm to what Amir suggested.
 > >  > > - Adjust iosize to start from 1K.
 > 
 > Chengguang,
 > 
 > Sorry, I did't notice that you did that. Why?
 > As you can see below, this change has a very bad impact on test run time.
 > Any reason not to use _get_block_size?

Use _get_block_size cannot mitigate the effect perfectly, 
in the worst case that we formatted fs with blocksize=1K,
the test will  take long time and also test time is not fixed.

 > 
 > 
 > >  > > - Remove from quick test group.
 > >  >
 > >  > Why? you said it takes 7s without the kernel patch.
 > >  > The test overlay/001 is in quick group and it copies up 2*4GB
 > >  > sparse files.
 > >
 > > I noticed that after changed to start from 1K iosize the test took about 23s.
 > > I'm afraid maybe it will take more time on low performance VM env.
 > >
 > > The test overlay/001 took 8s/1s with/without kernel patch, so mainly test time
 > > wasted on creating test files on test overlay/066.
 > 
 > You are correct about the time spent on creating the files, but...
 > 
 > On my low perf VM, the test runs 95s with overlay over xfs+reflink
 > 
 > But if I set start iosize=4 (which what my fs block size is) the test
 > runs only 30s.
 > 
 > IOW, most of the test time is spent on creating the files with small iosize
 > below fs block size, which doesn't test copy up of holes at all.
 > 
 > If I further change file size to be a multiply of iosize (x10),
 > test run time drops to 6s!
 > I don't think we loose too much test coverage if we do that?
 > If anything we gain testing different file sizes.

hmm, for small iosize the file size is even smaller than
copy-up CHUNK SIZE(1M),  so that all contents(data+hole)
will be passed at once, I'm not very sure is it helpful for
hole copy-up logic in kernel patch. What do you think? 


 > 
 > The disk space requirement formula for ${iosize}K_holefiles becomes:
 > 10*(2^0 + 2^11)K*12/2 =~ 10 * 1024 * 12

That's the mean of 12/2 ?

 > same as before, just needs explaining.
 > (the formula assumes the worst case of min_iosize=1)
 > 
 > -------------
 >  #
 >  # |-- hole --|-- data --| ... |-- data --|-- hole --|
 > 
 > -iosize=1
 > +min_iosize=$(($(_get_block_size "${lowerdir}") / 1024 ))
 > +iosize=$min_iosize
 >  max_iosize=2048
 > -file_size=10240
 > -max_pos=`expr $file_size - $max_iosize`
 > 
 >  while [ $iosize -le $max_iosize ]; do
 > +       file_size=$((10*$iosize))
 > +       max_pos=`expr $file_size - $iosize`
 > +       date >>$seqres.full

That's the purpose for putting data info here?

 > +       echo "Creating ${testfile}_iosize${iosize}K_holefile..." >>$seqres.full
 >         pos=$iosize
 >         $XFS_IO_PROG -fc "truncate ${file_size}K" \
 >                 "${lowerdir}/${testfile}_iosize${iosize}K_holefile"
 > >>$seqres.full
 > -----------
 > 
 > 
 > >
 > >  >
 > >  > Tests that are not in quick group are far less likely to be run
 > >  > regularly by developers.
 > >
 > > hmm...well, lets add 'quick' group again and remove it  if anyone complains later.
 > >
 > 
 > I am now complaining ;-), but after fixes above, test is really quick
 > 
 > Please send a fix patch (to already merged test) to fir test runtime
 > and possibly use _get_block_size.
 > 
 > Thanks,
 > Amir.
 >
Amir Goldstein Oct. 28, 2019, 12:29 p.m. UTC | #5
On Mon, Oct 28, 2019 at 2:09 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
>
>  ---- 在 星期日, 2019-10-27 21:59:36 Amir Goldstein <amir73il@gmail.com> 撰写 ----
>  > On Fri, Oct 25, 2019 at 4:19 AM Chengguang Xu <cgxu519@mykernel.net> wrote:
>  > >
>  > >  ---- 在 星期五, 2019-10-25 05:02:07 Amir Goldstein <amir73il@gmail.com> 撰写 ----
>  > >  > On Thu, Oct 24, 2019 at 3:29 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
>  > >  > >
>  > >  > > This is intensive copy-up test for sparse files,
>  > >  > > these cases will be mainly used for regression test
>  > >  > > of copy-up improvement for sparse files.
>  > >  > >
>  > >  > > Reviewed-by: Amir Goldstein <amir73il@gmail.com>
>  > >  > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>
>  > >  > >
>  > >  > > ---
>  > >  > > v1->v2:
>  > >  > > - Call _get_block_size to get fs block size.
>  > >  > > - Add comment for test space requirement.
>  > >  > > - Print meaningful error message when copy-up fail.
>  > >  > > - Adjust random hole range to 1M~5M.
>  > >  > > - Fix typo.
>  > >  > >
>  > >  > > v2->v3:
>  > >  > > - Fix space requiremnt for test.
>  > >  > > - Add more descriptions for test files and hole patterns.
>  > >  > > - Define well named variables to replace unexplained numbers.
>  > >  > > - Fix random hole algorithm to what Amir suggested.
>  > >  > > - Adjust iosize to start from 1K.
>  >
>  > Chengguang,
>  >
>  > Sorry, I did't notice that you did that. Why?
>  > As you can see below, this change has a very bad impact on test run time.
>  > Any reason not to use _get_block_size?
>
> Use _get_block_size cannot mitigate the effect perfectly,
> in the worst case that we formatted fs with blocksize=1K,
> the test will  take long time and also test time is not fixed.
>
>  >
>  >
>  > >  > > - Remove from quick test group.
>  > >  >
>  > >  > Why? you said it takes 7s without the kernel patch.
>  > >  > The test overlay/001 is in quick group and it copies up 2*4GB
>  > >  > sparse files.
>  > >
>  > > I noticed that after changed to start from 1K iosize the test took about 23s.
>  > > I'm afraid maybe it will take more time on low performance VM env.
>  > >
>  > > The test overlay/001 took 8s/1s with/without kernel patch, so mainly test time
>  > > wasted on creating test files on test overlay/066.
>  >
>  > You are correct about the time spent on creating the files, but...
>  >
>  > On my low perf VM, the test runs 95s with overlay over xfs+reflink
>  >
>  > But if I set start iosize=4 (which what my fs block size is) the test
>  > runs only 30s.
>  >
>  > IOW, most of the test time is spent on creating the files with small iosize
>  > below fs block size, which doesn't test copy up of holes at all.
>  >
>  > If I further change file size to be a multiply of iosize (x10),
>  > test run time drops to 6s!
>  > I don't think we loose too much test coverage if we do that?
>  > If anything we gain testing different file sizes.
>
> hmm, for small iosize the file size is even smaller than
> copy-up CHUNK SIZE(1M),  so that all contents(data+hole)
> will be passed at once, I'm not very sure is it helpful for
> hole copy-up logic in kernel patch. What do you think?
>

Not sure. I don't think we should target the test by what we know your
patch does, but by maximizing test coverage in a cost effective way.

Creating a 10M file with so many small holes doesn't add much to test
coverage IMO. If you feel those are needed, you should use a C helper
to create those files more efficiently.

BTW I think what is missing from test coverage is small holes
that are not aligned to 1M boundary.


>
>  >
>  > The disk space requirement formula for ${iosize}K_holefiles becomes:
>  > 10*(2^0 + 2^11)K*12/2 =~ 10 * 1024 * 12
>
> That's the mean of 12/2 ?

it's the formula to the sum of the series:
2^0+2^2+...2^11 = (2^0 + 2^11)*12/2

>
>  > same as before, just needs explaining.
>  > (the formula assumes the worst case of min_iosize=1)
>  >
>  > -------------
>  >  #
>  >  # |-- hole --|-- data --| ... |-- data --|-- hole --|
>  >
>  > -iosize=1
>  > +min_iosize=$(($(_get_block_size "${lowerdir}") / 1024 ))
>  > +iosize=$min_iosize
>  >  max_iosize=2048
>  > -file_size=10240
>  > -max_pos=`expr $file_size - $max_iosize`
>  >
>  >  while [ $iosize -le $max_iosize ]; do
>  > +       file_size=$((10*$iosize))
>  > +       max_pos=`expr $file_size - $iosize`
>  > +       date >>$seqres.full
>
> That's the purpose for putting data info here?

leftover from my debugging patch to figure out what takes so much time
You don't need it

Thanks,
Amir.
Chengguang Xu Oct. 28, 2019, 2:11 p.m. UTC | #6
---- 在 星期一, 2019-10-28 20:29:03 Amir Goldstein <amir73il@gmail.com> 撰写 ----
 > On Mon, Oct 28, 2019 at 2:09 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
 > >
 > >  ---- 在 星期日, 2019-10-27 21:59:36 Amir Goldstein <amir73il@gmail.com> 撰写 ----
 > >  > On Fri, Oct 25, 2019 at 4:19 AM Chengguang Xu <cgxu519@mykernel.net> wrote:
 > >  > >
 > >  > >  ---- 在 星期五, 2019-10-25 05:02:07 Amir Goldstein <amir73il@gmail.com> 撰写 ----
 > >  > >  > On Thu, Oct 24, 2019 at 3:29 PM Chengguang Xu <cgxu519@mykernel.net> wrote:
 > >  > >  > >
 > >  > >  > > This is intensive copy-up test for sparse files,
 > >  > >  > > these cases will be mainly used for regression test
 > >  > >  > > of copy-up improvement for sparse files.
 > >  > >  > >
 > >  > >  > > Reviewed-by: Amir Goldstein <amir73il@gmail.com>
 > >  > >  > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net>
 > >  > >  > >
 > >  > >  > > ---
 > >  > >  > > v1->v2:
 > >  > >  > > - Call _get_block_size to get fs block size.
 > >  > >  > > - Add comment for test space requirement.
 > >  > >  > > - Print meaningful error message when copy-up fail.
 > >  > >  > > - Adjust random hole range to 1M~5M.
 > >  > >  > > - Fix typo.
 > >  > >  > >
 > >  > >  > > v2->v3:
 > >  > >  > > - Fix space requiremnt for test.
 > >  > >  > > - Add more descriptions for test files and hole patterns.
 > >  > >  > > - Define well named variables to replace unexplained numbers.
 > >  > >  > > - Fix random hole algorithm to what Amir suggested.
 > >  > >  > > - Adjust iosize to start from 1K.
 > >  >
 > >  > Chengguang,
 > >  >
 > >  > Sorry, I did't notice that you did that. Why?
 > >  > As you can see below, this change has a very bad impact on test run time.
 > >  > Any reason not to use _get_block_size?
 > >
 > > Use _get_block_size cannot mitigate the effect perfectly,
 > > in the worst case that we formatted fs with blocksize=1K,
 > > the test will  take long time and also test time is not fixed.
 > >
 > >  >
 > >  >
 > >  > >  > > - Remove from quick test group.
 > >  > >  >
 > >  > >  > Why? you said it takes 7s without the kernel patch.
 > >  > >  > The test overlay/001 is in quick group and it copies up 2*4GB
 > >  > >  > sparse files.
 > >  > >
 > >  > > I noticed that after changed to start from 1K iosize the test took about 23s.
 > >  > > I'm afraid maybe it will take more time on low performance VM env.
 > >  > >
 > >  > > The test overlay/001 took 8s/1s with/without kernel patch, so mainly test time
 > >  > > wasted on creating test files on test overlay/066.
 > >  >
 > >  > You are correct about the time spent on creating the files, but...
 > >  >
 > >  > On my low perf VM, the test runs 95s with overlay over xfs+reflink
 > >  >
 > >  > But if I set start iosize=4 (which what my fs block size is) the test
 > >  > runs only 30s.
 > >  >
 > >  > IOW, most of the test time is spent on creating the files with small iosize
 > >  > below fs block size, which doesn't test copy up of holes at all.
 > >  >
 > >  > If I further change file size to be a multiply of iosize (x10),
 > >  > test run time drops to 6s!
 > >  > I don't think we loose too much test coverage if we do that?
 > >  > If anything we gain testing different file sizes.
 > >
 > > hmm, for small iosize the file size is even smaller than
 > > copy-up CHUNK SIZE(1M),  so that all contents(data+hole)
 > > will be passed at once, I'm not very sure is it helpful for
 > > hole copy-up logic in kernel patch. What do you think?
 > >
 > 
 > Not sure. I don't think we should target the test by what we know your
 > patch does, but by maximizing test coverage in a cost effective way.
 > 
 > Creating a 10M file with so many small holes doesn't add much to test
 > coverage IMO. If you feel those are needed, you should use a C helper
 > to create those files more efficiently.
 > 
 > BTW I think what is missing from test coverage is small holes
 > that are not aligned to 1M boundary.
 > 

Agreed.

So how about change test pattern to below, it will cover  most of the
cases that we want. I haven't done test for the performance(test time)
but I think it will be fast enough.


One 4K empty file.
One 4M empty file.
One 10M file with random small holes (4K~512K)
One 100M file with random big holes (1M~5M)


 > 
 > >
 > >  >
 > >  > The disk space requirement formula for ${iosize}K_holefiles becomes:
 > >  > 10*(2^0 + 2^11)K*12/2 =~ 10 * 1024 * 12
 > >
 > > That's the mean of 12/2 ?
 > 
 > it's the formula to the sum of the series:
 > 2^0+2^2+...2^11 = (2^0 + 2^11)*12/2

oh, sorry, I misread the formula. 

 > 
 > >
 > >  > same as before, just needs explaining.
 > >  > (the formula assumes the worst case of min_iosize=1)
 > >  >
 > >  > -------------
 > >  >  #
 > >  >  # |-- hole --|-- data --| ... |-- data --|-- hole --|
 > >  >
 > >  > -iosize=1
 > >  > +min_iosize=$(($(_get_block_size "${lowerdir}") / 1024 ))
 > >  > +iosize=$min_iosize
 > >  >  max_iosize=2048
 > >  > -file_size=10240
 > >  > -max_pos=`expr $file_size - $max_iosize`
 > >  >
 > >  >  while [ $iosize -le $max_iosize ]; do
 > >  > +       file_size=$((10*$iosize))
 > >  > +       max_pos=`expr $file_size - $iosize`
 > >  > +       date >>$seqres.full
 > >
 > > That's the purpose for putting data info here?
 > 
 > leftover from my debugging patch to figure out what takes so much time
 > You don't need it
 > 
 > Thanks,
 > Amir.
 >
Amir Goldstein Oct. 28, 2019, 2:34 p.m. UTC | #7
>  >
>  > Not sure. I don't think we should target the test by what we know your
>  > patch does, but by maximizing test coverage in a cost effective way.
>  >
>  > Creating a 10M file with so many small holes doesn't add much to test
>  > coverage IMO. If you feel those are needed, you should use a C helper
>  > to create those files more efficiently.
>  >
>  > BTW I think what is missing from test coverage is small holes
>  > that are not aligned to 1M boundary.
>  >
>
> Agreed.
>
> So how about change test pattern to below, it will cover  most of the
> cases that we want. I haven't done test for the performance(test time)
> but I think it will be fast enough.
>
>
> One 4K empty file.
> One 4M empty file.
> One 10M file with random small holes (4K~512K)
> One 100M file with random big holes (1M~5M)
>
>

It sounds like a good addition, but maybe lets keep
the files with aligned holes iosize*10 in size?

Thanks,
Amir.
diff mbox series

Patch

diff --git a/tests/overlay/066 b/tests/overlay/066
new file mode 100755
index 00000000..285a5aff
--- /dev/null
+++ b/tests/overlay/066
@@ -0,0 +1,130 @@ 
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Chengguang Xu <cgxu519@mykernel.net>
+# All Rights Reserved.
+#
+# FS QA Test 066
+#
+# Test overlayfs copy-up function for variant sparse files.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks.
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test.
+rm -f $seqres.full
+
+# real QA test starts here.
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch
+
+# Remove all files from previous tests
+_scratch_mkfs
+
+# We have totally 14 test files in this test.
+# The detail as below:
+# 1 empty file(10M) + 2^0(K)..2^11(K) hole size files(each 10M) + 1 random hole size file(100M).
+#
+# Considering both upper and lower fs will fill zero when copy-up
+# hole area in the file, this test at least requires double disk
+# space of the sum of above test files' size.
+
+_require_fs_space $OVL_BASE_SCRATCH_MNT $(((10*1024*13 + 100*1024*1) * 2))
+
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
+testfile="copyup_sparse_test"
+
+# Create a completely empty hole file(10M).
+file_size=10240
+$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_holefile" \
+		 >>$seqres.full
+
+# Create 2^0(K)..2^11(K) hole size test files(each 10M).
+#
+# The pattern is like below, both hole and data are equal to
+# iosize except last hole.
+#
+# |-- hole --|-- data --| ... |-- data --|-- hole --|
+
+iosize=1
+max_iosize=2048
+file_size=10240
+max_pos=`expr $file_size - $max_iosize`
+
+while [ $iosize -le $max_iosize ]; do
+	pos=$iosize
+	$XFS_IO_PROG -fc "truncate ${file_size}K" \
+		"${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full
+	while [ $pos -lt $max_pos ]; do
+		$XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \
+		"${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full
+		pos=`expr $pos + $iosize + $iosize`
+	done
+	iosize=`expr $iosize + $iosize`
+done
+
+# Create test file with many random holes(hole size is between 1M and 5M),
+# total file size is 100M.
+
+pos=2048
+max_pos=81920
+file_size=102400
+min_hole=1024
+max_hole=5120
+
+$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_holefile" \
+		>>$seqres.full
+
+while [ $pos -le $max_pos ]; do
+	iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole))
+	$XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \
+		"${lowerdir}/${testfile}_random_holefile" >>$seqres.full
+	pos=`expr $pos + $iosize + $iosize`
+done
+
+_scratch_mount
+
+# Open the test files, no errors are expected.
+for f in $SCRATCH_MNT/*; do
+	$XFS_IO_PROG -c "open" $f >>$seqres.full
+done
+
+echo "Silence is golden"
+
+# Check all copy-up files in upper layer.
+iosize=1
+while [ $iosize -le 2048 ]; do
+	diff "${lowerdir}/${testfile}_iosize${iosize}K_holefile" \
+		"${upperdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full ||\
+		echo "${upperdir}/${testfile}_iosize${iosize}K_holefile" copy up failed!
+	iosize=`expr $iosize + $iosize`
+done
+
+diff "${lowerdir}/${testfile}_empty_holefile"  "${upperdir}/${testfile}_empty_holefile"  \
+	>>$seqres.full || echo "${upperdir}/${testfile}_empty_holefile" copy up failed!
+diff "${lowerdir}/${testfile}_random_holefile" "${upperdir}/${testfile}_random_holefile" \
+	>>$seqres.full || echo "${upperdir}/${testfile}_random_holefile" copy up failed!
+
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/066.out b/tests/overlay/066.out
new file mode 100644
index 00000000..b60cc24c
--- /dev/null
+++ b/tests/overlay/066.out
@@ -0,0 +1,2 @@ 
+QA output created by 066
+Silence is golden
diff --git a/tests/overlay/group b/tests/overlay/group
index ef8517a1..e22134df 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -68,3 +68,4 @@ 
 063 auto quick whiteout
 064 auto quick copyup
 065 auto quick mount
+066 auto copyup