| Message ID | 20190415012229.20689-1-ming.lei@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | nvme/012 & 013: avoid extremely slow xfs IO | expand |
Thanks for the patch Ming. Couple of comments below. On 4/14/19 6:22 PM, Ming Lei wrote: > It is observed that nvme/012 may take ~17 minutes to complete on aarch64, > even worse it may trigger IO timeout on nvme-loop. > > Eric and Dave replied that it is because of too small log size on small > disk. > > So pass '-l size=32m' to avoid the issue. > > With this patch, nvme/012 can be completed in one minute. > Then we should set the QUICK=1 if its taking shorter time. > Cc: Eric Sandeen <esandeen@redhat.com> > Cc: Dave Chinner <dchinner@redhat.com> > Cc: "Darrick J. Wong" <darrick.wong@oracle.com> > Cc: linux-xfs@vger.kernel.org > Signed-off-by: Ming Lei <ming.lei@redhat.com> > --- > tests/nvme/012 | 2 +- > tests/nvme/013 | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tests/nvme/012 b/tests/nvme/012 > index 9a6801511df7..d7a8751ec752 100755 > --- a/tests/nvme/012 > +++ b/tests/nvme/012 > @@ -46,7 +46,7 @@ test() { > > umount ${mount_dir} > /dev/null 2>&1 > > - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything. > mount /dev/"${nvmedev}n1" "${mount_dir}" > > diff --git a/tests/nvme/013 b/tests/nvme/013 > index e346cd4baa35..bc435f752dca 100755 > --- a/tests/nvme/013 > +++ b/tests/nvme/013 > @@ -44,7 +44,7 @@ test() { > > umount ${mount_dir} > /dev/null 2>&1 > > - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > Extra space here but can be ignored once we move mkfs to the helper call. > mount /dev/"${nvmedev}n1" "${mount_dir}" > >
On 4/14/19 7:01 PM, Chaitanya Kulkarni wrote: > Thanks for the patch Ming. Couple of comments below. > On 4/14/19 6:22 PM, Ming Lei wrote: >> It is observed that nvme/012 may take ~17 minutes to complete on aarch64, >> even worse it may trigger IO timeout on nvme-loop. >> >> Eric and Dave replied that it is because of too small log size on small >> disk. >> >> So pass '-l size=32m' to avoid the issue. >> >> With this patch, nvme/012 can be completed in one minute. >> > Then we should set the QUICK=1 if its taking shorter time. > >> Cc: Eric Sandeen <esandeen@redhat.com> >> Cc: Dave Chinner <dchinner@redhat.com> >> Cc: "Darrick J. Wong" <darrick.wong@oracle.com> >> Cc: linux-xfs@vger.kernel.org >> Signed-off-by: Ming Lei <ming.lei@redhat.com> >> --- >> tests/nvme/012 | 2 +- >> tests/nvme/013 | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/tests/nvme/012 b/tests/nvme/012 >> index 9a6801511df7..d7a8751ec752 100755 >> --- a/tests/nvme/012 >> +++ b/tests/nvme/012 >> @@ -46,7 +46,7 @@ test() { >> >> umount ${mount_dir} > /dev/null 2>&1 >> >> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> > As a part of this series lets move this to the helper mkfs in the > nvme/rc and use that call in all the file-backed ns related testcases. > Let me know if you want me to do that or you want me to do that as a > part of this series. I'm okay with anything. >> mount /dev/"${nvmedev}n1" "${mount_dir}" >> >> diff --git a/tests/nvme/013 b/tests/nvme/013 >> index e346cd4baa35..bc435f752dca 100755 >> --- a/tests/nvme/013 >> +++ b/tests/nvme/013 >> @@ -44,7 +44,7 @@ test() { >> >> umount ${mount_dir} > /dev/null 2>&1 >> >> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> > Extra space here but can be ignored once we move mkfs to the helper call. >> mount /dev/"${nvmedev}n1" "${mount_dir}" >> >> > >
On Mon, Apr 15, 2019 at 10:02 AM Chaitanya Kulkarni <Chaitanya.Kulkarni@wdc.com> wrote: > > Thanks for the patch Ming. Couple of comments below. > On 4/14/19 6:22 PM, Ming Lei wrote: > > It is observed that nvme/012 may take ~17 minutes to complete on aarch64, > > even worse it may trigger IO timeout on nvme-loop. > > > > Eric and Dave replied that it is because of too small log size on small > > disk. > > > > So pass '-l size=32m' to avoid the issue. > > > > With this patch, nvme/012 can be completed in one minute. > > > Then we should set the QUICK=1 if its taking shorter time. That should have been a QUICK test, it is fine to set it, but it shouldn't belong to this patch. > > > Cc: Eric Sandeen <esandeen@redhat.com> > > Cc: Dave Chinner <dchinner@redhat.com> > > Cc: "Darrick J. Wong" <darrick.wong@oracle.com> > > Cc: linux-xfs@vger.kernel.org > > Signed-off-by: Ming Lei <ming.lei@redhat.com> > > --- > > tests/nvme/012 | 2 +- > > tests/nvme/013 | 2 +- > > 2 files changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/tests/nvme/012 b/tests/nvme/012 > > index 9a6801511df7..d7a8751ec752 100755 > > --- a/tests/nvme/012 > > +++ b/tests/nvme/012 > > @@ -46,7 +46,7 @@ test() { > > > > umount ${mount_dir} > /dev/null 2>&1 > > > > - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > > + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > > > As a part of this series lets move this to the helper mkfs in the > nvme/rc and use that call in all the file-backed ns related testcases. > Let me know if you want me to do that or you would like to do that as a > part of this series. I'm okay with anything. I am fine with either way. Thanks, Ming Lei
On 4/14/19 7:11 PM, Ming Lei wrote: > On Mon, Apr 15, 2019 at 10:02 AM Chaitanya Kulkarni > <Chaitanya.Kulkarni@wdc.com> wrote: >> >> Thanks for the patch Ming. Couple of comments below. >> On 4/14/19 6:22 PM, Ming Lei wrote: >>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64, >>> even worse it may trigger IO timeout on nvme-loop. >>> >>> Eric and Dave replied that it is because of too small log size on small >>> disk. >>> >>> So pass '-l size=32m' to avoid the issue. >>> >>> With this patch, nvme/012 can be completed in one minute. >>> >> Then we should set the QUICK=1 if its taking shorter time. > > That should have been a QUICK test, it is fine to set it, but it shouldn't > belong to this patch. > >> >>> Cc: Eric Sandeen <esandeen@redhat.com> >>> Cc: Dave Chinner <dchinner@redhat.com> >>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com> >>> Cc: linux-xfs@vger.kernel.org >>> Signed-off-by: Ming Lei <ming.lei@redhat.com> >>> --- >>> tests/nvme/012 | 2 +- >>> tests/nvme/013 | 2 +- >>> 2 files changed, 2 insertions(+), 2 deletions(-) >>> >>> diff --git a/tests/nvme/012 b/tests/nvme/012 >>> index 9a6801511df7..d7a8751ec752 100755 >>> --- a/tests/nvme/012 >>> +++ b/tests/nvme/012 >>> @@ -46,7 +46,7 @@ test() { >>> >>> umount ${mount_dir} > /dev/null 2>&1 >>> >>> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >>> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >>> >> As a part of this series lets move this to the helper mkfs in the >> nvme/rc and use that call in all the file-backed ns related testcases. >> Let me know if you want me to do that or you would like to do that as a >> part of this series. I'm okay with anything. > > I am fine with either way. > Okay, let's get this in, will send out the refactoring. > Thanks, > Ming Lei >
On 4/14/19 9:01 PM, Chaitanya Kulkarni wrote: > Thanks for the patch Ming. Couple of comments below. > On 4/14/19 6:22 PM, Ming Lei wrote: >> It is observed that nvme/012 may take ~17 minutes to complete on aarch64, >> even worse it may trigger IO timeout on nvme-loop. >> >> Eric and Dave replied that it is because of too small log size on small >> disk. >> >> So pass '-l size=32m' to avoid the issue. >> >> With this patch, nvme/012 can be completed in one minute. >> > Then we should set the QUICK=1 if its taking shorter time. > >> Cc: Eric Sandeen <esandeen@redhat.com> >> Cc: Dave Chinner <dchinner@redhat.com> >> Cc: "Darrick J. Wong" <darrick.wong@oracle.com> >> Cc: linux-xfs@vger.kernel.org >> Signed-off-by: Ming Lei <ming.lei@redhat.com> >> --- >> tests/nvme/012 | 2 +- >> tests/nvme/013 | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/tests/nvme/012 b/tests/nvme/012 >> index 9a6801511df7..d7a8751ec752 100755 >> --- a/tests/nvme/012 >> +++ b/tests/nvme/012 >> @@ -46,7 +46,7 @@ test() { >> >> umount ${mount_dir} > /dev/null 2>&1 >> >> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> > As a part of this series lets move this to the helper mkfs in the > nvme/rc and use that call in all the file-backed ns related testcases. > Let me know if you want me to do that or you would like to do that as a > part of this series. I'm okay with anything. We also discussed making the log size larger by default, on small fielsystems, but that won't help you yet. My only caution is that manually setting the log to 32m may actually create a smaller than default log if your backing file happens to be very large. Just something to consider. If the backing files are typically around the 4G size of this test (?) then 32m seems reasonable, 128m certainly would not hurt. -Eric >> mount /dev/"${nvmedev}n1" "${mount_dir}" >> >> diff --git a/tests/nvme/013 b/tests/nvme/013 >> index e346cd4baa35..bc435f752dca 100755 >> --- a/tests/nvme/013 >> +++ b/tests/nvme/013 >> @@ -44,7 +44,7 @@ test() { >> >> umount ${mount_dir} > /dev/null 2>&1 >> >> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 >> > Extra space here but can be ignored once we move mkfs to the helper call. >> mount /dev/"${nvmedev}n1" "${mount_dir}" >> >> >
On Mon, Apr 15, 2019 at 09:46:02AM -0400, Eric Sandeen wrote: > On 4/14/19 9:01 PM, Chaitanya Kulkarni wrote: > > Thanks for the patch Ming. Couple of comments below. > > On 4/14/19 6:22 PM, Ming Lei wrote: > >> It is observed that nvme/012 may take ~17 minutes to complete on aarch64, > >> even worse it may trigger IO timeout on nvme-loop. > >> > >> Eric and Dave replied that it is because of too small log size on small > >> disk. > >> > >> So pass '-l size=32m' to avoid the issue. > >> > >> With this patch, nvme/012 can be completed in one minute. > >> > > Then we should set the QUICK=1 if its taking shorter time. > > > >> Cc: Eric Sandeen <esandeen@redhat.com> > >> Cc: Dave Chinner <dchinner@redhat.com> > >> Cc: "Darrick J. Wong" <darrick.wong@oracle.com> > >> Cc: linux-xfs@vger.kernel.org > >> Signed-off-by: Ming Lei <ming.lei@redhat.com> > >> --- > >> tests/nvme/012 | 2 +- > >> tests/nvme/013 | 2 +- > >> 2 files changed, 2 insertions(+), 2 deletions(-) > >> > >> diff --git a/tests/nvme/012 b/tests/nvme/012 > >> index 9a6801511df7..d7a8751ec752 100755 > >> --- a/tests/nvme/012 > >> +++ b/tests/nvme/012 > >> @@ -46,7 +46,7 @@ test() { > >> > >> umount ${mount_dir} > /dev/null 2>&1 > >> > >> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > >> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 > >> > > As a part of this series lets move this to the helper mkfs in the > > nvme/rc and use that call in all the file-backed ns related testcases. > > Let me know if you want me to do that or you would like to do that as a > > part of this series. I'm okay with anything. > > We also discussed making the log size larger by default, on small fielsystems, > but that won't help you yet. > > My only caution is that manually setting the log to 32m may actually create > a smaller than default log if your backing file happens to be very large. > Just something to consider. If the backing files are typically around the > 4G size of this test (?) then 32m seems reasonable, 128m certainly would > not hurt. In the two tests, the device size is 1G, and looks 32m log size works just fine. thanks, Ming
On Mon, Apr 15, 2019 at 09:22:29AM +0800, Ming Lei wrote: > It is observed that nvme/012 may take ~17 minutes to complete on aarch64, > even worse it may trigger IO timeout on nvme-loop. > > Eric and Dave replied that it is because of too small log size on small > disk. > > So pass '-l size=32m' to avoid the issue. > > With this patch, nvme/012 can be completed in one minute. Nice: nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed] runtime 402.605s ... 27.431s Thanks, Ming, applied.
diff --git a/tests/nvme/012 b/tests/nvme/012 index 9a6801511df7..d7a8751ec752 100755 --- a/tests/nvme/012 +++ b/tests/nvme/012 @@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 mount /dev/"${nvmedev}n1" "${mount_dir}" diff --git a/tests/nvme/013 b/tests/nvme/013 index e346cd4baa35..bc435f752dca 100755 --- a/tests/nvme/013 +++ b/tests/nvme/013 @@ -44,7 +44,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 mount /dev/"${nvmedev}n1" "${mount_dir}"
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop. Eric and Dave replied that it is because of too small log size on small disk. So pass '-l size=32m' to avoid the issue. With this patch, nvme/012 can be completed in one minute. Cc: Eric Sandeen <esandeen@redhat.com> Cc: Dave Chinner <dchinner@redhat.com> Cc: "Darrick J. Wong" <darrick.wong@oracle.com> Cc: linux-xfs@vger.kernel.org Signed-off-by: Ming Lei <ming.lei@redhat.com> --- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)