| Message ID | 20130226044039.GM5551@dastard (mailing list archive) |
|---|---|
| State | Deferred, archived |
| Headers | show |
Dave Chinner <david@fromorbit.com> writes: > All of the evidence points of a problem caused by the pvmove > operation. I'd suggest that you don't use it until we get to the > bottom of the problem (i.e. where the EAGAIN is coming from and > why).... Dave, thanks for your quick reply and detailed analysis. For the benefit of other readers, I wish to add that the same problem occurs when creating lvm mirrors (e.g. "lvconvert -m1 vg/lvol"), which is not surprising since pvmove works itself by creating a mirror. Consequently, mirror creation should also be avoided until the root cause of the problem is eliminated. Cheers, Matteo -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
(Quick pointers that might be relevant) EAGAIN, I'm not aware of dm itself returning that on the i/o path. For 3.8 issues, read dm-devel around https://www.redhat.com/archives/dm-devel/2013-February/msg00086.html (I queued the dm-side fixes for linux-next earlier today) For pvmove, check exactly which version and whether discards are enabled: there was a userspace bug for a short period some time ago when discards were enabled. Alasdair -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Wed, Feb 27, 2013 at 01:04:14AM +0000, Alasdair G Kergon wrote: > (Quick pointers that might be relevant) > > EAGAIN, I'm not aware of dm itself returning that on the i/o path. Neither am I, but it's coming from somewhere in the IO path... > For 3.8 issues, read dm-devel around https://www.redhat.com/archives/dm-devel/2013-February/msg00086.html > (I queued the dm-side fixes for linux-next earlier today) It's reproducable on lots of different kernels, apparently - 3.8, 3.4.33, CentOS 6.3, debian sid/wheezy and Fedora 18 were mentioned specifically by the OP - so it doesn't look like a recent regression or constrained to a specific kernel. > For pvmove, check exactly which version and whether discards are enabled: there > was a userspace bug for a short period some time ago when discards were enabled. The version I used to reproduce on a 3.8.0 kernel was: $ pvmove --version LVM version: 2.02.95(2) (2012-03-06) Library version: 1.02.74 (2012-03-06) Driver version: 4.23.1 >From Debian unstable. Cheers, Dave.
Dave Chinner <david@fromorbit.com> writes: > On Wed, Feb 27, 2013 at 01:04:14AM +0000, Alasdair G Kergon wrote: >> (Quick pointers that might be relevant) >> >> EAGAIN, I'm not aware of dm itself returning that on the i/o path. > > Neither am I, but it's coming from somewhere in the IO path... Well, I don't really know anything about this topic, so I may be completely off the mark, but dm-raid1.c:mirror_map() does indeed return EWOULDBLOCK, and EWOULDBLOCK is #define'd to be EAGAIN, so it seems to me that dm-raid1 does indeed return EAGAIN for "rw == READA" (which I assume is read-ahead) if the "region is not in-sync": /* * If region is not in-sync queue the bio. */ if (!r || (r == -EWOULDBLOCK)) { if (rw == READA) return -EWOULDBLOCK; queue_bio(ms, bio, rw); return DM_MAPIO_SUBMITTED; } Regards, MF -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Tue, Feb 26, 2013 at 09:21:44PM -0500, Matteo Frigo wrote: > Dave Chinner <david@fromorbit.com> writes: > > > On Wed, Feb 27, 2013 at 01:04:14AM +0000, Alasdair G Kergon wrote: > >> (Quick pointers that might be relevant) > >> > >> EAGAIN, I'm not aware of dm itself returning that on the i/o path. > > > > Neither am I, but it's coming from somewhere in the IO path... > > Well, I don't really know anything about this topic, so I may be > completely off the mark, but dm-raid1.c:mirror_map() does indeed return > EWOULDBLOCK, and EWOULDBLOCK is #define'd to be EAGAIN, so it seems to > me that dm-raid1 does indeed return EAGAIN for "rw == READA" (which I > assume is read-ahead) if the "region is not in-sync": > > /* > * If region is not in-sync queue the bio. > */ > if (!r || (r == -EWOULDBLOCK)) { > if (rw == READA) > return -EWOULDBLOCK; > > queue_bio(ms, bio, rw); > return DM_MAPIO_SUBMITTED; > } Trees, forest.... Thanks for pointing out the obvious, Matteo. :) Cheers, Dave.
On Tue, Feb 26 2013 at 8:49pm -0500, Dave Chinner <david@fromorbit.com> wrote: > On Wed, Feb 27, 2013 at 01:04:14AM +0000, Alasdair G Kergon wrote: > > (Quick pointers that might be relevant) > > > > EAGAIN, I'm not aware of dm itself returning that on the i/o path. > > Neither am I, but it's coming from somewhere in the IO path... > > > For 3.8 issues, read dm-devel around https://www.redhat.com/archives/dm-devel/2013-February/msg00086.html > > (I queued the dm-side fixes for linux-next earlier today) > > It's reproducable on lots of different kernels, apparently - 3.8, > 3.4.33, CentOS 6.3, debian sid/wheezy and Fedora 18 were mentioned > specifically by the OP - so it doesn't look like a recent > regression or constrained to a specific kernel. > > > For pvmove, check exactly which version and whether discards are enabled: there > > was a userspace bug for a short period some time ago when discards were enabled. > > The version I used to reproduce on a 3.8.0 kernel was: > > $ pvmove --version > LVM version: 2.02.95(2) (2012-03-06) > Library version: 1.02.74 (2012-03-06) > Driver version: 4.23.1 Was issue_discards enabled in lvm.conf? If so, as Alasdair said, this lvm2 2.02.97 fix is needed: http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=07a25c249b3e -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
I have "issue_discards = 0" in lvm.conf, so this not appear to be the problem. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Wed, Feb 27, 2013 at 10:07:15AM -0500, Mike Snitzer wrote: > On Tue, Feb 26 2013 at 8:49pm -0500, > Dave Chinner <david@fromorbit.com> wrote: > > > On Wed, Feb 27, 2013 at 01:04:14AM +0000, Alasdair G Kergon wrote: > > > (Quick pointers that might be relevant) > > > > > > EAGAIN, I'm not aware of dm itself returning that on the i/o path. > > > > Neither am I, but it's coming from somewhere in the IO path... > > > > > For 3.8 issues, read dm-devel around https://www.redhat.com/archives/dm-devel/2013-February/msg00086.html > > > (I queued the dm-side fixes for linux-next earlier today) > > > > It's reproducable on lots of different kernels, apparently - 3.8, > > 3.4.33, CentOS 6.3, debian sid/wheezy and Fedora 18 were mentioned > > specifically by the OP - so it doesn't look like a recent > > regression or constrained to a specific kernel. > > > > > For pvmove, check exactly which version and whether discards are enabled: there > > > was a userspace bug for a short period some time ago when discards were enabled. > > > > The version I used to reproduce on a 3.8.0 kernel was: > > > > $ pvmove --version > > LVM version: 2.02.95(2) (2012-03-06) > > Library version: 1.02.74 (2012-03-06) > > Driver version: 4.23.1 > > Was issue_discards enabled in lvm.conf? $ grep issue_discards /etc/lvm/lvm.conf issue_discards = 0 $ Cheers, Dave.
Matteo Frigo <athena@fftw.org> writes: > Well, I don't really know anything about this topic, so I may be > completely off the mark, but dm-raid1.c:mirror_map() does indeed return > EWOULDBLOCK, and EWOULDBLOCK is #define'd to be EAGAIN, so it seems to > me that dm-raid1 does indeed return EAGAIN for "rw == READA" (which I > assume is read-ahead) if the "region is not in-sync": > > /* > * If region is not in-sync queue the bio. > */ > if (!r || (r == -EWOULDBLOCK)) { > if (rw == READA) > return -EWOULDBLOCK; > > queue_bio(ms, bio, rw); > return DM_MAPIO_SUBMITTED; > } Dave (and others), do you have any suggestion on what should be done to fix this bug? I have tried returning -EIO instead of -EWOULDBLOCK, but xfs does not like that. dm-zero.c:zero_map() appears to return -EIO too, so this is another potential issue. I have verified that removing the READA special case, treating READA like READ, fixes the problem: if(0) if (rw == READA) return -EWOULDBLOCK; Of course this "fix" throws away the baby with the bath water. I am willing to write and submit a patch, but I would appreciate directions as to what the correct protocol between xfs and dm is supposed to be. Regards, MF -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Thu, Mar 07, 2013 at 07:13:27AM -0500, Matteo Frigo wrote: > Matteo Frigo <athena@fftw.org> writes: > > > Well, I don't really know anything about this topic, so I may be > > completely off the mark, but dm-raid1.c:mirror_map() does indeed return > > EWOULDBLOCK, and EWOULDBLOCK is #define'd to be EAGAIN, so it seems to > > me that dm-raid1 does indeed return EAGAIN for "rw == READA" (which I > > assume is read-ahead) if the "region is not in-sync": > > > > /* > > * If region is not in-sync queue the bio. > > */ > > if (!r || (r == -EWOULDBLOCK)) { > > if (rw == READA) > > return -EWOULDBLOCK; > > > > queue_bio(ms, bio, rw); > > return DM_MAPIO_SUBMITTED; > > } > > Dave (and others), > > do you have any suggestion on what should be done to fix this bug? > > I have tried returning -EIO instead of -EWOULDBLOCK, but xfs does not > like that. dm-zero.c:zero_map() appears to return -EIO too, so this is > another potential issue. You need the XFS patch I posted so that readahead buffer verification is avoided in the case of an error being returned from the readahead. Cheers, Dave.
On Fri, Mar 08, 2013 at 09:31:40AM +1100, Dave Chinner wrote: > On Thu, Mar 07, 2013 at 07:13:27AM -0500, Matteo Frigo wrote: > > Matteo Frigo <athena@fftw.org> writes: > > > > > Well, I don't really know anything about this topic, so I may be > > > completely off the mark, but dm-raid1.c:mirror_map() does indeed return > > > EWOULDBLOCK, and EWOULDBLOCK is #define'd to be EAGAIN, so it seems to > > > me that dm-raid1 does indeed return EAGAIN for "rw == READA" (which I > > > assume is read-ahead) if the "region is not in-sync": > > > > > > /* > > > * If region is not in-sync queue the bio. > > > */ > > > if (!r || (r == -EWOULDBLOCK)) { > > > if (rw == READA) > > > return -EWOULDBLOCK; > > > > > > queue_bio(ms, bio, rw); > > > return DM_MAPIO_SUBMITTED; > > > } > > > > Dave (and others), > > > > do you have any suggestion on what should be done to fix this bug? > > > > I have tried returning -EIO instead of -EWOULDBLOCK, but xfs does not > > like that. dm-zero.c:zero_map() appears to return -EIO too, so this is > > another potential issue. > > You need the XFS patch I posted so that readahead buffer > verification is avoided in the case of an error being returned from > the readahead. I don't recall if that patch was sent to this thread, so here it is: http://oss.sgi.com/archives/xfs/2013-02/msg00516.html Cheers, Dave.
--- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1022,7 +1022,9 @@ xfs_buf_iodone_work( bool read = !!(bp->b_flags & XBF_READ); bp->b_flags &= ~(XBF_READ | XBF_WRITE | XBF_READ_AHEAD); - if (read && bp->b_ops) + + /* validate buffers that were read without errors */ + if (read && bp->b_ops && !bp->b_error && (bp->b_flags & XBF_DONE)) bp->b_ops->verify_read(bp); if (bp->b_iodone)