| Message ID | 20180615014314.27574-1-david@fromorbit.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 6/14/18 8:43 PM, Dave Chinner wrote: > From: Dave Chinner <dchinner@redhat.com> > > A log recovery failure has been reproduced where a symlink inode has > a zero length in extent form. It was caused by a shutdown during a > combined fstress+fsmark workload. > > To fix it, we have to allow zero length symlink inodes through > xfs_dinode_verify() during log recovery. We already specifically > check and allow this case in the shortform symlink fork verifier, > but in this case we don't get that far, and the inode is not in > shortform format. > > Update the dinode verifier to handle this case, and change the > symlink fork verifier to only allow this case to exist during log > recovery. > > Signed-off-by: Dave Chinner <dchinner@redhat.com> Looks reasonable. Reviewed-by: Eric Sandeen <sandeen@redhat.com> > --- > fs/xfs/libxfs/xfs_inode_buf.c | 16 +++++++++++++--- > fs/xfs/libxfs/xfs_symlink_remote.c | 12 +++++++++--- > fs/xfs/xfs_log.c | 11 +++++++++++ > fs/xfs/xfs_log.h | 3 ++- > 4 files changed, 35 insertions(+), 7 deletions(-) > > diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c > index d38d724534c4..bec9178377e3 100644 > --- a/fs/xfs/libxfs/xfs_inode_buf.c > +++ b/fs/xfs/libxfs/xfs_inode_buf.c > @@ -19,6 +19,7 @@ > #include "xfs_trans.h" > #include "xfs_ialloc.h" > #include "xfs_dir2.h" > +#include "xfs_log.h" > > #include <linux/iversion.h> > > @@ -411,9 +412,18 @@ xfs_dinode_verify( > if (mode && xfs_mode_to_ftype(mode) == XFS_DIR3_FT_UNKNOWN) > return __this_address; > > - /* No zero-length symlinks/dirs. */ > - if ((S_ISLNK(mode) || S_ISDIR(mode)) && di_size == 0) > - return __this_address; > + /* > + * Normally both symlinks and dirs cannot be zero length. However, if a > + * symlink is in the process of being torn down and there's a > + * shutdown/crash, the symlink on disk may have a zero size. Hence we > + * only allow zero length symlinks during log recovery. > + */ > + if (di_size == 0) { > + if (S_ISDIR(mode)) > + return __this_address; > + if (S_ISLNK(mode) && !xfs_log_in_recovery(mp)) > + return __this_address; > + } > > /* Fork checks carried over from xfs_iformat_fork */ > if (mode && > diff --git a/fs/xfs/libxfs/xfs_symlink_remote.c b/fs/xfs/libxfs/xfs_symlink_remote.c > index 95374ab2dee7..b1f0dd14f805 100644 > --- a/fs/xfs/libxfs/xfs_symlink_remote.c > +++ b/fs/xfs/libxfs/xfs_symlink_remote.c > @@ -215,9 +215,15 @@ xfs_symlink_shortform_verify( > size = ifp->if_bytes; > endp = sfp + size; > > - /* Zero length symlinks can exist while we're deleting a remote one. */ > - if (size == 0) > - return NULL; > + /* > + * Zero length symlinks can only pass through here during log recovery > + * while recovering deletion of a remote symlink. > + */ > + if (size == 0) { > + if (xfs_log_in_recovery(ip->i_mount)) > + return NULL; > + return __this_address; > + } > > /* No negative sizes or overly long symlink targets. */ > if (size < 0 || size > XFS_SYMLINK_MAXLEN) > diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c > index 5e56f3b93d4b..012397f6ec5a 100644 > --- a/fs/xfs/xfs_log.c > +++ b/fs/xfs/xfs_log.c > @@ -4083,3 +4083,14 @@ xfs_log_check_lsn( > > return valid; > } > + > +bool > +xfs_log_in_recovery( > + struct xfs_mount *mp) > +{ > + if (!mp->m_log) > + return false; > + if (mp->m_log->l_flags & XLOG_ACTIVE_RECOVERY) > + return true; > + return false; > +} > diff --git a/fs/xfs/xfs_log.h b/fs/xfs/xfs_log.h > index 3c1f6a8b4b70..410d4b3a20d3 100644 > --- a/fs/xfs/xfs_log.h > +++ b/fs/xfs/xfs_log.h > @@ -152,6 +152,7 @@ bool xfs_log_item_in_current_chkpt(struct xfs_log_item *lip); > > void xfs_log_work_queue(struct xfs_mount *mp); > void xfs_log_quiesce(struct xfs_mount *mp); > -bool xfs_log_check_lsn(struct xfs_mount *, xfs_lsn_t); > +bool xfs_log_check_lsn(struct xfs_mount *mp, xfs_lsn_t lsn); > +bool xfs_log_in_recovery(struct xfs_mount *mp); > > #endif /* __XFS_LOG_H__ */ > -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 6/14/18 8:57 PM, Eric Sandeen wrote: > On 6/14/18 8:43 PM, Dave Chinner wrote: >> From: Dave Chinner <dchinner@redhat.com> >> >> A log recovery failure has been reproduced where a symlink inode has >> a zero length in extent form. It was caused by a shutdown during a >> combined fstress+fsmark workload. >> >> To fix it, we have to allow zero length symlink inodes through >> xfs_dinode_verify() during log recovery. We already specifically >> check and allow this case in the shortform symlink fork verifier, >> but in this case we don't get that far, and the inode is not in >> shortform format. >> >> Update the dinode verifier to handle this case, and change the >> symlink fork verifier to only allow this case to exist during log >> recovery. >> >> Signed-off-by: Dave Chinner <dchinner@redhat.com> > > Looks reasonable. > > Reviewed-by: Eric Sandeen <sandeen@redhat.com> Also probably worth tagging with Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=200063 -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jun 14, 2018 at 09:02:11PM -0500, Eric Sandeen wrote: > > > On 6/14/18 8:57 PM, Eric Sandeen wrote: > > On 6/14/18 8:43 PM, Dave Chinner wrote: > >> From: Dave Chinner <dchinner@redhat.com> > >> > >> A log recovery failure has been reproduced where a symlink inode has > >> a zero length in extent form. It was caused by a shutdown during a > >> combined fstress+fsmark workload. > >> > >> To fix it, we have to allow zero length symlink inodes through > >> xfs_dinode_verify() during log recovery. We already specifically > >> check and allow this case in the shortform symlink fork verifier, > >> but in this case we don't get that far, and the inode is not in > >> shortform format. > >> > >> Update the dinode verifier to handle this case, and change the > >> symlink fork verifier to only allow this case to exist during log > >> recovery. > >> > >> Signed-off-by: Dave Chinner <dchinner@redhat.com> > > > > Looks reasonable. > > > > Reviewed-by: Eric Sandeen <sandeen@redhat.com> > > Also probably worth tagging with > > Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=200063 Why? I didn't know that existed until you pointed it out, and I still haven't looked at it as I write this.... Cheers, Dave.
On Fri, Jun 15, 2018 at 11:43:14AM +1000, Dave Chinner wrote: > From: Dave Chinner <dchinner@redhat.com> > > A log recovery failure has been reproduced where a symlink inode has > a zero length in extent form. It was caused by a shutdown during a > combined fstress+fsmark workload. > > To fix it, we have to allow zero length symlink inodes through > xfs_dinode_verify() during log recovery. We already specifically > check and allow this case in the shortform symlink fork verifier, > but in this case we don't get that far, and the inode is not in > shortform format. > > Update the dinode verifier to handle this case, and change the > symlink fork verifier to only allow this case to exist during log > recovery. > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- Seems Ok to me, but before we restrict some of the existing checks to log recovery I am curious about one thing. xfs_inactive_symlink() has this: /* * Zero length symlinks _can_ exist. */ pathlen = (int)ip->i_d.di_size; if (!pathlen) { xfs_iunlock(ip, XFS_ILOCK_EXCL); return 0; } I'm not quite sure what case that covers, but it seems slightly inconsistent with the fork verifer change (simply because that path is not exclusive to the read from disk case), at least. Any idea? Brian > fs/xfs/libxfs/xfs_inode_buf.c | 16 +++++++++++++--- > fs/xfs/libxfs/xfs_symlink_remote.c | 12 +++++++++--- > fs/xfs/xfs_log.c | 11 +++++++++++ > fs/xfs/xfs_log.h | 3 ++- > 4 files changed, 35 insertions(+), 7 deletions(-) > > diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c > index d38d724534c4..bec9178377e3 100644 > --- a/fs/xfs/libxfs/xfs_inode_buf.c > +++ b/fs/xfs/libxfs/xfs_inode_buf.c > @@ -19,6 +19,7 @@ > #include "xfs_trans.h" > #include "xfs_ialloc.h" > #include "xfs_dir2.h" > +#include "xfs_log.h" > > #include <linux/iversion.h> > > @@ -411,9 +412,18 @@ xfs_dinode_verify( > if (mode && xfs_mode_to_ftype(mode) == XFS_DIR3_FT_UNKNOWN) > return __this_address; > > - /* No zero-length symlinks/dirs. */ > - if ((S_ISLNK(mode) || S_ISDIR(mode)) && di_size == 0) > - return __this_address; > + /* > + * Normally both symlinks and dirs cannot be zero length. However, if a > + * symlink is in the process of being torn down and there's a > + * shutdown/crash, the symlink on disk may have a zero size. Hence we > + * only allow zero length symlinks during log recovery. > + */ > + if (di_size == 0) { > + if (S_ISDIR(mode)) > + return __this_address; > + if (S_ISLNK(mode) && !xfs_log_in_recovery(mp)) > + return __this_address; > + } > > /* Fork checks carried over from xfs_iformat_fork */ > if (mode && > diff --git a/fs/xfs/libxfs/xfs_symlink_remote.c b/fs/xfs/libxfs/xfs_symlink_remote.c > index 95374ab2dee7..b1f0dd14f805 100644 > --- a/fs/xfs/libxfs/xfs_symlink_remote.c > +++ b/fs/xfs/libxfs/xfs_symlink_remote.c > @@ -215,9 +215,15 @@ xfs_symlink_shortform_verify( > size = ifp->if_bytes; > endp = sfp + size; > > - /* Zero length symlinks can exist while we're deleting a remote one. */ > - if (size == 0) > - return NULL; > + /* > + * Zero length symlinks can only pass through here during log recovery > + * while recovering deletion of a remote symlink. > + */ > + if (size == 0) { > + if (xfs_log_in_recovery(ip->i_mount)) > + return NULL; > + return __this_address; > + } > > /* No negative sizes or overly long symlink targets. */ > if (size < 0 || size > XFS_SYMLINK_MAXLEN) > diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c > index 5e56f3b93d4b..012397f6ec5a 100644 > --- a/fs/xfs/xfs_log.c > +++ b/fs/xfs/xfs_log.c > @@ -4083,3 +4083,14 @@ xfs_log_check_lsn( > > return valid; > } > + > +bool > +xfs_log_in_recovery( > + struct xfs_mount *mp) > +{ > + if (!mp->m_log) > + return false; > + if (mp->m_log->l_flags & XLOG_ACTIVE_RECOVERY) > + return true; > + return false; > +} > diff --git a/fs/xfs/xfs_log.h b/fs/xfs/xfs_log.h > index 3c1f6a8b4b70..410d4b3a20d3 100644 > --- a/fs/xfs/xfs_log.h > +++ b/fs/xfs/xfs_log.h > @@ -152,6 +152,7 @@ bool xfs_log_item_in_current_chkpt(struct xfs_log_item *lip); > > void xfs_log_work_queue(struct xfs_mount *mp); > void xfs_log_quiesce(struct xfs_mount *mp); > -bool xfs_log_check_lsn(struct xfs_mount *, xfs_lsn_t); > +bool xfs_log_check_lsn(struct xfs_mount *mp, xfs_lsn_t lsn); > +bool xfs_log_in_recovery(struct xfs_mount *mp); > > #endif /* __XFS_LOG_H__ */ > -- > 2.17.0 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jun 15, 2018 at 07:31:26AM -0400, Brian Foster wrote: > On Fri, Jun 15, 2018 at 11:43:14AM +1000, Dave Chinner wrote: > > From: Dave Chinner <dchinner@redhat.com> > > > > A log recovery failure has been reproduced where a symlink inode has > > a zero length in extent form. It was caused by a shutdown during a > > combined fstress+fsmark workload. > > > > To fix it, we have to allow zero length symlink inodes through > > xfs_dinode_verify() during log recovery. We already specifically > > check and allow this case in the shortform symlink fork verifier, > > but in this case we don't get that far, and the inode is not in > > shortform format. > > > > Update the dinode verifier to handle this case, and change the > > symlink fork verifier to only allow this case to exist during log > > recovery. > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > > --- > > Seems Ok to me, but before we restrict some of the existing checks to > log recovery I am curious about one thing. xfs_inactive_symlink() has > this: > > /* > * Zero length symlinks _can_ exist. > */ > pathlen = (int)ip->i_d.di_size; > if (!pathlen) { > xfs_iunlock(ip, XFS_ILOCK_EXCL); > return 0; > } > > I'm not quite sure what case that covers, but it seems slightly > inconsistent with the fork verifer change (simply because that path is > not exclusive to the read from disk case), at least. Any idea? Yeah, that's what I'm trying to chase down right now. I had the verifier fire on inode writeback during generic/269. I don't know yet where these zero length symlinks are coming from, and none of the comments (there's a couple that say the above) actually give any hint to their source. Cheers, Dave.
On Sat, Jun 16, 2018 at 10:10:34AM +1000, Dave Chinner wrote: > On Fri, Jun 15, 2018 at 07:31:26AM -0400, Brian Foster wrote: > > On Fri, Jun 15, 2018 at 11:43:14AM +1000, Dave Chinner wrote: > > > From: Dave Chinner <dchinner@redhat.com> > > > > > > A log recovery failure has been reproduced where a symlink inode has > > > a zero length in extent form. It was caused by a shutdown during a > > > combined fstress+fsmark workload. > > > > > > To fix it, we have to allow zero length symlink inodes through > > > xfs_dinode_verify() during log recovery. We already specifically > > > check and allow this case in the shortform symlink fork verifier, > > > but in this case we don't get that far, and the inode is not in > > > shortform format. > > > > > > Update the dinode verifier to handle this case, and change the > > > symlink fork verifier to only allow this case to exist during log > > > recovery. > > > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > > > --- > > > > Seems Ok to me, but before we restrict some of the existing checks to > > log recovery I am curious about one thing. xfs_inactive_symlink() has > > this: > > > > /* > > * Zero length symlinks _can_ exist. > > */ > > pathlen = (int)ip->i_d.di_size; > > if (!pathlen) { > > xfs_iunlock(ip, XFS_ILOCK_EXCL); > > return 0; > > } > > > > I'm not quite sure what case that covers, but it seems slightly > > inconsistent with the fork verifer change (simply because that path is > > not exclusive to the read from disk case), at least. Any idea? > > Yeah, that's what I'm trying to chase down right now. I had the > verifier fire on inode writeback during generic/269. I don't know > yet where these zero length symlinks are coming from, and none of > the comments (there's a couple that say the above) > actually give any hint to their source. Ok, so there is this comment in fs/namei.c w.r.t. symlink handling before getname_flags(): * POSIX.1 2.4: an empty pathname is invalid (ENOENT). So the call chain is sys_symlink(oldname ....) do_symlinkat(oldname ...) getname(oldname) getname_flags(oldname, 0, NULL) len = strncpy_from_user(... oldname ....) .... if (!len) { if (!(flags & LOOKUP_EMPTY)) return -ENOENT; } So we should never see a zero length symlink from userspace as flags is always zero. Hence if we are seeing zero length symlinks on disk, then that's an XFS implementation issue, not a user API requirement. There's two issues in the symlink code that can lead to zero length symlinks firing the verifiers. They are symptoms of the same core issue in xfs_inactive_symlink(): the inode is unlocked between the symlink inactivation/truncation and the inode being freed. This opens a window for the inode to be written to disk before it xfs_ifree() removes it from the unlinked list, marks it free in the inobt and zeros the mode. The first, and simplest to solve issue is the shortform verifier. This verifier doesn't actually verify on disk state - it verifies *in memory inode fork state*. Specifically, it checks for a zero length inode fork (ifp->if_bytes) and says specifically "this can happen". The only place it can happen in in the window between xfs_inactive_symlink() and xfs_ifree() because xfs_inactive_symlink() tears down the data fork. It doesn't, however, change the inode size, so if the inode is written back to disk in this window, it's written with a non-zero size, leaving the data fork in the inode untouched. i.e. the inode on disk is still a valid symlink. To fix this is easy. xfs_ifree() actually cleans up the in-memory data and attr fork structures, and so there is absolutely no need to do it in xfs_inactive_symlink(). With that change, the symlink verifier error goes away. Which leaves remote symlink inactivation. This runs a transaction that truncates away the symlink extent and sets the inode size to zero. IOWs, it creates an actual path for zero length symlinks on disk. However, the symlink inode at this point is unreferenced by userspace and is on the unlinked list, and hence userspace can never see a zero length symlink inode. This does, however, create a problem - we can get zero length symlinks on disk in log recovery because the inode size is set to zero at the same time the EFI intents are recorded. hence if log recovery then reads the inode off disk to replay EFIs or other non-completed intents, it can see a symlink inode with zero length. So we have a choice here: either special case log recovery for symlink inode verification, or prevent zero length extent for symlink inodes from existing on disk. The former is essentially the patch I posted, the latter requires discussion. If we want to avoid zero length extent form symlinks on disk we either need to make the inactivation and freeing atomic, or we can make symlink inactivation change the type of inode to something that allows zero length. The former is complex and a major undertaking (new deferred op, a bunch of new intents, log recovery work, etc) while the latter is one line of code. i.e. we simply change the mode of the inode to a regular file at the same time we set the size to zero. If the transaction with the EFIs and zero size goes to disk, we don't really care what the inode type is. it's on the unlinked list, can't be seen from userspace, and we just need to run extent removal and freeing on it in log recovery. Hence if we change it to be a regular file inode, then we maintain the "no zero length symlinks on disk" rule, and we get cleanup occurring without any new code or concerns being created. Thoughts? Cheers, Dave.
On Fri, Jun 15, 2018 at 11:43:14AM +1000, Dave Chinner wrote: > From: Dave Chinner <dchinner@redhat.com> > > A log recovery failure has been reproduced where a symlink inode has > a zero length in extent form. It was caused by a shutdown during a > combined fstress+fsmark workload. > > To fix it, we have to allow zero length symlink inodes through > xfs_dinode_verify() during log recovery. We already specifically > check and allow this case in the shortform symlink fork verifier, > but in this case we don't get that far, and the inode is not in > shortform format. > > Update the dinode verifier to handle this case, and change the > symlink fork verifier to only allow this case to exist during log > recovery. > > Signed-off-by: Dave Chinner <dchinner@redhat.com> Self-NAK on this - I've got a patch that prevents zero length symlinks from going to disk, so allowing them through log recovery is not necessary - they are always invalid..... Cheers, Dave.
diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index d38d724534c4..bec9178377e3 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -19,6 +19,7 @@ #include "xfs_trans.h" #include "xfs_ialloc.h" #include "xfs_dir2.h" +#include "xfs_log.h" #include <linux/iversion.h> @@ -411,9 +412,18 @@ xfs_dinode_verify( if (mode && xfs_mode_to_ftype(mode) == XFS_DIR3_FT_UNKNOWN) return __this_address; - /* No zero-length symlinks/dirs. */ - if ((S_ISLNK(mode) || S_ISDIR(mode)) && di_size == 0) - return __this_address; + /* + * Normally both symlinks and dirs cannot be zero length. However, if a + * symlink is in the process of being torn down and there's a + * shutdown/crash, the symlink on disk may have a zero size. Hence we + * only allow zero length symlinks during log recovery. + */ + if (di_size == 0) { + if (S_ISDIR(mode)) + return __this_address; + if (S_ISLNK(mode) && !xfs_log_in_recovery(mp)) + return __this_address; + } /* Fork checks carried over from xfs_iformat_fork */ if (mode && diff --git a/fs/xfs/libxfs/xfs_symlink_remote.c b/fs/xfs/libxfs/xfs_symlink_remote.c index 95374ab2dee7..b1f0dd14f805 100644 --- a/fs/xfs/libxfs/xfs_symlink_remote.c +++ b/fs/xfs/libxfs/xfs_symlink_remote.c @@ -215,9 +215,15 @@ xfs_symlink_shortform_verify( size = ifp->if_bytes; endp = sfp + size; - /* Zero length symlinks can exist while we're deleting a remote one. */ - if (size == 0) - return NULL; + /* + * Zero length symlinks can only pass through here during log recovery + * while recovering deletion of a remote symlink. + */ + if (size == 0) { + if (xfs_log_in_recovery(ip->i_mount)) + return NULL; + return __this_address; + } /* No negative sizes or overly long symlink targets. */ if (size < 0 || size > XFS_SYMLINK_MAXLEN) diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c index 5e56f3b93d4b..012397f6ec5a 100644 --- a/fs/xfs/xfs_log.c +++ b/fs/xfs/xfs_log.c @@ -4083,3 +4083,14 @@ xfs_log_check_lsn( return valid; } + +bool +xfs_log_in_recovery( + struct xfs_mount *mp) +{ + if (!mp->m_log) + return false; + if (mp->m_log->l_flags & XLOG_ACTIVE_RECOVERY) + return true; + return false; +} diff --git a/fs/xfs/xfs_log.h b/fs/xfs/xfs_log.h index 3c1f6a8b4b70..410d4b3a20d3 100644 --- a/fs/xfs/xfs_log.h +++ b/fs/xfs/xfs_log.h @@ -152,6 +152,7 @@ bool xfs_log_item_in_current_chkpt(struct xfs_log_item *lip); void xfs_log_work_queue(struct xfs_mount *mp); void xfs_log_quiesce(struct xfs_mount *mp); -bool xfs_log_check_lsn(struct xfs_mount *, xfs_lsn_t); +bool xfs_log_check_lsn(struct xfs_mount *mp, xfs_lsn_t lsn); +bool xfs_log_in_recovery(struct xfs_mount *mp); #endif /* __XFS_LOG_H__ */