| Message ID | 1452219742-18507-1-git-send-email-jeff.layton@primarydata.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, 7 Jan 2016 21:22:22 -0500 Jeff Layton <jlayton@poochiereds.net> wrote: > Dmitry reported that he was able to reproduce the WARN_ON_ONCE that > fires in locks_free_lock_context when the flc_posix list isn't empty. > > The problem turns out to be that we're basically rebuilding the > file_lock from scratch in fcntl_setlk when we discover that the setlk > has raced with a close. If the l_whence field is SEEK_CUR or SEEK_END, > then we may end up with fl_start and fl_end values that differ from > when the lock was initially set, if the file position or length of the > file has changed in the interim. > > Fix this by just reusing the same lock request structure, and simply > override fl_type value with F_UNLCK as appropriate. That ensures that > we really are unlocking the lock that was initially set. > > While we're there, make sure that we do pop a WARN_ON_ONCE if the > removal ever fails. Also return -EBADF in this event, since that's > what we would have returned if the close had happened earlier. > > Cc: <stable@vger.kernel.org> > Reported-by: Dmitry Vyukov <dvyukov@google.com> > Signed-off-by: Jeff Layton <jeff.layton@primarydata.com> > --- > fs/locks.c | 19 ++++++++++--------- > 1 file changed, 10 insertions(+), 9 deletions(-) > > diff --git a/fs/locks.c b/fs/locks.c > index 593dca300b29..0db640e4ced4 100644 > --- a/fs/locks.c > +++ b/fs/locks.c > @@ -2181,7 +2181,6 @@ int fcntl_setlk(unsigned int fd, struct file *filp, unsigned int cmd, > goto out; > } > > -again: > error = flock_to_posix_lock(filp, file_lock, &flock); > if (error) > goto out; > @@ -2231,9 +2230,11 @@ again: > spin_lock(¤t->files->file_lock); > f = fcheck(fd); > spin_unlock(¤t->files->file_lock); > - if (!error && f != filp && flock.l_type != F_UNLCK) { > - flock.l_type = F_UNLCK; > - goto again; > + if (!error && f != filp && file_lock->fl_type != F_UNLCK) { > + file_lock->fl_type = F_UNLCK; > + error = do_lock_file_wait(filp, cmd, file_lock); > + WARN_ON_ONCE(error); > + error = -EBADF; > } > > out: > @@ -2321,7 +2322,6 @@ int fcntl_setlk64(unsigned int fd, struct file *filp, unsigned int cmd, > goto out; > } > > -again: > error = flock64_to_posix_lock(filp, file_lock, &flock); > if (error) > goto out; > @@ -2366,11 +2366,12 @@ again: > spin_lock(¤t->files->file_lock); > f = fcheck(fd); > spin_unlock(¤t->files->file_lock); > - if (!error && f != filp && flock.l_type != F_UNLCK) { > - flock.l_type = F_UNLCK; > - goto again; > + if (!error && f != filp && file_lock->fl_type != F_UNLCK) { > + file_lock->fl_type = F_UNLCK; > + error = do_lock_file_wait(filp, cmd, file_lock); > + WARN_ON_ONCE(error); > + error = -EBADF; > } > - > out: > locks_free_lock(file_lock); > return error; While this does fix Dmitri's reproducer, I think the basic concept of removing locks like this after they are set is racy. Consider where we have two threads: Thread1 Thread2 ---------------------------------------------------------------------------- fd1 = memfd_create(...); fd2 = dup(fd1); fcntl(fd2, F_SETLK); (Here we call fcntl, and lock is set, but task gets scheduled out before fcheck) close(fd2) fcntl(fd1, F_SETLK...); Task scheduled back in, does fcheck for fd2 and finds that it's gone. Removes the lock that Thread1 just set. So that seems wrong...in the face of the race above we can end up with no lock set on the file, even though Thread1 thinks it has one. It is a pretty unlikely race, but I don't see anything that prevents it. The fix for filesystems that do not define their own ->lock op would be pretty simple. We could do a fcheck after taking the flc_lock, but before setting the lock on the file. The flc_lock should be enough to prevent that race (though we may need to revisit some of the lockless checks in locks_remove_posix). That wouldn't work for filesystems that do set ->lock though, and I think we really do need a more general solution there. The good news is that OFD locks should be exempt from that fcheck altogether. I'll spin up another patch for that, so we can at least ensure that they aren't subject to that race. Any thoughts on how to fix the above for traditional POSIX locks though?
The first patch in this patchset fixes a very long-standing bug in the handling of races between setlk and close. That one should be appropriate for all stable kernels and should apply to most kernels as-is. The second patch exempts OFD locks from setlk/close race handling since they shouldn't need it anyway. The rest of the patches add some better debugging for these problems and do a little function name cleanup. I'm planning to go ahead and put these into linux-next today and send them to Linus for the 4.5 merge, unless there are any objections... Jeff Layton (6): locks: fix unlock when fcntl_setlk races with a close locks: don't check for race with close when setting OFD lock locks: sprinkle some tracepoints around the file locking code locks: pass inode pointer to locks_free_lock_context locks: prink more detail when there are leaked locks locks: rename __posix_lock_file to posix_lock_inode fs/inode.c | 2 +- fs/locks.c | 123 +++++++++++++++++++++++++++------------- include/linux/fs.h | 4 +- include/trace/events/filelock.h | 77 +++++++++++++++++++++++++ 4 files changed, 165 insertions(+), 41 deletions(-)
On Fri, Jan 08, 2016 at 07:48:04AM -0500, Jeff Layton wrote: > On Thu, 7 Jan 2016 21:22:22 -0500 > Jeff Layton <jlayton@poochiereds.net> wrote: > > > Dmitry reported that he was able to reproduce the WARN_ON_ONCE that > > fires in locks_free_lock_context when the flc_posix list isn't empty. > > > > The problem turns out to be that we're basically rebuilding the > > file_lock from scratch in fcntl_setlk when we discover that the setlk > > has raced with a close. If the l_whence field is SEEK_CUR or SEEK_END, > > then we may end up with fl_start and fl_end values that differ from > > when the lock was initially set, if the file position or length of the > > file has changed in the interim. > > > > Fix this by just reusing the same lock request structure, and simply > > override fl_type value with F_UNLCK as appropriate. That ensures that > > we really are unlocking the lock that was initially set. > > > > While we're there, make sure that we do pop a WARN_ON_ONCE if the > > removal ever fails. Also return -EBADF in this event, since that's > > what we would have returned if the close had happened earlier. > > > > Cc: <stable@vger.kernel.org> > > Reported-by: Dmitry Vyukov <dvyukov@google.com> > > Signed-off-by: Jeff Layton <jeff.layton@primarydata.com> > > --- > > fs/locks.c | 19 ++++++++++--------- > > 1 file changed, 10 insertions(+), 9 deletions(-) > > > > diff --git a/fs/locks.c b/fs/locks.c > > index 593dca300b29..0db640e4ced4 100644 > > --- a/fs/locks.c > > +++ b/fs/locks.c > > @@ -2181,7 +2181,6 @@ int fcntl_setlk(unsigned int fd, struct file *filp, unsigned int cmd, > > goto out; > > } > > > > -again: > > error = flock_to_posix_lock(filp, file_lock, &flock); > > if (error) > > goto out; > > @@ -2231,9 +2230,11 @@ again: > > spin_lock(¤t->files->file_lock); > > f = fcheck(fd); > > spin_unlock(¤t->files->file_lock); > > - if (!error && f != filp && flock.l_type != F_UNLCK) { > > - flock.l_type = F_UNLCK; > > - goto again; > > + if (!error && f != filp && file_lock->fl_type != F_UNLCK) { > > + file_lock->fl_type = F_UNLCK; > > + error = do_lock_file_wait(filp, cmd, file_lock); > > + WARN_ON_ONCE(error); > > + error = -EBADF; > > } > > > > out: > > @@ -2321,7 +2322,6 @@ int fcntl_setlk64(unsigned int fd, struct file *filp, unsigned int cmd, > > goto out; > > } > > > > -again: > > error = flock64_to_posix_lock(filp, file_lock, &flock); > > if (error) > > goto out; > > @@ -2366,11 +2366,12 @@ again: > > spin_lock(¤t->files->file_lock); > > f = fcheck(fd); > > spin_unlock(¤t->files->file_lock); > > - if (!error && f != filp && flock.l_type != F_UNLCK) { > > - flock.l_type = F_UNLCK; > > - goto again; > > + if (!error && f != filp && file_lock->fl_type != F_UNLCK) { > > + file_lock->fl_type = F_UNLCK; > > + error = do_lock_file_wait(filp, cmd, file_lock); > > + WARN_ON_ONCE(error); > > + error = -EBADF; > > } > > - > > out: > > locks_free_lock(file_lock); > > return error; > > While this does fix Dmitri's reproducer, I think the basic concept of > removing locks like this after they are set is racy. Consider where we > have two threads: > > Thread1 Thread2 > ---------------------------------------------------------------------------- > fd1 = memfd_create(...); > fd2 = dup(fd1); > fcntl(fd2, F_SETLK); > (Here we call fcntl, and lock is set, but > task gets scheduled out before fcheck) > close(fd2) > fcntl(fd1, F_SETLK...); > > Task scheduled back in, does fcheck for fd2 > and finds that it's gone. Removes the lock > that Thread1 just set. > > So that seems wrong...in the face of the race above we can end up with > no lock set on the file, even though Thread1 thinks it has one. It is a > pretty unlikely race, but I don't see anything that prevents it. > > The fix for filesystems that do not define their own ->lock op would be > pretty simple. We could do a fcheck after taking the flc_lock, but > before setting the lock on the file. The flc_lock should be enough to > prevent that race (though we may need to revisit some of the lockless > checks in locks_remove_posix). That wouldn't work for filesystems that > do set ->lock though, and I think we really do need a more general > solution there. > > The good news is that OFD locks should be exempt from that fcheck > altogether. I'll spin up another patch for that, so we can at least > ensure that they aren't subject to that race. > > Any thoughts on how to fix the above for traditional POSIX locks though? This logic seems to fall into a common trap by assuming that the result of a posix lock followed by an unlock is a no-op. The assumption is false because the region, or parts of it, may have been locked by the same owner before the initial lock. You might think you're free of that logic since closing is a scorched-earth "remove every lock owned by this owner" event, except that as you point out the lock+unlock isn't atomic here.... OK, I'm just repeating what you've said really. I don't know how to fix it. --b. -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/fs/locks.c b/fs/locks.c index 593dca300b29..0db640e4ced4 100644 --- a/fs/locks.c +++ b/fs/locks.c @@ -2181,7 +2181,6 @@ int fcntl_setlk(unsigned int fd, struct file *filp, unsigned int cmd, goto out; } -again: error = flock_to_posix_lock(filp, file_lock, &flock); if (error) goto out; @@ -2231,9 +2230,11 @@ again: spin_lock(¤t->files->file_lock); f = fcheck(fd); spin_unlock(¤t->files->file_lock); - if (!error && f != filp && flock.l_type != F_UNLCK) { - flock.l_type = F_UNLCK; - goto again; + if (!error && f != filp && file_lock->fl_type != F_UNLCK) { + file_lock->fl_type = F_UNLCK; + error = do_lock_file_wait(filp, cmd, file_lock); + WARN_ON_ONCE(error); + error = -EBADF; } out: @@ -2321,7 +2322,6 @@ int fcntl_setlk64(unsigned int fd, struct file *filp, unsigned int cmd, goto out; } -again: error = flock64_to_posix_lock(filp, file_lock, &flock); if (error) goto out; @@ -2366,11 +2366,12 @@ again: spin_lock(¤t->files->file_lock); f = fcheck(fd); spin_unlock(¤t->files->file_lock); - if (!error && f != filp && flock.l_type != F_UNLCK) { - flock.l_type = F_UNLCK; - goto again; + if (!error && f != filp && file_lock->fl_type != F_UNLCK) { + file_lock->fl_type = F_UNLCK; + error = do_lock_file_wait(filp, cmd, file_lock); + WARN_ON_ONCE(error); + error = -EBADF; } - out: locks_free_lock(file_lock); return error;
Dmitry reported that he was able to reproduce the WARN_ON_ONCE that fires in locks_free_lock_context when the flc_posix list isn't empty. The problem turns out to be that we're basically rebuilding the file_lock from scratch in fcntl_setlk when we discover that the setlk has raced with a close. If the l_whence field is SEEK_CUR or SEEK_END, then we may end up with fl_start and fl_end values that differ from when the lock was initially set, if the file position or length of the file has changed in the interim. Fix this by just reusing the same lock request structure, and simply override fl_type value with F_UNLCK as appropriate. That ensures that we really are unlocking the lock that was initially set. While we're there, make sure that we do pop a WARN_ON_ONCE if the removal ever fails. Also return -EBADF in this event, since that's what we would have returned if the close had happened earlier. Cc: <stable@vger.kernel.org> Reported-by: Dmitry Vyukov <dvyukov@google.com> Signed-off-by: Jeff Layton <jeff.layton@primarydata.com> --- fs/locks.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)