| Message ID | 20230922-ctime-v8-0-45f0c236ede1@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | fs: multigrain timestamps for XFS's change_cookie | expand |
On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > My initial goal was to implement multigrain timestamps on most major > filesystems, so we could present them to userland, and use them for > NFSv3, etc. > > With the current implementation however, we can't guarantee that a file > with a coarse grained timestamp modified after one with a fine grained > timestamp will always appear to have a later value. This could confuse > some programs like make, rsync, find, etc. that depend on strict > ordering requirements for timestamps. > > The goal of this version is more modest: fix XFS' change attribute. > XFS's change attribute is bumped on atime updates in addition to other > deliberate changes. This makes it unsuitable for export via nfsd. > > Jan Kara suggested keeping this functionality internal-only for now and > plumbing the fine grained timestamps through getattr [1]. This set takes > a slightly different approach and has XFS use the fine-grained attr to > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > While we keep fine-grained timestamps in struct inode, when presenting > the timestamps via getattr, we truncate them at a granularity of number > of ns per jiffy, That's not good, because user explicitly set granular mtime would be truncated too and booting with different kernels (HZ) would change the observed timestamps of files. > which allows us to smooth over the fuzz that causes > ordering problems. > The reported ordering problems (i.e. cp -u) is not even limited to the scope of a single fs, right? Thinking out loud - if the QERIED bit was not per inode timestamp but instead in a global fs_multigrain_ts variable, then all the inodes of all the mgtime fs would be using globally ordered timestamps That should eliminate the reported issues with time reorder for fine vs coarse grained timestamps. The risk of extra unneeded "change cookie" updates compared to per inode QUERIED bit may exist, but I think it is a rather small overhead and maybe worth the tradeoff of having to maintain a real per inode "change cookie" in addition to a "globally ordered mgtime"? If this idea is acceptable, you may still be able to salvage the reverted ctime series for 6.7, because the change to use global mgtime should be quite trivial? Thanks, Amir.
On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > My initial goal was to implement multigrain timestamps on most major > > filesystems, so we could present them to userland, and use them for > > NFSv3, etc. > > > > With the current implementation however, we can't guarantee that a file > > with a coarse grained timestamp modified after one with a fine grained > > timestamp will always appear to have a later value. This could confuse > > some programs like make, rsync, find, etc. that depend on strict > > ordering requirements for timestamps. > > > > The goal of this version is more modest: fix XFS' change attribute. > > XFS's change attribute is bumped on atime updates in addition to other > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > Jan Kara suggested keeping this functionality internal-only for now and > > plumbing the fine grained timestamps through getattr [1]. This set takes > > a slightly different approach and has XFS use the fine-grained attr to > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > While we keep fine-grained timestamps in struct inode, when presenting > > the timestamps via getattr, we truncate them at a granularity of number > > of ns per jiffy, > > That's not good, because user explicitly set granular mtime would be > truncated too and booting with different kernels (HZ) would change > the observed timestamps of files. > That's a very good point. > > which allows us to smooth over the fuzz that causes > > ordering problems. > > > > The reported ordering problems (i.e. cp -u) is not even limited to the > scope of a single fs, right? > It isn't. Most of the tools we're concerned with don't generally care about filesystem boundaries. > Thinking out loud - if the QERIED bit was not per inode timestamp > but instead in a global fs_multigrain_ts variable, then all the inodes > of all the mgtime fs would be using globally ordered timestamps > > That should eliminate the reported issues with time reorder for > fine vs coarse grained timestamps. > > The risk of extra unneeded "change cookie" updates compared to > per inode QUERIED bit may exist, but I think it is a rather small overhead > and maybe worth the tradeoff of having to maintain a real per inode > "change cookie" in addition to a "globally ordered mgtime"? > > If this idea is acceptable, you may still be able to salvage the reverted > ctime series for 6.7, because the change to use global mgtime should > be quite trivial? > This is basically the idea I was going to look at next once I got some other stuff settled here: Basically, when we apply a fine-grained timestamp to an inode, we'd advance the coarse-grained clock that filesystems use to that value. It could cause some write amplification: if you are streaming writes to a bunch of files at the same time and someone stats one of them, then they'd all end up getting an extra inode transaction. That doesn't sound _too_ bad on its face, but I probably need to implement it and then run some numbers to see.
On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > My initial goal was to implement multigrain timestamps on most major > > filesystems, so we could present them to userland, and use them for > > NFSv3, etc. > > > > With the current implementation however, we can't guarantee that a file > > with a coarse grained timestamp modified after one with a fine grained > > timestamp will always appear to have a later value. This could confuse > > some programs like make, rsync, find, etc. that depend on strict > > ordering requirements for timestamps. > > > > The goal of this version is more modest: fix XFS' change attribute. > > XFS's change attribute is bumped on atime updates in addition to other > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > Jan Kara suggested keeping this functionality internal-only for now and > > plumbing the fine grained timestamps through getattr [1]. This set takes > > a slightly different approach and has XFS use the fine-grained attr to > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > While we keep fine-grained timestamps in struct inode, when presenting > > the timestamps via getattr, we truncate them at a granularity of number > > of ns per jiffy, > > That's not good, because user explicitly set granular mtime would be > truncated too and booting with different kernels (HZ) would change > the observed timestamps of files. > Thinking about this some more, I think the first problem is easily addressable: The ctime isn't explicitly settable and with this set, we're already not truncating the atime. We haven't used any of the extra bits in the mtime yet, so we could just carve out a flag in there that says "this mtime was explicitly set and shouldn't be truncated before presentation". The second problem (booting to older kernel) is a bit tougher to deal with however. I'll have to think about that one a bit more.
On Sat, Sep 23, 2023 at 1:46 PM Jeff Layton <jlayton@kernel.org> wrote: > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > My initial goal was to implement multigrain timestamps on most major > > > filesystems, so we could present them to userland, and use them for > > > NFSv3, etc. > > > > > > With the current implementation however, we can't guarantee that a file > > > with a coarse grained timestamp modified after one with a fine grained > > > timestamp will always appear to have a later value. This could confuse > > > some programs like make, rsync, find, etc. that depend on strict > > > ordering requirements for timestamps. > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > XFS's change attribute is bumped on atime updates in addition to other > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > a slightly different approach and has XFS use the fine-grained attr to > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > the timestamps via getattr, we truncate them at a granularity of number > > > of ns per jiffy, > > > > That's not good, because user explicitly set granular mtime would be > > truncated too and booting with different kernels (HZ) would change > > the observed timestamps of files. > > > > Thinking about this some more, I think the first problem is easily > addressable: > > The ctime isn't explicitly settable and with this set, we're already not > truncating the atime. We haven't used any of the extra bits in the mtime > yet, so we could just carve out a flag in there that says "this mtime > was explicitly set and shouldn't be truncated before presentation". > I thought about this option too. But note that the "mtime was explicitly set" flag needs to be persisted to disk so you cannot store it in the high nsec bits. At least XFS won't store those bits if you use them - they have to be translated to an XFS inode flag and I don't know if changing XFS on-disk format was on your wish list. Thanks, Amir.
On Sat, Sep 23, 2023 at 1:22 PM Jeff Layton <jlayton@kernel.org> wrote: > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > My initial goal was to implement multigrain timestamps on most major > > > filesystems, so we could present them to userland, and use them for > > > NFSv3, etc. > > > > > > With the current implementation however, we can't guarantee that a file > > > with a coarse grained timestamp modified after one with a fine grained > > > timestamp will always appear to have a later value. This could confuse > > > some programs like make, rsync, find, etc. that depend on strict > > > ordering requirements for timestamps. > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > XFS's change attribute is bumped on atime updates in addition to other > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > a slightly different approach and has XFS use the fine-grained attr to > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > the timestamps via getattr, we truncate them at a granularity of number > > > of ns per jiffy, > > > > That's not good, because user explicitly set granular mtime would be > > truncated too and booting with different kernels (HZ) would change > > the observed timestamps of files. > > > > That's a very good point. > > > > which allows us to smooth over the fuzz that causes > > > ordering problems. > > > > > > > The reported ordering problems (i.e. cp -u) is not even limited to the > > scope of a single fs, right? > > > > It isn't. Most of the tools we're concerned with don't generally care > about filesystem boundaries. > > > Thinking out loud - if the QERIED bit was not per inode timestamp > > but instead in a global fs_multigrain_ts variable, then all the inodes > > of all the mgtime fs would be using globally ordered timestamps > > > > That should eliminate the reported issues with time reorder for > > fine vs coarse grained timestamps. > > > > The risk of extra unneeded "change cookie" updates compared to > > per inode QUERIED bit may exist, but I think it is a rather small overhead > > and maybe worth the tradeoff of having to maintain a real per inode > > "change cookie" in addition to a "globally ordered mgtime"? > > > > If this idea is acceptable, you may still be able to salvage the reverted > > ctime series for 6.7, because the change to use global mgtime should > > be quite trivial? > > > > This is basically the idea I was going to look at next once I got some > other stuff settled here: Basically, when we apply a fine-grained > timestamp to an inode, we'd advance the coarse-grained clock that > filesystems use to that value. > > It could cause some write amplification: if you are streaming writes to > a bunch of files at the same time and someone stats one of them, then > they'd all end up getting an extra inode transaction. That doesn't sound > _too_ bad on its face, but I probably need to implement it and then run > some numbers to see. > Several journal transactions within a single jiffie tick? If ctime/change_cookie of an inode is updated once within the scope of a single running transaction, I don't think it matters how many times it would be updated, but maybe I am missing something. The problem is probably going to be that the seqlock of the coarse grained clock is going to be invalidated way too frequently to be "read mostly" in the presence of ls -lR workload, but again, I did not study the implementation, so I may be way off. Thanks, Amir.
On Sat, Sep 23, 2023 at 1:46 PM Jeff Layton <jlayton@kernel.org> wrote: > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > My initial goal was to implement multigrain timestamps on most major > > > filesystems, so we could present them to userland, and use them for > > > NFSv3, etc. > > > > > > With the current implementation however, we can't guarantee that a file > > > with a coarse grained timestamp modified after one with a fine grained > > > timestamp will always appear to have a later value. This could confuse > > > some programs like make, rsync, find, etc. that depend on strict > > > ordering requirements for timestamps. > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > XFS's change attribute is bumped on atime updates in addition to other > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > a slightly different approach and has XFS use the fine-grained attr to > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > the timestamps via getattr, we truncate them at a granularity of number > > > of ns per jiffy, > > > > That's not good, because user explicitly set granular mtime would be > > truncated too and booting with different kernels (HZ) would change > > the observed timestamps of files. > > > > Thinking about this some more, I think the first problem is easily > addressable: > > The ctime isn't explicitly settable and with this set, we're already not > truncating the atime. We haven't used any of the extra bits in the mtime > yet, so we could just carve out a flag in there that says "this mtime > was explicitly set and shouldn't be truncated before presentation". > > The second problem (booting to older kernel) is a bit tougher to deal > with however. I'll have to think about that one a bit more. There is a straightforward solution to both these problems - opt in with a mount option to truncate user visible timestamps to 100ns (and not jiffy) resolution as Linus is trying to promote. Thanks, Amir.
> My initial goal was to implement multigrain timestamps on most major > filesystems, so we could present them to userland, and use them for > NFSv3, etc. If there's no clear users and workloads depending on this other than for the sake of NFS then we shouldn't expose this to userspace. We've tried this and I'm not convinced we're getting anything other than regressions out of it. Keep it internal and confined to the filesystem that actually needs this.
On Sat, Sep 23, 2023 at 05:52:36PM +0300, Amir Goldstein wrote: > On Sat, Sep 23, 2023 at 1:46 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > My initial goal was to implement multigrain timestamps on most major > > > > filesystems, so we could present them to userland, and use them for > > > > NFSv3, etc. > > > > > > > > With the current implementation however, we can't guarantee that a file > > > > with a coarse grained timestamp modified after one with a fine grained > > > > timestamp will always appear to have a later value. This could confuse > > > > some programs like make, rsync, find, etc. that depend on strict > > > > ordering requirements for timestamps. > > > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > > XFS's change attribute is bumped on atime updates in addition to other > > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > > a slightly different approach and has XFS use the fine-grained attr to > > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > > the timestamps via getattr, we truncate them at a granularity of number > > > > of ns per jiffy, > > > > > > That's not good, because user explicitly set granular mtime would be > > > truncated too and booting with different kernels (HZ) would change > > > the observed timestamps of files. > > > > > > > Thinking about this some more, I think the first problem is easily > > addressable: > > > > The ctime isn't explicitly settable and with this set, we're already not > > truncating the atime. We haven't used any of the extra bits in the mtime > > yet, so we could just carve out a flag in there that says "this mtime > > was explicitly set and shouldn't be truncated before presentation". > > > > I thought about this option too. > But note that the "mtime was explicitly set" flag needs > to be persisted to disk so you cannot store it in the high nsec bits. > At least XFS won't store those bits if you use them - they have to > be translated to an XFS inode flag and I don't know if changing > XFS on-disk format was on your wish list. Remember: this multi-grain timestamp thing was an idea to solve the NFS change attribute problem without requiring *any* filesystem with sub-jiffie timestamp capability to change their on-disk format to implement a persistent change attribute that matches the new requires of the kernel nfsd. If we now need to change the on-disk format to support some whacky new timestamp semantic to do this, then people have completely lost sight of what problem the multi-grain timestamp idea was supposed to address. -Dave.
On Sun, 24 Sep 2023, Christian Brauner wrote: > > My initial goal was to implement multigrain timestamps on most major > > filesystems, so we could present them to userland, and use them for > > NFSv3, etc. > > If there's no clear users and workloads depending on this other than for > the sake of NFS then we shouldn't expose this to userspace. We've tried > this and I'm not convinced we're getting anything other than regressions > out of it. Keep it internal and confined to the filesystem that actually > needs this. > Some NFS servers run in userspace, and they would a "clear user" of this functionality. NeilBrown
On Sat, 2023-09-23 at 17:58 +0300, Amir Goldstein wrote: > On Sat, Sep 23, 2023 at 1:22 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > My initial goal was to implement multigrain timestamps on most major > > > > filesystems, so we could present them to userland, and use them for > > > > NFSv3, etc. > > > > > > > > With the current implementation however, we can't guarantee that a file > > > > with a coarse grained timestamp modified after one with a fine grained > > > > timestamp will always appear to have a later value. This could confuse > > > > some programs like make, rsync, find, etc. that depend on strict > > > > ordering requirements for timestamps. > > > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > > XFS's change attribute is bumped on atime updates in addition to other > > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > > a slightly different approach and has XFS use the fine-grained attr to > > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > > the timestamps via getattr, we truncate them at a granularity of number > > > > of ns per jiffy, > > > > > > That's not good, because user explicitly set granular mtime would be > > > truncated too and booting with different kernels (HZ) would change > > > the observed timestamps of files. > > > > > > > That's a very good point. > > > > > > which allows us to smooth over the fuzz that causes > > > > ordering problems. > > > > > > > > > > The reported ordering problems (i.e. cp -u) is not even limited to the > > > scope of a single fs, right? > > > > > > > It isn't. Most of the tools we're concerned with don't generally care > > about filesystem boundaries. > > > > > Thinking out loud - if the QERIED bit was not per inode timestamp > > > but instead in a global fs_multigrain_ts variable, then all the inodes > > > of all the mgtime fs would be using globally ordered timestamps > > > > > > That should eliminate the reported issues with time reorder for > > > fine vs coarse grained timestamps. > > > > > > The risk of extra unneeded "change cookie" updates compared to > > > per inode QUERIED bit may exist, but I think it is a rather small overhead > > > and maybe worth the tradeoff of having to maintain a real per inode > > > "change cookie" in addition to a "globally ordered mgtime"? > > > > > > If this idea is acceptable, you may still be able to salvage the reverted > > > ctime series for 6.7, because the change to use global mgtime should > > > be quite trivial? > > > > > > > This is basically the idea I was going to look at next once I got some > > other stuff settled here: Basically, when we apply a fine-grained > > timestamp to an inode, we'd advance the coarse-grained clock that > > filesystems use to that value. > > > > It could cause some write amplification: if you are streaming writes to > > a bunch of files at the same time and someone stats one of them, then > > they'd all end up getting an extra inode transaction. That doesn't sound > > _too_ bad on its face, but I probably need to implement it and then run > > some numbers to see. > > > > Several journal transactions within a single jiffie tick? > If ctime/change_cookie of an inode is updated once within the scope > of a single running transaction, I don't think it matters how many > times it would be updated, but maybe I am missing something. > > The problem is probably going to be that the seqlock of the coarse > grained clock is going to be invalidated way too frequently to be > "read mostly" in the presence of ls -lR workload, but again, I did > not study the implementation, so I may be way off. > That may end up being the case, but I think if we can minimize the number of fine-grained updates, then the number of invalidations will be minimal too. I haven't rolled an implementation of this yet. This is all very much still in the "waving of hands" stage anyway. Once the dust settles from the atime and mtime API rework, I may still take a stab at doing this.
On Mon, 2023-09-25 at 08:18 +1000, Dave Chinner wrote: > On Sat, Sep 23, 2023 at 05:52:36PM +0300, Amir Goldstein wrote: > > On Sat, Sep 23, 2023 at 1:46 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > > > My initial goal was to implement multigrain timestamps on most major > > > > > filesystems, so we could present them to userland, and use them for > > > > > NFSv3, etc. > > > > > > > > > > With the current implementation however, we can't guarantee that a file > > > > > with a coarse grained timestamp modified after one with a fine grained > > > > > timestamp will always appear to have a later value. This could confuse > > > > > some programs like make, rsync, find, etc. that depend on strict > > > > > ordering requirements for timestamps. > > > > > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > > > XFS's change attribute is bumped on atime updates in addition to other > > > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > > > a slightly different approach and has XFS use the fine-grained attr to > > > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > > > the timestamps via getattr, we truncate them at a granularity of number > > > > > of ns per jiffy, > > > > > > > > That's not good, because user explicitly set granular mtime would be > > > > truncated too and booting with different kernels (HZ) would change > > > > the observed timestamps of files. > > > > > > > > > > Thinking about this some more, I think the first problem is easily > > > addressable: > > > > > > The ctime isn't explicitly settable and with this set, we're already not > > > truncating the atime. We haven't used any of the extra bits in the mtime > > > yet, so we could just carve out a flag in there that says "this mtime > > > was explicitly set and shouldn't be truncated before presentation". > > > > > > > I thought about this option too. > > But note that the "mtime was explicitly set" flag needs > > to be persisted to disk so you cannot store it in the high nsec bits. > > At least XFS won't store those bits if you use them - they have to > > be translated to an XFS inode flag and I don't know if changing > > XFS on-disk format was on your wish list. > > Remember: this multi-grain timestamp thing was an idea to solve the > NFS change attribute problem without requiring *any* filesystem with > sub-jiffie timestamp capability to change their on-disk format to > implement a persistent change attribute that matches the new > requires of the kernel nfsd. > > If we now need to change the on-disk format to support > some whacky new timestamp semantic to do this, then people have > completely lost sight of what problem the multi-grain timestamp idea > was supposed to address. > Yep. The main impetus for all of this was to fix XFS's change attribute without requiring an on-disk format change. If we have to rev the on- disk format, we're probably better off plumbing in a proper i_version counter and tossing this idea aside. That said, I think all we'd need for this scheme is a single flag per inode (to indicate that the mtime shouldn't be truncated before presentation). If that's possible to do without fully revving the inode format, then we could still pursue this. I take it that's probably not the case though.
On Mon, 2023-09-25 at 08:44 +1000, NeilBrown wrote: > On Sun, 24 Sep 2023, Christian Brauner wrote: > > > My initial goal was to implement multigrain timestamps on most major > > > filesystems, so we could present them to userland, and use them for > > > NFSv3, etc. > > > > If there's no clear users and workloads depending on this other than for > > the sake of NFS then we shouldn't expose this to userspace. We've tried > > this and I'm not convinced we're getting anything other than regressions > > out of it. Keep it internal and confined to the filesystem that actually > > needs this. > > > > Some NFS servers run in userspace, and they would a "clear user" of this > functionality. > Indeed. Also, all of the programs that we're concerned about breaking here (make, rsync, etc.) could benefit from proper fine-grained timestamps: Today, when they see two identical timestamps on files, these programs have to assume the worst: rsync has to do the copy, make has to update the target, etc. With a real distinguishable fine-grained timestamps, these programs would likely be more efficient and some of these unneeded operations would be avoided.
On Mon, Sep 25, 2023 at 06:14:05AM -0400, Jeff Layton wrote: > On Mon, 2023-09-25 at 08:18 +1000, Dave Chinner wrote: > > On Sat, Sep 23, 2023 at 05:52:36PM +0300, Amir Goldstein wrote: > > > On Sat, Sep 23, 2023 at 1:46 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > > > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > > > > > My initial goal was to implement multigrain timestamps on most major > > > > > > filesystems, so we could present them to userland, and use them for > > > > > > NFSv3, etc. > > > > > > > > > > > > With the current implementation however, we can't guarantee that a file > > > > > > with a coarse grained timestamp modified after one with a fine grained > > > > > > timestamp will always appear to have a later value. This could confuse > > > > > > some programs like make, rsync, find, etc. that depend on strict > > > > > > ordering requirements for timestamps. > > > > > > > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > > > > XFS's change attribute is bumped on atime updates in addition to other > > > > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > > > > a slightly different approach and has XFS use the fine-grained attr to > > > > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > > > > the timestamps via getattr, we truncate them at a granularity of number > > > > > > of ns per jiffy, > > > > > > > > > > That's not good, because user explicitly set granular mtime would be > > > > > truncated too and booting with different kernels (HZ) would change > > > > > the observed timestamps of files. > > > > > > > > > > > > > Thinking about this some more, I think the first problem is easily > > > > addressable: > > > > > > > > The ctime isn't explicitly settable and with this set, we're already not > > > > truncating the atime. We haven't used any of the extra bits in the mtime > > > > yet, so we could just carve out a flag in there that says "this mtime > > > > was explicitly set and shouldn't be truncated before presentation". > > > > > > > > > > I thought about this option too. > > > But note that the "mtime was explicitly set" flag needs > > > to be persisted to disk so you cannot store it in the high nsec bits. > > > At least XFS won't store those bits if you use them - they have to > > > be translated to an XFS inode flag and I don't know if changing > > > XFS on-disk format was on your wish list. > > > > Remember: this multi-grain timestamp thing was an idea to solve the > > NFS change attribute problem without requiring *any* filesystem with > > sub-jiffie timestamp capability to change their on-disk format to > > implement a persistent change attribute that matches the new > > requires of the kernel nfsd. > > > > If we now need to change the on-disk format to support > > some whacky new timestamp semantic to do this, then people have > > completely lost sight of what problem the multi-grain timestamp idea > > was supposed to address. > > > > Yep. The main impetus for all of this was to fix XFS's change attribute > without requiring an on-disk format change. If we have to rev the on- > disk format, we're probably better off plumbing in a proper i_version > counter and tossing this idea aside. > > That said, I think all we'd need for this scheme is a single flag per > inode (to indicate that the mtime shouldn't be truncated before > presentation). If that's possible to do without fully revving the inode > format, then we could still pursue this. I take it that's probably not > the case though. Older kernels that don't know what the flag means, but that should be OK for an inode flag. The bigger issue is that none of the userspace tools (xfs_db, xfs_repair, etc) know about it, so they would have to be taught about it. And then there's testing it, which likely means userspace needs visibility of the flag (e.g. FS_XFLAG for it) and then there's more work.... It's really not worth it. I think that Linus's suggestion of the in-memory inode timestamp always being a 64bit, 100ns granularity value instead of a timespec that gets truncated at sample time has merit as a general solution. We also must not lose sight of the fact that the lazytime mount option makes atime updates on XFS behave exactly as the nfsd/NFS client application wants. That is, XFS will do in-memory atime updates unless the atime update also sets S_VERSION to explicitly bump the i_version counter if required. That leads to another potential nfsd specific solution without requiring filesystems to change on disk formats: the nfsd explicitly asks operations for lazy atime updates... And we must also keep in sight the fact that io_uring wants non-blocking timestamp updates to be possible (for all types of updates). Hence it looks to me like we have more than one use case for per-operation/application specific timestamp update semantics. Perhaps there's a generic solution to this problem (e.g. operation specific non-blocking, in-memory pure timestamp updates) that does what everyone needs... -Dave.
On Tue, 2023-09-26 at 08:32 +1000, Dave Chinner wrote: > On Mon, Sep 25, 2023 at 06:14:05AM -0400, Jeff Layton wrote: > > On Mon, 2023-09-25 at 08:18 +1000, Dave Chinner wrote: > > > On Sat, Sep 23, 2023 at 05:52:36PM +0300, Amir Goldstein wrote: > > > > On Sat, Sep 23, 2023 at 1:46 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > > > On Sat, 2023-09-23 at 10:15 +0300, Amir Goldstein wrote: > > > > > > On Fri, Sep 22, 2023 at 8:15 PM Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > > > > > > > My initial goal was to implement multigrain timestamps on most major > > > > > > > filesystems, so we could present them to userland, and use them for > > > > > > > NFSv3, etc. > > > > > > > > > > > > > > With the current implementation however, we can't guarantee that a file > > > > > > > with a coarse grained timestamp modified after one with a fine grained > > > > > > > timestamp will always appear to have a later value. This could confuse > > > > > > > some programs like make, rsync, find, etc. that depend on strict > > > > > > > ordering requirements for timestamps. > > > > > > > > > > > > > > The goal of this version is more modest: fix XFS' change attribute. > > > > > > > XFS's change attribute is bumped on atime updates in addition to other > > > > > > > deliberate changes. This makes it unsuitable for export via nfsd. > > > > > > > > > > > > > > Jan Kara suggested keeping this functionality internal-only for now and > > > > > > > plumbing the fine grained timestamps through getattr [1]. This set takes > > > > > > > a slightly different approach and has XFS use the fine-grained attr to > > > > > > > fake up STATX_CHANGE_COOKIE in its getattr routine itself. > > > > > > > > > > > > > > While we keep fine-grained timestamps in struct inode, when presenting > > > > > > > the timestamps via getattr, we truncate them at a granularity of number > > > > > > > of ns per jiffy, > > > > > > > > > > > > That's not good, because user explicitly set granular mtime would be > > > > > > truncated too and booting with different kernels (HZ) would change > > > > > > the observed timestamps of files. > > > > > > > > > > > > > > > > Thinking about this some more, I think the first problem is easily > > > > > addressable: > > > > > > > > > > The ctime isn't explicitly settable and with this set, we're already not > > > > > truncating the atime. We haven't used any of the extra bits in the mtime > > > > > yet, so we could just carve out a flag in there that says "this mtime > > > > > was explicitly set and shouldn't be truncated before presentation". > > > > > > > > > > > > > I thought about this option too. > > > > But note that the "mtime was explicitly set" flag needs > > > > to be persisted to disk so you cannot store it in the high nsec bits. > > > > At least XFS won't store those bits if you use them - they have to > > > > be translated to an XFS inode flag and I don't know if changing > > > > XFS on-disk format was on your wish list. > > > > > > Remember: this multi-grain timestamp thing was an idea to solve the > > > NFS change attribute problem without requiring *any* filesystem with > > > sub-jiffie timestamp capability to change their on-disk format to > > > implement a persistent change attribute that matches the new > > > requires of the kernel nfsd. > > > > > > If we now need to change the on-disk format to support > > > some whacky new timestamp semantic to do this, then people have > > > completely lost sight of what problem the multi-grain timestamp idea > > > was supposed to address. > > > > > > > Yep. The main impetus for all of this was to fix XFS's change attribute > > without requiring an on-disk format change. If we have to rev the on- > > disk format, we're probably better off plumbing in a proper i_version > > counter and tossing this idea aside. > > > > That said, I think all we'd need for this scheme is a single flag per > > inode (to indicate that the mtime shouldn't be truncated before > > presentation). If that's possible to do without fully revving the inode > > format, then we could still pursue this. I take it that's probably not > > the case though. > > Older kernels that don't know what the flag means, but that should > be OK for an inode flag. The bigger issue is that none of the > userspace tools (xfs_db, xfs_repair, etc) know about it, so they > would have to be taught about it. And then there's testing it, which > likely means userspace needs visibility of the flag (e.g. FS_XFLAG > for it) and then there's more work.... > > It's really not worth it. > > > I think that Linus's suggestion of the in-memory inode timestamp > always being a 64bit, 100ns granularity value instead of a timespec > that gets truncated at sample time has merit as a general solution. > Changing how we store timestamps in struct inode is a good idea, and reducing the effective granularity to 100ns seems reasonable, but that alone won't fix XFS's i_version counter, or the ordering problems that we hit with the multigrain series that had to be reverted. > We also must not lose sight of the fact that the lazytime mount > option makes atime updates on XFS behave exactly as the nfsd/NFS > client application wants. That is, XFS will do in-memory atime > updates unless the atime update also sets S_VERSION to explicitly > bump the i_version counter if required. That leads to another > potential nfsd specific solution without requiring filesystems to > change on disk formats: the nfsd explicitly asks operations for lazy > atime updates... > Not exactly. The problem with XFS's i_version is that it also bumps it on atime updates. lazytime reduces the number of atime updates to ~1/day. To be exactly what nfsd wants, you'd need to make that 0. I suppose you can work around it with noatime, but that has problems of its own. > And we must also keep in sight the fact that io_uring wants > non-blocking timestamp updates to be possible (for all types of > updates). Hence it looks to me like we have more than one use case > for per-operation/application specific timestamp update semantics. > Perhaps there's a generic solution to this problem (e.g. operation > specific non-blocking, in-memory pure timestamp updates) that does > what everyone needs...
> > Some NFS servers run in userspace, and they would a "clear user" of this > > functionality. > > > > Indeed. Also, all of the programs that we're concerned about breaking > here (make, rsync, etc.) could benefit from proper fine-grained > timestamps: > > Today, when they see two identical timestamps on files, these programs > have to assume the worst: rsync has to do the copy, make has to update > the target, etc. With a real distinguishable fine-grained timestamps, > these programs would likely be more efficient and some of these unneeded > operations would be avoided. The whole sales pitch falls flat if we end up with wrong ordering of timestamps which caused us to revert this. So unless this is fixed we shouldn't expose this to userspace again.
> > If there's no clear users and workloads depending on this other than for > > the sake of NFS then we shouldn't expose this to userspace. We've tried > > Some NFS servers run in userspace, and they would a "clear user" of this > functionality. See my comment above. We did thist mostly for the sake of NFS as there was in itself nothing wrong with timestamps that needed urgent fixing. The end result has been that we caused a regression for four other major filesystems when they were switched to fine-grained timestamps. So NFS servers in userspace isn't a sufficient argument to just try again with a slightly tweaked solution but without a wholesale fix of the actual ordering problem. The bar to merge this will naturally be higher the second time around. That's orthogonal to improving the general timestamp infrastructure in struct inode ofc.
On Tue, 2023-09-26 at 14:18 +0200, Christian Brauner wrote: > > > If there's no clear users and workloads depending on this other than for > > > the sake of NFS then we shouldn't expose this to userspace. We've tried > > > > > Some NFS servers run in userspace, and they would a "clear user" of this > > functionality. > > See my comment above. We did thist mostly for the sake of NFS as there > was in itself nothing wrong with timestamps that needed urgent fixing. > > The end result has been that we caused a regression for four other major > filesystems when they were switched to fine-grained timestamps. > > So NFS servers in userspace isn't a sufficient argument to just try > again with a slightly tweaked solution but without a wholesale fix of > the actual ordering problem. The bar to merge this will naturally be > higher the second time around. > > That's orthogonal to improving the general timestamp infrastructure in > struct inode ofc. There are multiple proposals in flight here, and they all sort of dovetail on one another. I'm not proposing we expose any changes to the timestamps to users in any way, unless we can fix the ordering issues, and ensure that we can preserve existing behavior re: utimensat(). I think it's possible to do that, but I'm going to table that work for the moment, and finish up the atime/mtime accessor conversions first. That makes experimenting with all of this a lot simpler. I think I can also put together a nicer implementation of multigrain timestamps too, if we can first convert the current timespec64's in struct inode into a single 64-bit word.
On Tue, Sep 26, 2023 at 08:51:32AM -0400, Jeff Layton wrote: > On Tue, 2023-09-26 at 14:18 +0200, Christian Brauner wrote: > > > > If there's no clear users and workloads depending on this other than for > > > > the sake of NFS then we shouldn't expose this to userspace. We've tried > > > > > > > > Some NFS servers run in userspace, and they would a "clear user" of this > > > functionality. > > > > See my comment above. We did thist mostly for the sake of NFS as there > > was in itself nothing wrong with timestamps that needed urgent fixing. > > > > The end result has been that we caused a regression for four other major > > filesystems when they were switched to fine-grained timestamps. > > > > So NFS servers in userspace isn't a sufficient argument to just try > > again with a slightly tweaked solution but without a wholesale fix of > > the actual ordering problem. The bar to merge this will naturally be > > higher the second time around. > > > > That's orthogonal to improving the general timestamp infrastructure in > > struct inode ofc. > > There are multiple proposals in flight here, and they all sort of > dovetail on one another. I'm not proposing we expose any changes to the > timestamps to users in any way, unless we can fix the ordering issues, > and ensure that we can preserve existing behavior re: utimensat(). Yeah, I know you're aware of that and I'm mostly clarifying the conditions for this work for the ones not that closely involved. > I think it's possible to do that, but I'm going to table that work for > the moment, and finish up the atime/mtime accessor conversions first. That sounds great. > That makes experimenting with all of this a lot simpler. I think I can Oh that's good then.
On Tue, Sep 26, 2023 at 07:31:55AM -0400, Jeff Layton wrote: > On Tue, 2023-09-26 at 08:32 +1000, Dave Chinner wrote: > > We also must not lose sight of the fact that the lazytime mount > > option makes atime updates on XFS behave exactly as the nfsd/NFS > > client application wants. That is, XFS will do in-memory atime > > updates unless the atime update also sets S_VERSION to explicitly > > bump the i_version counter if required. That leads to another > > potential nfsd specific solution without requiring filesystems to > > change on disk formats: the nfsd explicitly asks operations for lazy > > atime updates... > > > > Not exactly. The problem with XFS's i_version is that it also bumps it > on atime updates. lazytime reduces the number of atime updates to > ~1/day. To be exactly what nfsd wants, you'd need to make that 0. As long as there are future modifications going to those files, lazytime completely elides the visibility of atime updates as they get silently aggregated into future modifications and so there are 0 i_version changes as a resutl of pure atime updates in those cases. If there are no future modifications, then just like relatime, there is a timestamp update every 24hrs. That's no big deal, nobody is complaining about this being a problem. It's the "persistent atime update after modification" heuristic implemented by relatime that is causing all the problems here. If that behaviour is elided on the server side, then most of the client side invalidation problems with these workloads go away. IOWs, nfsd needs direct control over how atime updates should be treated by the VFS/filesystem (i.e. as pure in-memory updates) rather than leaving it to some heuristic that may do the exact opposite of what the nfsd application needs. That's the point I was making: we have emerging requirements for per-operation timestamp update behaviour control with io_uring and other non-blocking applications. The nfsd application also has specific semantics it wants the VFS/filesystem to implement (non-persistent atime unless something else changes).... My point is that we've now failed a couple of times now to implement what NFSD requires via trying to change VFS and/or filesystem infrastructure to provide i_version or ctime semantics the nfsd requires. That's a fairly good sign that we might not be approaching this problem from the right direction, and so doubling down and considering changing the timestamp infrastructure from the ground up just to solve a relatively niche, filesystem specific issue doesn't seem like the best approach. OTOH, having the application actually tell the timestamp updates exactly what semantics it needs (non blocking, persistent vs in memory, etc) will allow the VFS and filesystems can do the right thing for the application without having to worry about general heuristics that sometimes do exactly the wrong thing.... -Dave.
On Wed, 2023-09-27 at 09:33 +1000, Dave Chinner wrote: > On Tue, Sep 26, 2023 at 07:31:55AM -0400, Jeff Layton wrote: > > On Tue, 2023-09-26 at 08:32 +1000, Dave Chinner wrote: > > > We also must not lose sight of the fact that the lazytime mount > > > option makes atime updates on XFS behave exactly as the nfsd/NFS > > > client application wants. That is, XFS will do in-memory atime > > > updates unless the atime update also sets S_VERSION to explicitly > > > bump the i_version counter if required. That leads to another > > > potential nfsd specific solution without requiring filesystems to > > > change on disk formats: the nfsd explicitly asks operations for lazy > > > atime updates... > > > > > > > Not exactly. The problem with XFS's i_version is that it also bumps it > > on atime updates. lazytime reduces the number of atime updates to > > ~1/day. To be exactly what nfsd wants, you'd need to make that 0. > > As long as there are future modifications going to those files, > lazytime completely elides the visibility of atime updates as they > get silently aggregated into future modifications and so there are > 0 i_version changes as a resutl of pure atime updates in those cases. > > If there are no future modifications, then just like relatime, there > is a timestamp update every 24hrs. That's no big deal, nobody is > complaining about this being a problem. > Right. The main issue here is that (with relatime) we'll still end up with a cache invalidation once every 24 hours for any r/o files that have been accessed. It's not a _huge_ problem on most workloads; it's just not ideal. > It's the "persistent atime update after modification" heuristic > implemented by relatime that is causing all the problems here. If > that behaviour is elided on the server side, then most of the client > side invalidation problems with these workloads go away. > > IOWs, nfsd needs direct control over how atime updates should be > treated by the VFS/filesystem (i.e. as pure in-memory updates) > rather than leaving it to some heuristic that may do the exact > opposite of what the nfsd application needs. > > That's the point I was making: we have emerging requirements for > per-operation timestamp update behaviour control with io_uring and > other non-blocking applications. The nfsd application also has > specific semantics it wants the VFS/filesystem to implement > (non-persistent atime unless something else changes).... > > My point is that we've now failed a couple of times now to implement > what NFSD requires via trying to change VFS and/or filesystem > infrastructure to provide i_version or ctime semantics the nfsd > requires. That's a fairly good sign that we might not be approaching > this problem from the right direction, and so doubling down and > considering changing the timestamp infrastructure from the ground up > just to solve a relatively niche, filesystem specific issue doesn't > seem like the best approach. > > OTOH, having the application actually tell the timestamp updates > exactly what semantics it needs (non blocking, persistent vs in > memory, etc) will allow the VFS and filesystems can do the right > thing for the application without having to worry about general > heuristics that sometimes do exactly the wrong thing.... > I'm a little unclear on exactly what you're proposing here, but I think that's overstating what's needed. nfsd's needs are pretty simple: it wants a change attribute that changes any time the ctime would change. btrfs, ext4 and tmpfs have this. xfs does not because its change attribute changes when the atime changes as well. With the right mount options, that problem can be mitigated to some degree, but it's still not ideal. We have a couple of options: try to make the ctime behave the way we need, or just implement a proper change attribute in xfs (which involves revving the on-disk format).
My initial goal was to implement multigrain timestamps on most major filesystems, so we could present them to userland, and use them for NFSv3, etc. With the current implementation however, we can't guarantee that a file with a coarse grained timestamp modified after one with a fine grained timestamp will always appear to have a later value. This could confuse some programs like make, rsync, find, etc. that depend on strict ordering requirements for timestamps. The goal of this version is more modest: fix XFS' change attribute. XFS's change attribute is bumped on atime updates in addition to other deliberate changes. This makes it unsuitable for export via nfsd. Jan Kara suggested keeping this functionality internal-only for now and plumbing the fine grained timestamps through getattr [1]. This set takes a slightly different approach and has XFS use the fine-grained attr to fake up STATX_CHANGE_COOKIE in its getattr routine itself. While we keep fine-grained timestamps in struct inode, when presenting the timestamps via getattr, we truncate them at a granularity of number of ns per jiffy, which allows us to smooth over the fuzz that causes ordering problems. This set only converts XFS to use this scheme. All of the other commonly-exported local filesystems have a native change attribute and wouldn't clearly benefit from mgtime support at this time. Still, it should be possible to add this support to other filesystems in the future, as the need arises (bcachefs?). I'd like to see this go in for v6.7 if possible, so getting it into linux-next now would be great if there are no objections. [1]: https://lore.kernel.org/linux-fsdevel/20230920124823.ghl6crb5sh4x2pmt@quack3/ Signed-off-by: Jeff Layton <jlayton@kernel.org> --- Jeff Layton (5): fs: add infrastructure for multigrain timestamps fs: optimize away some fine-grained timestamp updates fs: have setattr_copy handle multigrain timestamps appropriately fs: add timestamp_truncate_to_gran helper xfs: switch to multigrain timestamps fs/attr.c | 52 ++++++++++++-- fs/inode.c | 151 ++++++++++++++++++++++++++++++++++++---- fs/xfs/libxfs/xfs_trans_inode.c | 6 +- fs/xfs/xfs_iops.c | 26 +++++-- fs/xfs/xfs_super.c | 2 +- include/linux/fs.h | 64 ++++++++++++++++- 6 files changed, 269 insertions(+), 32 deletions(-) --- base-commit: f8f2d6d669b91ea98ec8f182c22e06d3d0663e15 change-id: 20230921-ctime-5b7a628a4b95 Best regards,