| Message ID | 20230314042109.82161-1-catherine.hoang@oracle.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | setting uuid of online filesystems | expand |
On Mon, Mar 13, 2023 at 09:21:05PM -0700, Catherine Hoang wrote: > Hi all, > > This series of patches implements a new ioctl to set the uuid of mounted > filesystems. Eventually this will be used by the 'xfs_io fsuuid' command > to allow userspace to update the uuid. > > Comments and feedback appreciated! What's the use case for this? What are the pro's and cons of the implementation? Some problems I see: * How does this interact with pNFS exports (i.e. CONFIG_EXPORTFS_BLOCK_OPS). XFS hands the sb_uuid directly to pNFS server (and remote clients, I think) so that incoming mapping requests can be directed to the correct block device hosting the XFS filesystem the mapping information is for. IIRC, the pNFS data path is just given a byte offset into the device where the UUID in the superblock lives, and if it matches it allows the remote IO to go ahead - it doesn't actually know that there is a filesystem on that device at all... * IIRC, the nfsd export table can also use the filesystem uuid to identify the filesystem being exported, and IIRC that gets encoded in the filehandle. Does changing the filesystem UUID then cause problems with export indentification and/or file handle creation/resolution? * Is the VFS prepared for sb->s_uuid changing underneath running operations on mounted filesystems? I can see that this might cause problems with any sort of fscrypt implementation as it may encode the s_uuid into encryption keys held in xattrs, similarly IMA and EVM integrity protection keys. * Should the VFS superblock sb->s_uuid use the XFS sb->sb_meta_uuid value and never change because we can encode it into other objects stored in the active filesystem? What does that mean for tools that identify block devices or filesystems by the VFS uuid rather than the filesystem uuid? There's a whole can-o-worms here - the ability to dynamically change the UUID of a filesystem while it is mounted mean we need to think about these things - it's no longer as simple as "can only do it offline" which is typically only used to relabel a writeable snapshot of a golden image file during new VM deployment preparation..... > > Catherine > > Catherine Hoang (4): > xfs: refactor xfs_uuid_mount and xfs_uuid_unmount > xfs: implement custom freeze/thaw functions The "custom" parts that XFS requires need to be added to the VFS level freeze/thaw functions, not duplicate the VFS functionality within XFS and then slight modify it for this additional feature. Doing this results in unmaintainable code over the long term. > xfs: add XFS_IOC_SETFSUUID ioctl > xfs: export meta uuid via xfs_fsop_geom For what purpose does userspace ever need to know the sb_meta_uuid? Cheers, Dave.
> On Mar 13, 2023, at 11:28 PM, Dave Chinner <david@fromorbit.com> wrote: > > On Mon, Mar 13, 2023 at 09:21:05PM -0700, Catherine Hoang wrote: >> Hi all, >> >> This series of patches implements a new ioctl to set the uuid of mounted >> filesystems. Eventually this will be used by the 'xfs_io fsuuid' command >> to allow userspace to update the uuid. >> >> Comments and feedback appreciated! > > What's the use case for this? We want to be able to change the uuid on newly mounted clone vm images so that each deployed system has a different uuid. We need to do this the first time the system boots, but after the root fs is mounted so that fsuuid can run in parallel with other service startup to minimize deployment times. > > What are the pro's and cons of the implementation? > > Some problems I see: > > * How does this interact with pNFS exports (i.e. > CONFIG_EXPORTFS_BLOCK_OPS). XFS hands the sb_uuid directly to pNFS > server (and remote clients, I think) so that incoming mapping > requests can be directed to the correct block device hosting the XFS > filesystem the mapping information is for. IIRC, the pNFS data path > is just given a byte offset into the device where the UUID in the > superblock lives, and if it matches it allows the remote IO to go > ahead - it doesn't actually know that there is a filesystem on that > device at all... > > * IIRC, the nfsd export table can also use the filesystem uuid to > identify the filesystem being exported, and IIRC that gets encoded > in the filehandle. Does changing the filesystem UUID then cause > problems with export indentification and/or file handle > creation/resolution? > > * Is the VFS prepared for sb->s_uuid changing underneath running > operations on mounted filesystems? I can see that this might cause > problems with any sort of fscrypt implementation as it may encode > the s_uuid into encryption keys held in xattrs, similarly IMA and > EVM integrity protection keys. > > * Should the VFS superblock sb->s_uuid use the XFS > sb->sb_meta_uuid value and never change because we can encode it > into other objects stored in the active filesystem? What does that > mean for tools that identify block devices or filesystems by the VFS > uuid rather than the filesystem uuid? > > There's a whole can-o-worms here - the ability to dynamically change > the UUID of a filesystem while it is mounted mean we need to think > about these things - it's no longer as simple as "can only do it > offline" which is typically only used to relabel a writeable > snapshot of a golden image file during new VM deployment > preparation..... > >> >> Catherine >> >> Catherine Hoang (4): >> xfs: refactor xfs_uuid_mount and xfs_uuid_unmount >> xfs: implement custom freeze/thaw functions > > The "custom" parts that XFS requires need to be added to the VFS > level freeze/thaw functions, not duplicate the VFS functionality > within XFS and then slight modify it for this additional feature. > Doing this results in unmaintainable code over the long term. > >> xfs: add XFS_IOC_SETFSUUID ioctl >> xfs: export meta uuid via xfs_fsop_geom > > For what purpose does userspace ever need to know the sb_meta_uuid? Userspace would need to know the meta uuid if we want to restore the original uuid after it has been changed. > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com
On Tue, Mar 14, 2023 at 05:28:47PM +1100, Dave Chinner wrote: > On Mon, Mar 13, 2023 at 09:21:05PM -0700, Catherine Hoang wrote: > > Hi all, > > > > This series of patches implements a new ioctl to set the uuid of mounted > > filesystems. Eventually this will be used by the 'xfs_io fsuuid' command > > to allow userspace to update the uuid. > > > > Comments and feedback appreciated! > > What's the use case for this? > > What are the pro's and cons of the implementation? > > Some problems I see: > > * How does this interact with pNFS exports (i.e. > CONFIG_EXPORTFS_BLOCK_OPS). XFS hands the sb_uuid directly to pNFS > server (and remote clients, I think) so that incoming mapping > requests can be directed to the correct block device hosting the XFS > filesystem the mapping information is for. IIRC, the pNFS data path > is just given a byte offset into the device where the UUID in the > superblock lives, and if it matches it allows the remote IO to go > ahead - it doesn't actually know that there is a filesystem on that > device at all... I think we're going to have to detect the presence of pNFS exports and EAGAIN if there are any active. That probably involves incrementing a counter during ->map_blocks and decreasing it during ->commit blocks. That might still invite races between someone setting the fsuuid and exporting via NFS. > * IIRC, the nfsd export table can also use the filesystem uuid to > identify the filesystem being exported, and IIRC that gets encoded > in the filehandle. Does changing the filesystem UUID then cause > problems with export indentification and/or file handle > creation/resolution? I thought NFS (when not being used for block layouts and pnfs stuff) still used the fsid that's also in the superblock? Presumably we'd leave the old m_fixedfsid unchanged while the fs is still mounted, and document the caveat that handles will not work after a remount. On some level, we might simply have to document that changing the user-visible uuid will break file handles and pnfs exports, so sysadmins had better get that done early. OTOH that is an argument for leaving the xfs_db-based version that we have now. > * Is the VFS prepared for sb->s_uuid changing underneath running > operations on mounted filesystems? I can see that this might cause > problems with any sort of fscrypt implementation as it may encode > the s_uuid into encryption keys held in xattrs, similarly IMA and > EVM integrity protection keys. I would hope it's not so hard to detect that EVM/fscrypt are active on a given xfs mount. EVM seems pretty hard to detect since it operates above the fs. fscrypt might not be so difficult, since we likely need to add support in the ondisk metadata, which means xfs will know. IMA seems to be using it for rule matching, so I'd say that if the sysadmin changes the fsuuid, they had better update the IMA rulebook too. Come to think of it, perhaps reading a filesystem's uuid (whether via handle export, direct read of s_uuid, nfs, evm, ima, or fscrypt) should set a superblock flag that someone has accessed it; and only if nobody's yet accessed it do we allow the fsuuid update? > * Should the VFS superblock sb->s_uuid use the XFS > sb->sb_meta_uuid value and never change because we can encode it > into other objects stored in the active filesystem? What does that > mean for tools that identify block devices or filesystems by the VFS > uuid rather than the filesystem uuid? I don't know of any other than the ones you found. > There's a whole can-o-worms here - the ability to dynamically change > the UUID of a filesystem while it is mounted mean we need to think > about these things - it's no longer as simple as "can only do it > offline" which is typically only used to relabel a writeable > snapshot of a golden image file during new VM deployment > preparation..... Agreed. > > > > Catherine > > > > Catherine Hoang (4): > > xfs: refactor xfs_uuid_mount and xfs_uuid_unmount > > xfs: implement custom freeze/thaw functions > > The "custom" parts that XFS requires need to be added to the VFS > level freeze/thaw functions, not duplicate the VFS functionality > within XFS and then slight modify it for this additional feature. > Doing this results in unmaintainable code over the long term. Yeah, I think for the next iteration we ought to try to pull in Luis's kernel-initiated-exclusive-freeze patch. > > xfs: add XFS_IOC_SETFSUUID ioctl > > xfs: export meta uuid via xfs_fsop_geom > > For what purpose does userspace ever need to know the sb_meta_uuid? (No idea; once someone sets a fsuuid they usually don't want to go back.) --D > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com
On Sat, Mar 18, 2023 at 2:24 AM Darrick J. Wong <djwong@kernel.org> wrote: > > On Tue, Mar 14, 2023 at 05:28:47PM +1100, Dave Chinner wrote: > > On Mon, Mar 13, 2023 at 09:21:05PM -0700, Catherine Hoang wrote: > > > Hi all, > > > > > > This series of patches implements a new ioctl to set the uuid of mounted > > > filesystems. Eventually this will be used by the 'xfs_io fsuuid' command > > > to allow userspace to update the uuid. > > > > > > Comments and feedback appreciated! > > > > What's the use case for this? > > > > What are the pro's and cons of the implementation? > > > > Some problems I see: > > > > * How does this interact with pNFS exports (i.e. > > CONFIG_EXPORTFS_BLOCK_OPS). XFS hands the sb_uuid directly to pNFS > > server (and remote clients, I think) so that incoming mapping > > requests can be directed to the correct block device hosting the XFS > > filesystem the mapping information is for. IIRC, the pNFS data path > > is just given a byte offset into the device where the UUID in the > > superblock lives, and if it matches it allows the remote IO to go > > ahead - it doesn't actually know that there is a filesystem on that > > device at all... > > I think we're going to have to detect the presence of pNFS exports and > EAGAIN if there are any active. That probably involves incrementing a > counter during ->map_blocks and decreasing it during ->commit blocks. > > That might still invite races between someone setting the fsuuid and > exporting via NFS. > > > * IIRC, the nfsd export table can also use the filesystem uuid to > > identify the filesystem being exported, and IIRC that gets encoded > > in the filehandle. Does changing the filesystem UUID then cause > > problems with export indentification and/or file handle > > creation/resolution? > > I thought NFS (when not being used for block layouts and pnfs stuff) > still used the fsid that's also in the superblock? Presumably we'd > leave the old m_fixedfsid unchanged while the fs is still mounted, and > document the caveat that handles will not work after a remount. > > On some level, we might simply have to document that changing the > user-visible uuid will break file handles and pnfs exports, so sysadmins > had better get that done early. > > OTOH that is an argument for leaving the xfs_db-based version that we > have now. > > > * Is the VFS prepared for sb->s_uuid changing underneath running > > operations on mounted filesystems? I can see that this might cause > > problems with any sort of fscrypt implementation as it may encode > > the s_uuid into encryption keys held in xattrs, similarly IMA and > > EVM integrity protection keys. > > I would hope it's not so hard to detect that EVM/fscrypt are active on a > given xfs mount. EVM seems pretty hard to detect since it operates > above the fs. > > fscrypt might not be so difficult, since we likely need to add support > in the ondisk metadata, which means xfs will know. > > IMA seems to be using it for rule matching, so I'd say that if the > sysadmin changes the fsuuid, they had better update the IMA rulebook > too. > > Come to think of it, perhaps reading a filesystem's uuid (whether via > handle export, direct read of s_uuid, nfs, evm, ima, or fscrypt) should > set a superblock flag that someone has accessed it; and only if nobody's > yet accessed it do we allow the fsuuid update? > > > * Should the VFS superblock sb->s_uuid use the XFS > > sb->sb_meta_uuid value and never change because we can encode it > > into other objects stored in the active filesystem? What does that > > mean for tools that identify block devices or filesystems by the VFS > > uuid rather than the filesystem uuid? > > I don't know of any other than the ones you found. FWIW, overlayfs also uses s_uuid in a similar manner to nfsd as a persistent reference to the origin lower file after copy up. Like nfsd, the overlayfs persistent origin handle (fsuuid+filehandle) will break when changing lower fs uuid offline, but I don't see much difference from changing s_uuid online - worse case that I can think of is that an overlayfs inode will change its st_ino after evict/lookup on a mounted overlayfs. Thanks, Amir.
On Thu, Mar 16, 2023 at 08:41:14PM +0000, Catherine Hoang wrote: > > On Mar 13, 2023, at 11:28 PM, Dave Chinner <david@fromorbit.com> wrote: > > > > On Mon, Mar 13, 2023 at 09:21:05PM -0700, Catherine Hoang wrote: > >> Hi all, > >> > >> This series of patches implements a new ioctl to set the uuid of mounted > >> filesystems. Eventually this will be used by the 'xfs_io fsuuid' command > >> to allow userspace to update the uuid. > >> > >> Comments and feedback appreciated! > > > > What's the use case for this? > > We want to be able to change the uuid on newly mounted clone vm images > so that each deployed system has a different uuid. We need to do this the > first time the system boots, but after the root fs is mounted so that fsuuid > can run in parallel with other service startup to minimize deployment times. Why can't you do it offline immediately after the offline clone of the golden image? I mean, cloning images and setting up their contents is something the external orchestration software does and will always have to do, so i don't really understand why UUID needs to be modified at first mount vs at clone time. Can you describe why it actually needs to be done after first mount? > >> xfs: add XFS_IOC_SETFSUUID ioctl > >> xfs: export meta uuid via xfs_fsop_geom > > > > For what purpose does userspace ever need to know the sb_meta_uuid? > > Userspace would need to know the meta uuid if we want to restore > the original uuid after it has been changed. I don't understand why you'd want to restore the original UUID given the use case you've describe. Can you explain the situation where you want to return a cloned image to the original golden image UUID? -Dave.
On Sun, Mar 19, 2023 at 11:16:48AM +1100, Dave Chinner wrote: > On Thu, Mar 16, 2023 at 08:41:14PM +0000, Catherine Hoang wrote: > > > On Mar 13, 2023, at 11:28 PM, Dave Chinner <david@fromorbit.com> wrote: > > > > > > On Mon, Mar 13, 2023 at 09:21:05PM -0700, Catherine Hoang wrote: > > >> Hi all, > > >> > > >> This series of patches implements a new ioctl to set the uuid of mounted > > >> filesystems. Eventually this will be used by the 'xfs_io fsuuid' command > > >> to allow userspace to update the uuid. > > >> > > >> Comments and feedback appreciated! > > > > > > What's the use case for this? > > > > We want to be able to change the uuid on newly mounted clone vm images > > so that each deployed system has a different uuid. We need to do this the > > first time the system boots, but after the root fs is mounted so that fsuuid > > can run in parallel with other service startup to minimize deployment times. > > Why can't you do it offline immediately after the offline clone of > the golden image? I mean, cloning images and setting up their That /is/ how they do it currently. The goal here was to reduce the number of scripts that then must be worked into every distro's bespoke initrd generation system (update-initramfs, dracut, etc.). Doing this from systemd bypasses that, and it means it can get done in the background during first boot, instead of rewriting superblocks in the singlethreaded initramfs hot path during deployments. They need the fsuuid to change after the VM starts up, but it doesn't have to get done until just before we start handing out NFS leases. Granted that was before we hit the wall of "lots of subsystems encode the fs uuid into things and hand them out"... > contents is something the external orchestration software does > and will always have to do, so i don't really understand why UUID > needs to be modified at first mount vs at clone time. Can you > describe why it actually needs to be done after first mount? > > > >> xfs: add XFS_IOC_SETFSUUID ioctl > > >> xfs: export meta uuid via xfs_fsop_geom > > > > > > For what purpose does userspace ever need to know the sb_meta_uuid? > > > > Userspace would need to know the meta uuid if we want to restore > > the original uuid after it has been changed. > > I don't understand why you'd want to restore the original UUID given > the use case you've describe. Can you explain the situation where > you want to return a cloned image to the original golden image UUID? I can't think of any, and was assuming that Catherine did that to maintain parity with the offline fsuuid setting command... --D > > -Dave. > -- > Dave Chinner > david@fromorbit.com