| Message ID | 20130807195718.GC31381@wotan.suse.de (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
On Wed, Aug 07, 2013 at 12:57:18PM -0700, Mark Fasheh wrote: > stat(2) on btrfs returns a custom device, but proc uses s_dev from the super > block. This causes problems (abi breakage) because software (and users) are > not expecting the kernel to return different devices from these calls. So fix stat on btrfs to return the proper device instead. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Aug 07, 2013 at 01:18:26PM -0700, Christoph Hellwig wrote: > On Wed, Aug 07, 2013 at 12:57:18PM -0700, Mark Fasheh wrote: > > stat(2) on btrfs returns a custom device, but proc uses s_dev from the super > > block. This causes problems (abi breakage) because software (and users) are > > not expecting the kernel to return different devices from these calls. > > So fix stat on btrfs to return the proper device instead. > Not possible, this will break other things as subvolumes have their own inode space, it will confuse applications that get multiples of an inode number for different devices with the same st_dev. Each subvolume has it's own anonymous dev to segregate things. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Aug 07, 2013 at 04:51:46PM -0400, Josef Bacik wrote: > Not possible, this will break other things as subvolumes have their own inode > space, it will confuse applications that get multiples of an inode number for > different devices with the same st_dev. Each subvolume has it's own anonymous > dev to segregate things. Thanks, Yes, it's the same old issue of btrfs volumes misbehaving, and the solution is still the same as 5 years ago: make sure each subvolume has it's own sb, vfsmount and gets automounted, similar to what nfs4 does for this case. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 08, 2013 at 05:13:49AM -0700, Christoph Hellwig wrote: > On Wed, Aug 07, 2013 at 04:51:46PM -0400, Josef Bacik wrote: > > Not possible, this will break other things as subvolumes have their own inode > > space, it will confuse applications that get multiples of an inode number for > > different devices with the same st_dev. Each subvolume has it's own anonymous > > dev to segregate things. Thanks, > > Yes, it's the same old issue of btrfs volumes misbehaving, and the > solution is still the same as 5 years ago: make sure each subvolume > has it's own sb, vfsmount and gets automounted, similar to what nfs4 > does for this case. This won't work, try having 10000 subvolumes with dirty inodes and do sync then go skiing, you'll have time :). Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: > This won't work, try having 10000 subvolumes with dirty inodes and do sync then > go skiing, you'll have time :). Thanks, Why would the dirty inodes make any difference? If you share the bdi between the subvolumes the sync workflow should be exactly the same still. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 08, 2013 at 06:48:05AM -0700, Christoph Hellwig wrote: > On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: > > This won't work, try having 10000 subvolumes with dirty inodes and do sync then > > go skiing, you'll have time :). Thanks, > > Why would the dirty inodes make any difference? If you share the bdi > between the subvolumes the sync workflow should be exactly the same > still. > The inodes are in the per-sb list, so we may start all the writing but we don't wait all at once, so in the case of btrfs we will write all the dirty inodes, and then wait on the ones in whatever sb we have, and then sync, which will commit the transaction. Then we go to the next sb and wait on those inodes which will dirty metadata which means we'll have another transaction and we'll commit the transaction and so on and so forth. This means we write the superblock 10000 times for one sync when we could have just done it once. Now we could probably get around this by having ->sync_fs wait itself for all of the inodes to complete and then commit the transaction once, but we're still going to get called the 99999 times for the same damned file system that has already had everything done. And this is just one example, IIRC there were a few other issues that popped up because we assume sb == completely separate file system, freeze I think is one of those things. I'm sure there were other ones but the last time I tried to do this was 2010/2011 and many brain cells have died since then. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 08, 2013 at 06:48:05AM -0700, Christoph Hellwig wrote: > On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: > > This won't work, try having 10000 subvolumes with dirty inodes and do sync then > > go skiing, you'll have time :). Thanks, > > Why would the dirty inodes make any difference? If you share the bdi > between the subvolumes the sync workflow should be exactly the same > still. > If we could dis-entangle vfsmounts from sb's and have it so you could have multiple vfsmounts with just one sb that would solve at least the in-kernel confusion, but I think we still have the userspace confusion. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 08, 2013 at 11:44:54AM -0400, Josef Bacik wrote: > On Thu, Aug 08, 2013 at 06:48:05AM -0700, Christoph Hellwig wrote: > > On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: > > > This won't work, try having 10000 subvolumes with dirty inodes and do sync then > > > go skiing, you'll have time :). Thanks, > > > > Why would the dirty inodes make any difference? If you share the bdi > > between the subvolumes the sync workflow should be exactly the same > > still. > > > > If we could dis-entangle vfsmounts from sb's and have it so you could have > multiple vfsmounts with just one sb that would solve at least the in-kernel > confusion, but I think we still have the userspace confusion. Thanks, I think it would mostly solve userspace confusion, as userspace only sees mounts and the device names. But please fix this up properly instead of propagating the effects of the nasty btrfs hack that should never have been merged in that form further up the stack. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Aug 12, 2013 at 04:47:52AM -0700, Christoph Hellwig wrote: > On Thu, Aug 08, 2013 at 11:44:54AM -0400, Josef Bacik wrote: > > On Thu, Aug 08, 2013 at 06:48:05AM -0700, Christoph Hellwig wrote: > > > On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: > > > > This won't work, try having 10000 subvolumes with dirty inodes and do sync then > > > > go skiing, you'll have time :). Thanks, > > > > > > Why would the dirty inodes make any difference? If you share the bdi > > > between the subvolumes the sync workflow should be exactly the same > > > still. > > > > > > > If we could dis-entangle vfsmounts from sb's and have it so you could have > > multiple vfsmounts with just one sb that would solve at least the in-kernel > > confusion, but I think we still have the userspace confusion. Thanks, > > I think it would mostly solve userspace confusion, as userspace only > sees mounts and the device names. > > But please fix this up properly instead of propagating the effects of > the nasty btrfs hack that should never have been merged in that form > further up the stack. Can one of you explain how this solves the problem that userspace is getting different devices for the same inode? Seriously, I've been looking into it and I'm a bit lost. I followed the converstaion until here but I don't see how any of the proposed changes actually *fix* anything? Also, what is the relationship between vfsmounts and sb today? Wouldn't a bind mount produce the situation of more than 1 vfsmount per sb that is described above? Sincerely, someone who would like to fix this ABI breakage that has been going on for years. --Mark -- Mark Fasheh -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Sep 10, 2013 at 08:36:55AM -0700, Mark Fasheh wrote: > On Mon, Aug 12, 2013 at 04:47:52AM -0700, Christoph Hellwig wrote: > > On Thu, Aug 08, 2013 at 11:44:54AM -0400, Josef Bacik wrote: > > > On Thu, Aug 08, 2013 at 06:48:05AM -0700, Christoph Hellwig wrote: > > > > On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: > > > > > This won't work, try having 10000 subvolumes with dirty inodes and do sync then > > > > > go skiing, you'll have time :). Thanks, > > > > > > > > Why would the dirty inodes make any difference? If you share the bdi > > > > between the subvolumes the sync workflow should be exactly the same > > > > still. > > > > > > > > > > If we could dis-entangle vfsmounts from sb's and have it so you could have > > > multiple vfsmounts with just one sb that would solve at least the in-kernel > > > confusion, but I think we still have the userspace confusion. Thanks, > > > > I think it would mostly solve userspace confusion, as userspace only > > sees mounts and the device names. > > > > But please fix this up properly instead of propagating the effects of > > the nasty btrfs hack that should never have been merged in that form > > further up the stack. > > Can one of you explain how this solves the problem that userspace is getting > different devices for the same inode? > > Seriously, I've been looking into it and I'm a bit lost. I followed the > converstaion until here but I don't see how any of the proposed changes > actually *fix* anything? Also, what is the relationship between vfsmounts > and sb today? Wouldn't a bind mount produce the situation of more than 1 > vfsmount per sb that is described above? > > Sincerely, someone who would like to fix this ABI breakage that has been > going on for years. And let me restate the problem so we're all on the same page. Btrfs has subvolumes, completely separate trees within the file system. These trees get their own object numbering, which in turn is how we do our inode numbers. So if you have multiple subvolumes, they will likely have the same inode numbers within the same file system. This screws up things like rsync which say "hey look, these two inodes are the same, lets skip them." So we have an anonymous dev so we can make them look different. Now if we were to make each subvol its own vfsmount (essentially a bind mount) and remove the anonymous device that wouldn't fix the problem _at all_. The file system would appear to be the same to rsync and it wouldn't back stuff up. So we still need some way of telling userspace that this object is different. I'm not convinced vfsmounts is the way to do this, it doesn't do anything other than add a whole lot of complexity to our mounting/subvolume mechanism that is already relatively complex. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 9/10/13 11:56 AM, Josef Bacik wrote: > On Tue, Sep 10, 2013 at 08:36:55AM -0700, Mark Fasheh wrote: >> On Mon, Aug 12, 2013 at 04:47:52AM -0700, Christoph Hellwig wrote: >>> On Thu, Aug 08, 2013 at 11:44:54AM -0400, Josef Bacik wrote: >>>> On Thu, Aug 08, 2013 at 06:48:05AM -0700, Christoph Hellwig wrote: >>>>> On Thu, Aug 08, 2013 at 09:02:07AM -0400, Josef Bacik wrote: >>>>>> This won't work, try having 10000 subvolumes with dirty inodes and do sync then >>>>>> go skiing, you'll have time :). Thanks, >>>>> >>>>> Why would the dirty inodes make any difference? If you share the bdi >>>>> between the subvolumes the sync workflow should be exactly the same >>>>> still. >>>>> >>>> >>>> If we could dis-entangle vfsmounts from sb's and have it so you could have >>>> multiple vfsmounts with just one sb that would solve at least the in-kernel >>>> confusion, but I think we still have the userspace confusion. Thanks, >>> >>> I think it would mostly solve userspace confusion, as userspace only >>> sees mounts and the device names. >>> >>> But please fix this up properly instead of propagating the effects of >>> the nasty btrfs hack that should never have been merged in that form >>> further up the stack. >> >> Can one of you explain how this solves the problem that userspace is getting >> different devices for the same inode? >> >> Seriously, I've been looking into it and I'm a bit lost. I followed the >> converstaion until here but I don't see how any of the proposed changes >> actually *fix* anything? Also, what is the relationship between vfsmounts >> and sb today? Wouldn't a bind mount produce the situation of more than 1 >> vfsmount per sb that is described above? >> >> Sincerely, someone who would like to fix this ABI breakage that has been >> going on for years. > > And let me restate the problem so we're all on the same page. > > Btrfs has subvolumes, completely separate trees within the file system. These > trees get their own object numbering, which in turn is how we do our inode > numbers. So if you have multiple subvolumes, they will likely have the same > inode numbers within the same file system. This screws up things like rsync > which say "hey look, these two inodes are the same, lets skip them." So we have > an anonymous dev so we can make them look different. > > Now if we were to make each subvol its own vfsmount (essentially a bind mount) > and remove the anonymous device that wouldn't fix the problem _at all_. The > file system would appear to be the same to rsync and it wouldn't back stuff up. > So we still need some way of telling userspace that this object is different. > > I'm not convinced vfsmounts is the way to do this, it doesn't do anything other > than add a whole lot of complexity to our mounting/subvolume mechanism that is > already relatively complex. Thanks, Agreed. It's hugely wasteful as well. We can have thousands of subvolumes even on modest systems like workstations when automated snapshots are involved. Using a vfsmount for each subvolume would make /proc/mounts pretty useless. Having a separate superblock for each one, at 1k a pop, would waste a ton of memory considering that they'll be identical except for the dev_t. The only way vfsmounts would work is if we added a dev_t there, which would usually be set to ->mnt_sb->s_dev except for the btrfs case. That still doesn't solve the polluted /proc/mounts, though. -Jeff
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index f0857e0..da7c5f5 100644 --- a/fs/btrfs/super.c +++ b/fs/btrfs/super.c @@ -822,6 +822,7 @@ static int btrfs_fill_super(struct super_block *sb, sb->s_flags |= MS_POSIXACL; #endif sb->s_flags |= MS_I_VERSION; + sb->s_flags |= MS_STAT_FOR_DEV; err = open_ctree(sb, fs_devices, (char *)data); if (err) { printk("btrfs: open_ctree failed\n"); diff --git a/fs/proc/generic.c b/fs/proc/generic.c index a2596af..ed9df4a 100644 --- a/fs/proc/generic.c +++ b/fs/proc/generic.c @@ -24,6 +24,8 @@ #include <linux/spinlock.h> #include <linux/completion.h> #include <asm/uaccess.h> +#include <linux/fs.h> +#include <linux/dcache.h> #include "internal.h" @@ -637,3 +639,16 @@ void *PDE_DATA(const struct inode *inode) return __PDE_DATA(inode); } EXPORT_SYMBOL(PDE_DATA); + +dev_t proc_get_map_dev(struct dentry *dentry) +{ + struct inode *inode = dentry->d_inode; + struct kstat kstat; + + if (inode->i_sb->s_flags & MS_STAT_FOR_DEV && + inode->i_op->getattr && + inode->i_op->getattr(NULL, dentry, &kstat) == 0) + return kstat.dev; + + return inode->i_sb->s_dev; +} diff --git a/fs/proc/internal.h b/fs/proc/internal.h index d600fb0..afc9424 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -192,6 +192,7 @@ static inline struct proc_dir_entry *pde_get(struct proc_dir_entry *pde) return pde; } extern void pde_put(struct proc_dir_entry *); +dev_t proc_get_map_dev(struct dentry *dentry); /* * inode.c diff --git a/fs/proc/nommu.c b/fs/proc/nommu.c index ccfd99b..fe22ff6 100644 --- a/fs/proc/nommu.c +++ b/fs/proc/nommu.c @@ -46,7 +46,7 @@ static int nommu_region_show(struct seq_file *m, struct vm_region *region) if (file) { struct inode *inode = file_inode(region->vm_file); - dev = inode->i_sb->s_dev; + dev = proc_get_map_dev(vma->vm_file->f_path->dentry); ino = inode->i_ino; } diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 3e636d8..4d9d1d9 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -272,7 +272,7 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma, int is_pid) if (file) { struct inode *inode = file_inode(vma->vm_file); - dev = inode->i_sb->s_dev; + dev = proc_get_map_dev(vma->vm_file->f_path.dentry); ino = inode->i_ino; pgoff = ((loff_t)vma->vm_pgoff) << PAGE_SHIFT; } diff --git a/fs/proc/task_nommu.c b/fs/proc/task_nommu.c index 56123a6..0995249 100644 --- a/fs/proc/task_nommu.c +++ b/fs/proc/task_nommu.c @@ -150,7 +150,7 @@ static int nommu_vma_show(struct seq_file *m, struct vm_area_struct *vma, if (file) { struct inode *inode = file_inode(vma->vm_file); - dev = inode->i_sb->s_dev; + dev = proc_get_map_dev(vma->vm_file->f_path.dentry); ino = inode->i_ino; pgoff = (loff_t)vma->vm_pgoff << PAGE_SHIFT; } diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index a4ed56c..918054d 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -88,6 +88,7 @@ struct inodes_stat_t { #define MS_STRICTATIME (1<<24) /* Always perform atime updates */ /* These sb flags are internal to the kernel */ +#define MS_STAT_FOR_DEV (1<<27) #define MS_NOSEC (1<<28) #define MS_BORN (1<<29) #define MS_ACTIVE (1<<30)
stat(2) on btrfs returns a custom device, but proc uses s_dev from the super block. This causes problems (abi breakage) because software (and users) are not expecting the kernel to return different devices from these calls. This patch fixes the problem by adding a new superblock flag, MS_STAT_FOR_DEV. When the proc code sees this flag, it will call the file systems ->getattr() method to extract a device as opposed to getting it directly from s_dev. This problem has been discussed several times before but still with no resolution: http://comments.gmane.org/gmane.comp.file-systems.btrfs/9682 http://thr3ads.net/btrfs-devel/2011/05/2346176-RFC-PATCH-0-2-btrfs-vfs-Return-same-device-in-stat-2-and-proc-pid-maps http://thread.gmane.org/gmane.comp.file-systems.btrfs/26786 Andrew Vagin's e-mail (third one in the list) links a couple bugzilla entries about this problem too. Signed-off-by: Mark Fasheh <mfasheh@suse.de> --- fs/btrfs/super.c | 1 + fs/proc/generic.c | 15 +++++++++++++++ fs/proc/internal.h | 1 + fs/proc/nommu.c | 2 +- fs/proc/task_mmu.c | 2 +- fs/proc/task_nommu.c | 2 +- include/uapi/linux/fs.h | 1 + 7 files changed, 21 insertions(+), 3 deletions(-)