[GIT,PULL] Mount and superblock notifications
mbox series

Message ID 1449543.1585579014@warthog.procyon.org.uk
State New
Headers show
Series
  • [GIT,PULL] Mount and superblock notifications
Related show

Pull-request

git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/notifications-fs-20200330

Message

David Howells March 30, 2020, 2:36 p.m. UTC
Hi Linus,

If you could consider pulling this - or would you prefer it to go through
Al?  It adds a couple of VFS-related event sources for the general
notification mechanism:

 (1) Mount topology events, such as mounting, unmounting, mount expiry,
     mount reconfiguration.

 (2) Superblock events, such as R/W<->R/O changes, quota overrun and I/O
     errors (not complete yet).


WHY
===

 (1) Mount notifications.

     This one is wanted to avoid repeated trawling of /proc/mounts or
     similar to work out changes to the mount object attributes and mount
     topology.  I'm told that the proc file holding the namespace_sem is a
     point of contention, especially as the process of generating the text
     descriptions of the mounts/superblocks can be quite involved.

     Whilst you can use poll() on /proc/mounts, it doesn't give you any
     clues as to what changed.  The notification generated here directly
     indicates the mounts involved in any particular event and gives an
     idea of what the change was.

     This is combined with a new fsinfo() system call that allows, amongst
     other things, the ability to retrieve in one go an { id,
     change_counter } tuple from all the children of a specified mount,
     allowing buffer overruns to be dealt with quickly.

     This can be used by systemd to improve efficiency:

	https://lore.kernel.org/linux-fsdevel/20200227151421.3u74ijhqt6ekbiss@ws.net.home/

     And it's not just Red Hat that's potentially interested in this:

	https://lore.kernel.org/linux-fsdevel/293c9bd3-f530-d75e-c353-ddeabac27cf6@6wind.com/

     Also, this can be used to improve management of containers by allowing
     watches to be set in foreign mount namespaces, such as are in a
     container.

 (2) Superblock notifications.

     This one is provided to allow systemd or the desktop to more easily
     detect events such as I/O errors and EDQUOT/ENOSPC.  This would be of
     interest to Postgres:

	https://lore.kernel.org/linux-fsdevel/20200211005626.7yqjf5rbs3vbwagd@alap3.anarazel.de/

     But could also be used to indicate to systemd when a superblock has
     had its configuration changed.

Thanks,
David
---
The following changes since commit 694435dbde3d1da79aafaf4cd680802f9eb229b7:

  smack: Implement the watch_key and post_notification hooks (2020-03-19 17:31:09 +0000)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/notifications-fs-20200330

for you to fetch changes up to 8dbf1aa122da5bbb4ede0f363a8a18dfc723be33:

  watch_queue: sample: Display superblock notifications (2020-03-19 17:31:09 +0000)

----------------------------------------------------------------
Filesystem notifications

----------------------------------------------------------------
David Howells (6):
      watch_queue: Add security hooks to rule on setting mount and sb watches
      watch_queue: Implement mount topology and attribute change notifications
      watch_queue: sample: Display mount tree change notifications
      watch_queue: Introduce a non-repeating system-unique superblock ID
      watch_queue: Add superblock notifications
      watch_queue: sample: Display superblock notifications

 Documentation/watch_queue.rst               |  24 ++-
 arch/alpha/kernel/syscalls/syscall.tbl      |   2 +
 arch/arm/tools/syscall.tbl                  |   2 +
 arch/arm64/include/asm/unistd.h             |   2 +-
 arch/arm64/include/asm/unistd32.h           |   4 +
 arch/ia64/kernel/syscalls/syscall.tbl       |   2 +
 arch/m68k/kernel/syscalls/syscall.tbl       |   2 +
 arch/microblaze/kernel/syscalls/syscall.tbl |   2 +
 arch/mips/kernel/syscalls/syscall_n32.tbl   |   2 +
 arch/mips/kernel/syscalls/syscall_n64.tbl   |   2 +
 arch/mips/kernel/syscalls/syscall_o32.tbl   |   2 +
 arch/parisc/kernel/syscalls/syscall.tbl     |   2 +
 arch/powerpc/kernel/syscalls/syscall.tbl    |   2 +
 arch/s390/kernel/syscalls/syscall.tbl       |   2 +
 arch/sh/kernel/syscalls/syscall.tbl         |   2 +
 arch/sparc/kernel/syscalls/syscall.tbl      |   2 +
 arch/x86/entry/syscalls/syscall_32.tbl      |   2 +
 arch/x86/entry/syscalls/syscall_64.tbl      |   2 +
 arch/xtensa/kernel/syscalls/syscall.tbl     |   2 +
 fs/Kconfig                                  |  21 +++
 fs/Makefile                                 |   1 +
 fs/internal.h                               |   1 +
 fs/mount.h                                  |  21 +++
 fs/mount_notify.c                           | 228 ++++++++++++++++++++++++++++
 fs/namespace.c                              |  22 +++
 fs/super.c                                  | 205 +++++++++++++++++++++++++
 include/linux/dcache.h                      |   1 +
 include/linux/fs.h                          |  62 ++++++++
 include/linux/lsm_hooks.h                   |  24 +++
 include/linux/security.h                    |  16 ++
 include/linux/syscalls.h                    |   4 +
 include/uapi/asm-generic/unistd.h           |   6 +-
 include/uapi/linux/watch_queue.h            |  65 +++++++-
 kernel/sys_ni.c                             |   6 +
 samples/watch_queue/watch_test.c            |  81 +++++++++-
 security/security.c                         |  14 ++
 36 files changed, 835 insertions(+), 5 deletions(-)
 create mode 100644 fs/mount_notify.c

Comments

Linus Torvalds April 4, 2020, 9:13 p.m. UTC | #1
On Mon, Mar 30, 2020 at 7:37 AM David Howells <dhowells@redhat.com> wrote:
>
> If you could consider pulling this - or would you prefer it to go through
> Al?  It adds a couple of VFS-related event sources for the general
> notification mechanism:

<y issue with these remains the same it was last time, so I'll just
quote what I said back then:

 "So I no longer hate the implementation, but I do want to see the
  actual user space users come out of the woodwork and try this out for
  their use cases.

  I'd hate to see a new event queue interface that people then can't
  really use due to it not fulfilling their needs, or can't use for some
  other reason."

I want to see somebody step up enough to say "yes, I actually use
this, and have the patches for the user space side, and it helps my
load by 3000%, and here are the numbers, and the event overflow case
isn't an issue because Y"

Or whatever. It doesn't have to be performance, but the separate
discussion I've seen has been about that being the reason for it.

I just don't want it to be a _hypothetical_ reason. I want it to be a
tested reason where people said "yeah, this is easy to use and
actually fixes the problems".

Because if what happens is that when the events overflow, and maybe
people fall back on the old model (or whatever) then that probably
just means that you do better up until a point where you start doing
_worse_ than we used to.

Or people find out that they needed more information anyway, and the
event model doesn't work when you restart your special server because
you've lost the original state. Or any other number of "cool feature,
but I can't really use it".

IOW, I really want to know that yes, the design is what people will
then use and it actually fixes real-world issues.

And it needs to be interesting and pressing enough that those people
actually at least do a working prototype on top of a patch-set that
hasn't made it into the kernel yet.

Now, I realize that other projects won't _upstream_ their support
before the kernel has the infrastructure, so I'm not looking for
_that_ kind of "yeah, look, project XYZ already does this and Red Hat
ships it". No, I'm looking for those outside developers who say more
than "this is a pet peeve of mine with the existing interface". I want
to see some actual use - even if it's just in a development
environment - that shows that it's (a) sufficient and (b) actually
fixes problems.

             Linus
Andres Freund April 5, 2020, 10:52 p.m. UTC | #2
Hi,

On 2020-04-04 14:13:03 -0700, Linus Torvalds wrote:
> And it needs to be interesting and pressing enough that those people
> actually at least do a working prototype on top of a patch-set that
> hasn't made it into the kernel yet.
> 
> Now, I realize that other projects won't _upstream_ their support
> before the kernel has the infrastructure, so I'm not looking for
> _that_ kind of "yeah, look, project XYZ already does this and Red Hat
> ships it". No, I'm looking for those outside developers who say more
> than "this is a pet peeve of mine with the existing interface". I want
> to see some actual use - even if it's just in a development
> environment - that shows that it's (a) sufficient and (b) actually
> fixes problems.

FWIW, postgres remains interested in using the per-superblock events.

On 2020-03-30 15:36:54 +0100, David Howells wrote:
>  (2) Superblock notifications.
> 
>      This one is provided to allow systemd or the desktop to more easily
>      detect events such as I/O errors and EDQUOT/ENOSPC.  This would be of
>      interest to Postgres:
> 
> 	https://lore.kernel.org/linux-fsdevel/20200211005626.7yqjf5rbs3vbwagd@alap3.anarazel.de/
> 
>      But could also be used to indicate to systemd when a superblock has
>      had its configuration changed.

What prevents me from coming up with a prototype is that the error
handling pieces aren't complete, as far as I can tell:

On 2020-03-30 15:36:54 +0100, David Howells wrote:
>  (2) Superblock events, such as R/W<->R/O changes, quota overrun and I/O
>      errors (not complete yet).

There's afaict no notify_sb_error() callers, making it hard for me to
actually test anything.

The important issue for us is I/O errors, but EDQUOT/ENOSPC could also
be useful (but is not urgent).

Greetings,

Andres Freund