[0/5,RFC] MUSE: Userspace backed MTD

Message

Richard Weinberger Nov. 19, 2020, 2:16 p.m. UTC

When working with flash devices a common task is emulating them to run various
tests or inspect dumps from real hardware. To achieve that we have plenty of
emulators in the mtd subsystem: mtdram, block2mtd, nandsim.

Each of them implements a adhoc MTD and have various drawbacks.
Over the last years some developers tried to extend them but these attempts
often got rejected because they added just more adhoc feature instead of
addressing overall problems.

MUSE is a novel approach to address the need of advanced MTD emulators.
Advanced means in this context supporting different (vendor specific) image
formats, different ways for fault injection (fuzzing) and recoding/replaying
IOs to emulate power cuts.

The core goal of MUSE is having the complexity on the userspace side and
only a small MTD driver in kernelspace.
While playing with different approaches I realized that FUSE offers everything
we need. So MUSE is a little like CUSE except that it does not implement a
bare character device but an MTD.

To get early feedback I'm sending this series as RFC, so don't consider it as
ready to merge yet.

Open issues are:

1. Dummy file object
The logic around fuse_direct_io() expects a file object.
Unlike FUSE or CUSE we don't have such an object in MUSE because usually an
MTD is not opened by userspace. The kernel uses the MTD and makes it available
to filesystems or other layers such as mtdblock, mtdchar or UBI.
Currently a anon inode is (ab)used for that.
Maybe there is a better way?

2. Init parameter passing
Currently parameter passing borrowed the logic from CUSE and parameters are
passed as stringy key value pairs.
Most MTD paramerters are numbers (erase size, etc..) so passing them via
struct muse_init_out seems more natural.
But I plan to pass also pure string parameters such as an mtdparts command line.
What is the perffered way these days in FUSE?
Am I allowed to embed structs such as struct mtd_info_user (mtd-abi.h) in
muse_init_out?

3. MUSE specific FUSE ops
At this time MUSE_INIT, FUSE_READ, FUSE_WRITE, FUSE_FSYNC and MUSE_ERASE are
used.

I plan to get rid of FUSE_READ and FUSE_WRITE too. On NAND'ish MTDs there is
out of band (OOB) data which can be read and written. Strictly speaking for
testing UBI/UBIFS OOB is not needed but for JFFS2 it is.

FUSE_READ/WRITE also consider every non-zero return code as fatal and abort
the transfer. In MTD, -EUCLEAN and -EBADMSG are not fatal, a driver is
expected to return possible corrupted data and let the next layer deal
with it.
So far I found no good way how to encode this in FUSE_READ. Maybe you can
point me in the right direction?

This series can also be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rw/misc.git muse_v1

Richard Weinberger (5):
  fuse: Rename FUSE_DIO_CUSE
  fuse: Export fuse_simple_request
  fuse: Make cuse_parse_one a common helper
  mtd: Add MTD_MUSE flag
  fuse: Implement MUSE: MTD in userspace

 fs/fuse/Kconfig            |  15 ++
 fs/fuse/Makefile           |   2 +
 fs/fuse/cuse.c             |  62 +----
 fs/fuse/dev.c              |   1 +
 fs/fuse/file.c             |   4 +-
 fs/fuse/fuse_i.h           |   7 +-
 fs/fuse/helper.c           |  68 ++++++
 fs/fuse/muse.c             | 450 +++++++++++++++++++++++++++++++++++++
 include/uapi/linux/fuse.h  |  25 ++-
 include/uapi/mtd/mtd-abi.h |   1 +
 10 files changed, 571 insertions(+), 64 deletions(-)
 create mode 100644 fs/fuse/helper.c
 create mode 100644 fs/fuse/muse.c

Comments

Richard Weinberger Nov. 25, 2020, 11:13 p.m. UTC | #1

----- Ursprüngliche Mail -----
> When working with flash devices a common task is emulating them to run various
> tests or inspect dumps from real hardware. To achieve that we have plenty of
> emulators in the mtd subsystem: mtdram, block2mtd, nandsim.
> 
> Each of them implements a adhoc MTD and have various drawbacks.
> Over the last years some developers tried to extend them but these attempts
> often got rejected because they added just more adhoc feature instead of
> addressing overall problems.
> 
> MUSE is a novel approach to address the need of advanced MTD emulators.
> Advanced means in this context supporting different (vendor specific) image
> formats, different ways for fault injection (fuzzing) and recoding/replaying
> IOs to emulate power cuts.
> 
> The core goal of MUSE is having the complexity on the userspace side and
> only a small MTD driver in kernelspace.
> While playing with different approaches I realized that FUSE offers everything
> we need. So MUSE is a little like CUSE except that it does not implement a
> bare character device but an MTD.
> 
> To get early feedback I'm sending this series as RFC, so don't consider it as
> ready to merge yet.
> 
> Open issues are:
> 
> 1. Dummy file object
> The logic around fuse_direct_io() expects a file object.
> Unlike FUSE or CUSE we don't have such an object in MUSE because usually an
> MTD is not opened by userspace. The kernel uses the MTD and makes it available
> to filesystems or other layers such as mtdblock, mtdchar or UBI.
> Currently a anon inode is (ab)used for that.
> Maybe there is a better way?

FYI, I'll send an updated series soon. I rewrote the MUSE IO path to not use fuse_direct_io()
which made things much simpler and all hacks go away.

Thanks,
//richard