mbox series

[v21,00/10] NTFS read-write driver GPL implementation by Paragon Software

Message ID 20210212162416.2756937-1-almaz.alexandrovich@paragon-software.com (mailing list archive)
Headers show
Series NTFS read-write driver GPL implementation by Paragon Software | expand

Message

Konstantin Komarov Feb. 12, 2021, 4:24 p.m. UTC
This patch adds NTFS Read-Write driver to fs/ntfs3.

Having decades of expertise in commercial file systems development and huge
test coverage, we at Paragon Software GmbH want to make our contribution to
the Open Source Community by providing implementation of NTFS Read-Write
driver for the Linux Kernel.

This is fully functional NTFS Read-Write driver. Current version works with
NTFS(including v3.1) and normal/compressed/sparse files and supports journal replaying.

We plan to support this version after the codebase once merged, and add new
features and fix bugs. For example, full journaling support over JBD will be
added in later updates.

v2:
 - patch splitted to chunks (file-wise)
 - build issues fixed
 - sparse and checkpatch.pl errors fixed
 - NULL pointer dereference on mkfs.ntfs-formatted volume mount fixed
 - cosmetics + code cleanup

v3:
 - added acl, noatime, no_acs_rules, prealloc mount options
 - added fiemap support
 - fixed encodings support
 - removed typedefs
 - adapted Kernel-way logging mechanisms
 - fixed typos and corner-case issues

v4:
 - atomic_open() refactored
 - code style updated
 - bugfixes

v5:
- nls/nls_alt mount options added
- Unicode conversion fixes
- Improved very fragmented files operations
- logging cosmetics

v6:
- Security Descriptors processing changed
  added system.ntfs_security xattr to set
  SD
- atomic_open() optimized
- cosmetics

v7:
- Security Descriptors validity checks added (by Mark Harmstone)
- atomic_open() fixed for the compressed file creation with directio
  case
- remount support
- temporarily removed readahead usage
- cosmetics

v8:
- Compressed files operations fixed

v9:
- Further cosmetics applied as suggested
by Joe Perches

v10:
- operations with compressed/sparse files on very fragmented volumes improved
- reduced memory consumption for above cases

v11:
- further compressed files optimizations: reads/writes are now skipping bufferization
- journal wipe to the initial state optimized (bufferization is also skipped)
- optimized run storage (re-packing cluster metainformation)
- fixes based on Matthew Wilcox feedback to the v10
- compressed/sparse/normal could be set for empty files with 'system.ntfs_attrib' xattr

v12:
- nls_alt mount option removed after discussion with Pali Rohar
- fixed ni_repack()
- fixed resident files transition to non-resident when size increasing

v13:
- nested_lock fix (lockdep)
- out-of-bounds read fix (KASAN warning)
- resident->nonresident transition fixed for compressed files
- load_nls() missed fix applied
- some sparse utility warnings fixes

v14:
- support for additional compression types (we've adapted WIMLIB's
  implementation, authored by Eric Biggers, into ntfs3)

v15:
- kernel test robot warnings fixed
- lzx/xpress compression license headers updated

v16:
- lzx/xpress moved to initial ntfs-3g plugin code
- mutexes instead of a global spinlock for compresions
- FALLOC_FL_PUNCH_HOLE and FALLOC_FL_COLLAPSE_RANGE implemented
- CONFIG_NTFS3_FS_POSIX_ACL added

v17:
- FALLOC_FL_COLLAPSE_RANGE fixed
- fixes for Mattew Wilcox's and Andy Lavr's concerns

v18:
- ntfs_alloc macro splitted into two ntfs_malloc + ntfs_zalloc
- attrlist.c: always use ntfs_cmp_names instead of memcmp; compare entry names
  only for entry with vcn == 0
- dir.c: remove unconditional ni_lock in ntfs_readdir
- fslog.c: corrected error case behavior
- index.c: refactored due to modification of ntfs_cmp_names; use rw_semaphore
  for read/write access to alloc_run and bitmap_run while ntfs_readdir
- run.c: separated big/little endian code in functions
- upcase.c: improved ntfs_cmp_names, thanks to Kari Argillander for idea
  and 'bothcase' implementation

v19:
- fixed directory bitmap for 2MB cluster size
- fixed rw_semaphore init for directories

v20:
- fixed issue with incorrect hidden/system attribute setting on
  root subdirectories
- use kvmalloc instead of kmalloc for runs array
- fixed index behavior on volumes with cluster size more than 4k
- current build info is added into module info instead of printing on insmod

v21:
- fixes for clang CFI checks
- fixed sb->s_maxbytes for 32bit clusters
- user.DOSATTRIB is no more intercepted by ntfs3
- corrected xattr limits;  is used
- corrected CONFIG_NTFS3_64BIT_CLUSTER usage
- info about current build is added into module info and printing
on insmod (by Andy Lavr's request)
note: v21 is applicable for 'linux-next' not older than 2021.01.28

Konstantin Komarov (10):
  fs/ntfs3: Add headers and misc files
  fs/ntfs3: Add initialization of super block
  fs/ntfs3: Add bitmap
  fs/ntfs3: Add file operations and implementation
  fs/ntfs3: Add attrib operations
  fs/ntfs3: Add compression
  fs/ntfs3: Add NTFS journal
  fs/ntfs3: Add Kconfig, Makefile and doc
  fs/ntfs3: Add NTFS3 in fs/Kconfig and fs/Makefile
  fs/ntfs3: Add MAINTAINERS

 Documentation/filesystems/ntfs3.rst |  107 +
 MAINTAINERS                         |    7 +
 fs/Kconfig                          |    1 +
 fs/Makefile                         |    1 +
 fs/ntfs3/Kconfig                    |   45 +
 fs/ntfs3/Makefile                   |   31 +
 fs/ntfs3/attrib.c                   | 2085 +++++++++++
 fs/ntfs3/attrlist.c                 |  457 +++
 fs/ntfs3/bitfunc.c                  |  135 +
 fs/ntfs3/bitmap.c                   | 1495 ++++++++
 fs/ntfs3/debug.h                    |   64 +
 fs/ntfs3/dir.c                      |  583 +++
 fs/ntfs3/file.c                     | 1130 ++++++
 fs/ntfs3/frecord.c                  | 3083 ++++++++++++++++
 fs/ntfs3/fslog.c                    | 5204 +++++++++++++++++++++++++++
 fs/ntfs3/fsntfs.c                   | 2535 +++++++++++++
 fs/ntfs3/index.c                    | 2646 ++++++++++++++
 fs/ntfs3/inode.c                    | 2058 +++++++++++
 fs/ntfs3/lib/decompress_common.c    |  332 ++
 fs/ntfs3/lib/decompress_common.h    |  352 ++
 fs/ntfs3/lib/lib.h                  |   26 +
 fs/ntfs3/lib/lzx_decompress.c       |  683 ++++
 fs/ntfs3/lib/xpress_decompress.c    |  155 +
 fs/ntfs3/lznt.c                     |  452 +++
 fs/ntfs3/namei.c                    |  592 +++
 fs/ntfs3/ntfs.h                     | 1236 +++++++
 fs/ntfs3/ntfs_fs.h                  | 1073 ++++++
 fs/ntfs3/record.c                   |  609 ++++
 fs/ntfs3/run.c                      | 1120 ++++++
 fs/ntfs3/super.c                    | 1502 ++++++++
 fs/ntfs3/upcase.c                   |  100 +
 fs/ntfs3/xattr.c                    | 1050 ++++++
 32 files changed, 30949 insertions(+)
 create mode 100644 Documentation/filesystems/ntfs3.rst
 create mode 100644 fs/ntfs3/Kconfig
 create mode 100644 fs/ntfs3/Makefile
 create mode 100644 fs/ntfs3/attrib.c
 create mode 100644 fs/ntfs3/attrlist.c
 create mode 100644 fs/ntfs3/bitfunc.c
 create mode 100644 fs/ntfs3/bitmap.c
 create mode 100644 fs/ntfs3/debug.h
 create mode 100644 fs/ntfs3/dir.c
 create mode 100644 fs/ntfs3/file.c
 create mode 100644 fs/ntfs3/frecord.c
 create mode 100644 fs/ntfs3/fslog.c
 create mode 100644 fs/ntfs3/fsntfs.c
 create mode 100644 fs/ntfs3/index.c
 create mode 100644 fs/ntfs3/inode.c
 create mode 100644 fs/ntfs3/lib/decompress_common.c
 create mode 100644 fs/ntfs3/lib/decompress_common.h
 create mode 100644 fs/ntfs3/lib/lib.h
 create mode 100644 fs/ntfs3/lib/lzx_decompress.c
 create mode 100644 fs/ntfs3/lib/xpress_decompress.c
 create mode 100644 fs/ntfs3/lznt.c
 create mode 100644 fs/ntfs3/namei.c
 create mode 100644 fs/ntfs3/ntfs.h
 create mode 100644 fs/ntfs3/ntfs_fs.h
 create mode 100644 fs/ntfs3/record.c
 create mode 100644 fs/ntfs3/run.c
 create mode 100644 fs/ntfs3/super.c
 create mode 100644 fs/ntfs3/upcase.c
 create mode 100644 fs/ntfs3/xattr.c


base-commit: dcc0b49040c70ad827a7f3d58a21b01fdb14e749

Comments

Oleksandr Natalenko Feb. 12, 2021, 9:27 p.m. UTC | #1
Hi.

On Fri, Feb 12, 2021 at 07:24:06PM +0300, Konstantin Komarov wrote:
> This patch adds NTFS Read-Write driver to fs/ntfs3.
> …
> v21:
> - fixes for clang CFI checks
> - fixed sb->s_maxbytes for 32bit clusters
> - user.DOSATTRIB is no more intercepted by ntfs3
> - corrected xattr limits;  is used
> - corrected CONFIG_NTFS3_64BIT_CLUSTER usage
> - info about current build is added into module info and printing
> on insmod (by Andy Lavr's request)
> note: v21 is applicable for 'linux-next' not older than 2021.01.28

For those who use this on v5.10/v5.11, there's an extra patch available
that applies on top of this submission: [1].

Hanabishi, babam (both in Cc), here [2] you've reported some issues with
accessing some files and with hidden attributes. You may reply to this
email of mine with detailed description of your issues, and maybe
developers will answer you.

Thanks.

[1] https://gitlab.com/post-factum/pf-kernel/-/commit/e487427ef07c735fdc711a56d1ceac6629c34dcf.patch
[2] https://aur.archlinux.org/packages/ntfs3-dkms/
Hanabishi Recca Feb. 13, 2021, 7 p.m. UTC | #2
On Sat, Feb 13, 2021 at 2:27 AM Oleksandr Natalenko
<oleksandr@natalenko.name> wrote:

> Hanabishi, babam (both in Cc), here [2] you've reported some issues with
> accessing some files and with hidden attributes. You may reply to this
> email of mine with detailed description of your issues, and maybe
> developers will answer you.

There is strange files access issue since v18 update. Some random
files on partition became inaccessible, can't be read or even deleted.
For example:

# ls -la
ls: cannot access 'NlsStrings.js': No such file or directory
total 176
drwxrwxrwx 1 root root  4096 Oct 20 10:41 .
drwxrwxrwx 1 root root 12288 Oct 20 10:42 ..
-rwxrwxrwx 1 root root  8230 Oct 19 17:02 Layer.js < this file is ok
-????????? ? ?    ?        ?            ? NlsStrings.js < this file is
inaccessible
...

To reproduce the issue try to mount a NTFS partition with deep
structure and large files amout. Then run on it some recursive file
command, e.g. 'du -sh', it will list all access errors.
Can't say what exactly causes it. Filesystem itself is not damaged,
when mounting it via ntfs-3g, ntfs3 <18 or in Windows it works
normally. The files is not damaged and chkdsk report no errors.
kasep pisan Feb. 15, 2021, 7:07 a.m. UTC | #3
The bug can be reproduced easily with following step:
find mountpoint -exec ls -d {} + 1>/dev/null


2021-02-14 2:00 GMT+07.00, Hanabishi Recca <irecca.kun@gmail.com>:
> On Sat, Feb 13, 2021 at 2:27 AM Oleksandr Natalenko
> <oleksandr@natalenko.name> wrote:
>
>> Hanabishi, babam (both in Cc), here [2] you've reported some issues with
>> accessing some files and with hidden attributes. You may reply to this
>> email of mine with detailed description of your issues, and maybe
>> developers will answer you.
>
> There is strange files access issue since v18 update. Some random
> files on partition became inaccessible, can't be read or even deleted.
> For example:
>
> # ls -la
> ls: cannot access 'NlsStrings.js': No such file or directory
> total 176
> drwxrwxrwx 1 root root  4096 Oct 20 10:41 .
> drwxrwxrwx 1 root root 12288 Oct 20 10:42 ..
> -rwxrwxrwx 1 root root  8230 Oct 19 17:02 Layer.js < this file is ok
> -????????? ? ?    ?        ?            ? NlsStrings.js < this file is
> inaccessible
> ...
>
> To reproduce the issue try to mount a NTFS partition with deep
> structure and large files amout. Then run on it some recursive file
> command, e.g. 'du -sh', it will list all access errors.
> Can't say what exactly causes it. Filesystem itself is not damaged,
> when mounting it via ntfs-3g, ntfs3 <18 or in Windows it works
> normally. The files is not damaged and chkdsk report no errors.
>