[RFC,v2,0/8] implement containerized syncfs for overlayfs

Message

Chengguang Xu Oct. 25, 2020, 3:41 a.m. UTC

Current syncfs(2) syscall on overlayfs just calls sync_filesystem()
on upper_sb to synchronize whole dirty inodes in upper filesystem
regardless of the overlay ownership of the inode. In the use case of
container, when multiple containers using the same underlying upper
filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary
inodes for target overlayfs.

(2) Interference
Unplanned synchronization will probably impact IO performance of
unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that
only sync target dirty upper inodes which are belong to specific overlayfs
instance. By doing this, it is able to reduce cost of synchronization and
will not seriously impact IO performance of unrelated processes. 

v1->v2:
- Mark overlayfs' inode dirty itself instead of adding notification
  mechanism to vfs inode.

Chengguang Xu (8):
  ovl: setup overlayfs' private bdi
  ovl: implement ->writepages operation
  ovl: implement overlayfs' ->evict_inode operation
  ovl: mark overlayfs' inode dirty on modification
  ovl: mark overlayfs' inode dirty on shared writable mmap
  ovl: implement overlayfs' ->write_inode operation
  ovl: cache dirty overlayfs' inode
  ovl: implement containerized syncfs for overlayfs

 fs/overlayfs/file.c      |  4 +++
 fs/overlayfs/inode.c     | 27 +++++++++++++++++++
 fs/overlayfs/overlayfs.h |  4 +++
 fs/overlayfs/super.c     | 57 +++++++++++++++++++++++++++++++++++++---
 fs/overlayfs/util.c      | 14 ++++++++++
 5 files changed, 102 insertions(+), 4 deletions(-)

Comments

Miklos Szeredi Oct. 30, 2020, 3:46 p.m. UTC | #1

On Sun, Oct 25, 2020 at 4:42 AM Chengguang Xu <[e-mail]> wrote:
>
> Current syncfs(2) syscall on overlayfs just calls sync_filesystem()
> on upper_sb to synchronize whole dirty inodes in upper filesystem
> regardless of the overlay ownership of the inode. In the use case of
> container, when multiple containers using the same underlying upper
> filesystem, it has some shortcomings as below.
>
> (1) Performance
> Synchronization is probably heavy because it actually syncs unnecessary
> inodes for target overlayfs.
>
> (2) Interference
> Unplanned synchronization will probably impact IO performance of
> unrelated container processes on the other overlayfs.
>
> This series try to implement containerized syncfs for overlayfs so that
> only sync target dirty upper inodes which are belong to specific overlayfs
> instance. By doing this, it is able to reduce cost of synchronization and
> will not seriously impact IO performance of unrelated processes.

Series looks good at first glance.  Still need to do an in-depth review.

In the meantime can you post some numbers showing the performance improvements?

Thanks,
Miklos

Chengguang Xu Oct. 31, 2020, 12:22 p.m. UTC | #2

---- 在 星期五, 2020-10-30 23:46:00 Miklos Szeredi <[e-mail]> 撰写 ----
 > On Sun, Oct 25, 2020 at 4:42 AM Chengguang Xu <[e-mail]> wrote:
 > >
 > > Current syncfs(2) syscall on overlayfs just calls sync_filesystem()
 > > on upper_sb to synchronize whole dirty inodes in upper filesystem
 > > regardless of the overlay ownership of the inode. In the use case of
 > > container, when multiple containers using the same underlying upper
 > > filesystem, it has some shortcomings as below.
 > >
 > > (1) Performance
 > > Synchronization is probably heavy because it actually syncs unnecessary
 > > inodes for target overlayfs.
 > >
 > > (2) Interference
 > > Unplanned synchronization will probably impact IO performance of
 > > unrelated container processes on the other overlayfs.
 > >
 > > This series try to implement containerized syncfs for overlayfs so that
 > > only sync target dirty upper inodes which are belong to specific overlayfs
 > > instance. By doing this, it is able to reduce cost of synchronization and
 > > will not seriously impact IO performance of unrelated processes.
 > 
 > Series looks good at first glance.  Still need to do an in-depth review.
 > 
 > In the meantime can you post some numbers showing the performance improvements?
 > 

Actually, the performance improvement depends on workload and the number of overlay instances.

I did a very simple test on my dev box (intel NUC ) and the result seems quite obvious.

I made two overlay instances and untar a kernel source to an overlay then compare

umount time(sys part) of two instances. 

------------------------------------------
test env:
Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz  8core
32GB memory
Samsung NVMe SSD 970 pro

test script:

[root@localhost ovl]# cat do.sh
#!/bin/sh

rm -rf upper/* &> /dev/null
echo 3 > /proc/sys/vm/drop_caches

mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2

tar Jxvf linux-5.9.tar.xz -C merged &> /dev/null

time umount merged2
time umount merged

---------------------------------
test result:

[root@localhost ovl]# seq 3 | while read line ; do sh -x do.sh ; done
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2

real    0m0.437s
user    0m0.001s
sys     0m0.001s
+ umount merged

real    0m2.107s
user    0m0.002s
sys     0m0.090s
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2

real    0m0.443s
user    0m0.002s
sys     0m0.001s
+ umount merged

real    0m2.032s
user    0m0.001s
sys     0m0.105s
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2

real    0m0.263s
user    0m0.001s
sys     0m0.001s
+ umount merged

real    0m2.094s
user    0m0.000s
sys     0m0.083s
[root@localhost o




Thanks,
Chengguang