diff mbox

[1/2] ovl: support freeze/thaw super

Message ID 1484828008-27507-2-git-send-email-amir73il@gmail.com (mailing list archive)
State New, archived
Headers show

Commit Message

Amir Goldstein Jan. 19, 2017, 12:13 p.m. UTC
freeze/thaw of upper is all that is needed.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 fs/overlayfs/super.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

Comments

Amir Goldstein Jan. 23, 2017, 8:47 a.m. UTC | #1
On Thu, Jan 19, 2017 at 2:13 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> freeze/thaw of upper is all that is needed.
>

Miklos,

Looking at it again, I believe that not even that is needed.
Having fixed ovl_sync_fs() with patch #2, ovl_freeze()
and ovl_unfreeze() need to be NOP. Am I right?

In fact, freezing upper fs, when many overlayfs mounts
share the same base fs (a-la docker) would be quite lame.

WRT other use cases of freezing overlayfs, I am not sure
if it is needed for correctness of docker checkpoint/restart?

Pavel?

> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
>  fs/overlayfs/super.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index 6792bb7..0f8a9c8 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -160,6 +160,20 @@ static void ovl_put_super(struct super_block *sb)
>         kfree(ufs);
>  }
>
> +static int ovl_freeze(struct super_block *sb)
> +{
> +       struct ovl_fs *ufs = sb->s_fs_info;
> +
> +       return ufs->upper_mnt ? freeze_super(ufs->upper_mnt->mnt_sb) : 0;
> +}
> +
> +static int ovl_unfreeze(struct super_block *sb)
> +{
> +       struct ovl_fs *ufs = sb->s_fs_info;
> +
> +       return ufs->upper_mnt ? thaw_super(ufs->upper_mnt->mnt_sb) : 0;
> +}
> +
>  /**
>   * ovl_statfs
>   * @sb: The overlayfs super block
> @@ -222,6 +236,8 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
>
>  static const struct super_operations ovl_super_operations = {
>         .put_super      = ovl_put_super,
> +       .freeze_fs      = ovl_freeze,
> +       .unfreeze_fs    = ovl_unfreeze,
>         .statfs         = ovl_statfs,
>         .show_options   = ovl_show_options,
>         .remount_fs     = ovl_remount,
> --
> 2.7.4
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Pavel Emelyanov Jan. 23, 2017, 1:23 p.m. UTC | #2
On 01/23/2017 11:47 AM, Amir Goldstein wrote:
> On Thu, Jan 19, 2017 at 2:13 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> freeze/thaw of upper is all that is needed.
>>
> 
> Miklos,
> 
> Looking at it again, I believe that not even that is needed.
> Having fixed ovl_sync_fs() with patch #2, ovl_freeze()
> and ovl_unfreeze() need to be NOP. Am I right?
> 
> In fact, freezing upper fs, when many overlayfs mounts
> share the same base fs (a-la docker) would be quite lame.
> 
> WRT other use cases of freezing overlayfs, I am not sure
> if it is needed for correctness of docker checkpoint/restart?
> 
> Pavel?

Thanks for the heads-up :)

Good question. Actually we haven't yet experimented with doing _full_
(with FS) snapshot of containers, but at the first glance I can't
see the need for OVL freeze :( We stop all the processes in entry.S
effectively, so no in-flight IO can be happening.

>> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>> ---
>>  fs/overlayfs/super.c | 16 ++++++++++++++++
>>  1 file changed, 16 insertions(+)
>>
>> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
>> index 6792bb7..0f8a9c8 100644
>> --- a/fs/overlayfs/super.c
>> +++ b/fs/overlayfs/super.c
>> @@ -160,6 +160,20 @@ static void ovl_put_super(struct super_block *sb)
>>         kfree(ufs);
>>  }
>>
>> +static int ovl_freeze(struct super_block *sb)
>> +{
>> +       struct ovl_fs *ufs = sb->s_fs_info;
>> +
>> +       return ufs->upper_mnt ? freeze_super(ufs->upper_mnt->mnt_sb) : 0;
>> +}
>> +
>> +static int ovl_unfreeze(struct super_block *sb)
>> +{
>> +       struct ovl_fs *ufs = sb->s_fs_info;
>> +
>> +       return ufs->upper_mnt ? thaw_super(ufs->upper_mnt->mnt_sb) : 0;
>> +}
>> +
>>  /**
>>   * ovl_statfs
>>   * @sb: The overlayfs super block
>> @@ -222,6 +236,8 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
>>
>>  static const struct super_operations ovl_super_operations = {
>>         .put_super      = ovl_put_super,
>> +       .freeze_fs      = ovl_freeze,
>> +       .unfreeze_fs    = ovl_unfreeze,
>>         .statfs         = ovl_statfs,
>>         .show_options   = ovl_show_options,
>>         .remount_fs     = ovl_remount,
>> --
>> 2.7.4
>>
> .
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Amir Goldstein April 3, 2017, 11:27 a.m. UTC | #3
On Mon, Jan 23, 2017 at 3:23 PM, Pavel Emelyanov <xemul@virtuozzo.com> wrote:
> On 01/23/2017 11:47 AM, Amir Goldstein wrote:
>> On Thu, Jan 19, 2017 at 2:13 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>>> freeze/thaw of upper is all that is needed.
>>>
>>
>> Miklos,
>>
>> Looking at it again, I believe that not even that is needed.
>> Having fixed ovl_sync_fs() with patch #2, ovl_freeze()
>> and ovl_unfreeze() need to be NOP. Am I right?
>>
>> In fact, freezing upper fs, when many overlayfs mounts
>> share the same base fs (a-la docker) would be quite lame.
>>
>> WRT other use cases of freezing overlayfs, I am not sure
>> if it is needed for correctness of docker checkpoint/restart?
>>
>> Pavel?
>
> Thanks for the heads-up :)
>
> Good question. Actually we haven't yet experimented with doing _full_
> (with FS) snapshot of containers, but at the first glance I can't
> see the need for OVL freeze :( We stop all the processes in entry.S
> effectively, so no in-flight IO can be happening.
>

Serge, Stephan,

Same question.

Would lxc-snapshot gain anything from the ability to fsfreeze an overlay
mount?

Basically, it should give you the ability to create a consistent snapshot
of overlayfs upper dir from a running container, but maybe you already do
that by freezing the container processes?
I couldn't figure that out from lxc-snapshot documentation.

The context is that I implemented overlayfs fsfreeze for my own needs,
but I need other use cases to justify merging the feature.

Thanks,
Amir.
Serge E. Hallyn April 4, 2017, 5:47 p.m. UTC | #4
Quoting Amir Goldstein (amir73il@gmail.com):
> On Mon, Jan 23, 2017 at 3:23 PM, Pavel Emelyanov <xemul@virtuozzo.com> wrote:
> > On 01/23/2017 11:47 AM, Amir Goldstein wrote:
> >> On Thu, Jan 19, 2017 at 2:13 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >>> freeze/thaw of upper is all that is needed.
> >>>
> >>
> >> Miklos,
> >>
> >> Looking at it again, I believe that not even that is needed.
> >> Having fixed ovl_sync_fs() with patch #2, ovl_freeze()
> >> and ovl_unfreeze() need to be NOP. Am I right?
> >>
> >> In fact, freezing upper fs, when many overlayfs mounts
> >> share the same base fs (a-la docker) would be quite lame.
> >>
> >> WRT other use cases of freezing overlayfs, I am not sure
> >> if it is needed for correctness of docker checkpoint/restart?
> >>
> >> Pavel?
> >
> > Thanks for the heads-up :)
> >
> > Good question. Actually we haven't yet experimented with doing _full_
> > (with FS) snapshot of containers, but at the first glance I can't
> > see the need for OVL freeze :( We stop all the processes in entry.S
> > effectively, so no in-flight IO can be happening.
> >
> 
> Serge, Stephan,
> 
> Same question.
> 
> Would lxc-snapshot gain anything from the ability to fsfreeze an overlay
> mount?

lxc-snapshot only works on stopped containers.  'lxc snapshot' can do live
snapshots using criu.  Tycho, does that do anything right now to freeze the
fs?  I'm not sure that freezing all the tasks is necessarily enough to settle
the fs, but I assume you're doing something about that already?

> Basically, it should give you the ability to create a consistent snapshot
> of overlayfs upper dir from a running container, but maybe you already do
> that by freezing the container processes?
> I couldn't figure that out from lxc-snapshot documentation.
> 
> The context is that I implemented overlayfs fsfreeze for my own needs,
> but I need other use cases to justify merging the feature.
> 
> Thanks,
> Amir.
Tycho Andersen April 4, 2017, 6:01 p.m. UTC | #5
On Tue, Apr 04, 2017 at 12:47:52PM -0500, Serge E. Hallyn wrote:
> > Would lxc-snapshot gain anything from the ability to fsfreeze an overlay
> > mount?
> 
> lxc-snapshot only works on stopped containers.  'lxc snapshot' can do live
> snapshots using criu.  Tycho, does that do anything right now to freeze the
> fs?

Not that I'm aware of (CRIU might, but we don't in liblxc).

> I'm not sure that freezing all the tasks is necessarily enough to settle
> the fs, but I assume you're doing something about that already?

I suspect it's not, but we're not doing anything besides freezing the
tasks. In fact, we freeze the tasks by using the freezer cgroup,
which itself is buggy, since the freezer cgroup can race with various
filesystems. So, freezing tasks is hard, and I haven't even thought
about how to freeze the fs for real :)

But in any case, an fs freezing primitive does sound useful for
checkpoint restore, assuming that we're right and freezing the tasks
is simply not enough.

Tycho
Amir Goldstein April 4, 2017, 6:59 p.m. UTC | #6
On Tue, Apr 4, 2017 at 9:01 PM, Tycho Andersen <tycho@docker.com> wrote:
> On Tue, Apr 04, 2017 at 12:47:52PM -0500, Serge E. Hallyn wrote:
>> > Would lxc-snapshot gain anything from the ability to fsfreeze an overlay
>> > mount?
>>
>> lxc-snapshot only works on stopped containers.  'lxc snapshot' can do live
>> snapshots using criu.  Tycho, does that do anything right now to freeze the
>> fs?
>
> Not that I'm aware of (CRIU might, but we don't in liblxc).
>
>> I'm not sure that freezing all the tasks is necessarily enough to settle
>> the fs, but I assume you're doing something about that already?
>
> I suspect it's not, but we're not doing anything besides freezing the
> tasks. In fact, we freeze the tasks by using the freezer cgroup,
> which itself is buggy, since the freezer cgroup can race with various
> filesystems. So, freezing tasks is hard, and I haven't even thought
> about how to freeze the fs for real :)
>
> But in any case, an fs freezing primitive does sound useful for
> checkpoint restore, assuming that we're right and freezing the tasks
> is simply not enough.
>

So I already asked Pavel that question and he said that freezing
the tasks is enough. I am not convinced it is really enough to bring
a file system image (i.e. underlying blockdev) to a quiescent state,
but I think it may be enough for getting a stable view of the mounted
file system, so the files could be dumped somewhere.
I am guessing is what lxc snapshot does?

I still didn't understand wrt lxc snapshot, is there a use case for
taking live snapshots without using CRIU? (because freezer cgroup
mentioned races or whatnot?).

It's definitely possible with btrfs and if my overlayfs freeze patches
are not terribly wrong, then it should be easy with overlayfs as well.
Does lxc snapshot already support live snapshot of btrfs container?

If there is interest, I can try to do a POC of live lxc snapshot of an
overlayfs container.

Amir.
Tycho Andersen April 4, 2017, 7:07 p.m. UTC | #7
On Tue, Apr 04, 2017 at 09:59:16PM +0300, Amir Goldstein wrote:
> On Tue, Apr 4, 2017 at 9:01 PM, Tycho Andersen <tycho@docker.com> wrote:
> > On Tue, Apr 04, 2017 at 12:47:52PM -0500, Serge E. Hallyn wrote:
> >> > Would lxc-snapshot gain anything from the ability to fsfreeze an overlay
> >> > mount?
> >>
> >> lxc-snapshot only works on stopped containers.  'lxc snapshot' can do live
> >> snapshots using criu.  Tycho, does that do anything right now to freeze the
> >> fs?
> >
> > Not that I'm aware of (CRIU might, but we don't in liblxc).
> >
> >> I'm not sure that freezing all the tasks is necessarily enough to settle
> >> the fs, but I assume you're doing something about that already?
> >
> > I suspect it's not, but we're not doing anything besides freezing the
> > tasks. In fact, we freeze the tasks by using the freezer cgroup,
> > which itself is buggy, since the freezer cgroup can race with various
> > filesystems. So, freezing tasks is hard, and I haven't even thought
> > about how to freeze the fs for real :)
> >
> > But in any case, an fs freezing primitive does sound useful for
> > checkpoint restore, assuming that we're right and freezing the tasks
> > is simply not enough.
> >
> 
> So I already asked Pavel that question and he said that freezing
> the tasks is enough. I am not convinced it is really enough to bring
> a file system image (i.e. underlying blockdev) to a quiescent state,
> but I think it may be enough for getting a stable view of the mounted
> file system, so the files could be dumped somewhere.
> I am guessing is what lxc snapshot does?

Yes, lxc snapshot is basically just a frontend for CRIU.

> I still didn't understand wrt lxc snapshot, is there a use case for
> taking live snapshots without using CRIU? (because freezer cgroup
> mentioned races or whatnot?).

No, I think CRIU is the only project that will ever attempt to do
checkpoint restore this way ;-). CRIU supports two different ways of
freezing tasks: one using the freezer cgroup and one without. The one
without doesn't work against fork bombs very well, and the one with
doesn't work because of some filesystems. So it's mostly a container
engine implementation choice which to use.

> It's definitely possible with btrfs and if my overlayfs freeze patches
> are not terribly wrong, then it should be easy with overlayfs as well.
> Does lxc snapshot already support live snapshot of btrfs container?

Yes, it does. It freezes the tasks via the cgroup freezer and then
does a btrfs snapshot of the filesystem once the tasks are frozen.

Tycho
diff mbox

Patch

diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index 6792bb7..0f8a9c8 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -160,6 +160,20 @@  static void ovl_put_super(struct super_block *sb)
 	kfree(ufs);
 }
 
+static int ovl_freeze(struct super_block *sb)
+{
+	struct ovl_fs *ufs = sb->s_fs_info;
+
+	return ufs->upper_mnt ? freeze_super(ufs->upper_mnt->mnt_sb) : 0;
+}
+
+static int ovl_unfreeze(struct super_block *sb)
+{
+	struct ovl_fs *ufs = sb->s_fs_info;
+
+	return ufs->upper_mnt ? thaw_super(ufs->upper_mnt->mnt_sb) : 0;
+}
+
 /**
  * ovl_statfs
  * @sb: The overlayfs super block
@@ -222,6 +236,8 @@  static int ovl_remount(struct super_block *sb, int *flags, char *data)
 
 static const struct super_operations ovl_super_operations = {
 	.put_super	= ovl_put_super,
+	.freeze_fs	= ovl_freeze,
+	.unfreeze_fs	= ovl_unfreeze,
 	.statfs		= ovl_statfs,
 	.show_options	= ovl_show_options,
 	.remount_fs	= ovl_remount,