diff mbox series

[isar-cip-core,3/3] Added support for rootfs-overlay.

Message ID 20250217100024.42995-4-alexander.heinisch@siemens.com (mailing list archive)
State New
Headers show
Series Added support for rootfs-overlay (for development). | expand

Commit Message

Heinisch, Alexander Feb. 17, 2025, 10 a.m. UTC
From: Alexander Heinisch <alexander.heinisch@siemens.com>

In addition to the overlays for e.g. /etc we now support persistent
overlays for rootfs as well.
To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"

This helps when developing or debugging an existing image, while
still maintaining being as close to the prod image as possible.
That way all image features like swupdate / verity / aso. remain
intact with the sole exeception of having the possibility to
work on a rw rootfs.
Bare in mind while this comes in handy for debugging or developing
new features, such feature has to be used with care. e.g. when
updating the rootfs, overwritten files from the overlay could
potentially cause unexpected results!

Compared to typical overlays we have to move the storage device used
for persistence off the previous rootfs. Otherwise, the recursion in
the overlay setup would cause panics when finally switching root.
In addition to that we move all mountpoints setup prior to overlaying
to the new overlayed root.

Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
---
 .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
 .../initramfs-overlay-hook_0.3.bb             |  2 +-
 2 files changed, 83 insertions(+), 11 deletions(-)

Comments

Quirin Gylstorff Feb. 18, 2025, 8:39 a.m. UTC | #1
On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
> From: Alexander Heinisch <alexander.heinisch@siemens.com>
> 
> In addition to the overlays for e.g. /etc we now support persistent
> overlays for rootfs as well.
> To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
> or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> 

I would like to avoid that anti pattern of creating an overlay over the 
complete rootfs. What are the debug case you need a full overlay?

Quirin

> This helps when developing or debugging an existing image, while
> still maintaining being as close to the prod image as possible.
> That way all image features like swupdate / verity / aso. remain
> intact with the sole exeception of having the possibility to
> work on a rw rootfs.
> Bare in mind while this comes in handy for debugging or developing
> new features, such feature has to be used with care. e.g. when
> updating the rootfs, overwritten files from the overlay could
> potentially cause unexpected results!
> 
> Compared to typical overlays we have to move the storage device used
> for persistence off the previous rootfs. Otherwise, the recursion in
> the overlay setup would cause panics when finally switching root.
> In addition to that we move all mountpoints setup prior to overlaying
> to the new overlayed root.
> 
> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
> ---
>   .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
>   .../initramfs-overlay-hook_0.3.bb             |  2 +-
>   2 files changed, 83 insertions(+), 11 deletions(-)
> 
> diff --git a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> index 71cc63c..a17e4a4 100644
> --- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> +++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> @@ -6,14 +6,19 @@
>   # Authors:
>   #  Jan Kiszka <jan.kiszka@siemens.com>
>   #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
> +#  Alexander Heinisch <alexander.heinisch@siemens.com>
>   #
>   
>   ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
>   ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
>   
> -root_mount_storage=${rootmnt}${ovl_storage_path}
> +root_mount_storage=${ovl_storage_path}
>   storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
>   
> +# Setup mountpoint for persistent storage
> +# Either, it already got setup by a previous initramfs script,
> +# then we have to move it from $rootmnt to $storage_mount_point or it has not
> +# been setup yet, then check it and mount it under $storage_mount_point
>   if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>   	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
>   	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
> @@ -21,6 +26,8 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>   
>   	partition_fstype=$(get_fstype "${ovl_partition_device}")
>   
> +	# FS checks should be moved to a separate hook / or at least initramfs lib
> +	# so generic functionality can be reused here!
>   	case $partition_fstype in
>   	ext*)
>   		fsck_ret=0
> @@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>   		fi
>   		;;
>   	esac
> +
> +	# Mount overlay device to initramfs - will be moved to rootfs at the end
> +	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
>   	if ! mount -t ${partition_fstype} \
>   		 -o ${ovl_mount_option} \
>   		 ${ovl_partition_device} \
> -		 ${rootmnt}${storage_mount_point}; then
> +		 ${storage_mount_point}; then
>   		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
>   	fi
> +else
> +	# If the storage device is already mounted on the final rootfs
> +	# move it to the initramfs. Otherwise, we cannot move back the
> +	# storage mount to the resulting rootfs in the end.
> +
> +	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
> +	mount -n -o move ${rootmnt}${storage_mount_point} ${storage_mount_point}
>   fi
>   
> +[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
> +
>   for ovl_lower_dir in ${ovl_lower_dirs}; do
> -	# remove the first '/' and replace all '/' with '_'
> -	# on variable $ovl_lower_dir
> -	work_dir_name=$(awk -v var=$ovl_lower_dir \
> -		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> +	root_moved=false
> +
> +	if [ $ovl_lower_dir != "/" ]; then
> +		log_begin_msg "Mounting overlay $ovl_lower_dir"
> +
> +		# remove the first '/' and replace all '/' with '_'
> +		# on variable $ovl_lower_dir
> +		work_dir_name=$(awk -v var=$ovl_lower_dir \
> +			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> +
> +		lower_dir=${rootmnt}${ovl_lower_dir}
> +		upper_dir=${root_mount_storage}${ovl_lower_dir}
> +		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> +		mount_dir=${lower_dir}
> +	else
> +		log_begin_msg "Mounting root overlay"
> +
> +		# When overlaying root we need to move the mountpoint
> +		# from ${rootmnt} to a dedicated mountpoint inside
> +		# the storage dir to prevent a "recursion" within
> +		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
> +		# While this is not a limitation of overlay itself
> +		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
> +		# outside the tree of new root which panics when switching root
> +
> +		# create lower dir to move ro-rootfs to
> +		lower_dir=${root_mount_storage}/lower-root
> +		mkdir -p ${lower_dir}
> +
> +		# make the readonly root available
> +		mount -n -o move ${rootmnt} ${lower_dir}
> +		root_moved=true
>   
> -	lower_dir=${rootmnt}${ovl_lower_dir}
> -	upper_dir=${root_mount_storage}${ovl_lower_dir}
> -	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> +		upper_dir=${root_mount_storage}/root
> +		work_dir=${root_mount_storage}/.root-atomic
> +
> +		mount_dir=${rootmnt}
> +	fi
>   
>   	mkdir -p ${upper_dir}
>   	mkdir -p ${work_dir}
>   
> +	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
>   	if ! mount -t overlay \
>   		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
> -		   overlay ${lower_dir}; then
> +		   overlay ${mount_dir}; then
>   		panic "Can't mount overlay for '$ovl_lower_dir' !"
>   	fi
> +
> +	if ${root_moved}; then
> +		# Loop over all mountpoints already setup before we established the root overlay
> +		# and move them to the resulting overlayed rootfs
> +		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
> +		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
> +			while read -r old_mountpoint
> +			do
> +				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
> +
> +				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
> +				mount -n -o move ${old_mountpoint} ${new_mountpoint}
> +			done
> +	fi
> +
> +	log_end_msg
>   done
>   
> +# Finally, move the overlay workdir from initramfs to the final rootfs
> +[ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
> +mount -n -o move ${storage_mount_point} ${rootmnt}${storage_mount_point}
> +
> +log_success_msg "Overlays setup successfully"
> +
>   exit 0
> diff --git a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
> index ec7f85b..af610e5 100644
> --- a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
> +++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
> @@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
>   DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
>   
>   HOOK_ADD_MODULES = "overlay"
> -HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
> +HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
>   
>   SCRIPT_PREREQ="crypt"
>
Heinisch, Alexander Feb. 20, 2025, 1 p.m. UTC | #2
> On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
> > From: Alexander Heinisch <alexander.heinisch@siemens.com>
> > 
> > In addition to the overlays for e.g. /etc we now support persistent 
> > overlays for rootfs as well.
> > To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
> > or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> > 
> 
> I would like to avoid that anti pattern of creating an overlay over the complete rootfs. What are the debug case you need a full overlay?
>
To be very clear: This is not meant for debugging in the field! Never, ever.

The intention of having a root-overlay for us was to have an easy way
to develop and extend recipes based on our existing images. That means,
we wanted to have the possibility to install packages with apt, or copy 
additional binaries, services, configs on the device without having to 
rebuild our image all the time, still maintaining to have a working environment
as close as possible to the production image. 
That enables us to develop features in a debian native environment, still being
close to the features present in the final target image. 
After being confident, the steps done and changes reflected on the overlay can 
be transferred fairly easy to a recipe and end up in a production image (image without the overlay).

BR Alexander
 
> Quirin
> 
> > This helps when developing or debugging an existing image, while still 
> > maintaining being as close to the prod image as possible.
> > That way all image features like swupdate / verity / aso. remain 
> > intact with the sole exeception of having the possibility to work on a 
> > rw rootfs.
> > Bare in mind while this comes in handy for debugging or developing new 
> > features, such feature has to be used with care. e.g. when updating 
> > the rootfs, overwritten files from the overlay could potentially cause 
> > unexpected results!
> > 
> > Compared to typical overlays we have to move the storage device used 
> > for persistence off the previous rootfs. Otherwise, the recursion in 
> > the overlay setup would cause panics when finally switching root.
> > In addition to that we move all mountpoints setup prior to overlaying 
> > to the new overlayed root.
> > 
> > Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
> > ---
> >   .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
> >   .../initramfs-overlay-hook_0.3.bb             |  2 +-
> >   2 files changed, 83 insertions(+), 11 deletions(-)
> > 
> > diff --git 
> > a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl 
> > b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> > index 71cc63c..a17e4a4 100644
> > --- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> > +++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> > @@ -6,14 +6,19 @@
> >   # Authors:
> >   #  Jan Kiszka <jan.kiszka@siemens.com>
> >   #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
> > +#  Alexander Heinisch <alexander.heinisch@siemens.com>
> >   #
> >   
> >   ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
> >   ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
> >   
> > -root_mount_storage=${rootmnt}${ovl_storage_path}
> > +root_mount_storage=${ovl_storage_path}
> >   storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
> >   
> > +# Setup mountpoint for persistent storage # Either, it already got 
> > +setup by a previous initramfs script, # then we have to move it from 
> > +$rootmnt to $storage_mount_point or it has not # been setup yet, then 
> > +check it and mount it under $storage_mount_point
> >   if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
> >   	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
> >   	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
> > @@ -21,6 +26,8 @@ if ! mountpoint -q 
> > "${rootmnt}${storage_mount_point}"; then
> >   
> >   	partition_fstype=$(get_fstype "${ovl_partition_device}")
> >   
> > +	# FS checks should be moved to a separate hook / or at least initramfs lib
> > +	# so generic functionality can be reused here!
> >   	case $partition_fstype in
> >   	ext*)
> >   		fsck_ret=0
> > @@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
> >   		fi
> >   		;;
> >   	esac
> > +
> > +	# Mount overlay device to initramfs - will be moved to rootfs at the end
> > +	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
> >   	if ! mount -t ${partition_fstype} \
> >   		 -o ${ovl_mount_option} \
> >   		 ${ovl_partition_device} \
> > -		 ${rootmnt}${storage_mount_point}; then
> > +		 ${storage_mount_point}; then
> >   		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
> >   	fi
> > +else
> > +	# If the storage device is already mounted on the final rootfs
> > +	# move it to the initramfs. Otherwise, we cannot move back the
> > +	# storage mount to the resulting rootfs in the end.
> > +
> > +	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
> > +	mount -n -o move ${rootmnt}${storage_mount_point} 
> > +${storage_mount_point}
> >   fi
> >   
> > +[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
> > +
> >   for ovl_lower_dir in ${ovl_lower_dirs}; do
> > -	# remove the first '/' and replace all '/' with '_'
> > -	# on variable $ovl_lower_dir
> > -	work_dir_name=$(awk -v var=$ovl_lower_dir \
> > -		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> > +	root_moved=false
> > +
> > +	if [ $ovl_lower_dir != "/" ]; then
> > +		log_begin_msg "Mounting overlay $ovl_lower_dir"
> > +
> > +		# remove the first '/' and replace all '/' with '_'
> > +		# on variable $ovl_lower_dir
> > +		work_dir_name=$(awk -v var=$ovl_lower_dir \
> > +			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> > +
> > +		lower_dir=${rootmnt}${ovl_lower_dir}
> > +		upper_dir=${root_mount_storage}${ovl_lower_dir}
> > +		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> > +		mount_dir=${lower_dir}
> > +	else
> > +		log_begin_msg "Mounting root overlay"
> > +
> > +		# When overlaying root we need to move the mountpoint
> > +		# from ${rootmnt} to a dedicated mountpoint inside
> > +		# the storage dir to prevent a "recursion" within
> > +		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
> > +		# While this is not a limitation of overlay itself
> > +		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
> > +		# outside the tree of new root which panics when switching root
> > +
> > +		# create lower dir to move ro-rootfs to
> > +		lower_dir=${root_mount_storage}/lower-root
> > +		mkdir -p ${lower_dir}
> > +
> > +		# make the readonly root available
> > +		mount -n -o move ${rootmnt} ${lower_dir}
> > +		root_moved=true
> >   
> > -	lower_dir=${rootmnt}${ovl_lower_dir}
> > -	upper_dir=${root_mount_storage}${ovl_lower_dir}
> > -	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> > +		upper_dir=${root_mount_storage}/root
> > +		work_dir=${root_mount_storage}/.root-atomic
> > +
> > +		mount_dir=${rootmnt}
> > +	fi
> >   
> >   	mkdir -p ${upper_dir}
> >   	mkdir -p ${work_dir}
> >   
> > +	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
> >   	if ! mount -t overlay \
> >   		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
> > -		   overlay ${lower_dir}; then
> > +		   overlay ${mount_dir}; then
> >   		panic "Can't mount overlay for '$ovl_lower_dir' !"
> >   	fi
> > +
> > +	if ${root_moved}; then
> > +		# Loop over all mountpoints already setup before we established the root overlay
> > +		# and move them to the resulting overlayed rootfs
> > +		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
> > +		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
> > +			while read -r old_mountpoint
> > +			do
> > +				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
> > +
> > +				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
> > +				mount -n -o move ${old_mountpoint} ${new_mountpoint}
> > +			done
> > +	fi
> > +
> > +	log_end_msg
> >   done
> >   
> > +# Finally, move the overlay workdir from initramfs to the final 
> > +rootfs [ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
> > +mount -n -o move ${storage_mount_point} 
> > +${rootmnt}${storage_mount_point}
> > +
> > +log_success_msg "Overlays setup successfully"
> > +
> >   exit 0
> > diff --git 
> > a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
> > bb 
> > b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
> > bb
> > index ec7f85b..af610e5 100644
> > --- 
> > a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
> > bb
> > +++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_
> > +++ 0.3.bb
> > @@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
> >   DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
> >   
> >   HOOK_ADD_MODULES = "overlay"
> > -HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
> > +HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
> >   
> >   SCRIPT_PREREQ="crypt"
> >
Nussel, Ludwig Feb. 20, 2025, 1:11 p.m. UTC | #3
On Thu, 2025-02-20 at 13:00 +0000, Heinisch, Alexander via lists.cip-
project.org wrote:
> > On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
> > > From: Alexander Heinisch <alexander.heinisch@siemens.com>
> > > 
> > > In addition to the overlays for e.g. /etc we now support
> > > persistent 
> > > overlays for rootfs as well.
> > > To enable this behaviour you can either set
> > > INITRAMFS_OVERLAY_PATHS = "/"
> > > or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> > > 
> > 
> > I would like to avoid that anti pattern of creating an overlay over
> > the complete rootfs. What are the debug case you need a full
> > overlay?
> > 
> To be very clear: This is not meant for debugging in the field!
> Never, ever.
> 
> The intention of having a root-overlay for us was to have an easy way
> to develop and extend recipes based on our existing images. That
> means,
> we wanted to have the possibility to install packages with apt, or
> copy 
> additional binaries, services, configs on the device without having
> to 
> rebuild our image all the time, still maintaining to have a working
> environment
> as close as possible to the production image. 
> That enables us to develop features in a debian native environment,
> still being
> close to the features present in the final target image. 
> After being confident, the steps done and changes reflected on the
> overlay can 
> be transferred fairly easy to a recipe and end up in a production
> image (image without the overlay).

For that it should still be sufficient to overlay /usr only as that one
holds the read-only OS parts for any halfway modern OS that has
usrmerge. /etc already has a separate overlay.

What we came up with is to trigger the /usr overlay based on presence
of a file in /etc. The backing store dir in /var is then versioned
based on the revision/snapshot of the OS in /usr (eg BUILD_ID in os-
release). That way the overlay is only used for a given build of the
OS. A new build of the image can then be deployed without old overlays
interfering.

IIRC recently systemd-sysext also gained some options to provide rw
overlays. That might be the way going forward.


cu
Ludwig
diff mbox series

Patch

diff --git a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
index 71cc63c..a17e4a4 100644
--- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
+++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
@@ -6,14 +6,19 @@ 
 # Authors:
 #  Jan Kiszka <jan.kiszka@siemens.com>
 #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#  Alexander Heinisch <alexander.heinisch@siemens.com>
 #
 
 ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
 ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
 
-root_mount_storage=${rootmnt}${ovl_storage_path}
+root_mount_storage=${ovl_storage_path}
 storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
 
+# Setup mountpoint for persistent storage
+# Either, it already got setup by a previous initramfs script,
+# then we have to move it from $rootmnt to $storage_mount_point or it has not
+# been setup yet, then check it and mount it under $storage_mount_point
 if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
 	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
@@ -21,6 +26,8 @@  if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 
 	partition_fstype=$(get_fstype "${ovl_partition_device}")
 
+	# FS checks should be moved to a separate hook / or at least initramfs lib
+	# so generic functionality can be reused here!
 	case $partition_fstype in
 	ext*)
 		fsck_ret=0
@@ -33,32 +40,97 @@  if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 		fi
 		;;
 	esac
+
+	# Mount overlay device to initramfs - will be moved to rootfs at the end
+	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
 	if ! mount -t ${partition_fstype} \
 		 -o ${ovl_mount_option} \
 		 ${ovl_partition_device} \
-		 ${rootmnt}${storage_mount_point}; then
+		 ${storage_mount_point}; then
 		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
 	fi
+else
+	# If the storage device is already mounted on the final rootfs
+	# move it to the initramfs. Otherwise, we cannot move back the
+	# storage mount to the resulting rootfs in the end.
+
+	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
+	mount -n -o move ${rootmnt}${storage_mount_point} ${storage_mount_point}
 fi
 
+[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
+
 for ovl_lower_dir in ${ovl_lower_dirs}; do
-	# remove the first '/' and replace all '/' with '_'
-	# on variable $ovl_lower_dir
-	work_dir_name=$(awk -v var=$ovl_lower_dir \
-		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
+	root_moved=false
+
+	if [ $ovl_lower_dir != "/" ]; then
+		log_begin_msg "Mounting overlay $ovl_lower_dir"
+
+		# remove the first '/' and replace all '/' with '_'
+		# on variable $ovl_lower_dir
+		work_dir_name=$(awk -v var=$ovl_lower_dir \
+			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
+
+		lower_dir=${rootmnt}${ovl_lower_dir}
+		upper_dir=${root_mount_storage}${ovl_lower_dir}
+		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
+		mount_dir=${lower_dir}
+	else
+		log_begin_msg "Mounting root overlay"
+
+		# When overlaying root we need to move the mountpoint
+		# from ${rootmnt} to a dedicated mountpoint inside
+		# the storage dir to prevent a "recursion" within
+		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
+		# While this is not a limitation of overlay itself
+		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
+		# outside the tree of new root which panics when switching root
+
+		# create lower dir to move ro-rootfs to
+		lower_dir=${root_mount_storage}/lower-root
+		mkdir -p ${lower_dir}
+
+		# make the readonly root available
+		mount -n -o move ${rootmnt} ${lower_dir}
+		root_moved=true
 
-	lower_dir=${rootmnt}${ovl_lower_dir}
-	upper_dir=${root_mount_storage}${ovl_lower_dir}
-	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
+		upper_dir=${root_mount_storage}/root
+		work_dir=${root_mount_storage}/.root-atomic
+
+		mount_dir=${rootmnt}
+	fi
 
 	mkdir -p ${upper_dir}
 	mkdir -p ${work_dir}
 
+	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
 	if ! mount -t overlay \
 		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
-		   overlay ${lower_dir}; then
+		   overlay ${mount_dir}; then
 		panic "Can't mount overlay for '$ovl_lower_dir' !"
 	fi
+
+	if ${root_moved}; then
+		# Loop over all mountpoints already setup before we established the root overlay
+		# and move them to the resulting overlayed rootfs
+		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
+		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
+			while read -r old_mountpoint
+			do
+				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
+
+				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
+				mount -n -o move ${old_mountpoint} ${new_mountpoint}
+			done
+	fi
+
+	log_end_msg
 done
 
+# Finally, move the overlay workdir from initramfs to the final rootfs
+[ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
+mount -n -o move ${storage_mount_point} ${rootmnt}${storage_mount_point}
+
+log_success_msg "Overlays setup successfully"
+
 exit 0
diff --git a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
index ec7f85b..af610e5 100644
--- a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
+++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
@@ -45,7 +45,7 @@  TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
 DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
 
 HOOK_ADD_MODULES = "overlay"
-HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
+HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
 
 SCRIPT_PREREQ="crypt"