| Message ID | 20230414092408.679342-1-felix.moessbauer@siemens.com (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [isar-cip-core,v2] add support to limit resource consumption of squashfs creation | expand |
On Fri, 2023-04-14 at 11:33 +0200, Jan Kiszka wrote: > On 14.04.23 11:24, Felix Moessbauer wrote: > > This patch makes it possible to control the number of CPUs and the > > memory usage when generating a squashfs. The variable names used > > follow the imagetype variable scheme. > > > > Especially when compressing a squashfs internally with cpu and > > memory > > heavy compressors like xz, we inherit the issues we had with the > > image > > compression in the past. On CI systems, where many tasks might run > > in > > parallel, this often led to OOMs and poor performance. By providing > > these options similiar to the other imagetype options, we give CI > > administrators an easy way to control the memory consumption, > > independent of the other options passed to mksquashfs. > > If this is targeting CI tunings, we should also export the related > variables via kas to allow pickup from the environment, not? We don't do that for the other variables like XZ_MEMORY either. IMHO we should also not do this, as usually no further customizations are required. But if they are required, people can just add the pass- through in the kas file on a per use-case basis (e.g. without value, so the corresponding env var is picked up). The more important thing is, that we provide clear interfaces. And exactly that is what this patch does. Felix > > Jan > > > > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > > --- > > Changes since v1: > > > > - add reasoning in the commit message > > - no functional changes > > > > classes/squashfs.bbclass | 9 ++++++++- > > 1 file changed, 8 insertions(+), 1 deletion(-) > > > > diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass > > index a06c501..b14768c 100644 > > --- a/classes/squashfs.bbclass > > +++ b/classes/squashfs.bbclass > > @@ -15,6 +15,13 @@ SQUASHFS_EXCLUDE_DIRS ?= "" > > SQUASHFS_CONTENT ?= "${PP_ROOTFS}" > > SQUASHFS_CREATION_ARGS ?= "" > > > > +SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > > +SQUASHFS_THREADS[vardepvalue] = "1" > > +# default according to mksquasfs docs > > +SQUASHFS_MEMLIMIT ?= "7982M" > > +SQUASHFS_DEFAULTS ?= "-mem ${SQUASHFS_MEMLIMIT} -processors > > ${SQUASHFS_THREADS}" > > +SQUASHFS_DEFAULTS[vardepsexclude] += "SQUASHFS_MEMLIMIT > > SQUASHFS_THREADS" > > + > > python __anonymous() { > > exclude_directories = > > d.getVar('SQUASHFS_EXCLUDE_DIRS').split() > > if len(exclude_directories) == 0: > > @@ -31,5 +38,5 @@ IMAGE_CMD:squashfs[depends] = > > "${PN}:do_transform_template" > > IMAGE_CMD:squashfs() { > > ${SUDO_CHROOT} /bin/mksquashfs \ > > '${SQUASHFS_CONTENT}' '${IMAGE_FILE_CHROOT}' \ > > - -noappend ${SQUASHFS_CREATION_ARGS} > > + -noappend ${SQUASHFS_DEFAULTS} ${SQUASHFS_CREATION_ARGS} > > } >
On 14.04.23 14:55, Moessbauer, Felix (T CED INW-CN) wrote: > On Fri, 2023-04-14 at 11:33 +0200, Jan Kiszka wrote: >> On 14.04.23 11:24, Felix Moessbauer wrote: >>> This patch makes it possible to control the number of CPUs and the >>> memory usage when generating a squashfs. The variable names used >>> follow the imagetype variable scheme. >>> >>> Especially when compressing a squashfs internally with cpu and >>> memory >>> heavy compressors like xz, we inherit the issues we had with the >>> image >>> compression in the past. On CI systems, where many tasks might run >>> in >>> parallel, this often led to OOMs and poor performance. By providing >>> these options similiar to the other imagetype options, we give CI >>> administrators an easy way to control the memory consumption, >>> independent of the other options passed to mksquashfs. >> >> If this is targeting CI tunings, we should also export the related >> variables via kas to allow pickup from the environment, not? > > We don't do that for the other variables like XZ_MEMORY either. > IMHO we should also not do this, as usually no further customizations > are required. But if they are required, people can just add the pass- > through in the kas file on a per use-case basis (e.g. without value, so > the corresponding env var is picked up). There are several aspects: XZ_* are variables of Isar upstream, and Isar upstream has no kas support yet - isar-cip-core does have. If you wanted to run isar-cip-core in CI and should run into the need to customize those tunables, you would have to patch it. I agree, it's more likely that your downstream layer will have that need and could then perfectly add the tuning vars downstream to kas. But then when looking at Isar, the other question would be if those vars should not eventually be part of Isar's bitbake.conf as well. This class here is downstream (/wrt Isar) so far, but maybe now another trigger to change that. > > The more important thing is, that we provide clear interfaces. > And exactly that is what this patch does. I don't disagree on that, and I'm fine with moving forward as-is for now. Applied to next, thanks. Jan
diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass index a06c501..b14768c 100644 --- a/classes/squashfs.bbclass +++ b/classes/squashfs.bbclass @@ -15,6 +15,13 @@ SQUASHFS_EXCLUDE_DIRS ?= "" SQUASHFS_CONTENT ?= "${PP_ROOTFS}" SQUASHFS_CREATION_ARGS ?= "" +SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" +SQUASHFS_THREADS[vardepvalue] = "1" +# default according to mksquasfs docs +SQUASHFS_MEMLIMIT ?= "7982M" +SQUASHFS_DEFAULTS ?= "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}" +SQUASHFS_DEFAULTS[vardepsexclude] += "SQUASHFS_MEMLIMIT SQUASHFS_THREADS" + python __anonymous() { exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split() if len(exclude_directories) == 0: @@ -31,5 +38,5 @@ IMAGE_CMD:squashfs[depends] = "${PN}:do_transform_template" IMAGE_CMD:squashfs() { ${SUDO_CHROOT} /bin/mksquashfs \ '${SQUASHFS_CONTENT}' '${IMAGE_FILE_CHROOT}' \ - -noappend ${SQUASHFS_CREATION_ARGS} + -noappend ${SQUASHFS_DEFAULTS} ${SQUASHFS_CREATION_ARGS} }
This patch makes it possible to control the number of CPUs and the memory usage when generating a squashfs. The variable names used follow the imagetype variable scheme. Especially when compressing a squashfs internally with cpu and memory heavy compressors like xz, we inherit the issues we had with the image compression in the past. On CI systems, where many tasks might run in parallel, this often led to OOMs and poor performance. By providing these options similiar to the other imagetype options, we give CI administrators an easy way to control the memory consumption, independent of the other options passed to mksquashfs. Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- Changes since v1: - add reasoning in the commit message - no functional changes classes/squashfs.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)