| Message ID | 5728b6ca-8c6a-9eb3-3926-e99cf7081414@siemens.com (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [isar-cip-core] squashfs: Calculate a smarter default SQUASHFS_MEMLIMIT | expand |
On Sat, 2023-05-27 at 18:48 +0200, Jan Kiszka wrote: > From: Jan Kiszka <jan.kiszka@siemens.com> > > This should obsolete most - if not all - manual tuning in constrained > setups. > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > --- > classes/squashfs.bbclass | 14 ++++++++++++-- > 1 file changed, 12 insertions(+), 2 deletions(-) > > diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass > index 1f2a7595..b39be0cd 100644 > --- a/classes/squashfs.bbclass > +++ b/classes/squashfs.bbclass > @@ -9,6 +9,17 @@ > # SPDX-License-Identifier: MIT > # > > +def get_free_mem(): > + try: > + with open('/proc/meminfo') as meminfo: > + lines = meminfo.readlines() > + for line in lines: > + if line.startswith('MemAvailable:'): > + return int(line.split()[1]) * 1024 > + except FileNotFoundError: > + pass > + return 4*1024*1024*1024 # 4G > + > IMAGER_INSTALL:squashfs += "squashfs-tools" > > SQUASHFS_EXCLUDE_DIRS ?= "" > @@ -16,8 +27,7 @@ SQUASHFS_CONTENT ?= "${PP_ROOTFS}" > SQUASHFS_CREATION_ARGS ?= "" > > SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > -# default according to mksquasfs docs > -SQUASHFS_MEMLIMIT ?= "7982M" > +SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" Did you test this regarding reproducible builds? I'm almost certainly sure that this will break reproducability. Felix > SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors > ${SQUASHFS_THREADS}" > > python __anonymous() {
On 29.05.23 12:27, MOESSBAUER, Felix (T CED INW-CN) wrote: > On Sat, 2023-05-27 at 18:48 +0200, Jan Kiszka wrote: >> From: Jan Kiszka <jan.kiszka@siemens.com> >> >> This should obsolete most - if not all - manual tuning in constrained >> setups. >> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> >> --- >> classes/squashfs.bbclass | 14 ++++++++++++-- >> 1 file changed, 12 insertions(+), 2 deletions(-) >> >> diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass >> index 1f2a7595..b39be0cd 100644 >> --- a/classes/squashfs.bbclass >> +++ b/classes/squashfs.bbclass >> @@ -9,6 +9,17 @@ >> # SPDX-License-Identifier: MIT >> # >> >> +def get_free_mem(): >> + try: >> + with open('/proc/meminfo') as meminfo: >> + lines = meminfo.readlines() >> + for line in lines: >> + if line.startswith('MemAvailable:'): >> + return int(line.split()[1]) * 1024 >> + except FileNotFoundError: >> + pass >> + return 4*1024*1024*1024 # 4G >> + >> IMAGER_INSTALL:squashfs += "squashfs-tools" >> >> SQUASHFS_EXCLUDE_DIRS ?= "" >> @@ -16,8 +27,7 @@ SQUASHFS_CONTENT ?= "${PP_ROOTFS}" >> SQUASHFS_CREATION_ARGS ?= "" >> >> SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" >> -# default according to mksquasfs docs >> -SQUASHFS_MEMLIMIT ?= "7982M" >> +SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" > > Did you test this regarding reproducible builds? > I'm almost certainly sure that this will break reproducability. If it did, the current version would do as well because it already encodes the number the runner CPUs into the squashfs run. But why should the CPUs and memory size influence the output? We don't specify '-not-reproducible' here. Jan
diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass index 1f2a7595..b39be0cd 100644 --- a/classes/squashfs.bbclass +++ b/classes/squashfs.bbclass @@ -9,6 +9,17 @@ # SPDX-License-Identifier: MIT # +def get_free_mem(): + try: + with open('/proc/meminfo') as meminfo: + lines = meminfo.readlines() + for line in lines: + if line.startswith('MemAvailable:'): + return int(line.split()[1]) * 1024 + except FileNotFoundError: + pass + return 4*1024*1024*1024 # 4G + IMAGER_INSTALL:squashfs += "squashfs-tools" SQUASHFS_EXCLUDE_DIRS ?= "" @@ -16,8 +27,7 @@ SQUASHFS_CONTENT ?= "${PP_ROOTFS}" SQUASHFS_CREATION_ARGS ?= "" SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" -# default according to mksquasfs docs -SQUASHFS_MEMLIMIT ?= "7982M" +SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}" python __anonymous() {