| Message ID | 1619197235-13860-1-git-send-email-eberman@codeaurora.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [RESEND,v2] Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed | expand |
On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <eberman@codeaurora.org> wrote: Sorry for the delay. This patch is over-engineering. I will apply this. https://patchwork.kernel.org/project/linux-kbuild/patch/20210425062407.1183801-5-masahiroy@kernel.org/ The 'cmp' command is not expensive. md5sum is unneeded. > If you update the timestamp of KCONFIG_CONFIG without actually changing > anything, config_data.gz is re-generated and causes vmlinux to re-link. > When Link Time Optimization is enabled, unnecessary re-linking of > vmlinux is highly desirable since it adds several minutes to build time. > > Avoid touching config_data.gz by using a script to compare the existing > config_data.gz, KCONFIG_CONFIG, or script itself to update only if any > is mismatched. The script follows gen_kheaders.sh approach for > determing in update is needed. The script intentionally avoids > re-compressing KCONFIG_CONFIG. > > The .config can be touched, for instance, by a build script which > installs the default defconfig and then applies a defconfig fragment on > top. > > For a simple example on my x86 machine, I modified x86 default defconfig to set > CONFIG_IKCONFIG=y and run: > make -j50 defconfig tiny.config vmlinux > make -j50 defconfig tiny.config vmlinux > With this patch, vmlinux is not re-built as a result of config_data.gz > change. > > Changes in v2: > - Use md5 checksum to compare .config instead of gzip'ing again > > Signed-off-by: Elliot Berman <eberman@codeaurora.org> > --- > kernel/.gitignore | 1 + > kernel/Makefile | 4 +++- > kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ > 3 files changed, 35 insertions(+), 1 deletion(-) > create mode 100755 kernel/gen_config_data.sh > > diff --git a/kernel/.gitignore b/kernel/.gitignore > index 78701ea..a191136 100644 > --- a/kernel/.gitignore > +++ b/kernel/.gitignore > @@ -1,4 +1,5 @@ > # SPDX-License-Identifier: GPL-2.0-only > +config_data.gz.md5 > kheaders.md5 > timeconst.h > hz.bc > diff --git a/kernel/Makefile b/kernel/Makefile > index 320f1f3..0784bf3d 100644 > --- a/kernel/Makefile > +++ b/kernel/Makefile > @@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o > $(obj)/configs.o: $(obj)/config_data.gz > > targets += config_data.gz > +quiet_cmd_genicfg = CHK $(obj)/config_data.gz > + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< > $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE > - $(call if_changed,gzip) > + $(call cmd,genicfg) > > $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz > > diff --git a/kernel/gen_config_data.sh b/kernel/gen_config_data.sh > new file mode 100755 > index 00000000..e9ff193 > --- /dev/null > +++ b/kernel/gen_config_data.sh > @@ -0,0 +1,31 @@ > +#!/bin/sh > +# SPDX-License-Identifier: GPL-2.0 > + > +# This script generates a compressed version of .config, if its checksum has changed > +set -e > + > +this_file="$(readlink -f "$0")" > +outfile=$1 > +infile=$2 > + > +config_md5="$(md5sum $infile | cut -d ' ' -f1)" > +# Any changes to this script will also cause a rebuild of config_data. > +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" > +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi > + > +if [ -f $outfile.md5 ] && > + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && > + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && > + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then > + exit > +fi > + > +if [ "${quiet}" != "silent_" ]; then > + echo " GEN $outfile" > +fi > + > +${KGZIP} -c -n -f -9 $infile > $outfile > + > +echo "$config_md5" > $outfile.md5 > +echo "$this_file_md5" >> $outfile.md5 > +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5 > -- > The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, > a Linux Foundation Collaborative Project >
On 4/24/2021 11:50 PM, Masahiro Yamada wrote: > On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <eberman@codeaurora.org> wrote: > > > Sorry for the delay. No problem! > > This patch is over-engineering. I thought so, too. > > I will apply this. > https://patchwork.kernel.org/project/linux-kbuild/patch/20210425062407.1183801-5-masahiroy@kernel.org/ > > The 'cmp' command is not expensive. > md5sum is unneeded. Thanks! > >> If you update the timestamp of KCONFIG_CONFIG without actually changing >> anything, config_data.gz is re-generated and causes vmlinux to re-link. >> When Link Time Optimization is enabled, unnecessary re-linking of >> vmlinux is highly desirable since it adds several minutes to build time. >> >> Avoid touching config_data.gz by using a script to compare the existing >> config_data.gz, KCONFIG_CONFIG, or script itself to update only if any >> is mismatched. The script follows gen_kheaders.sh approach for >> determing in update is needed. The script intentionally avoids >> re-compressing KCONFIG_CONFIG. >> >> The .config can be touched, for instance, by a build script which >> installs the default defconfig and then applies a defconfig fragment on >> top. >> >> For a simple example on my x86 machine, I modified x86 default defconfig to set >> CONFIG_IKCONFIG=y and run: >> make -j50 defconfig tiny.config vmlinux >> make -j50 defconfig tiny.config vmlinux >> With this patch, vmlinux is not re-built as a result of config_data.gz >> change. >> >> Changes in v2: >> - Use md5 checksum to compare .config instead of gzip'ing again >> >> Signed-off-by: Elliot Berman <eberman@codeaurora.org> >> --- >> kernel/.gitignore | 1 + >> kernel/Makefile | 4 +++- >> kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ >> 3 files changed, 35 insertions(+), 1 deletion(-) >> create mode 100755 kernel/gen_config_data.sh >> >> diff --git a/kernel/.gitignore b/kernel/.gitignore >> index 78701ea..a191136 100644 >> --- a/kernel/.gitignore >> +++ b/kernel/.gitignore >> @@ -1,4 +1,5 @@ >> # SPDX-License-Identifier: GPL-2.0-only >> +config_data.gz.md5 >> kheaders.md5 >> timeconst.h >> hz.bc >> diff --git a/kernel/Makefile b/kernel/Makefile >> index 320f1f3..0784bf3d 100644 >> --- a/kernel/Makefile >> +++ b/kernel/Makefile >> @@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o >> $(obj)/configs.o: $(obj)/config_data.gz >> >> targets += config_data.gz >> +quiet_cmd_genicfg = CHK $(obj)/config_data.gz >> + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< >> $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE >> - $(call if_changed,gzip) >> + $(call cmd,genicfg) >> >> $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz >> >> diff --git a/kernel/gen_config_data.sh b/kernel/gen_config_data.sh >> new file mode 100755 >> index 00000000..e9ff193 >> --- /dev/null >> +++ b/kernel/gen_config_data.sh >> @@ -0,0 +1,31 @@ >> +#!/bin/sh >> +# SPDX-License-Identifier: GPL-2.0 >> + >> +# This script generates a compressed version of .config, if its checksum has changed >> +set -e >> + >> +this_file="$(readlink -f "$0")" >> +outfile=$1 >> +infile=$2 >> + >> +config_md5="$(md5sum $infile | cut -d ' ' -f1)" >> +# Any changes to this script will also cause a rebuild of config_data. >> +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" >> +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi >> + >> +if [ -f $outfile.md5 ] && >> + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && >> + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && >> + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then >> + exit >> +fi >> + >> +if [ "${quiet}" != "silent_" ]; then >> + echo " GEN $outfile" >> +fi >> + >> +${KGZIP} -c -n -f -9 $infile > $outfile >> + >> +echo "$config_md5" > $outfile.md5 >> +echo "$this_file_md5" >> $outfile.md5 >> +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5 >> -- >> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, >> a Linux Foundation Collaborative Project >> > >
diff --git a/kernel/.gitignore b/kernel/.gitignore index 78701ea..a191136 100644 --- a/kernel/.gitignore +++ b/kernel/.gitignore @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +config_data.gz.md5 kheaders.md5 timeconst.h hz.bc diff --git a/kernel/Makefile b/kernel/Makefile index 320f1f3..0784bf3d 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz +quiet_cmd_genicfg = CHK $(obj)/config_data.gz + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call cmd,genicfg) $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz diff --git a/kernel/gen_config_data.sh b/kernel/gen_config_data.sh new file mode 100755 index 00000000..e9ff193 --- /dev/null +++ b/kernel/gen_config_data.sh @@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 + +# This script generates a compressed version of .config, if its checksum has changed +set -e + +this_file="$(readlink -f "$0")" +outfile=$1 +infile=$2 + +config_md5="$(md5sum $infile | cut -d ' ' -f1)" +# Any changes to this script will also cause a rebuild of config_data. +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi + +if [ -f $outfile.md5 ] && + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then + exit +fi + +if [ "${quiet}" != "silent_" ]; then + echo " GEN $outfile" +fi + +${KGZIP} -c -n -f -9 $infile > $outfile + +echo "$config_md5" > $outfile.md5 +echo "$this_file_md5" >> $outfile.md5 +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time. Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG. The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top. For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run: make -j50 defconfig tiny.config vmlinux make -j50 defconfig tiny.config vmlinux With this patch, vmlinux is not re-built as a result of config_data.gz change. Changes in v2: - Use md5 checksum to compare .config instead of gzip'ing again Signed-off-by: Elliot Berman <eberman@codeaurora.org> --- kernel/.gitignore | 1 + kernel/Makefile | 4 +++- kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100755 kernel/gen_config_data.sh