diff mbox series

scripts: Fix incremental build header re-generation

Message ID 1614608540-1025-1-git-send-email-jshriram@codeaurora.org (mailing list archive)
State New
Headers show
Series scripts: Fix incremental build header re-generation | expand

Commit Message

Jeevan Shriram March 1, 2021, 2:22 p.m. UTC
compile.h and autoconf.h are ignored when checking headers sha as they
are always re-generated for every kernel compilation. However,
these two headers are packaged into kheaders tar. During incremental
compilation of kernel, kheaders tar file is always generated and re-packaged
irrespective of the changes in headers.

Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
---
 kernel/gen_kheaders.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Comments

Masahiro Yamada March 2, 2021, 3:36 a.m. UTC | #1
On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@codeaurora.org> wrote:
>
> compile.h and autoconf.h are ignored when checking headers sha as they
> are always re-generated for every kernel compilation. However,
> these two headers are packaged into kheaders tar. During incremental
> compilation of kernel, kheaders tar file is always generated and re-packaged
> irrespective of the changes in headers.


I do not see this problem.
Could you describe the steps to reproduce it, please?







> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
> Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
> ---
>  kernel/gen_kheaders.sh | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index c1510f0..5499f72 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
>  if [ -f kernel/kheaders.md5 ] &&
>         [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
>                 exit
>  fi
>
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>
Jeevan Shriram March 18, 2021, 2:54 a.m. UTC | #2
On 3/1/2021 7:36 PM, Masahiro Yamada wrote:
> On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@codeaurora.org> wrote:
>> compile.h and autoconf.h are ignored when checking headers sha as they
>> are always re-generated for every kernel compilation. However,
>> these two headers are packaged into kheaders tar. During incremental
>> compilation of kernel, kheaders tar file is always generated and re-packaged
>> irrespective of the changes in headers.
>
> I do not see this problem.
> Could you describe the steps to reproduce it, please?
>
Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes.
I have added following log in gen_kheaders.sh script for confirming the hash differences.

diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index b7425a0..ee542a0 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
  # Any changes to this script will also cause a rebuild of the archive.
  this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
+
+echo "Old Tar file $tarfile_md5"
+echo "New Tar file hash $this_file_md5"
+
  if [ -f kernel/kheaders.md5 ] &&
         [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
         [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&

log output :
89306 19:29:02.109961   CHK     kernel/kheaders_data.tar.xz
89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e
89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d

>> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
>> Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
>> ---
>>   kernel/gen_kheaders.sh | 3 +--
>>   1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
>> index c1510f0..5499f72 100755
>> --- a/kernel/gen_kheaders.sh
>> +++ b/kernel/gen_kheaders.sh
>> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
>>   if [ -f kernel/kheaders.md5 ] &&
>>          [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
>> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
>> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
>> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
>>                  exit
>>   fi
>>
>> --
>> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
>> a Linux Foundation Collaborative Project
>>
>
Masahiro Yamada March 24, 2021, 6:27 a.m. UTC | #3
On Thu, Mar 18, 2021 at 11:55 AM Jeevan Shriram <jshriram@codeaurora.org> wrote:
>
>
> On 3/1/2021 7:36 PM, Masahiro Yamada wrote:
> > On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@codeaurora.org> wrote:
> >> compile.h and autoconf.h are ignored when checking headers sha as they
> >> are always re-generated for every kernel compilation. However,
> >> these two headers are packaged into kheaders tar. During incremental
> >> compilation of kernel, kheaders tar file is always generated and re-packaged
> >> irrespective of the changes in headers.
> >
> > I do not see this problem.
> > Could you describe the steps to reproduce it, please?
> >
> Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes.
> I have added following log in gen_kheaders.sh script for confirming the hash differences.


Of course they are different because you are comparing
the hashes of two different files.



>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index b7425a0..ee542a0 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
>   # Any changes to this script will also cause a rebuild of the archive.
>   this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
> +
> +echo "Old Tar file $tarfile_md5"


This is the hash of kernel/kheaders_data.tar.xz


> +echo "New Tar file hash $this_file_md5"


This is the hash of kernel/gen_kheaders.sh




> +
>   if [ -f kernel/kheaders.md5 ] &&
>          [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
>          [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&
>
> log output :
> 89306 19:29:02.109961   CHK     kernel/kheaders_data.tar.xz
> 89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e
> 89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d


I do not understand what your claim is.



CHK     kernel/kheaders_data.tar.xz

is displayed when it is checking the hash in order to
determine if the tarball should be updated.


GEN     kernel/kheaders_data.tar.xz

is displayed when the tarball is really updated.





[Incremental build with no change]


masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
Kernel: arch/x86/boot/bzImage is ready  (#2)


[Touch one header and rebuild]

masahiro@oscar:~/ref/linux$ touch include/uapi/drm/exynos_drm.h
masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
  GEN     kernel/kheaders_data.tar.xz
  CC      kernel/kheaders.o
  AR      kernel/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  MODPOST Module.symvers
  CC      arch/x86/boot/version.o
  VOFFSET arch/x86/boot/compressed/../voffset.h
  OBJCOPY arch/x86/boot/compressed/vmlinux.bin
  RELOCS  arch/x86/boot/compressed/vmlinux.relocs
  CC      arch/x86/boot/compressed/kaslr.o
  CC      arch/x86/boot/compressed/misc.o
  GZIP    arch/x86/boot/compressed/vmlinux.bin.gz
  MKPIGGY arch/x86/boot/compressed/piggy.S
  AS      arch/x86/boot/compressed/piggy.o
  LD      arch/x86/boot/compressed/vmlinux
  ZOFFSET arch/x86/boot/zoffset.h
  OBJCOPY arch/x86/boot/vmlinux.bin
  AS      arch/x86/boot/header.o
  LD      arch/x86/boot/setup.elf
  OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#3)







> >> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
> >> Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
> >> ---
> >>   kernel/gen_kheaders.sh | 3 +--
> >>   1 file changed, 1 insertion(+), 2 deletions(-)
> >>
> >> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> >> index c1510f0..5499f72 100755
> >> --- a/kernel/gen_kheaders.sh
> >> +++ b/kernel/gen_kheaders.sh
> >> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
> >>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
> >>   if [ -f kernel/kheaders.md5 ] &&
> >>          [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
> >> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
> >> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
> >> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
> >>                  exit
> >>   fi
> >>
> >> --
> >> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> >> a Linux Foundation Collaborative Project
> >>
> >
--
Best Regards
Masahiro Yamada
diff mbox series

Patch

diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index c1510f0..5499f72 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -51,8 +51,7 @@  this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
 if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
 if [ -f kernel/kheaders.md5 ] &&
 	[ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
-	[ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
-	[ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
+	[ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
 		exit
 fi