| Message ID | 20220602212234.344394-1-Jason@zx2c4.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | ARM: initialize jump labels before setup_machine_fdt() | expand |
On Thu, 2 Jun 2022 at 23:22, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Stephen reported that a static key warning splat appears during early > boot on arm64 systems that credit randomness from device trees that > contain an "rng-seed" property, because setup_machine_fdt() is called > before jump_label_init() during setup_arch(), which was fixed by > 73e2d827a501 ("arm64: Initialize jump labels before > setup_machine_fdt()"). > > Upon cursory inspection, the same basic issue appears to apply to arm32 > as well. In this case, we reorder setup_arch() to do things in the same > order as is now the case on arm64. > > Reported-by: Stephen Boyd <swboyd@chromium.org> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Ard Biesheuvel <ardb@kernel.org> > Cc: stable@vger.kernel.org > Fixes: f5bda35fba61 ("random: use static branch for crng_ready()") Wouldn't it be better to defer the static_branch_enable(&crng_is_ready) call to later in the boot (e.g., using an initcall()), rather than going around 'fixing' fragile, working early boot code across multiple architectures? > Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> > --- > arch/arm/kernel/setup.c | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c > index 1e8a50a97edf..ef40d9f5d5a7 100644 > --- a/arch/arm/kernel/setup.c > +++ b/arch/arm/kernel/setup.c > @@ -1097,10 +1097,15 @@ void __init setup_arch(char **cmdline_p) > const struct machine_desc *mdesc = NULL; > void *atags_vaddr = NULL; > > + setup_initial_init_mm(_text, _etext, _edata, _end); > + setup_processor(); > + early_fixmap_init(); > + early_ioremap_init(); > + jump_label_init(); > + Is it really necessary to reorder all these calls? What does jump_label_init() actually need? If this is related to the code patching, I wonder whether it wouldn't be better not to rewrite all the NOPs (this is a x86-ism as every new x86 uarch appears to have a better [faster?] NOP than the previous one) The issue with changes like these is that we might end up with bug report in ~3 months' time that 'obscure platform X no longer boots or produces any output'. In the best case, we'll have a bisect report identifying this patch, but we won't be able to simply revert it as it would reintroduce this issue into a kernel that is now stable. > if (__atags_pointer) > atags_vaddr = FDT_VIRT_BASE(__atags_pointer); > > - setup_processor(); > if (atags_vaddr) { > mdesc = setup_machine_fdt(atags_vaddr); > if (mdesc) > @@ -1125,15 +1130,10 @@ void __init setup_arch(char **cmdline_p) > if (mdesc->reboot_mode != REBOOT_HARD) > reboot_mode = mdesc->reboot_mode; > > - setup_initial_init_mm(_text, _etext, _edata, _end); > - > /* populate cmd_line too for later use, preserving boot_command_line */ > strlcpy(cmd_line, boot_command_line, COMMAND_LINE_SIZE); > *cmdline_p = cmd_line; > > - early_fixmap_init(); > - early_ioremap_init(); > - > parse_early_param(); > > #ifdef CONFIG_MMU > -- > 2.35.1 >
Hi Ard, On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: > On Thu, 2 Jun 2022 at 23:22, Jason A. Donenfeld <Jason@zx2c4.com> wrote: >> >> Stephen reported that a static key warning splat appears during early >> boot on arm64 systems that credit randomness from device trees that >> contain an "rng-seed" property, because setup_machine_fdt() is called >> before jump_label_init() during setup_arch(), which was fixed by >> 73e2d827a501 ("arm64: Initialize jump labels before >> setup_machine_fdt()"). >> >> Upon cursory inspection, the same basic issue appears to apply to arm32 >> as well. In this case, we reorder setup_arch() to do things in the same >> order as is now the case on arm64. >> >> Reported-by: Stephen Boyd <swboyd@chromium.org> >> Cc: Catalin Marinas <catalin.marinas@arm.com> >> Cc: Ard Biesheuvel <ardb@kernel.org> >> Cc: stable@vger.kernel.org >> Fixes: f5bda35fba61 ("random: use static branch for crng_ready()") > > Wouldn't it be better to defer the > static_branch_enable(&crng_is_ready) call to later in the boot (e.g., > using an initcall()), rather than going around 'fixing' fragile, > working early boot code across multiple architectures? Yes, maybe. It's just more book keeping that's potentially unnecessary, which would be nice to avoid. I wrote a patch for this before, but it wasn't beautiful. And Catalin got a pretty easy arm64 patch queued up sufficiently fast that I figured this was better. > >> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> >> --- >> arch/arm/kernel/setup.c | 12 ++++++------ >> 1 file changed, 6 insertions(+), 6 deletions(-) >> >> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c >> index 1e8a50a97edf..ef40d9f5d5a7 100644 >> --- a/arch/arm/kernel/setup.c >> +++ b/arch/arm/kernel/setup.c >> @@ -1097,10 +1097,15 @@ void __init setup_arch(char **cmdline_p) >> const struct machine_desc *mdesc = NULL; >> void *atags_vaddr = NULL; >> >> + setup_initial_init_mm(_text, _etext, _edata, _end); >> + setup_processor(); >> + early_fixmap_init(); >> + early_ioremap_init(); >> + jump_label_init(); >> + > > Is it really necessary to reorder all these calls? What does > jump_label_init() actually need? I'm not quite sure, but it matched how arm64 does things now. Was hoping somebody with deep arm32 knowledge (e.g. you or rmk) would be able to eyeball that to confirm. Jason
(+ Greg) On Fri, 3 Jun 2022 at 09:37, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Hi Ard, > > On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: > > On Thu, 2 Jun 2022 at 23:22, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > >> > >> Stephen reported that a static key warning splat appears during early > >> boot on arm64 systems that credit randomness from device trees that > >> contain an "rng-seed" property, because setup_machine_fdt() is called > >> before jump_label_init() during setup_arch(), which was fixed by > >> 73e2d827a501 ("arm64: Initialize jump labels before > >> setup_machine_fdt()"). > >> > >> Upon cursory inspection, the same basic issue appears to apply to arm32 > >> as well. In this case, we reorder setup_arch() to do things in the same > >> order as is now the case on arm64. > >> > >> Reported-by: Stephen Boyd <swboyd@chromium.org> > >> Cc: Catalin Marinas <catalin.marinas@arm.com> > >> Cc: Ard Biesheuvel <ardb@kernel.org> > >> Cc: stable@vger.kernel.org > >> Fixes: f5bda35fba61 ("random: use static branch for crng_ready()") > > > > Wouldn't it be better to defer the > > static_branch_enable(&crng_is_ready) call to later in the boot (e.g., > > using an initcall()), rather than going around 'fixing' fragile, > > working early boot code across multiple architectures? > > Yes, maybe. It's just more book keeping that's potentially > unnecessary, which would be nice to avoid. I wrote a patch for this > before, but it wasn't beautiful. And Catalin got a pretty easy arm64 > patch queued up sufficiently fast that I figured this was better. > The problem is that your original patch was already backported as far back as 5.10, and so this fix will need to be as well. Playing with the code that runs before the call to setup_machine_fdt() is risky because it implies that issues that are introduced are likely to limit the ability of the system to generate diagnostic output of any kind, given that the device tree is what describes the topology of the system to the kernel. Before that, there is no serial or graphical console, and the only way to figure out what goes on is to connect a JTAG debugger and single step through the code or dump the contents of __log_buf[]. I like the /dev/random work you have been doing but as you know, I was skeptical about the need to backport all of that work to -stable, and it appears my skepticism may have been justified. The patch in question is an unquantified performance optimization, which means it does not meet the stable-kernel-rules.rst criteria, but it was backported nonetheless. Now, we are in a situation where we must refactor very early boot code to address a regression introduced by that backport. > > > >> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> > >> --- > >> arch/arm/kernel/setup.c | 12 ++++++------ > >> 1 file changed, 6 insertions(+), 6 deletions(-) > >> > >> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c > >> index 1e8a50a97edf..ef40d9f5d5a7 100644 > >> --- a/arch/arm/kernel/setup.c > >> +++ b/arch/arm/kernel/setup.c > >> @@ -1097,10 +1097,15 @@ void __init setup_arch(char **cmdline_p) > >> const struct machine_desc *mdesc = NULL; > >> void *atags_vaddr = NULL; > >> > >> + setup_initial_init_mm(_text, _etext, _edata, _end); > >> + setup_processor(); > >> + early_fixmap_init(); > >> + early_ioremap_init(); > >> + jump_label_init(); > >> + > > > > Is it really necessary to reorder all these calls? What does > > jump_label_init() actually need? > > I'm not quite sure, but it matched how arm64 does things now. Was > hoping somebody with deep arm32 knowledge (e.g. you or rmk) would be > able to eyeball that to confirm. > As far as I can tell, the early patching code on ARM does not rely on the early fixmap code. Did you try just moving jump_label_init() earlier in the function? Also, how did you test this change?
On Fri, 3 Jun 2022 at 09:51, Ard Biesheuvel <ardb@kernel.org> wrote: > > (+ Greg) > > On Fri, 3 Jun 2022 at 09:37, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > Hi Ard, > > > > On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: > > > On Thu, 2 Jun 2022 at 23:22, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > >> > > >> Stephen reported that a static key warning splat appears during early > > >> boot on arm64 systems that credit randomness from device trees that > > >> contain an "rng-seed" property, because setup_machine_fdt() is called > > >> before jump_label_init() during setup_arch(), which was fixed by > > >> 73e2d827a501 ("arm64: Initialize jump labels before > > >> setup_machine_fdt()"). > > >> > > >> Upon cursory inspection, the same basic issue appears to apply to arm32 > > >> as well. In this case, we reorder setup_arch() to do things in the same > > >> order as is now the case on arm64. > > >> > > >> Reported-by: Stephen Boyd <swboyd@chromium.org> > > >> Cc: Catalin Marinas <catalin.marinas@arm.com> > > >> Cc: Ard Biesheuvel <ardb@kernel.org> > > >> Cc: stable@vger.kernel.org > > >> Fixes: f5bda35fba61 ("random: use static branch for crng_ready()") > > > > > > Wouldn't it be better to defer the > > > static_branch_enable(&crng_is_ready) call to later in the boot (e.g., > > > using an initcall()), rather than going around 'fixing' fragile, > > > working early boot code across multiple architectures? > > > > Yes, maybe. It's just more book keeping that's potentially > > unnecessary, which would be nice to avoid. I wrote a patch for this > > before, but it wasn't beautiful. And Catalin got a pretty easy arm64 > > patch queued up sufficiently fast that I figured this was better. > > > > The problem is that your original patch was already backported as far > back as 5.10, and so this fix will need to be as well. > > Playing with the code that runs before the call to setup_machine_fdt() > is risky because it implies that issues that are introduced are likely > to limit the ability of the system to generate diagnostic output of > any kind, given that the device tree is what describes the topology of > the system to the kernel. Before that, there is no serial or graphical > console, and the only way to figure out what goes on is to connect a > JTAG debugger and single step through the code or dump the contents of > __log_buf[]. > > I like the /dev/random work you have been doing but as you know, I was > skeptical about the need to backport all of that work to -stable, and > it appears my skepticism may have been justified. > > The patch in question is an unquantified performance optimization, > which means it does not meet the stable-kernel-rules.rst criteria, but > it was backported nonetheless. Now, we are in a situation where we > must refactor very early boot code to address a regression introduced > by that backport. > > > > > > >> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> > > >> --- > > >> arch/arm/kernel/setup.c | 12 ++++++------ > > >> 1 file changed, 6 insertions(+), 6 deletions(-) > > >> > > >> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c > > >> index 1e8a50a97edf..ef40d9f5d5a7 100644 > > >> --- a/arch/arm/kernel/setup.c > > >> +++ b/arch/arm/kernel/setup.c > > >> @@ -1097,10 +1097,15 @@ void __init setup_arch(char **cmdline_p) > > >> const struct machine_desc *mdesc = NULL; > > >> void *atags_vaddr = NULL; > > >> > > >> + setup_initial_init_mm(_text, _etext, _edata, _end); > > >> + setup_processor(); > > >> + early_fixmap_init(); > > >> + early_ioremap_init(); > > >> + jump_label_init(); > > >> + > > > > > > Is it really necessary to reorder all these calls? What does > > > jump_label_init() actually need? > > > > I'm not quite sure, but it matched how arm64 does things now. Was > > hoping somebody with deep arm32 knowledge (e.g. you or rmk) would be > > able to eyeball that to confirm. > > > > As far as I can tell, the early patching code on ARM does not rely on > the early fixmap code. Did you try just moving jump_label_init() > earlier in the function? > The below seems to work too: --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -1101,6 +1101,7 @@ void __init setup_arch(char **cmdline_p) atags_vaddr = FDT_VIRT_BASE(__atags_pointer); setup_processor(); + jump_label_init(); if (atags_vaddr) { mdesc = setup_machine_fdt(atags_vaddr); if (mdesc)
Hi Ard, On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: > The problem is that your original patch You remain extremely unpleasant to communicate with. Can we keep things on topic please? > As far as I can tell, the early patching code on ARM does not rely on > the early fixmap code. Did you try just moving jump_label_init() > earlier in the function? > > Also, how did you test this change? Just booting a few configs in QEMU. I don't have access to real hardware right now unfortunately. Let me give a try to just moving the jump_label_init() function alone. That'd certainly make this patch a lot more basic, which would be a good thing, and might assuage your well justified concerns that too much boot order churn will break something subtle. I was just afraid of complicated intermingling with the other stuff after I saw that arm64 did things in the other order. But maybe that's silly. I'll send a v2 if that works, and send an update here if it doesn't. Thanks for the suggestion. Jason
Hi Ard, On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: > On Fri, 3 Jun 2022 at 09:51, Ard Biesheuvel <ardb@kernel.org> wrote: >> >> (+ Greg) >> >> On Fri, 3 Jun 2022 at 09:37, Jason A. Donenfeld <Jason@zx2c4.com> wrote: >> > >> > Hi Ard, >> > >> > On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: >> > > On Thu, 2 Jun 2022 at 23:22, Jason A. Donenfeld <Jason@zx2c4.com> >> > > wrote: >> > >> >> > >> Stephen reported that a static key warning splat appears during >> > >> early >> > >> boot on arm64 systems that credit randomness from device trees that >> > >> contain an "rng-seed" property, because setup_machine_fdt() is >> > >> called >> > >> before jump_label_init() during setup_arch(), which was fixed by >> > >> 73e2d827a501 ("arm64: Initialize jump labels before >> > >> setup_machine_fdt()"). >> > >> >> > >> Upon cursory inspection, the same basic issue appears to apply to >> > >> arm32 >> > >> as well. In this case, we reorder setup_arch() to do things in the >> > >> same >> > >> order as is now the case on arm64. >> > >> >> > >> Reported-by: Stephen Boyd <swboyd@chromium.org> >> > >> Cc: Catalin Marinas <catalin.marinas@arm.com> >> > >> Cc: Ard Biesheuvel <ardb@kernel.org> >> > >> Cc: stable@vger.kernel.org >> > >> Fixes: f5bda35fba61 ("random: use static branch for crng_ready()") >> > > >> > > Wouldn't it be better to defer the >> > > static_branch_enable(&crng_is_ready) call to later in the boot (e.g., >> > > using an initcall()), rather than going around 'fixing' fragile, >> > > working early boot code across multiple architectures? >> > >> > Yes, maybe. It's just more book keeping that's potentially >> > unnecessary, which would be nice to avoid. I wrote a patch for this >> > before, but it wasn't beautiful. And Catalin got a pretty easy arm64 >> > patch queued up sufficiently fast that I figured this was better. >> > >> >> The problem is that your original patch was already backported as far >> back as 5.10, and so this fix will need to be as well. >> >> Playing with the code that runs before the call to setup_machine_fdt() >> is risky because it implies that issues that are introduced are likely >> to limit the ability of the system to generate diagnostic output of >> any kind, given that the device tree is what describes the topology of >> the system to the kernel. Before that, there is no serial or graphical >> console, and the only way to figure out what goes on is to connect a >> JTAG debugger and single step through the code or dump the contents of >> __log_buf[]. >> >> I like the /dev/random work you have been doing but as you know, I was >> skeptical about the need to backport all of that work to -stable, and >> it appears my skepticism may have been justified. >> >> The patch in question is an unquantified performance optimization, >> which means it does not meet the stable-kernel-rules.rst criteria, but >> it was backported nonetheless. Now, we are in a situation where we >> must refactor very early boot code to address a regression introduced >> by that backport. >> >> > > >> > >> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> >> > >> --- >> > >> arch/arm/kernel/setup.c | 12 ++++++------ >> > >> 1 file changed, 6 insertions(+), 6 deletions(-) >> > >> >> > >> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c >> > >> index 1e8a50a97edf..ef40d9f5d5a7 100644 >> > >> --- a/arch/arm/kernel/setup.c >> > >> +++ b/arch/arm/kernel/setup.c >> > >> @@ -1097,10 +1097,15 @@ void __init setup_arch(char **cmdline_p) >> > >> const struct machine_desc *mdesc = NULL; >> > >> void *atags_vaddr = NULL; >> > >> >> > >> + setup_initial_init_mm(_text, _etext, _edata, _end); >> > >> + setup_processor(); >> > >> + early_fixmap_init(); >> > >> + early_ioremap_init(); >> > >> + jump_label_init(); >> > >> + >> > > >> > > Is it really necessary to reorder all these calls? What does >> > > jump_label_init() actually need? >> > >> > I'm not quite sure, but it matched how arm64 does things now. Was >> > hoping somebody with deep arm32 knowledge (e.g. you or rmk) would be >> > able to eyeball that to confirm. >> > >> >> As far as I can tell, the early patching code on ARM does not rely on >> the early fixmap code. Did you try just moving jump_label_init() >> earlier in the function? >> > > The below seems to work too: > > --- a/arch/arm/kernel/setup.c > +++ b/arch/arm/kernel/setup.c > @@ -1101,6 +1101,7 @@ void __init setup_arch(char **cmdline_p) > atags_vaddr = FDT_VIRT_BASE(__atags_pointer); > > setup_processor(); > + jump_label_init(); > if (atags_vaddr) { > mdesc = setup_machine_fdt(atags_vaddr); > if (mdesc) > Oh, awesome, that's exactly what I was about to try when I got to my laptop in a few hours. Do you want to send the v2 with that, or should I do so after also testing it in a bit? Jason
On Fri, 3 Jun 2022 at 10:12, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Hi Ard, > > On 6/3/22, Ard Biesheuvel <ardb@kernel.org> wrote: > > The problem is that your original patch > > You remain extremely unpleasant to communicate with. Noted. > Can we keep > things on topic please? > Sure. Are you saying the original patch is off-topic? Isn't that the patch that caused the regression to begin with? > > As far as I can tell, the early patching code on ARM does not rely on > > the early fixmap code. Did you try just moving jump_label_init() > > earlier in the function? > > > > Also, how did you test this change? > > Just booting a few configs in QEMU. I don't have access to real > hardware right now unfortunately. > > Let me give a try to just moving the jump_label_init() function alone. > That'd certainly make this patch a lot more basic, which would be a > good thing, and might assuage your well justified concerns that too > much boot order churn will break something subtle. I was just afraid > of complicated intermingling with the other stuff after I saw that > arm64 did things in the other order. But maybe that's silly. > > I'll send a v2 if that works, and send an update here if it doesn't. > Thanks for the suggestion. > > Jason
diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index 1e8a50a97edf..ef40d9f5d5a7 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -1097,10 +1097,15 @@ void __init setup_arch(char **cmdline_p) const struct machine_desc *mdesc = NULL; void *atags_vaddr = NULL; + setup_initial_init_mm(_text, _etext, _edata, _end); + setup_processor(); + early_fixmap_init(); + early_ioremap_init(); + jump_label_init(); + if (__atags_pointer) atags_vaddr = FDT_VIRT_BASE(__atags_pointer); - setup_processor(); if (atags_vaddr) { mdesc = setup_machine_fdt(atags_vaddr); if (mdesc) @@ -1125,15 +1130,10 @@ void __init setup_arch(char **cmdline_p) if (mdesc->reboot_mode != REBOOT_HARD) reboot_mode = mdesc->reboot_mode; - setup_initial_init_mm(_text, _etext, _edata, _end); - /* populate cmd_line too for later use, preserving boot_command_line */ strlcpy(cmd_line, boot_command_line, COMMAND_LINE_SIZE); *cmdline_p = cmd_line; - early_fixmap_init(); - early_ioremap_init(); - parse_early_param(); #ifdef CONFIG_MMU
Stephen reported that a static key warning splat appears during early boot on arm64 systems that credit randomness from device trees that contain an "rng-seed" property, because setup_machine_fdt() is called before jump_label_init() during setup_arch(), which was fixed by 73e2d827a501 ("arm64: Initialize jump labels before setup_machine_fdt()"). Upon cursory inspection, the same basic issue appears to apply to arm32 as well. In this case, we reorder setup_arch() to do things in the same order as is now the case on arm64. Reported-by: Stephen Boyd <swboyd@chromium.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Ard Biesheuvel <ardb@kernel.org> Cc: stable@vger.kernel.org Fixes: f5bda35fba61 ("random: use static branch for crng_ready()") Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> --- arch/arm/kernel/setup.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)