Documentation/tracing: Mention that RESET_ATTACK_MITIGATION can clear memory

Commit Message

Steven Rostedt Sept. 26, 2024, 5:01 p.m. UTC

From: Steven Rostedt <[e-mail]>

At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
about the persistent buffer to display traces from previous boots. He
mentioned that UEFI can clear memory. In my own tests I have not seen
this. He later informed me that it requires the config option:

 CONFIG_RESET_ATTACK_MITIGATION

It appears that setting this will allow the memory to be cleared on boot
up, which will definitely clear out the trace of the previous boot.

Add this information under the trace_instance in kernel-parameters.txt
to let people know that this can cause issues.

Link: https://lore.kernel.org/all/[e-mail]/

Reported-by: Hans de Goede <[e-mail]>
Signed-off-by: Steven Rostedt (Google) <[e-mail]>
---
 Documentation/admin-guide/kernel-parameters.txt | 3 +++
 1 file changed, 3 insertions(+)

Comments

Hans de Goede Sept. 26, 2024, 5:54 p.m. UTC | #1

Hi,

On 26-Sep-24 7:01 PM, Steven Rostedt wrote:
> From: Steven Rostedt <[e-mail]>
> 
> At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
> about the persistent buffer to display traces from previous boots. He
> mentioned that UEFI can clear memory. In my own tests I have not seen
> this. He later informed me that it requires the config option:
> 
>  CONFIG_RESET_ATTACK_MITIGATION
> 
> It appears that setting this will allow the memory to be cleared on boot
> up, which will definitely clear out the trace of the previous boot.
> 
> Add this information under the trace_instance in kernel-parameters.txt
> to let people know that this can cause issues.
> 
> Link: https://lore.kernel.org/all/[e-mail]/
> 
> Reported-by: Hans de Goede <[e-mail]>
> Signed-off-by: Steven Rostedt (Google) <[e-mail]>

Thanks, patch looks good to me:

Reviewed-by: Hans de Goede <[e-mail]>

Regards,

Hans



> ---
>  Documentation/admin-guide/kernel-parameters.txt | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index bb48ae24ae69..f9b79294f84a 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -6850,6 +6850,9 @@
>  
>  				reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
>  
> +			Note, CONFIG_RESET_ATTACK_MITIGATION can force a memory reset on boot which
> +			will clear any trace that was stored.
> +
>  			See also Documentation/trace/debugging.rst
>  
>

Steven Rostedt Sept. 30, 2024, 5:14 p.m. UTC | #2

On Thu, 26 Sep 2024 19:54:48 +0200
Hans de Goede <[e-mail]> wrote:

> Hi,
> 
> On 26-Sep-24 7:01 PM, Steven Rostedt wrote:
> > From: Steven Rostedt <[e-mail]>
> > 
> > At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
> > about the persistent buffer to display traces from previous boots. He
> > mentioned that UEFI can clear memory. In my own tests I have not seen
> > this. He later informed me that it requires the config option:
> > 
> >  CONFIG_RESET_ATTACK_MITIGATION
> > 
> > It appears that setting this will allow the memory to be cleared on boot
> > up, which will definitely clear out the trace of the previous boot.
> > 
> > Add this information under the trace_instance in kernel-parameters.txt
> > to let people know that this can cause issues.
> > 
> > Link: https://lore.kernel.org/all/[e-mail]/
> > 
> > Reported-by: Hans de Goede <[e-mail]>
> > Signed-off-by: Steven Rostedt (Google) <[e-mail]>  
> 
> Thanks, patch looks good to me:
> 
> Reviewed-by: Hans de Goede <[e-mail]>

Thanks,

And I forgot to send this to the Documentation maintainers :-p

Jon, could you take this? Do you need me to resend, or can you just pull it
from lore?

   https://lore.kernel.org/linux-trace-kernel/[e-mail]/

-- Steve


> 
> Regards,
> 
> Hans
> 
> 
> 
> > ---
> >  Documentation/admin-guide/kernel-parameters.txt | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index bb48ae24ae69..f9b79294f84a 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -6850,6 +6850,9 @@
> >  
> >  				reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
> >  
> > +			Note, CONFIG_RESET_ATTACK_MITIGATION can force a memory reset on boot which
> > +			will clear any trace that was stored.
> > +
> >  			See also Documentation/trace/debugging.rst
> >  
> >

Jonathan Corbet Sept. 30, 2024, 5:20 p.m. UTC | #3

Steven Rostedt <[e-mail]> writes:

> On Thu, 26 Sep 2024 19:54:48 +0200
> Hans de Goede <[e-mail]> wrote:
>
>> Hi,
>> 
>> On 26-Sep-24 7:01 PM, Steven Rostedt wrote:
>> > From: Steven Rostedt <[e-mail]>
>> > 
>> > At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
>> > about the persistent buffer to display traces from previous boots. He
>> > mentioned that UEFI can clear memory. In my own tests I have not seen
>> > this. He later informed me that it requires the config option:
>> > 
>> >  CONFIG_RESET_ATTACK_MITIGATION
>> > 
>> > It appears that setting this will allow the memory to be cleared on boot
>> > up, which will definitely clear out the trace of the previous boot.
>> > 
>> > Add this information under the trace_instance in kernel-parameters.txt
>> > to let people know that this can cause issues.
>> > 
>> > Link: https://lore.kernel.org/all/[e-mail]/
>> > 
>> > Reported-by: Hans de Goede <[e-mail]>
>> > Signed-off-by: Steven Rostedt (Google) <[e-mail]>  
>> 
>> Thanks, patch looks good to me:
>> 
>> Reviewed-by: Hans de Goede <[e-mail]>
>
> Thanks,
>
> And I forgot to send this to the Documentation maintainers :-p
>
> Jon, could you take this? Do you need me to resend, or can you just pull it
> from lore?

I'll grab it in a bit.

jon

Steven Rostedt Sept. 30, 2024, 5:24 p.m. UTC | #4

On Mon, 30 Sep 2024 11:20:48 -0600
Jonathan Corbet <[e-mail]> wrote:

> > And I forgot to send this to the Documentation maintainers :-p
> >
> > Jon, could you take this? Do you need me to resend, or can you just pull it
> > from lore?  
> 
> I'll grab it in a bit.

No rush.

Thanks!

-- Steve

Ard Biesheuvel Oct. 1, 2024, 6:17 a.m. UTC | #5

On Thu, 26 Sept 2024 at 19:02, Steven Rostedt <[e-mail]> wrote:
>
> From: Steven Rostedt <[e-mail]>
>
> At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
> about the persistent buffer to display traces from previous boots. He
> mentioned that UEFI can clear memory. In my own tests I have not seen
> this. He later informed me that it requires the config option:
>
>  CONFIG_RESET_ATTACK_MITIGATION
>
> It appears that setting this will allow the memory to be cleared on boot
> up, which will definitely clear out the trace of the previous boot.
>
> Add this information under the trace_instance in kernel-parameters.txt
> to let people know that this can cause issues.
>
> Link: https://lore.kernel.org/all/[e-mail]/
>
> Reported-by: Hans de Goede <[e-mail]>
> Signed-off-by: Steven Rostedt (Google) <[e-mail]>
> ---
>  Documentation/admin-guide/kernel-parameters.txt | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index bb48ae24ae69..f9b79294f84a 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -6850,6 +6850,9 @@
>
>                                 reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
>
> +                       Note, CONFIG_RESET_ATTACK_MITIGATION can force a memory reset on boot which
> +                       will clear any trace that was stored.
> +

CONFIG_RESET_ATTACK_MITIGATION can force a wipe of system RAM at warm
reboot on systems that have a TPM enabled, but disabling it does not
prevent it. Also, there are many other reasons why the trace buffer
region may be wiped and/or reused for other purposes, so singling out
CONFIG_RESET_ATTACK_MITIGATION like this is not that useful imo.

As I indicated when this feature was under review, it should be made
very clear that any kernel side changes that affect the system's
behavior in this regard should not be considered regressions. So
instead of mentioning CONFIG_RESET_ATTACK_MITIGATION here, perhaps it
would be better to document that system firmware generally makes no
guarantees about preserving memory contents, and so this feature may
break without warning.

Note that on UEFI systems, there is a so-called 'capsule' interface
which does allow the OS to preserve an arbitrary buffer across a warm
reboot. However, whether it works on all systems is anybody's guess.

Hans de Goede Oct. 1, 2024, 8:56 a.m. UTC | #6

Hi,

On 1-Oct-24 8:17 AM, Ard Biesheuvel wrote:
> On Thu, 26 Sept 2024 at 19:02, Steven Rostedt <[e-mail]> wrote:
>>
>> From: Steven Rostedt <[e-mail]>
>>
>> At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
>> about the persistent buffer to display traces from previous boots. He
>> mentioned that UEFI can clear memory. In my own tests I have not seen
>> this. He later informed me that it requires the config option:
>>
>>  CONFIG_RESET_ATTACK_MITIGATION
>>
>> It appears that setting this will allow the memory to be cleared on boot
>> up, which will definitely clear out the trace of the previous boot.
>>
>> Add this information under the trace_instance in kernel-parameters.txt
>> to let people know that this can cause issues.
>>
>> Link: https://lore.kernel.org/all/[e-mail]/
>>
>> Reported-by: Hans de Goede <[e-mail]>
>> Signed-off-by: Steven Rostedt (Google) <[e-mail]>
>> ---
>>  Documentation/admin-guide/kernel-parameters.txt | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>> index bb48ae24ae69..f9b79294f84a 100644
>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> @@ -6850,6 +6850,9 @@
>>
>>                                 reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
>>
>> +                       Note, CONFIG_RESET_ATTACK_MITIGATION can force a memory reset on boot which
>> +                       will clear any trace that was stored.
>> +
> 
> CONFIG_RESET_ATTACK_MITIGATION can force a wipe of system RAM at warm
> reboot on systems that have a TPM enabled, but disabling it does not
> prevent it. Also, there are many other reasons why the trace buffer
> region may be wiped and/or reused for other purposes, so singling out
> CONFIG_RESET_ATTACK_MITIGATION like this is not that useful imo.

Since the userspace parts to clear the CONFIG_RESET_ATTACK_MITIGATION
related EFI variable after cleaning cryptographic keys from RAM has
never materialized CONFIG_RESET_ATTACK_MITIGATION is pretty much
guaranteed to clear any traces on any modern machine (and at least
in Fedora's kernel config it is disabled because of this).

I agree that there are more ways the RAM might get cleared, but
since this will clear the RAM almost 100% of the time it is worth
documenting this IMHO.

I get the feeling you (Ard) see documenting this as some sorta bug
report against CONFIG_RESET_ATTACK_MITIGATION, that is not the intention.
Quite the opposite the documentation is there to let the user know
that CONFIG_RESET_ATTACK_MITIGATION works as advertised and that it
will (almost) always clear the RAM on reboot and thus conflicts with
keeping traces over reboot.

Regards,

Hans

Ard Biesheuvel Oct. 1, 2024, 9:35 a.m. UTC | #7

Hi Hans,

On Tue, 1 Oct 2024 at 10:57, Hans de Goede <[e-mail]> wrote:
>
> Hi,
>
> On 1-Oct-24 8:17 AM, Ard Biesheuvel wrote:
> > On Thu, 26 Sept 2024 at 19:02, Steven Rostedt <[e-mail]> wrote:
> >>
> >> From: Steven Rostedt <[e-mail]>
> >>
> >> At the 2024 Linux Plumbers Conference, I was talking with Hans de Goede
> >> about the persistent buffer to display traces from previous boots. He
> >> mentioned that UEFI can clear memory. In my own tests I have not seen
> >> this. He later informed me that it requires the config option:
> >>
> >>  CONFIG_RESET_ATTACK_MITIGATION
> >>
> >> It appears that setting this will allow the memory to be cleared on boot
> >> up, which will definitely clear out the trace of the previous boot.
> >>
> >> Add this information under the trace_instance in kernel-parameters.txt
> >> to let people know that this can cause issues.
> >>
> >> Link: https://lore.kernel.org/all/[e-mail]/
> >>
> >> Reported-by: Hans de Goede <[e-mail]>
> >> Signed-off-by: Steven Rostedt (Google) <[e-mail]>
> >> ---
> >>  Documentation/admin-guide/kernel-parameters.txt | 3 +++
> >>  1 file changed, 3 insertions(+)
> >>
> >> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> >> index bb48ae24ae69..f9b79294f84a 100644
> >> --- a/Documentation/admin-guide/kernel-parameters.txt
> >> +++ b/Documentation/admin-guide/kernel-parameters.txt
> >> @@ -6850,6 +6850,9 @@
> >>
> >>                                 reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
> >>
> >> +                       Note, CONFIG_RESET_ATTACK_MITIGATION can force a memory reset on boot which
> >> +                       will clear any trace that was stored.
> >> +
> >
> > CONFIG_RESET_ATTACK_MITIGATION can force a wipe of system RAM at warm
> > reboot on systems that have a TPM enabled, but disabling it does not
> > prevent it. Also, there are many other reasons why the trace buffer
> > region may be wiped and/or reused for other purposes, so singling out
> > CONFIG_RESET_ATTACK_MITIGATION like this is not that useful imo.
>
> Since the userspace parts to clear the CONFIG_RESET_ATTACK_MITIGATION
> related EFI variable after cleaning cryptographic keys from RAM has
> never materialized CONFIG_RESET_ATTACK_MITIGATION is pretty much
> guaranteed to clear any traces on any modern machine (and at least
> in Fedora's kernel config it is disabled because of this).
>

Any modern x86 PC, sure.

> I agree that there are more ways the RAM might get cleared, but
> since this will clear the RAM almost 100% of the time it is worth
> documenting this IMHO.
>

Fair enough.

> I get the feeling you (Ard) see documenting this as some sorta bug
> report against CONFIG_RESET_ATTACK_MITIGATION, that is not the intention.
> Quite the opposite the documentation is there to let the user know
> that CONFIG_RESET_ATTACK_MITIGATION works as advertised and that it
> will (almost) always clear the RAM on reboot and thus conflicts with
> keeping traces over reboot.
>

I am not against documenting this. CONFIG_RESET_ATTACK_MITIGATION is
simply incompatible with this feature so it makes sense calling that
out.

But there are so many other ways in which the trace buffer might get
clobbered, and mentioning CONFIG_RESET_ATTACK_MITIGATION in particular
might suggest to some that those other causes are due to kernel bugs.

All I am asking for is a line in the documentation that says that
clobbered trace buffers could occur at any time, regardless of whether
CONFIG_RESET_ATTACK_MITIGATION is enabled or not. That way, we have
something to refer to when people start filing bugs against the EFI
component when this breaks.

Steven Rostedt Oct. 1, 2024, 12:53 p.m. UTC | #8

On Tue, 1 Oct 2024 11:35:53 +0200
Ard Biesheuvel <[e-mail]> wrote:

> All I am asking for is a line in the documentation that says that
> clobbered trace buffers could occur at any time, regardless of whether
> CONFIG_RESET_ATTACK_MITIGATION is enabled or not. That way, we have
> something to refer to when people start filing bugs against the EFI
> component when this breaks.

How about if I change the comment to this?

        Note, saving the trace buffer across reboots does require that the system
        is set up to not wipe memory. For instance, CONFIG_RESET_ATTACK_MITIGATION
        can force a memory reset on boot which will clear any trace that was stored.
        This is just one of many ways that can clear memory. Make sure your system
        keeps the content of memory across reboots before relying on this option.

Would that be better?

I want to stress that this doesn't work for every setup. It just so happens
to work for mine and others. I do not want people thinking that it's a bug
if it doesn't work for them.

-- Steve

Ard Biesheuvel Oct. 1, 2024, 1:32 p.m. UTC | #9

On Tue, 1 Oct 2024 at 14:52, Steven Rostedt <[e-mail]> wrote:
>
> On Tue, 1 Oct 2024 11:35:53 +0200
> Ard Biesheuvel <[e-mail]> wrote:
>
> > All I am asking for is a line in the documentation that says that
> > clobbered trace buffers could occur at any time, regardless of whether
> > CONFIG_RESET_ATTACK_MITIGATION is enabled or not. That way, we have
> > something to refer to when people start filing bugs against the EFI
> > component when this breaks.
>
> How about if I change the comment to this?
>
>         Note, saving the trace buffer across reboots does require that the system
>         is set up to not wipe memory. For instance, CONFIG_RESET_ATTACK_MITIGATION
>         can force a memory reset on boot which will clear any trace that was stored.
>         This is just one of many ways that can clear memory. Make sure your system
>         keeps the content of memory across reboots before relying on this option.
>
> Would that be better?
>

Yes, that looks much better. Thanks.

Acked-by: Ard Biesheuvel <[e-mail]>

> I want to stress that this doesn't work for every setup. It just so happens
> to work for mine and others. I do not want people thinking that it's a bug
> if it doesn't work for them.

Exactly. I can see how this feature is very useful if it happens to
work on your system firmware, but it should be documented as not being
covered by the 'no user visible regressions' policy.

Patch

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index bb48ae24ae69..f9b79294f84a 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6850,6 +6850,9 @@ 
 
 				reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
 
+			Note, CONFIG_RESET_ATTACK_MITIGATION can force a memory reset on boot which
+			will clear any trace that was stored.
+
 			See also Documentation/trace/debugging.rst