[1/1] stackleak: Disable ftrace for stackleak.c

Commit Message

Alexander Popov Nov. 10, 2018, 10:05 p.m. UTC

The stackleak_erase() function is called on the trampoline stack at the
end of syscall. This stack is not big enough for ftrace operations,
e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Let's disable ftrace for stackleak.c to avoid such situations.

Reported-by: kernel test robot <[e-mail]>
Signed-off-by: Alexander Popov <[e-mail]>
Reviewed-by: Kees Cook <[e-mail]>
---
 kernel/Makefile | 1 +
 1 file changed, 1 insertion(+)

Comments

Steven Rostedt Nov. 10, 2018, 11:30 p.m. UTC | #1

On Sun, 11 Nov 2018 01:05:30 +0300
Alexander Popov <[e-mail]> wrote:

> The stackleak_erase() function is called on the trampoline stack at the
> end of syscall. This stack is not big enough for ftrace operations,
> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops
function tracing which I only want disabled if function tracing itself
is an issue, not for other things that may use the function tracing
infrastructure.

-- Steve

Alexander Popov Nov. 11, 2018, 10:19 a.m. UTC | #2

On 11.11.2018 2:30, Steven Rostedt wrote:
> On Sun, 11 Nov 2018 01:05:30 +0300
> Alexander Popov <[e-mail]> wrote:
> 
>> The stackleak_erase() function is called on the trampoline stack at the
>> end of syscall. This stack is not big enough for ftrace operations,
>> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
> 
> Is the issue with kprobes or with function tracing? Because this stops
> function tracing which I only want disabled if function tracing itself
> is an issue, not for other things that may use the function tracing
> infrastructure.

Hello Steven,

I believe that stackleak erasing is not compatible with function tracing itself.
That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1

I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Best regards,
Alexander

Steven Rostedt Nov. 12, 2018, 1:53 a.m. UTC | #3

On Sun, 11 Nov 2018 13:19:45 +0300
Alexander Popov <[e-mail]> wrote:

> On 11.11.2018 2:30, Steven Rostedt wrote:
> > On Sun, 11 Nov 2018 01:05:30 +0300
> > Alexander Popov <[e-mail]> wrote:
> >   
> >> The stackleak_erase() function is called on the trampoline stack at the
> >> end of syscall. This stack is not big enough for ftrace operations,
> >> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().  
> > 
> > Is the issue with kprobes or with function tracing? Because this stops
> > function tracing which I only want disabled if function tracing itself
> > is an issue, not for other things that may use the function tracing
> > infrastructure.  
> 
> Hello Steven,
> 
> I believe that stackleak erasing is not compatible with function tracing itself.
> That's what the kernel testing robot has hit:
> https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
> 
> I used kprobe_events just to reproduce the problem:
> https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Have you tried adding a "notrace" to stackleak_erase()?

Not tracing the entire file is a bit of overkill. There's no reason
ftrace can't trace stack_erasing_sysctl() or perhaps even
stackleak_track_stack() as that may be very interesting to trace.

-- Steve

Masami Hiramatsu (Google) Nov. 12, 2018, 2:50 a.m. UTC | #4

Hi Alexander and Steve,

On Sun, 11 Nov 2018 20:53:51 -0500
Steven Rostedt <[e-mail]> wrote:

> On Sun, 11 Nov 2018 13:19:45 +0300
> Alexander Popov <[e-mail]> wrote:
> 
> > On 11.11.2018 2:30, Steven Rostedt wrote:
> > > On Sun, 11 Nov 2018 01:05:30 +0300
> > > Alexander Popov <[e-mail]> wrote:
> > >   
> > >> The stackleak_erase() function is called on the trampoline stack at the
> > >> end of syscall. This stack is not big enough for ftrace operations,
> > >> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().  
> > > 
> > > Is the issue with kprobes or with function tracing? Because this stops
> > > function tracing which I only want disabled if function tracing itself
> > > is an issue, not for other things that may use the function tracing
> > > infrastructure.  
> > 
> > Hello Steven,
> > 
> > I believe that stackleak erasing is not compatible with function tracing itself.
> > That's what the kernel testing robot has hit:
> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
> > 
> > I used kprobe_events just to reproduce the problem:
> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
> 
> Have you tried adding a "notrace" to stackleak_erase()?
> 
> Not tracing the entire file is a bit of overkill. There's no reason
> ftrace can't trace stack_erasing_sysctl() or perhaps even
> stackleak_track_stack() as that may be very interesting to trace.

I think it is not enough for stopping kprobes. If you want to stop the kprobes
(int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase),
since kprobes can work without ftrace. 

Thank you,

Mathieu Desnoyers Nov. 12, 2018, 2:53 p.m. UTC | #5

----- On Nov 11, 2018, at 9:50 PM, Masami Hiramatsu [e-mail] wrote:

> Hi Alexander and Steve,
> 
> On Sun, 11 Nov 2018 20:53:51 -0500
> Steven Rostedt <[e-mail]> wrote:
> 
>> On Sun, 11 Nov 2018 13:19:45 +0300
>> Alexander Popov <[e-mail]> wrote:
>> 
>> > On 11.11.2018 2:30, Steven Rostedt wrote:
>> > > On Sun, 11 Nov 2018 01:05:30 +0300
>> > > Alexander Popov <[e-mail]> wrote:
>> > >   
>> > >> The stackleak_erase() function is called on the trampoline stack at the
>> > >> end of syscall. This stack is not big enough for ftrace operations,
>> > >> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
>> > > 
>> > > Is the issue with kprobes or with function tracing? Because this stops
>> > > function tracing which I only want disabled if function tracing itself
>> > > is an issue, not for other things that may use the function tracing
>> > > infrastructure.
>> > 
>> > Hello Steven,
>> > 
>> > I believe that stackleak erasing is not compatible with function tracing itself.
>> > That's what the kernel testing robot has hit:
>> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
>> > 
>> > I used kprobe_events just to reproduce the problem:
>> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
>> 
>> Have you tried adding a "notrace" to stackleak_erase()?
>> 
>> Not tracing the entire file is a bit of overkill. There's no reason
>> ftrace can't trace stack_erasing_sysctl() or perhaps even
>> stackleak_track_stack() as that may be very interesting to trace.
> 
> I think it is not enough for stopping kprobes. If you want to stop the kprobes
> (int3 version) on stackleak_erase(), you should use
> NOKPROBE_SYMBOL(stackleak_erase),
> since kprobes can work without ftrace.

Just to clarify: AFAIU you guys are recommending to add _both_ a "notrace"
annotation to stackleak_erase() _and_ a NOKPROBE_SYMBOL(stackleak_erase),
so neither function tracing nor kprobes can hook on that function.

Thanks,

Mathieu

Alexander Popov Nov. 12, 2018, 4:51 p.m. UTC | #6

Hello Steven and Masami,

Thanks for your comments.

On 12.11.2018 5:50, Masami Hiramatsu wrote:
> Hi Alexander and Steve,
> 
> On Sun, 11 Nov 2018 20:53:51 -0500
> Steven Rostedt <[e-mail]> wrote:
> 
>> On Sun, 11 Nov 2018 13:19:45 +0300
>> Alexander Popov <[e-mail]> wrote:
>>
>>> On 11.11.2018 2:30, Steven Rostedt wrote:
>>>> On Sun, 11 Nov 2018 01:05:30 +0300
>>>> Alexander Popov <[e-mail]> wrote:
>>>>   
>>>>> The stackleak_erase() function is called on the trampoline stack at the
>>>>> end of syscall. This stack is not big enough for ftrace operations,
>>>>> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().  
>>>>
>>>> Is the issue with kprobes or with function tracing? Because this stops
>>>> function tracing which I only want disabled if function tracing itself
>>>> is an issue, not for other things that may use the function tracing
>>>> infrastructure.  
>>>
>>> Hello Steven,
>>>
>>> I believe that stackleak erasing is not compatible with function tracing itself.
>>> That's what the kernel testing robot has hit:
>>> https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
>>>
>>> I used kprobe_events just to reproduce the problem:
>>> https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
>>
>> Have you tried adding a "notrace" to stackleak_erase()?
>>
>> Not tracing the entire file is a bit of overkill. There's no reason
>> ftrace can't trace stack_erasing_sysctl() or perhaps even
>> stackleak_track_stack() as that may be very interesting to trace.

Yes, thank you. It's much better.

> I think it is not enough for stopping kprobes. If you want to stop the kprobes
> (int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase),
> since kprobes can work without ftrace. 

Thanks!

I learned how to use kprobes without ftrace and managed to reproduce the problem
as well (I modified kprobe_example.c and kretprobe_example.c). NOKPROBE_SYMBOL()
allowed to avoid it.

I'll send the patch soon.

By the way, are there any other tracing/instrumentation mechanisms that should
be disabled?

Best regards,
Alexander

Steven Rostedt Nov. 12, 2018, 4:52 p.m. UTC | #7

On Mon, 12 Nov 2018 09:53:29 -0500 (EST)
Mathieu Desnoyers <[e-mail]> wrote:


> Just to clarify: AFAIU you guys are recommending to add _both_ a "notrace"
> annotation to stackleak_erase() _and_ a NOKPROBE_SYMBOL(stackleak_erase),
> so neither function tracing nor kprobes can hook on that function.

Correct.

-- Steve

Steven Rostedt Nov. 12, 2018, 5:21 p.m. UTC | #8

On Mon, 12 Nov 2018 19:51:00 +0300
Alexander Popov <[e-mail]> wrote:

> By the way, are there any other tracing/instrumentation mechanisms that should
> be disabled?

ftrace and kprobes are pretty much the only ones that currently do self
modification of code all over the kernel. Kprobes even more so than
ftrace.

-- Steve

Masami Hiramatsu (Google) Nov. 13, 2018, 6:21 p.m. UTC | #9

On Mon, 12 Nov 2018 12:21:48 -0500
Steven Rostedt <[e-mail]> wrote:

> On Mon, 12 Nov 2018 19:51:00 +0300
> Alexander Popov <[e-mail]> wrote:
> 
> > By the way, are there any other tracing/instrumentation mechanisms that should
> > be disabled?
> 
> ftrace and kprobes are pretty much the only ones that currently do self
> modification of code all over the kernel. Kprobes even more so than
> ftrace.

Right, since kprobes uses int3 or sw breakpoint exception for hooking into
the code, it consumes stack much more.

Thank you,

Patch

diff --git a/kernel/Makefile b/kernel/Makefile
index 7343b3a..0906f6d 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -18,6 +18,7 @@  obj-$(CONFIG_MULTIUSER) += groups.o
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace internal ftrace files
 CFLAGS_REMOVE_irq_work.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_stackleak.o = $(CC_FLAGS_FTRACE)
 endif
 
 # Prevents flicker of uninteresting __do_softirq()/__local_bh_disable_ip()