| Message ID | 20190906151123.1088455-1-arnd@arndb.de (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | net/mlx5: reduce stack usage in FW tracer | expand |
On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote: > It's generally not ok to put a 512 byte buffer on the stack, as > kernel > stack is a scarce resource: > > drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c:660:13: > error: stack frame size of 1032 bytes in function > 'mlx5_fw_tracer_handle_traces' [-Werror,-Wframe-larger-than=] > > This is done in a context that is allowed to sleep, so using > dynamic allocation is ok as well. I'm not too worried about > runtime overhead, as this already contains an snprintf() and > other expensive functions. > > Fixes: 70dd6fdb8987 ("net/mlx5: FW tracer, parse traces and kernel > tracing support") > Signed-off-by: Arnd Bergmann <arnd@arndb.de> > --- > .../mellanox/mlx5/core/diag/fw_tracer.c | 21 ++++++++++------- > -- > 1 file changed, 11 insertions(+), 10 deletions(-) > > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > index 2011eaf15cc5..d81e78060f9f 100644 > --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct > tracer_string_format *str_frmt, > struct mlx5_core_dev *dev, > u64 trace_timestamp) > { > - char tmp[512]; > - Hi Arnd, thanks for the patch, this function is very perfomance critical when fw traces are activated to pull some fw content on error situations, using kmalloc here might become a problem and stall the system further more if the problem was initially due to lack of memory. since this function only needs 512 bytes maybe we should mark it as noinline to avoid any extra stack usages on the caller function mlx5_fw_tracer_handle_traces ? > - snprintf(tmp, sizeof(tmp), str_frmt->string, > - str_frmt->params[0], > - str_frmt->params[1], > - str_frmt->params[2], > - str_frmt->params[3], > - str_frmt->params[4], > - str_frmt->params[5], > - str_frmt->params[6]); > + char *tmp = kasprintf(GFP_KERNEL, str_frmt->string, > + str_frmt->params[0], > + str_frmt->params[1], > + str_frmt->params[2], > + str_frmt->params[3], > + str_frmt->params[4], > + str_frmt->params[5], > + str_frmt->params[6]); > + if (!tmp) > + return; > > trace_mlx5_fw(dev->tracer, trace_timestamp, str_frmt->lost, > str_frmt->event_id, tmp); > @@ -576,6 +576,7 @@ static void mlx5_tracer_print_trace(struct > tracer_string_format *str_frmt, > > /* remove it from hash */ > mlx5_tracer_clean_message(str_frmt); > + kfree(tmp); > } > > static int mlx5_tracer_handle_string_trace(struct mlx5_fw_tracer > *tracer,
On Mon, Sep 9, 2019 at 9:39 PM Saeed Mahameed <saeedm@mellanox.com> wrote: > On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote: > > --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > > +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > > @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct > > tracer_string_format *str_frmt, > > struct mlx5_core_dev *dev, > > u64 trace_timestamp) > > { > > - char tmp[512]; > > - > > Hi Arnd, thanks for the patch, > this function is very perfomance critical when fw traces are activated > to pull some fw content on error situations, using kmalloc here might > become a problem and stall the system further more if the problem was > initially due to lack of memory. > > since this function only needs 512 bytes maybe we should mark it as > noinline to avoid any extra stack usages on the caller function > mlx5_fw_tracer_handle_traces ? That would shut up the warning, but doesn't sound right either. If it's performance critical indeed, maybe the best solution would be to also avoid the snprintf(), as that is also a rather heavyweight function? I could not find an easy solution for this, but I did notice the unusual way this deals with a variable format string passed into mlx5_tracer_print_trace along with a set of parameters, which opens up a set of possible format string vulnerabilities as well as making mlx5_tracer_print_trace() a bit expensive. You also take a mutex and free memory in there, which obviously then also got allocated in the fast path. To do this right, a better approach may be to just rely on ftrace, storing the (pointer to the) format string and the arguments in the buffer without creating a string. Would that be an option here? A more minimal approach might be to move what is now the on-stack buffer into the mlx5_fw_tracer function. I see that you already store a copy of the string in there from mlx5_fw_tracer_save_trace(), which conveniently also holds a mutex already that protects it from concurrent access. Arnd
On Mon, 2019-09-09 at 22:18 +0200, Arnd Bergmann wrote: > On Mon, Sep 9, 2019 at 9:39 PM Saeed Mahameed <saeedm@mellanox.com> > wrote: > > On Fri, 2019-09-06 at 17:11 +0200, Arnd Bergmann wrote: > > > --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > > > +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c > > > @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct > > > tracer_string_format *str_frmt, > > > struct mlx5_core_dev *dev, > > > u64 trace_timestamp) > > > { > > > - char tmp[512]; > > > - > > > > Hi Arnd, thanks for the patch, > > this function is very perfomance critical when fw traces are > > activated > > to pull some fw content on error situations, using kmalloc here > > might > > become a problem and stall the system further more if the problem > > was > > initially due to lack of memory. > > > > since this function only needs 512 bytes maybe we should mark it as > > noinline to avoid any extra stack usages on the caller function > > mlx5_fw_tracer_handle_traces ? > > That would shut up the warning, but doesn't sound right either. > > If it's performance critical indeed, maybe the best solution would > be to also avoid the snprintf(), as that is also a rather heavyweight > function? > > I could not find an easy solution for this, but I did notice the > unusual way > this deals with a variable format string passed into > mlx5_tracer_print_trace > along with a set of parameters, which opens up a set of possible > format string vulnerabilities as well as making > mlx5_tracer_print_trace() > a bit expensive. You also take a mutex and free memory in there, > which obviously then also got allocated in the fast path. > > To do this right, a better approach may be to just rely on ftrace, > storing > the (pointer to the) format string and the arguments in the buffer > without > creating a string. Would that be an option here? I am not sure how this would work, since the format parameters can changes depending on the FW string and the specific traces. > > A more minimal approach might be to move what is now the on-stack > buffer into the mlx5_fw_tracer function. I see that you already store > a copy of the string in there from mlx5_fw_tracer_save_trace(), > which conveniently also holds a mutex already that protects > it from concurrent access. > This sounds plausible. So for now let's do this or the noinline approach, Please let me know which one do you prefer, if it is the mutex protected buffer, i can do it myself. I will open an internal task and discussion then address your valuable points in a future submission, since we already in rc8 I don't want to take the risk now. Thanks for your feedback ! Saeed. > Arnd
On Mon, Sep 9, 2019 at 11:53 PM Saeed Mahameed <saeedm@mellanox.com> wrote: > On Mon, 2019-09-09 at 22:18 +0200, Arnd Bergmann wrote: > > To do this right, a better approach may be to just rely on ftrace, > > storing > > the (pointer to the) format string and the arguments in the buffer > > without > > creating a string. Would that be an option here? > > I am not sure how this would work, since the format parameters can > changes depending on the FW string and the specific traces. Ah, so the format string comes from the firmware? I didn't look at the code in enough detail to understand why it's done like this, only enough to notice that it's rather unusual. Possibly trace_mlx5_fw might still get away with copying the format string and the arguments, leaving the snprintf() to the time we read the buffer, but I don't know enough about ftrace to be sure that would actually work, and you'd need to duplicate it in mlx5_devlink_fmsg_fill_trace(). > > A more minimal approach might be to move what is now the on-stack > > buffer into the mlx5_fw_tracer function. I see that you already store > > a copy of the string in there from mlx5_fw_tracer_save_trace(), > > which conveniently also holds a mutex already that protects > > it from concurrent access. > > > > This sounds plausible. > > So for now let's do this or the noinline approach, Please let me know > which one do you prefer, if it is the mutex protected buffer, i can do > it myself. > > I will open an internal task and discussion then address your valuable > points in a future submission, since we already in rc8 I don't want to > take the risk now. Yes, that sounds like a good plan. If you can't avoid the snprintf entirely, then the mutex protected buffer should be helpful, and also avoid a strncpy() along with the stack buffer. Arnd
From: Arnd Bergmann > Sent: 10 September 2019 09:15 ... > > I am not sure how this would work, since the format parameters can > > changes depending on the FW string and the specific traces. > > Ah, so the format string comes from the firmware? I didn't look > at the code in enough detail to understand why it's done like this, > only enough to notice that it's rather unusual. If the format string comes from the firmware you really shouldn't pass it to any standard printf function. You must ensure that it doesn't contain any format effectors that might dereference parameters. (The code might try to do that.) Given that 'pointer' format effectors can't be used, the firmware must also supply the relevant integer ones? This should mean that all the processing is deferrable until the trace record is read. 'noinline' just papers over the cracks. Especially since vasprintf() is likely to use a lot of stack. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
On Tue, 2019-09-10 at 15:38 +0000, David Laight wrote: > From: Arnd Bergmann > > Sent: 10 September 2019 09:15 > ... > > > I am not sure how this would work, since the format parameters > > > can > > > changes depending on the FW string and the specific traces. > > > > Ah, so the format string comes from the firmware? I didn't look > > at the code in enough detail to understand why it's done like this, > > only enough to notice that it's rather unusual. > > If the format string comes from the firmware you really shouldn't > pass it to any standard printf function. > You must ensure that it doesn't contain any format effectors > that might dereference parameters. > (The code might try to do that.) > > Given that 'pointer' format effectors can't be used, the firmware > must also supply the relevant integer ones? > This should mean that all the processing is deferrable until the > trace record is read. > Point taken, i will discuss this with the team next week, since i am traveling this week. and we will provide a proper solution. for now, off the top of my head, FW strings and parameters are well defined and very simple, many of the problems here do not apply, but will verify. > 'noinline' just papers over the cracks. > Especially since vasprintf() is likely to use a lot of stack. Right i also tend to agree with noinline as a temporary solution until we address all the points provided here. > > David > > - > Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, > MK1 1PT, UK > Registration No: 1397386 (Wales)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c index 2011eaf15cc5..d81e78060f9f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c @@ -557,16 +557,16 @@ static void mlx5_tracer_print_trace(struct tracer_string_format *str_frmt, struct mlx5_core_dev *dev, u64 trace_timestamp) { - char tmp[512]; - - snprintf(tmp, sizeof(tmp), str_frmt->string, - str_frmt->params[0], - str_frmt->params[1], - str_frmt->params[2], - str_frmt->params[3], - str_frmt->params[4], - str_frmt->params[5], - str_frmt->params[6]); + char *tmp = kasprintf(GFP_KERNEL, str_frmt->string, + str_frmt->params[0], + str_frmt->params[1], + str_frmt->params[2], + str_frmt->params[3], + str_frmt->params[4], + str_frmt->params[5], + str_frmt->params[6]); + if (!tmp) + return; trace_mlx5_fw(dev->tracer, trace_timestamp, str_frmt->lost, str_frmt->event_id, tmp); @@ -576,6 +576,7 @@ static void mlx5_tracer_print_trace(struct tracer_string_format *str_frmt, /* remove it from hash */ mlx5_tracer_clean_message(str_frmt); + kfree(tmp); } static int mlx5_tracer_handle_string_trace(struct mlx5_fw_tracer *tracer,
It's generally not ok to put a 512 byte buffer on the stack, as kernel stack is a scarce resource: drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c:660:13: error: stack frame size of 1032 bytes in function 'mlx5_fw_tracer_handle_traces' [-Werror,-Wframe-larger-than=] This is done in a context that is allowed to sleep, so using dynamic allocation is ok as well. I'm not too worried about runtime overhead, as this already contains an snprintf() and other expensive functions. Fixes: 70dd6fdb8987 ("net/mlx5: FW tracer, parse traces and kernel tracing support") Signed-off-by: Arnd Bergmann <arnd@arndb.de> --- .../mellanox/mlx5/core/diag/fw_tracer.c | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-)