| Message ID | 20180220205924.2035765-1-arnd@arndb.de (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
Hi Arnd Bergmann, > -----Original Message----- > From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma- > owner@vger.kernel.org] On Behalf Of Arnd Bergmann > Sent: Tuesday, February 20, 2018 2:59 PM > To: Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca> > Cc: Arnd Bergmann <arnd@arndb.de>; Leon Romanovsky > <leonro@mellanox.com>; Sagi Grimberg <sagi@grimberg.me>; Bart Van Assche > <bart.vanassche@sandisk.com>; linux-rdma@vger.kernel.org; linux- > kernel@vger.kernel.org > Subject: [PATCH] RDMA/core: reduce IB_POLL_BATCH constant > > The ib_wc structure has grown to much that putting 16 of them on the stack hits > the warning limit for dangerous kernel stack consumption: > > drivers/infiniband/core/cq.c: In function 'ib_process_cq_direct': > drivers/infiniband/core/cq.c:78:1: error: the frame size of 1032 bytes is larger > than 1024 bytes [-Werror=frame-larger-than=] > > Using half that number brings us comfortably below that limit again. > > Fixes: 02d8883f520e ("RDMA/restrack: Add general infrastructure to track > RDMA resources") It is not clear to me how above commit 02d8883f520e introduced this stack issue. Bodong and I came across ib_wc size increase in [1] and it was fixed in [2]. Did you hit this error after/before applying patch [2]? [1] https://www.spinics.net/lists/linux-rdma/msg50754.html [2] https://patchwork.kernel.org/patch/10159623/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2018-02-20 at 21:59 +0100, Arnd Bergmann wrote: > /* # of WCs to poll for with a single call to ib_poll_cq */ > -#define IB_POLL_BATCH 16 > +#define IB_POLL_BATCH 8 The purpose of batch polling is to minimize contention on the cq spinlock. Reducing the IB_POLL_BATCH constant may affect performance negatively. Has the performance impact of this change been verified for all affected drivers (ib_srp, ib_srpt, ib_iser, ib_isert, NVMeOF, NVMeOF target, SMB Direct, NFS over RDMA, ...)? Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> On Feb 20, 2018, at 4:14 PM, Bart Van Assche <Bart.VanAssche@wdc.com> wrote: > > On Tue, 2018-02-20 at 21:59 +0100, Arnd Bergmann wrote: >> /* # of WCs to poll for with a single call to ib_poll_cq */ >> -#define IB_POLL_BATCH 16 >> +#define IB_POLL_BATCH 8 > > The purpose of batch polling is to minimize contention on the cq spinlock. > Reducing the IB_POLL_BATCH constant may affect performance negatively. Has > the performance impact of this change been verified for all affected drivers > (ib_srp, ib_srpt, ib_iser, ib_isert, NVMeOF, NVMeOF target, SMB Direct, NFS > over RDMA, ...)? Only the users of the DIRECT polling method use an on-stack array of ib_wc's. This is only the SRP drivers. The other two modes have use of a dynamically allocated array of ib_wc's that hangs off the ib_cq. These shouldn't need any reduction in the size of this array, and they are the common case. IMO a better solution would be to change ib_process_cq_direct to use a smaller on-stack array, and leave IB_POLL_BATCH alone. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Feb 20, 2018 at 10:14 PM, Parav Pandit <parav@mellanox.com> wrote: > Hi Arnd Bergmann, > >> -----Original Message----- >> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma- >> owner@vger.kernel.org] On Behalf Of Arnd Bergmann >> Sent: Tuesday, February 20, 2018 2:59 PM >> To: Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca> >> Cc: Arnd Bergmann <arnd@arndb.de>; Leon Romanovsky >> <leonro@mellanox.com>; Sagi Grimberg <sagi@grimberg.me>; Bart Van Assche >> <bart.vanassche@sandisk.com>; linux-rdma@vger.kernel.org; linux- >> kernel@vger.kernel.org >> Subject: [PATCH] RDMA/core: reduce IB_POLL_BATCH constant >> >> The ib_wc structure has grown to much that putting 16 of them on the stack hits >> the warning limit for dangerous kernel stack consumption: >> >> drivers/infiniband/core/cq.c: In function 'ib_process_cq_direct': >> drivers/infiniband/core/cq.c:78:1: error: the frame size of 1032 bytes is larger >> than 1024 bytes [-Werror=frame-larger-than=] >> >> Using half that number brings us comfortably below that limit again. >> >> Fixes: 02d8883f520e ("RDMA/restrack: Add general infrastructure to track >> RDMA resources") > > It is not clear to me how above commit 02d8883f520e introduced this stack issue. My mistake, I misread the git history. I did a proper bisection now and ended up with the commit that added the IB_POLL_BACK sized array on the stack, i.e. commit 246d8b184c10 ("IB/cq: Don't force IB_POLL_DIRECT poll context for ib_process_cq_direct") > Bodong and I came across ib_wc size increase in [1] and it was fixed in [2]. > Did you hit this error after/before applying patch [2]? > > [1] https://www.spinics.net/lists/linux-rdma/msg50754.html > [2] https://patchwork.kernel.org/patch/10159623/ I did the analysis a few weeks ago when I first hit the problem but didn't send it out at the time. Today I saw the problem still persists on mainline (4.16-rc2), which does contain the patch from [2]. What I see is that 'ib_wc' is now exactly 59 bytes on 32-bit ARM, plus 5 bytes of padding, so 16 of them gets us exactly the warning limit, and then there are a few bytes for the function itself. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2/20/2018 11:47 PM, Chuck Lever wrote: > > >> On Feb 20, 2018, at 4:14 PM, Bart Van Assche <Bart.VanAssche@wdc.com> wrote: >> >> On Tue, 2018-02-20 at 21:59 +0100, Arnd Bergmann wrote: >>> /* # of WCs to poll for with a single call to ib_poll_cq */ >>> -#define IB_POLL_BATCH 16 >>> +#define IB_POLL_BATCH 8 >> >> The purpose of batch polling is to minimize contention on the cq spinlock. >> Reducing the IB_POLL_BATCH constant may affect performance negatively. Has >> the performance impact of this change been verified for all affected drivers >> (ib_srp, ib_srpt, ib_iser, ib_isert, NVMeOF, NVMeOF target, SMB Direct, NFS >> over RDMA, ...)? > > Only the users of the DIRECT polling method use an on-stack > array of ib_wc's. This is only the SRP drivers. > > The other two modes have use of a dynamically allocated array > of ib_wc's that hangs off the ib_cq. These shouldn't need any > reduction in the size of this array, and they are the common > case. > > IMO a better solution would be to change ib_process_cq_direct > to use a smaller on-stack array, and leave IB_POLL_BATCH alone. Yup, good idea. you can define IB_DIRECT_POLL_BATCH to be 8 and use it in ib_process_cq_direct. *but* please make sure to use the right value in ib_poll_cq since the wcs array should be able to hold the requested amount of wcs. -Max. > > -- > Chuck Lever > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2/20/2018 10:59 PM, Arnd Bergmann wrote: > The ib_wc structure has grown to much that putting 16 of them on the stack > hits the warning limit for dangerous kernel stack consumption: is there any real reason we can't have ib_wc as 64B / one cache line? what is this reason? > > drivers/infiniband/core/cq.c: In function 'ib_process_cq_direct': > drivers/infiniband/core/cq.c:78:1: error: the frame size of 1032 bytes is larger than 1024 bytes [-Werror=frame-larger-than=] -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index bc79ca8215d7..2626adbb978e 100644 --- a/drivers/infiniband/core/cq.c +++ b/drivers/infiniband/core/cq.c @@ -16,7 +16,7 @@ #include <rdma/ib_verbs.h> /* # of WCs to poll for with a single call to ib_poll_cq */ -#define IB_POLL_BATCH 16 +#define IB_POLL_BATCH 8 /* # of WCs to iterate over before yielding */ #define IB_POLL_BUDGET_IRQ 256
The ib_wc structure has grown to much that putting 16 of them on the stack hits the warning limit for dangerous kernel stack consumption: drivers/infiniband/core/cq.c: In function 'ib_process_cq_direct': drivers/infiniband/core/cq.c:78:1: error: the frame size of 1032 bytes is larger than 1024 bytes [-Werror=frame-larger-than=] Using half that number brings us comfortably below that limit again. Fixes: 02d8883f520e ("RDMA/restrack: Add general infrastructure to track RDMA resources") Signed-off-by: Arnd Bergmann <arnd@arndb.de> --- drivers/infiniband/core/cq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)