From patchwork Wed Feb 21 13:44:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10232721 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 04205602A7 for ; Wed, 21 Feb 2018 13:45:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8EE3286C5 for ; Wed, 21 Feb 2018 13:45:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DBDAD28B99; Wed, 21 Feb 2018 13:45:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 86F82286C5 for ; Wed, 21 Feb 2018 13:45:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S936684AbeBUNor (ORCPT ); Wed, 21 Feb 2018 08:44:47 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:33660 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S936089AbeBUNop (ORCPT ); Wed, 21 Feb 2018 08:44:45 -0500 Received: by mail-wr0-f196.google.com with SMTP id s5so4650798wra.0; Wed, 21 Feb 2018 05:44:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=9pwgh7TXhYRIaDqDYNXm4/FBblrn9VsJYb1cGcQZ6KA=; b=cRKTmHYfE1zIsrSdriaEBpgG7bOy1j7slGGacnoUPWRrLybFu2V8zpuFSizBXu3rSu HGik66w2SD6h8Xw20tPffEQ1EqhAyo44c5/Mo2ENQxq3+kjP/5VFrS0+vtf9RA9ASfUi ppONIiQMClr74UCdXFGGDWapWADQyrKvfk2dBR267boYo0+TWGFvcQtMEEQrxTVhM8xv 0FZUkBmmMAyfSyFUnc+tfCPzPjQIJ0DCqMracaHJUITQkGV93XI9opfmEsrUyV5SYvqI S8lgeXx0p1Cm6HjzUQqotOHdzlKMBDmgSxE9VLYFKgJcdj3VPsvuNpyW/4avDrjLRFel jGxQ== X-Gm-Message-State: APf1xPDwVPTB1DaehwBJ4UdUGUMFiJ3Kuv1ZJRp5XJzrD2o+s9lwKFL6 gIkmP6B7oHStgaCwt1lEOq6q/GCc X-Google-Smtp-Source: AH8x227hjya6IXLP9mTuGDvlAHwrQ/CHITRezFRPS4Ceu8oMHd9FOSSkCpSo5DDjTYaM4u3AsUwO8w== X-Received: by 10.28.160.9 with SMTP id j9mr2134692wme.103.1519220683467; Wed, 21 Feb 2018 05:44:43 -0800 (PST) Received: from [192.168.64.105] (bzq-219-42-90.isdn.bezeqint.net. [62.219.42.90]) by smtp.gmail.com with ESMTPSA id v9sm27055146wre.8.2018.02.21.05.44.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 05:44:42 -0800 (PST) Subject: Re: [PATCH] RDMA/core: reduce IB_POLL_BATCH constant To: Chuck Lever , Bart Van Assche Cc: "jgg@ziepe.ca" , "arnd@arndb.de" , "dledford@redhat.com" , "linux-kernel@vger.kernel.org" , "leonro@mellanox.com" , "linux-rdma@vger.kernel.org" References: <20180220205924.2035765-1-arnd@arndb.de> <1519161268.3737.12.camel@wdc.com> From: Sagi Grimberg Message-ID: Date: Wed, 21 Feb 2018 15:44:40 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP >> On Tue, 2018-02-20 at 21:59 +0100, Arnd Bergmann wrote: >>> /* # of WCs to poll for with a single call to ib_poll_cq */ >>> -#define IB_POLL_BATCH 16 >>> +#define IB_POLL_BATCH 8 >> >> The purpose of batch polling is to minimize contention on the cq spinlock. >> Reducing the IB_POLL_BATCH constant may affect performance negatively. Has >> the performance impact of this change been verified for all affected drivers >> (ib_srp, ib_srpt, ib_iser, ib_isert, NVMeOF, NVMeOF target, SMB Direct, NFS >> over RDMA, ...)? > > Only the users of the DIRECT polling method use an on-stack > array of ib_wc's. This is only the SRP drivers. > > The other two modes have use of a dynamically allocated array > of ib_wc's that hangs off the ib_cq. These shouldn't need any > reduction in the size of this array, and they are the common > case. > > IMO a better solution would be to change ib_process_cq_direct > to use a smaller on-stack array, and leave IB_POLL_BATCH alone. The only reason why I added this array on-stack was to allow consumers that did not use ib_alloc_cq api to call it, but that seems like a wrong decision when thinking it over again (as probably these users did not set the wr_cqe correctly). How about we make ib_process_cq_direct use the cq wc array and add a WARN_ON statement (and fail it gracefully) if the caller used this API without calling ib_alloc_cq? --- * budget might be (-1) if the caller does not @@ -72,9 +72,9 @@ static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc) */ int ib_process_cq_direct(struct ib_cq *cq, int budget) { - struct ib_wc wcs[IB_POLL_BATCH]; - - return __ib_process_cq(cq, budget, wcs); + if (unlikely(WARN_ON_ONCE(!cq->wc))) + return 0; + return __ib_process_cq(cq, budget); } EXPORT_SYMBOL(ib_process_cq_direct); @@ -88,7 +88,7 @@ static int ib_poll_handler(struct irq_poll *iop, int budget) struct ib_cq *cq = container_of(iop, struct ib_cq, iop); int completed; - completed = __ib_process_cq(cq, budget, NULL); + completed = __ib_process_cq(cq, budget); if (completed < budget) { irq_poll_complete(&cq->iop); if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) @@ -108,7 +108,7 @@ static void ib_cq_poll_work(struct work_struct *work) struct ib_cq *cq = container_of(work, struct ib_cq, work); int completed; - completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, NULL); + completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE); if (completed >= IB_POLL_BUDGET_WORKQUEUE || ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) queue_work(ib_comp_wq, &cq->work); -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index bc79ca8215d7..cd3e9e124834 100644 --- a/drivers/infiniband/core/cq.c +++ b/drivers/infiniband/core/cq.c @@ -25,10 +25,10 @@ #define IB_POLL_FLAGS \ (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) -static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc) +static int __ib_process_cq(struct ib_cq *cq, int budget) { int i, n, completed = 0; - struct ib_wc *wcs = poll_wc ? : cq->wc; + struct ib_wc *wcs = cq->wc; /*