From patchwork Fri Jan 13 17:43:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 9516127 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7A28360762 for ; Fri, 13 Jan 2017 17:43:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C80728745 for ; Fri, 13 Jan 2017 17:43:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6161328765; Fri, 13 Jan 2017 17:43:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CDFAE28745 for ; Fri, 13 Jan 2017 17:43:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751086AbdAMRn2 (ORCPT ); Fri, 13 Jan 2017 12:43:28 -0500 Received: from mail-it0-f65.google.com ([209.85.214.65]:32907 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750947AbdAMRn1 (ORCPT ); Fri, 13 Jan 2017 12:43:27 -0500 Received: by mail-it0-f65.google.com with SMTP id v14so5807840itb.0; Fri, 13 Jan 2017 09:43:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=e5K67J/G1KfHQdrdDeBV1P5pCaJ6XSPNbZ2T/Gn+nmY=; b=bt9IiJpUlLBG3BZ+jJT7iE+L74LrYWCvcvGXw4XLZKa2RtZ6mKdWs8LLEdPYT2b2SD hoO4Q134OYXCJfBA21JjccXrHLJNk0A+Q20LIukcPkEUOcwHGx6zMAw7rt+lTEgiWWRf ZQL9dF6N8NeBNpuYD+WGbd+fUjUTgUp/4rhqOExfxfPqyzAiuSpk3jJtcF5dK7DP8gq0 x0K+5YRMMcvDnir4CQ4eFBAzalwuluXhXpFu10pgFlFq5jCT4iuOlSlidA5KbYLwQ7yV HW3xkSK4f6YR0CJBhHh1Zy8Ul12D+YwZgioQB7aakeFk0aq41GymiA4rEhKmIuahLexB vJbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=e5K67J/G1KfHQdrdDeBV1P5pCaJ6XSPNbZ2T/Gn+nmY=; b=ljVB9iJ+2zZWsZ6zP+J+BVnSdeQSAuqRmwswujCA4PA70LB3zbJUtHQfv+XjM4Zua3 JT44CVkHcU1Rn/8orMRmnhZPnG6Pc3Sd4agx9eZiSN7Lf5Mgq8AU3qozXT3NZzFqkjBq 0rQbsn0UpD35JytgB6ibl6nQTrY74a8tIgp9B1RZ2XPuknUkldP10H/h26p5Qt911wC6 W5Ko3qBdBY79MFGdRmMe5J0AqFOMRa5ehM9/GbMPWlL/BEocPRzydp7mu1buhlszVQKK ZVdKqT6xI5da+dwJ840iVIafBD02kyxsP8jsW9O71zQaWlIbEfVKZeEsy+/h3v9BQLzn H8Xg== X-Gm-Message-State: AIkVDXJaiV33EXpzAZLlRpnxSHFyLn9m1soLdWBEFzbQbzQ4VRE9j5kVPMOV95kLI6w7uQ== X-Received: by 10.36.4.2 with SMTP id 2mr3445535itb.116.1484329403472; Fri, 13 Jan 2017 09:43:23 -0800 (PST) Received: from manet.1015granger.net ([2604:8800:100:81fc:ec4:7aff:fe6c:1dce]) by smtp.gmail.com with ESMTPSA id 35sm6906859iot.6.2017.01.13.09.43.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Jan 2017 09:43:23 -0800 (PST) Subject: [PATCH v1 4/5] xprtrdma: Reduce required number of send SGEs From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Fri, 13 Jan 2017 12:43:22 -0500 Message-ID: <20170113174322.32692.66126.stgit@manet.1015granger.net> In-Reply-To: <20170113173023.32692.30661.stgit@manet.1015granger.net> References: <20170113173023.32692.30661.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The MAX_SEND_SGES check introduced in commit 655fec6987be ("xprtrdma: Use gathered Send for large inline messages") fails for devices that have a small max_sge. Instead of checking for a large fixed maximum number of SGEs, check for a minimum small number. RPC-over-RDMA will switch to using a Read chunk if an xdr_buf has more pages than can fit in the device's max_sge limit. This is better than failing all together to mount the server. This fix supports devices that have as few as three send SGEs available. Reported-By: Selvin Xavier Reported-By: Devesh Sharma Reported-by: Honggang Li Reported-by: Ram Amrani Fixes: 655fec6987be ("xprtrdma: Use gathered Send for large ...") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 23 ++++++++++++++++++++++- net/sunrpc/xprtrdma/verbs.c | 13 +++++++------ net/sunrpc/xprtrdma/xprt_rdma.h | 1 + 3 files changed, 30 insertions(+), 7 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 4909758..ab699f9 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -126,13 +126,34 @@ void rpcrdma_set_max_header_sizes(struct rpcrdma_xprt *r_xprt) * plus the RPC call fit under the transport's inline limit. If the * combined call message size exceeds that limit, the client must use * the read chunk list for this operation. + * + * A read chunk is also required if sending the RPC call inline would + * exceed this device's max_sge limit. */ static bool rpcrdma_args_inline(struct rpcrdma_xprt *r_xprt, struct rpc_rqst *rqst) { + struct xdr_buf *xdr = &rqst->rq_snd_buf; struct rpcrdma_ia *ia = &r_xprt->rx_ia; + unsigned int count, remaining, offset; + + if (xdr->len > ia->ri_max_inline_write) + return false; + + if (xdr->page_len) { + remaining = xdr->page_len; + offset = xdr->page_base & ~PAGE_MASK; + count = 0; + while (remaining) { + remaining -= min_t(unsigned int, + PAGE_SIZE - offset, remaining); + offset = 0; + if (++count > ia->ri_max_sgeno) + return false; + } + } - return rqst->rq_snd_buf.len <= ia->ri_max_inline_write; + return true; } /* The client can't know how large the actual reply will be. Thus it diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 12e8242..5dcdd0b 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -488,18 +488,19 @@ static void rpcrdma_destroy_id(struct rdma_cm_id *id) */ int rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, - struct rpcrdma_create_data_internal *cdata) + struct rpcrdma_create_data_internal *cdata) { struct rpcrdma_connect_private *pmsg = &ep->rep_cm_private; + unsigned int max_qp_wr, max_sge; struct ib_cq *sendcq, *recvcq; - unsigned int max_qp_wr; int rc; - if (ia->ri_device->attrs.max_sge < RPCRDMA_MAX_SEND_SGES) { - dprintk("RPC: %s: insufficient sge's available\n", - __func__); + max_sge = min(ia->ri_device->attrs.max_sge, RPCRDMA_MAX_SEND_SGES); + if (max_sge < 3) { + pr_warn("rpcrdma: HCA provides only %d send SGEs\n", max_sge); return -ENOMEM; } + ia->ri_max_sgeno = max_sge - 3; if (ia->ri_device->attrs.max_qp_wr <= RPCRDMA_BACKWARD_WRS) { dprintk("RPC: %s: insufficient wqe's available\n", @@ -524,7 +525,7 @@ static void rpcrdma_destroy_id(struct rdma_cm_id *id) ep->rep_attr.cap.max_recv_wr = cdata->max_requests; ep->rep_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS; ep->rep_attr.cap.max_recv_wr += 1; /* drain cqe */ - ep->rep_attr.cap.max_send_sge = RPCRDMA_MAX_SEND_SGES; + ep->rep_attr.cap.max_send_sge = max_sge; ep->rep_attr.cap.max_recv_sge = 1; ep->rep_attr.cap.max_inline_data = 0; ep->rep_attr.sq_sig_type = IB_SIGNAL_REQ_WR; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index f495df0c..c134d0b 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -74,6 +74,7 @@ struct rpcrdma_ia { unsigned int ri_max_frmr_depth; unsigned int ri_max_inline_write; unsigned int ri_max_inline_read; + unsigned int ri_max_sgeno; bool ri_reminv_expected; bool ri_implicit_padding; enum ib_mr_type ri_mrtype;