From patchwork Fri Jan 13 17:43:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 9516119 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CEBDD60762 for ; Fri, 13 Jan 2017 17:43:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF2D6286E3 for ; Fri, 13 Jan 2017 17:43:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B455D2875A; Fri, 13 Jan 2017 17:43:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43B30286E3 for ; Fri, 13 Jan 2017 17:43:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751063AbdAMRnJ (ORCPT ); Fri, 13 Jan 2017 12:43:09 -0500 Received: from mail-it0-f68.google.com ([209.85.214.68]:34871 "EHLO mail-it0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750969AbdAMRnI (ORCPT ); Fri, 13 Jan 2017 12:43:08 -0500 Received: by mail-it0-f68.google.com with SMTP id 203so5787956ith.2; Fri, 13 Jan 2017 09:43:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=v+mZga3OmCDUPxNFUAUB2E52BUpn2MUU5xmIQN8no/4=; b=q4hKWSxRe0i6Snuo4l7ReAbcCSpafJsGfLYstAria6+0Tz6luucYFpl/Xuq9JxUfPV m6DubDq6DbOq37jVsEdepbjoBAvQO/m8U+ccDVIKBlw8tGIIMs/sgRPVF8tBg+lKkFiI JJw2NTDvFOF/4C0ljWno0X5yASyMlpDkVClHcH+VeLKrYZPmbZsGnWpmF/jslkzK02FA 6zD/pyjXhsFQqDdNVoa2tXh+DlrtKqXgOW3RL28MENXFzC80kLvVpJTD0X0w2JtAAL36 jY4q8ieQCoN5zXqSNoHaC3JVxXD2yXQIAMB8H8jz73ESqmqrX0wFvOczG9AiQKdr6vlk 5EMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=v+mZga3OmCDUPxNFUAUB2E52BUpn2MUU5xmIQN8no/4=; b=sPUKV80d6u0yfTA5x0ygPZqsKzaeDEZMR2V7Okgqk8k3sXKEbKXfX0WAPpDfzxpWiL 9sVCmHaDUa0ecwkV26AGrpQy7LJ0jiIPdTniLbWFwXdOmlxTlWukF596DcEBoT4Mkf/I BqJvNhJZs5wmMtEWBPaJjnEU0lvFCvZ2QOQhGgqH7I5eGUgBznfXJI0BhA4ncUeT88HX WaNdgmuAF4FtwYGNvaJmV//+25g5uVfZKNHDwu8dh3tG+rMSN7+ssF49mN7mIYe1fOTp YhYS5nJKuiuqmQAfEegNaiXT++SkDj6QRbGB5UxKpzCbSdzXxBygVpX4RTQBjnT2wmB2 SsfQ== X-Gm-Message-State: AIkVDXJNMAkSXCeXTMUwD1TRFRuIupUcCJH/Gow0NFPSXvr/E3kKQpzD5Via6wr1l4G4Nw== X-Received: by 10.36.252.65 with SMTP id b62mr3553242ith.38.1484329386986; Fri, 13 Jan 2017 09:43:06 -0800 (PST) Received: from manet.1015granger.net ([2604:8800:100:81fc:ec4:7aff:fe6c:1dce]) by smtp.gmail.com with ESMTPSA id n3sm1226766itb.2.2017.01.13.09.43.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Jan 2017 09:43:06 -0800 (PST) Subject: [PATCH v1 2/5] xprtrdma: Per-connection pad optimization From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Fri, 13 Jan 2017 12:43:06 -0500 Message-ID: <20170113174305.32692.50131.stgit@manet.1015granger.net> In-Reply-To: <20170113173023.32692.30661.stgit@manet.1015granger.net> References: <20170113173023.32692.30661.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Pad optimization is changed by echoing into /proc/sys/sunrpc/rdma_pad_optimize. This is a global setting, affecting all RPC-over-RDMA connections to all servers. The marshaling code picks up that value and uses it for decisions about how to construct each RPC-over-RDMA frame. Having it change suddenly in mid-operation can result in unexpected failures. And some servers a client mounts may need pads, while others don't. So instead, copy the setting into each connection's rpcrdma_ia at mount time, and use the copy, which can't change during the life of the connection. This also removes a hack: rpcrdma_convert_iovs was using the remote-invalidation-expected flag to predict when it could leave out Write chunk padding. This is because the Linux server handles implicit XDR padding on Write chunks correctly, and only Linux servers can set the connection's remote-invalidation-expected flag. It's more sensible to use the pad optimization setting instead. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 28 ++++++++++++++-------------- net/sunrpc/xprtrdma/verbs.c | 1 + net/sunrpc/xprtrdma/xprt_rdma.h | 1 + 3 files changed, 16 insertions(+), 14 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index a524d3c..4909758 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -186,9 +186,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, */ static int -rpcrdma_convert_iovs(struct xdr_buf *xdrbuf, unsigned int pos, - enum rpcrdma_chunktype type, struct rpcrdma_mr_seg *seg, - bool reminv_expected) +rpcrdma_convert_iovs(struct rpcrdma_xprt *r_xprt, struct xdr_buf *xdrbuf, + unsigned int pos, enum rpcrdma_chunktype type, + struct rpcrdma_mr_seg *seg) { int len, n, p, page_base; struct page **ppages; @@ -229,14 +229,15 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, /* When encoding a Read chunk, the tail iovec contains an * XDR pad and may be omitted. */ - if (type == rpcrdma_readch && xprt_rdma_pad_optimize) + if (type == rpcrdma_readch && r_xprt->rx_ia.ri_implicit_padding) return n; - /* When encoding the Write list, some servers need to see an extra - * segment for odd-length Write chunks. The upper layer provides - * space in the tail iovec for this purpose. + /* When encoding a Write chunk, some servers need to see an + * extra segment for non-XDR-aligned Write chunks. The upper + * layer provides space in the tail iovec that may be used + * for this purpose. */ - if (type == rpcrdma_writech && reminv_expected) + if (type == rpcrdma_writech && r_xprt->rx_ia.ri_implicit_padding) return n; if (xdrbuf->tail[0].iov_len) { @@ -291,7 +292,8 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (rtype == rpcrdma_areadch) pos = 0; seg = req->rl_segments; - nsegs = rpcrdma_convert_iovs(&rqst->rq_snd_buf, pos, rtype, seg, false); + nsegs = rpcrdma_convert_iovs(r_xprt, &rqst->rq_snd_buf, pos, + rtype, seg); if (nsegs < 0) return ERR_PTR(nsegs); @@ -353,10 +355,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, } seg = req->rl_segments; - nsegs = rpcrdma_convert_iovs(&rqst->rq_rcv_buf, + nsegs = rpcrdma_convert_iovs(r_xprt, &rqst->rq_rcv_buf, rqst->rq_rcv_buf.head[0].iov_len, - wtype, seg, - r_xprt->rx_ia.ri_reminv_expected); + wtype, seg); if (nsegs < 0) return ERR_PTR(nsegs); @@ -421,8 +422,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, } seg = req->rl_segments; - nsegs = rpcrdma_convert_iovs(&rqst->rq_rcv_buf, 0, wtype, seg, - r_xprt->rx_ia.ri_reminv_expected); + nsegs = rpcrdma_convert_iovs(r_xprt, &rqst->rq_rcv_buf, 0, wtype, seg); if (nsegs < 0) return ERR_PTR(nsegs); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 11d0774..890cb3a 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -208,6 +208,7 @@ /* Default settings for RPC-over-RDMA Version One */ r_xprt->rx_ia.ri_reminv_expected = false; + r_xprt->rx_ia.ri_implicit_padding = xprt_rdma_pad_optimize; rsize = RPCRDMA_V1_DEF_INLINE_SIZE; wsize = RPCRDMA_V1_DEF_INLINE_SIZE; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index e35efd4..f495df0c 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -75,6 +75,7 @@ struct rpcrdma_ia { unsigned int ri_max_inline_write; unsigned int ri_max_inline_read; bool ri_reminv_expected; + bool ri_implicit_padding; enum ib_mr_type ri_mrtype; struct ib_qp_attr ri_qp_attr; struct ib_qp_init_attr ri_qp_init_attr;