From patchwork Tue May 28 18:20:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAB186C5 for ; Tue, 28 May 2019 18:21:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9825228640 for ; Tue, 28 May 2019 18:21:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B33B2864E; Tue, 28 May 2019 18:21:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 326E128640 for ; Tue, 28 May 2019 18:21:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728028AbfE1SU6 (ORCPT ); Tue, 28 May 2019 14:20:58 -0400 Received: from mail-it1-f196.google.com ([209.85.166.196]:40909 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SU5 (ORCPT ); Tue, 28 May 2019 14:20:57 -0400 Received: by mail-it1-f196.google.com with SMTP id h11so5487791itf.5; Tue, 28 May 2019 11:20:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=q4ANcePUORAcvpLHm6FS75fc8hbBquDvs08Hlz+shUc=; b=XESmdCYmTp/dY9FmD0w4v39ONKaXE/6wArAyHu3etrGaUUFzlMJuQ3Rc2GEe3Fdcan 0LHplpUmF3BC1xrT6TO9ItTZVRxEcxbKtozBDcIPEd9Xpf3ZXLXXKb1CzJ63fZiV7BBa 3pB53q3ZWWFEkTQkhV5oQvOfsF+hutizCw3QDkyGufdKkEK0S10j+5H+s/w1UnEuQV0Y J5w/YpBppDxhX6JUxLzVfajfA8DHRidOAL85X6Zy1uV4ecOxjt5pl9eQmuRiOU8qeJ1d f37XfvrB2og0aJResyG7CPQmAB9Y9fjTmAI9t47o0r48h0aqlh786uVw0bohYadEpXOn hOTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=q4ANcePUORAcvpLHm6FS75fc8hbBquDvs08Hlz+shUc=; b=pvtt1lQz5DuLVIDnPeNZ9FXFhAPcMWyYWNrxUTV+wsv8D7Kv/kdOOY0KuuEQlbGde9 AM9aMhl4IVNT/UCEhe+SmqagT7pzD63m1SQL5fQyLkHrEITpJ69JXDZbOUBdBkSR8enz 9f878/i4QwFfWAIJUmhITj1lF7QJ4yppmc9OtQCvLPEjawmF+OwngQDbepSjH7SMcwve ZErhv3HB3MTU0zYhLUGWRU6oUsRuGCq46jLIK5vL9PCCV6gGSGg4QuVmCJblDs3gNtCe PwDS5d44t4hBMVLi2+lcoPYBXZE03OEa/ltv/NZ9lPNbOUJMnV8RQZ81rlPCmvdAvU6W CZ5g== X-Gm-Message-State: APjAAAW3PNBG880ydosLV5wDld1H6qkTSat103bf+mnZsjhA4x207lqR xnz1TAdfgBbmf5w5DaKPW8iNVV9K X-Google-Smtp-Source: APXvYqww4W0RM1tXa00sxi9KrAimw2Bnv+VX2qZAmOWdvUrC9+l+CWSE2WXtotTzo6CsJ7+3s2dQ1A== X-Received: by 2002:a02:3506:: with SMTP id k6mr3850184jaa.41.1559067656622; Tue, 28 May 2019 11:20:56 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id u187sm861455iod.37.2019.05.28.11.20.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:20:56 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SIKt17014516; Tue, 28 May 2019 18:20:55 GMT Subject: [PATCH RFC 01/12] xprtrdma: Fix use-after-free in rpcrdma_post_recvs From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:20:55 -0400 Message-ID: <20190528182055.19012.95728.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Dereference wr->next /before/ the memory backing wr has been released. This issue was found by code inspection. It is not expected to be a significant problem because it is in an error path that is almost never executed. Fixes: 7c8d9e7c8863 ("xprtrdma: Move Receive posting to ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index bef5eac..a803e94 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1554,10 +1554,11 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) rc = ib_post_recv(r_xprt->rx_ia.ri_id->qp, wr, (const struct ib_recv_wr **)&bad_wr); if (rc) { - for (wr = bad_wr; wr; wr = wr->next) { + for (wr = bad_wr; wr;) { struct rpcrdma_rep *rep; rep = container_of(wr, struct rpcrdma_rep, rr_recv_wr); + wr = wr->next; rpcrdma_recv_buffer_put(rep); --count; } From patchwork Tue May 28 18:21:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965617 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12B206C5 for ; Tue, 28 May 2019 18:21:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 02E6428640 for ; Tue, 28 May 2019 18:21:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EBACF2864E; Tue, 28 May 2019 18:21:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 987B328640 for ; Tue, 28 May 2019 18:21:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728030AbfE1SVD (ORCPT ); Tue, 28 May 2019 14:21:03 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:36182 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVC (ORCPT ); Tue, 28 May 2019 14:21:02 -0400 Received: by mail-it1-f194.google.com with SMTP id e184so5513589ite.1; Tue, 28 May 2019 11:21:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=DuGm7Y8gW91gv6AuiLt2GdkmCz1p6fKJf4IyLcPAavM=; b=iXS6FAHYbpthazRI+rDMbtBCsC2AFlq/b5nY7bQa5os9MEc5bWcFNIassiS7A5avlK +oEHffKncbwNz7K/q7/kVtqQSEFOx5eaoclI6WmDwTX9xX7CwIgTqazz8bPQ1raaD/AX MyDw8dQXnSLdpIHLrt8i0a1gQbCJCU8a/yrZyQo1fG8yPhDFsOYqduDdXGbMPWERQTzt 9x6ZqPMQytNr6KIbJJNLDWinTZNMGvrw4SkfKLgCAt8fTxy9naikWuqitPtAQhbkzRja W4H/4j3w9oqKt5sRRN9ZClQNLkwSmtvB+crcU1ynfA3fQyMk/NVgNQ6E/YTFk/BFNgof 5d+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=DuGm7Y8gW91gv6AuiLt2GdkmCz1p6fKJf4IyLcPAavM=; b=NOG91hOTdTGe4K2V/VPR5kYFb4JQxWrTDmJ9L7C9kfUC2jNObcmV5/TA7l5W0eHFDi YgRUWFBBqpNIW/r84Pp/W9spXKOAwpfthQJ9Dnc7VfZt89cTaLpBSvblXmHuVckoMPsF S9OrQ96op9G1nn74YUoJDqJgivaSCl+HAUju3WW8b+flb19vV30yAD9OoJFK8QNs2q2n b9QgaIAa9gaamGYf/vyfD8NlU1mNCpyI/xOGFVU5NELqqjiSqu/tYfNg/DcVEfzo/W9m c2IrECPQ0pNpOvK5Kga/Tb3rfPEIH7efLgmhZVXFagJTonE6Inm8FDDgGvjGCOZrS8vx g7HQ== X-Gm-Message-State: APjAAAXzqJ5GU/653i+in1yfLQtHr7WowAtNO3pv+1Ar7fmsu1d9OiFC iAiK+rFw/nZmmXPnVpMmOGRQg8Hq X-Google-Smtp-Source: APXvYqxVTx6qZDfRAEOi0SziG5VIx5P350vVDq6hnm2ndrX3DxtnsEtlZAts+w2j3gwXm9/tqmqH/w== X-Received: by 2002:a02:ce50:: with SMTP id y16mr4961341jar.75.1559067661973; Tue, 28 May 2019 11:21:01 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id m67sm1514510itc.22.2019.05.28.11.21.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:01 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SIL08D014519; Tue, 28 May 2019 18:21:00 GMT Subject: [PATCH RFC 02/12] xprtrdma: Replace use of xdr_stream_pos in rpcrdma_marshal_req From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:00 -0400 Message-ID: <20190528182100.19012.59793.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is a latent bug. xdr_stream_pos works by subtracting xdr_stream::nwords from xdr_buf::len. But xdr_stream::nwords is not initialized by xdr_init_encode(). It works today only because all fields in rpcrdma_req::rl_stream are initialized to zero by rpcrdma_req_create, making the subtraction in xdr_stream_pos always a no-op. I found this issue via code inspection. It was introduced by commit 39f4cd9e9982 ("xprtrdma: Harden chunk list encoding against send buffer overflow"), but the code has changed enough since then that this fix can't be automatically applied to stable. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 85115a2..608b9ac 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -867,9 +867,9 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, if (ret) goto out_err; - trace_xprtrdma_marshal(rqst, xdr_stream_pos(xdr), rtype, wtype); + trace_xprtrdma_marshal(rqst, req->rl_hdrbuf.len, rtype, wtype); - ret = rpcrdma_prepare_send_sges(r_xprt, req, xdr_stream_pos(xdr), + ret = rpcrdma_prepare_send_sges(r_xprt, req, req->rl_hdrbuf.len, &rqst->rq_snd_buf, rtype); if (ret) goto out_err; From patchwork Tue May 28 18:21:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965621 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 625CA13AD for ; Tue, 28 May 2019 18:21:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5095528640 for ; Tue, 28 May 2019 18:21:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 452812865F; Tue, 28 May 2019 18:21:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 25E7728640 for ; Tue, 28 May 2019 18:21:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728031AbfE1SVI (ORCPT ); Tue, 28 May 2019 14:21:08 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:39784 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVI (ORCPT ); Tue, 28 May 2019 14:21:08 -0400 Received: by mail-it1-f193.google.com with SMTP id 9so5493264itf.4; Tue, 28 May 2019 11:21:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=1OszlyJLb8eGDXIDH++nzznHYMFVhNicKOFYHEfYLnk=; b=JYWB86EySmWhnyFauftikS6YmAbsuu7Q6ETLnBbI9rAa1kWCRNPPT3dsT+Okn4LK+Z dGAF6bL+AW1DFEk0JXblKQF+bd/1nBZfRf5Q1ilOBRhS/Y4uEvZw9ZHlKXSPBIYfkd0z zmEgV+s6+ukny0u3i1MWPWpdej4bK+2ojjxw2jE18d5UHdNgAZJDFmF5RjyeoTMOCWL2 lw4b0Pi0NLMFedKSEJIJZw4YW4iF3MifEAVBkhl04C5ljg3RFLM4d5faATtJHZEGgYP+ qbXXIapO/2NwqkfSY0JiMCszmeriHUhcFM2gJrSD03BsNAmp2kHHmxKsELimeNogdRFf bOrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=1OszlyJLb8eGDXIDH++nzznHYMFVhNicKOFYHEfYLnk=; b=TaH6zPMBfXOB5e14fUGQKXwH/7xU0iZAtuJBtIzmZneqiGC1rhDReHRyLQfdvUg54D 1qEK2+L2N880mdyTbAVgPRjoZDXgxyP3ul9YrAICoUrG8K1+JSXZ12z9HBp0zk2uM+14 Wqquz64wt/GVGwXFenAihyWHV6JhVncZkA4IygzoPoekQSL/gW3Ujj5pR22IsjECFN6W 488wrIiT4lcTjyq5RwHixqcLxXH6TPuoiZZeTvmN7NecINjXm4QXBgV+seOgQVyneDCS CNA7TQ8I9jSrqOnpVpnn34kduyhS9zQxEqt2SkjKdDHSDm0t37mFZNzsl5vaXGxHt/1w 2lWQ== X-Gm-Message-State: APjAAAWvR5fCT/bDd2QRYpA421YylHnQYiz6nR7GZIJn8xMy7VamnOr7 5HtlzlQaqFfbkp3ySnJDoDyD7VK9 X-Google-Smtp-Source: APXvYqzl1i3Q6IhIZG90e/cPy3yPKC4J9yPSXMt6ZVrKG913ETeN61RBHPZ3HsHzPNaJ/AUZrEDHRQ== X-Received: by 2002:a24:b30b:: with SMTP id e11mr3431049itf.99.1559067667439; Tue, 28 May 2019 11:21:07 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id w26sm4115479iom.59.2019.05.28.11.21.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:07 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SIL6Yo014522; Tue, 28 May 2019 18:21:06 GMT Subject: [PATCH RFC 03/12] xprtrdma: Fix occasional transport deadlock From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:06 -0400 Message-ID: <20190528182106.19012.43122.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Under high I/O workloads, I've noticed that an RPC/RDMA transport occasionally deadlocks (IOPS goes to zero, and doesn't recover). Diagnosis shows that the sendctx queue is empty, but when sendctxs are returned to the queue, the xprt_write_space wake-up never occurs. The wake-up logic in rpcrdma_sendctx_put_locked is racy. I noticed that both EMPTY_SCQ and XPRT_WRITE_SPACE are implemented via an atomic bit. Just one of those is sufficient. Removing EMPTY_SCQ in favor of the generic bit mechanism makes the deadlock un-reproducible. Without EMPTY_SCQ, rpcrdma_buffer::rb_flags is no longer used and is therefore removed. Unfortunately this patch does not apply cleanly to stable. If needed, someone will have to port it and test it. Fixes: 2fad659209d5 ("xprtrdma: Wait on empty sendctx queue") Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 27 +++++++++++++++++++++++++++ net/sunrpc/xprtrdma/frwr_ops.c | 6 +++++- net/sunrpc/xprtrdma/rpc_rdma.c | 26 ++++++++++++-------------- net/sunrpc/xprtrdma/verbs.c | 11 +++-------- net/sunrpc/xprtrdma/xprt_rdma.h | 6 ------ 5 files changed, 47 insertions(+), 29 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index df9851cb..a4ab3a2 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -538,6 +538,33 @@ ) ); +TRACE_EVENT(xprtrdma_prepsend_failed, + TP_PROTO(const struct rpc_rqst *rqst, + int ret + ), + + TP_ARGS(rqst, ret), + + TP_STRUCT__entry( + __field(unsigned int, task_id) + __field(unsigned int, client_id) + __field(u32, xid) + __field(int, ret) + ), + + TP_fast_assign( + __entry->task_id = rqst->rq_task->tk_pid; + __entry->client_id = rqst->rq_task->tk_client->cl_clid; + __entry->xid = be32_to_cpu(rqst->rq_xid); + __entry->ret = ret; + ), + + TP_printk("task:%u@%u xid=0x%08x: ret=%d", + __entry->task_id, __entry->client_id, __entry->xid, + __entry->ret + ) +); + TRACE_EVENT(xprtrdma_post_send, TP_PROTO( const struct rpcrdma_req *req, diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 794ba4c..ac47314 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -391,7 +391,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, rpcrdma_mr_recycle(mr); mr = rpcrdma_mr_get(r_xprt); if (!mr) - return ERR_PTR(-EAGAIN); + goto out_getmr_err; } while (mr->frwr.fr_state != FRWR_IS_INVALID); frwr = &mr->frwr; frwr->fr_state = FRWR_IS_VALID; @@ -448,6 +448,10 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, *out = mr; return seg; +out_getmr_err: + xprt_wait_for_buffer_space(&r_xprt->rx_xprt); + return ERR_PTR(-EAGAIN); + out_dmamap_err: mr->mr_dir = DMA_NONE; trace_xprtrdma_frwr_sgerr(mr, i); diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 608b9ac..d1f44f4 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -699,22 +699,28 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, u32 hdrlen, struct xdr_buf *xdr, enum rpcrdma_chunktype rtype) { + int ret; + + ret = -EAGAIN; req->rl_sendctx = rpcrdma_sendctx_get_locked(r_xprt); if (!req->rl_sendctx) - return -EAGAIN; + goto err; req->rl_sendctx->sc_wr.num_sge = 0; req->rl_sendctx->sc_unmap_count = 0; req->rl_sendctx->sc_req = req; __clear_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags); + ret = -EIO; if (!rpcrdma_prepare_hdr_sge(r_xprt, req, hdrlen)) - return -EIO; - + goto err; if (rtype != rpcrdma_areadch) if (!rpcrdma_prepare_msg_sges(r_xprt, req, xdr, rtype)) - return -EIO; - + goto err; return 0; + +err: + trace_xprtrdma_prepsend_failed(&req->rl_slot, ret); + return ret; } /** @@ -877,15 +883,7 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, out_err: trace_xprtrdma_marshal_failed(rqst, ret); - switch (ret) { - case -EAGAIN: - xprt_wait_for_buffer_space(rqst->rq_xprt); - break; - case -ENOBUFS: - break; - default: - r_xprt->rx_stats.failed_marshal_count++; - } + r_xprt->rx_stats.failed_marshal_count++; return ret; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index a803e94..78be985 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -902,7 +902,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) * completions recently. This is a sign the Send Queue is * backing up. Cause the caller to pause and try again. */ - set_bit(RPCRDMA_BUF_F_EMPTY_SCQ, &buf->rb_flags); + xprt_wait_for_buffer_space(&r_xprt->rx_xprt); r_xprt->rx_stats.empty_sendctx_q++; return NULL; } @@ -937,10 +937,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) /* Paired with READ_ONCE */ smp_store_release(&buf->rb_sc_tail, next_tail); - if (test_and_clear_bit(RPCRDMA_BUF_F_EMPTY_SCQ, &buf->rb_flags)) { - smp_mb__after_atomic(); - xprt_write_space(&sc->sc_xprt->rx_xprt); - } + xprt_write_space(&sc->sc_xprt->rx_xprt); } static void @@ -978,8 +975,6 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) r_xprt->rx_stats.mrs_allocated += count; spin_unlock(&buf->rb_mrlock); trace_xprtrdma_createmrs(r_xprt, count); - - xprt_write_space(&r_xprt->rx_xprt); } static void @@ -991,6 +986,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) rx_buf); rpcrdma_mrs_create(r_xprt); + xprt_write_space(&r_xprt->rx_xprt); } /** @@ -1090,7 +1086,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) struct rpcrdma_buffer *buf = &r_xprt->rx_buf; int i, rc; - buf->rb_flags = 0; buf->rb_max_requests = r_xprt->rx_ep.rep_max_requests; buf->rb_bc_srv_max_requests = 0; spin_lock_init(&buf->rb_mrlock); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index d1e0749..2c6c5d8 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -391,7 +391,6 @@ struct rpcrdma_buffer { struct list_head rb_recv_bufs; struct list_head rb_allreqs; - unsigned long rb_flags; u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ @@ -402,11 +401,6 @@ struct rpcrdma_buffer { struct delayed_work rb_refresh_worker; }; -/* rb_flags */ -enum { - RPCRDMA_BUF_F_EMPTY_SCQ = 0, -}; - /* * Statistics for RPCRDMA */ From patchwork Tue May 28 18:21:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965625 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A04456C5 for ; Tue, 28 May 2019 18:21:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DA2328640 for ; Tue, 28 May 2019 18:21:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8236E28657; Tue, 28 May 2019 18:21:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2044628640 for ; Tue, 28 May 2019 18:21:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728036AbfE1SVO (ORCPT ); Tue, 28 May 2019 14:21:14 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:38341 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727976AbfE1SVO (ORCPT ); Tue, 28 May 2019 14:21:14 -0400 Received: by mail-io1-f67.google.com with SMTP id x24so16569483ion.5; Tue, 28 May 2019 11:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=dtCtmMjHsjE9+kkAKoJ+QHjUK+M5LzkEDCShi/9x3lE=; b=KKr4p9YYrbTqgQDAGj/Bpx3x5ZsFeg53mO1YqlqIoxdDnILG+Pkd3obctftnPBqd4s xk19tyMZ1G1CjbjUS4icpPUsTvC9L+u8krCIkfUQkasKYQacr/9+HJeBxXuOmSxTaycu ecusTphT78ri4+7z5pMcdAihceMYOwZKZnl6/BbgU1zR5bNfFLpkL6GO5lJ76MHnsDjd FNQ03kWkVXaIEWeaB+hvdurg46vYPZ2jNPR0jPTmygk/ohGx+W5rn6HSz2Xg++DpYOJd turwkKWWuKEHgCdo78xuaTuk0Qqel8qQxrn3wH58mrhPvaGDoNTQsUCx5Lp/0eRgEWKd h8rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=dtCtmMjHsjE9+kkAKoJ+QHjUK+M5LzkEDCShi/9x3lE=; b=JikJsUkSMNTiC8QGc5hJEv/wTFUoXRpzL+VM591Jdgbykqwjuee1Redvn3CF11bpy+ CjN9Szf0GKogug1vStO31eHesOnXxqtZ7GvAisE1OxMCcwL9snlGkdtRdS2l3rVtACLn N5gMwXdrePgn+hZ6//Vzp+MZqcyHzK/bGQHCPCr8IJZP3JYqQMeC/a9FIxYN1ncnfn8F idIXtVkjycFZrmBmr8U+n9FnGA+aWq/8fGAlNxxhq81m9HvqtnYYH8YPv3VvAIp+EnJK Y7YK7lfqRiGbHSuoVhfDGg41A7QQ19rdYkuj6woaPwvoiEKbaJTfkB8fbAuDX0QVVmTq f3Jw== X-Gm-Message-State: APjAAAWW3eWQ15Gzcb+4UgM42ggXCmAVEjys2kaQkeIuwqeqzoBGMkgj kCCm+nQpbtfOShqU5BV5H3MGnkan X-Google-Smtp-Source: APXvYqyWWLW4/XiAWJcCtmCPFolMjm/f3NtCRzjFBGD9PsflA5Vx6U/e5UqC3efLgyJFr4mFKSL9Cg== X-Received: by 2002:a05:6602:1d6:: with SMTP id w22mr20582789iot.215.1559067673208; Tue, 28 May 2019 11:21:13 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id q72sm1592587ita.26.2019.05.28.11.21.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:12 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILBxI014525; Tue, 28 May 2019 18:21:11 GMT Subject: [PATCH RFC 04/12] xprtrdma: Remove the RPCRDMA_REQ_F_PENDING flag From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:11 -0400 Message-ID: <20190528182111.19012.50313.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit 9590d083c1bb ("xprtrdma: Use xprt_pin_rqst in rpcrdma_reply_handler") pins incoming RPC/RDMA replies so they can be left in the pending requests queue while they are being processed without introducing a race between ->buf_free and the transport's reply handler. Therefore RPCRDMA_REQ_F_PENDING is no longer necessary. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 1 - net/sunrpc/xprtrdma/transport.c | 4 +--- net/sunrpc/xprtrdma/xprt_rdma.h | 1 - 3 files changed, 1 insertion(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index d1f44f4..77fc1e4 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1371,7 +1371,6 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) } req->rl_reply = rep; rep->rr_rqst = rqst; - clear_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags); trace_xprtrdma_reply(rqst->rq_task, rep, req, credits); queue_work(buf->rb_completion_wq, &rep->rr_work); diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 1f73a6a..f84375d 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -618,8 +618,7 @@ static bool rpcrdma_check_regbuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(rqst->rq_xprt); struct rpcrdma_req *req = rpcr_to_rdmar(rqst); - if (test_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags)) - rpcrdma_release_rqst(r_xprt, req); + rpcrdma_release_rqst(r_xprt, req); trace_xprtrdma_op_free(task, req); } @@ -667,7 +666,6 @@ static bool rpcrdma_check_regbuf(struct rpcrdma_xprt *r_xprt, goto drop_connection; rqst->rq_xtime = ktime_get(); - __set_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags); if (rpcrdma_ep_post(&r_xprt->rx_ia, &r_xprt->rx_ep, req)) goto drop_connection; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 2c6c5d8..3c39aa3 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -344,7 +344,6 @@ struct rpcrdma_req { /* rl_flags */ enum { - RPCRDMA_REQ_F_PENDING = 0, RPCRDMA_REQ_F_TX_RESOURCES, }; From patchwork Tue May 28 18:21:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965629 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F40D76C5 for ; Tue, 28 May 2019 18:21:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0BC92864E for ; Tue, 28 May 2019 18:21:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D247328657; Tue, 28 May 2019 18:21:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D27B52864B for ; Tue, 28 May 2019 18:21:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728039AbfE1SVU (ORCPT ); Tue, 28 May 2019 14:21:20 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:51751 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVU (ORCPT ); Tue, 28 May 2019 14:21:20 -0400 Received: by mail-it1-f193.google.com with SMTP id m3so5983178itl.1; Tue, 28 May 2019 11:21:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=TfoKCNUVRcrQMRw6qSTU+DT1hc7JTFEqY6ELSFmrqbM=; b=KLNxdI5bbTTAMbIYjb1d+MwtbfErPuQjzh41ht5HOkM0LOkXFyRwTQfvHcy9Lg8XrR D92fV6vBS4reqr43+UbLVK0T7n9T60Q1SdDryYM4DBrV5SM+6zfbkCrazGVrEDQ12l2c Qk0+zxx0O2yaTNAqf1p0PNu2HuO4MeHL+T3rUeE1PvAEDF8Pu6xHbrRPad8Z8G1SZEvR ySJ1tQ9KL+Yu0lSvme14YfEr1xUaST8Gzd4AEa7D4n7zqw6iBXBwS334vB7LymRcxDJb 406D6sxHfGUWYAsY8XOWhygHZURH8jK+GKbgfUZTf0y+u+QdpinlCoDE/Se5a3+cCiaa OtHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=TfoKCNUVRcrQMRw6qSTU+DT1hc7JTFEqY6ELSFmrqbM=; b=t7f7sPSJJCgQWzLtTWc9fthidamOeBvHjBkOafOHfdD9A4xbyckREccLVTogoRugO2 AKIq9kGWkZScPlYikLTfeZJGtvl7/iv59XJmwvZ4FNnmuwjcjE4nlSG+1+CBRmVHXWpe Qw/6SAGup0e0XI/pAxehc9vGtpZjPxN4Tq++g6H1gaPoXili2cKM4JAY+gW0IhKB7orY INrw9xWo7N7E/4Foo1d3wmfddkza9jAxpRXTPsRBthsy1pzIb+tpcbGvibqjL6B1kf4D NLOChpYwPfvKFeSlzP6MCq7IVZapf006OLTH59Dj43qtiXuL0v/4jBnvr6FziVdnRmEC j32w== X-Gm-Message-State: APjAAAUy40ZY0vM4Y9Ta5h6h43/i4V/dByPSfRtVe2jWAHbbQtvRzGOr iQP45lHMypDbhI9tA1Zv5F+j899m X-Google-Smtp-Source: APXvYqyucteB0lN4lXcFWrna82Tz0MklW5mMDQCfgjKW/Hg/pVD/mkyiSM8m/3vi630n853v5ycw8A== X-Received: by 2002:a24:d486:: with SMTP id x128mr4191518itg.80.1559067678567; Tue, 28 May 2019 11:21:18 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id b196sm1568738itb.30.2019.05.28.11.21.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:18 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILHg8014528; Tue, 28 May 2019 18:21:17 GMT Subject: [PATCH RFC 05/12] xprtrdma: Remove fr_state From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:17 -0400 Message-ID: <20190528182116.19012.50268.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since both the Send and Receive completion queues are processed in a workqueue context, it should be safe to DMA unmap and return MRs to the free or recycle lists directly in the completion handlers. Doing this means rpcrdma_frwr no longer needs to track the state of each MR... a VALID or FLUSHED MR can no longer appear on an xprt's MR free list. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 19 ---- net/sunrpc/xprtrdma/frwr_ops.c | 202 ++++++++++++++++++--------------------- net/sunrpc/xprtrdma/rpc_rdma.c | 2 net/sunrpc/xprtrdma/xprt_rdma.h | 11 -- 4 files changed, 95 insertions(+), 139 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index a4ab3a2..7fe21ba 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -181,18 +181,6 @@ ), \ TP_ARGS(task, mr, nsegs)) -TRACE_DEFINE_ENUM(FRWR_IS_INVALID); -TRACE_DEFINE_ENUM(FRWR_IS_VALID); -TRACE_DEFINE_ENUM(FRWR_FLUSHED_FR); -TRACE_DEFINE_ENUM(FRWR_FLUSHED_LI); - -#define xprtrdma_show_frwr_state(x) \ - __print_symbolic(x, \ - { FRWR_IS_INVALID, "INVALID" }, \ - { FRWR_IS_VALID, "VALID" }, \ - { FRWR_FLUSHED_FR, "FLUSHED_FR" }, \ - { FRWR_FLUSHED_LI, "FLUSHED_LI" }) - DECLARE_EVENT_CLASS(xprtrdma_frwr_done, TP_PROTO( const struct ib_wc *wc, @@ -203,22 +191,19 @@ TP_STRUCT__entry( __field(const void *, mr) - __field(unsigned int, state) __field(unsigned int, status) __field(unsigned int, vendor_err) ), TP_fast_assign( __entry->mr = container_of(frwr, struct rpcrdma_mr, frwr); - __entry->state = frwr->fr_state; __entry->status = wc->status; __entry->vendor_err = __entry->status ? wc->vendor_err : 0; ), TP_printk( - "mr=%p state=%s: %s (%u/0x%x)", - __entry->mr, xprtrdma_show_frwr_state(__entry->state), - rdma_show_wc_status(__entry->status), + "mr=%p: %s (%u/0x%x)", + __entry->mr, rdma_show_wc_status(__entry->status), __entry->status, __entry->vendor_err ) ); diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index ac47314..99871fbf 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -168,7 +168,6 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) goto out_list_err; mr->frwr.fr_mr = frmr; - mr->frwr.fr_state = FRWR_IS_INVALID; mr->mr_dir = DMA_NONE; INIT_LIST_HEAD(&mr->mr_list); INIT_WORK(&mr->mr_recycle, frwr_mr_recycle_worker); @@ -298,65 +297,6 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) } /** - * frwr_wc_fastreg - Invoked by RDMA provider for a flushed FastReg WC - * @cq: completion queue (ignored) - * @wc: completed WR - * - */ -static void -frwr_wc_fastreg(struct ib_cq *cq, struct ib_wc *wc) -{ - struct ib_cqe *cqe = wc->wr_cqe; - struct rpcrdma_frwr *frwr = - container_of(cqe, struct rpcrdma_frwr, fr_cqe); - - /* WARNING: Only wr_cqe and status are reliable at this point */ - if (wc->status != IB_WC_SUCCESS) - frwr->fr_state = FRWR_FLUSHED_FR; - trace_xprtrdma_wc_fastreg(wc, frwr); -} - -/** - * frwr_wc_localinv - Invoked by RDMA provider for a flushed LocalInv WC - * @cq: completion queue (ignored) - * @wc: completed WR - * - */ -static void -frwr_wc_localinv(struct ib_cq *cq, struct ib_wc *wc) -{ - struct ib_cqe *cqe = wc->wr_cqe; - struct rpcrdma_frwr *frwr = container_of(cqe, struct rpcrdma_frwr, - fr_cqe); - - /* WARNING: Only wr_cqe and status are reliable at this point */ - if (wc->status != IB_WC_SUCCESS) - frwr->fr_state = FRWR_FLUSHED_LI; - trace_xprtrdma_wc_li(wc, frwr); -} - -/** - * frwr_wc_localinv_wake - Invoked by RDMA provider for a signaled LocalInv WC - * @cq: completion queue (ignored) - * @wc: completed WR - * - * Awaken anyone waiting for an MR to finish being fenced. - */ -static void -frwr_wc_localinv_wake(struct ib_cq *cq, struct ib_wc *wc) -{ - struct ib_cqe *cqe = wc->wr_cqe; - struct rpcrdma_frwr *frwr = container_of(cqe, struct rpcrdma_frwr, - fr_cqe); - - /* WARNING: Only wr_cqe and status are reliable at this point */ - if (wc->status != IB_WC_SUCCESS) - frwr->fr_state = FRWR_FLUSHED_LI; - trace_xprtrdma_wc_li_wake(wc, frwr); - complete(&frwr->fr_linv_done); -} - -/** * frwr_map - Register a memory region * @r_xprt: controlling transport * @seg: memory region co-ordinates @@ -378,23 +318,15 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, { struct rpcrdma_ia *ia = &r_xprt->rx_ia; bool holes_ok = ia->ri_mrtype == IB_MR_TYPE_SG_GAPS; - struct rpcrdma_frwr *frwr; struct rpcrdma_mr *mr; struct ib_mr *ibmr; struct ib_reg_wr *reg_wr; int i, n; u8 key; - mr = NULL; - do { - if (mr) - rpcrdma_mr_recycle(mr); - mr = rpcrdma_mr_get(r_xprt); - if (!mr) - goto out_getmr_err; - } while (mr->frwr.fr_state != FRWR_IS_INVALID); - frwr = &mr->frwr; - frwr->fr_state = FRWR_IS_VALID; + mr = rpcrdma_mr_get(r_xprt); + if (!mr) + goto out_getmr_err; if (nsegs > ia->ri_max_frwr_depth) nsegs = ia->ri_max_frwr_depth; @@ -423,7 +355,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, if (!mr->mr_nents) goto out_dmamap_err; - ibmr = frwr->fr_mr; + ibmr = mr->frwr.fr_mr; n = ib_map_mr_sg(ibmr, mr->mr_sg, mr->mr_nents, NULL, PAGE_SIZE); if (unlikely(n != mr->mr_nents)) goto out_mapmr_err; @@ -433,7 +365,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, key = (u8)(ibmr->rkey & 0x000000FF); ib_update_fast_reg_key(ibmr, ++key); - reg_wr = &frwr->fr_regwr; + reg_wr = &mr->frwr.fr_regwr; reg_wr->mr = ibmr; reg_wr->key = ibmr->rkey; reg_wr->access = writing ? @@ -465,6 +397,23 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, } /** + * frwr_wc_fastreg - Invoked by RDMA provider for a flushed FastReg WC + * @cq: completion queue (ignored) + * @wc: completed WR + * + */ +static void frwr_wc_fastreg(struct ib_cq *cq, struct ib_wc *wc) +{ + struct ib_cqe *cqe = wc->wr_cqe; + struct rpcrdma_frwr *frwr = + container_of(cqe, struct rpcrdma_frwr, fr_cqe); + + /* WARNING: Only wr_cqe and status are reliable at this point */ + trace_xprtrdma_wc_fastreg(wc, frwr); + /* The MR will get recycled when the associated req is retransmitted */ +} + +/** * frwr_send - post Send WR containing the RPC Call message * @ia: interface adapter * @req: Prepared RPC Call @@ -516,65 +465,104 @@ void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) if (mr->mr_handle == rep->rr_inv_rkey) { list_del_init(&mr->mr_list); trace_xprtrdma_mr_remoteinv(mr); - mr->frwr.fr_state = FRWR_IS_INVALID; rpcrdma_mr_unmap_and_put(mr); break; /* only one invalidated MR per RPC */ } } +static void __frwr_release_mr(struct ib_wc *wc, struct rpcrdma_mr *mr) +{ + if (wc->status != IB_WC_SUCCESS) + rpcrdma_mr_recycle(mr); + else + rpcrdma_mr_unmap_and_put(mr); +} + /** - * frwr_unmap_sync - invalidate memory regions that were registered for @req - * @r_xprt: controlling transport - * @mrs: list of MRs to process + * frwr_wc_localinv - Invoked by RDMA provider for a LOCAL_INV WC + * @cq: completion queue (ignored) + * @wc: completed WR * - * Sleeps until it is safe for the host CPU to access the - * previously mapped memory regions. + */ +static void frwr_wc_localinv(struct ib_cq *cq, struct ib_wc *wc) +{ + struct ib_cqe *cqe = wc->wr_cqe; + struct rpcrdma_frwr *frwr = + container_of(cqe, struct rpcrdma_frwr, fr_cqe); + struct rpcrdma_mr *mr = container_of(frwr, struct rpcrdma_mr, frwr); + + /* WARNING: Only wr_cqe and status are reliable at this point */ + __frwr_release_mr(wc, mr); + trace_xprtrdma_wc_li(wc, frwr); +} + +/** + * frwr_wc_localinv_wake - Invoked by RDMA provider for a LOCAL_INV WC + * @cq: completion queue (ignored) + * @wc: completed WR * - * Caller ensures that @mrs is not empty before the call. This - * function empties the list. + * Awaken anyone waiting for an MR to finish being fenced. */ -void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) +static void frwr_wc_localinv_wake(struct ib_cq *cq, struct ib_wc *wc) +{ + struct ib_cqe *cqe = wc->wr_cqe; + struct rpcrdma_frwr *frwr = + container_of(cqe, struct rpcrdma_frwr, fr_cqe); + struct rpcrdma_mr *mr = container_of(frwr, struct rpcrdma_mr, frwr); + + /* WARNING: Only wr_cqe and status are reliable at this point */ + __frwr_release_mr(wc, mr); + trace_xprtrdma_wc_li_wake(wc, frwr); + complete(&frwr->fr_linv_done); +} + +/** + * frwr_unmap_sync - invalidate memory regions that were registered for @req + * @r_xprt: controlling transport instance + * @req: rpcrdma_req with a non-empty list of MRs to process + * + * Sleeps until it is safe for the host CPU to access the previously mapped + * memory regions. + */ +void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) { struct ib_send_wr *first, **prev, *last; const struct ib_send_wr *bad_wr; - struct rpcrdma_ia *ia = &r_xprt->rx_ia; struct rpcrdma_frwr *frwr; struct rpcrdma_mr *mr; - int count, rc; + int rc; /* ORDER: Invalidate all of the MRs first * * Chain the LOCAL_INV Work Requests and post them with * a single ib_post_send() call. */ - frwr = NULL; - count = 0; prev = &first; - list_for_each_entry(mr, mrs, mr_list) { - mr->frwr.fr_state = FRWR_IS_INVALID; + while (!list_empty(&req->rl_registered)) { + mr = rpcrdma_mr_pop(&req->rl_registered); - frwr = &mr->frwr; trace_xprtrdma_mr_localinv(mr); + r_xprt->rx_stats.local_inv_needed++; + frwr = &mr->frwr; frwr->fr_cqe.done = frwr_wc_localinv; last = &frwr->fr_invwr; - memset(last, 0, sizeof(*last)); + last->next = NULL; last->wr_cqe = &frwr->fr_cqe; + last->sg_list = NULL; + last->num_sge = 0; last->opcode = IB_WR_LOCAL_INV; + last->send_flags = IB_SEND_SIGNALED; last->ex.invalidate_rkey = mr->mr_handle; - count++; *prev = last; prev = &last->next; } - if (!frwr) - goto unmap; /* Strong send queue ordering guarantees that when the * last WR in the chain completes, all WRs in the chain * are complete. */ - last->send_flags = IB_SEND_SIGNALED; frwr->fr_cqe.done = frwr_wc_localinv_wake; reinit_completion(&frwr->fr_linv_done); @@ -582,26 +570,18 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) * replaces the QP. The RPC reply handler won't call us * unless ri_id->qp is a valid pointer. */ - r_xprt->rx_stats.local_inv_needed++; bad_wr = NULL; - rc = ib_post_send(ia->ri_id->qp, first, &bad_wr); - if (bad_wr != first) - wait_for_completion(&frwr->fr_linv_done); - if (rc) - goto out_release; + rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); + trace_xprtrdma_post_send(req, rc); - /* ORDER: Now DMA unmap all of the MRs, and return - * them to the free MR list. + /* The final LOCAL_INV WR in the chain is supposed to + * do the wake. If it never gets posted, the wake will + * not happen, so don't wait in that case. */ -unmap: - while (!list_empty(mrs)) { - mr = rpcrdma_mr_pop(mrs); - rpcrdma_mr_unmap_and_put(mr); - } - return; - -out_release: - pr_err("rpcrdma: FRWR invalidate ib_post_send returned %i\n", rc); + if (bad_wr != first) + wait_for_completion(&frwr->fr_linv_done); + if (!rc) + return; /* Unmap and release the MRs in the LOCAL_INV WRs that did not * get posted. diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 77fc1e4..6c049fd 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1277,7 +1277,7 @@ void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) * RPC has relinquished all its Send Queue entries. */ if (!list_empty(&req->rl_registered)) - frwr_unmap_sync(r_xprt, &req->rl_registered); + frwr_unmap_sync(r_xprt, req); /* Ensure that any DMA mapped pages associated with * the Send of the RPC Call have been unmapped before diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 3c39aa3..a9de116 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -240,17 +240,9 @@ struct rpcrdma_sendctx { * An external memory region is any buffer or page that is registered * on the fly (ie, not pre-registered). */ -enum rpcrdma_frwr_state { - FRWR_IS_INVALID, /* ready to be used */ - FRWR_IS_VALID, /* in use */ - FRWR_FLUSHED_FR, /* flushed FASTREG WR */ - FRWR_FLUSHED_LI, /* flushed LOCALINV WR */ -}; - struct rpcrdma_frwr { struct ib_mr *fr_mr; struct ib_cqe fr_cqe; - enum rpcrdma_frwr_state fr_state; struct completion fr_linv_done; union { struct ib_reg_wr fr_regwr; @@ -567,8 +559,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr **mr); int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); -void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, - struct list_head *mrs); +void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req); /* * RPC/RDMA protocol calls - xprtrdma/rpc_rdma.c From patchwork Tue May 28 18:21:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DFCDA13AD for ; Tue, 28 May 2019 18:21:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB14A28640 for ; Tue, 28 May 2019 18:21:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF9AB28657; Tue, 28 May 2019 18:21:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F4F628640 for ; Tue, 28 May 2019 18:21:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728041AbfE1SVY (ORCPT ); Tue, 28 May 2019 14:21:24 -0400 Received: from mail-it1-f195.google.com ([209.85.166.195]:53963 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVY (ORCPT ); Tue, 28 May 2019 14:21:24 -0400 Received: by mail-it1-f195.google.com with SMTP id m141so5955079ita.3; Tue, 28 May 2019 11:21:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=uHzjr5ujV1e5HRY2u+3bqiCffUkZMSbWoK0OLOu3dko=; b=M1TTPT1viKY1QUhCeYxzWZ6zXgLU8aoobmfsoA7JwCGaNcrdCD+DwAOASUPZR3ay9R FPyN7hBM1lZYx2/qvqRdC/uYbORg14nEL16MNqRXJd2OQp40pXWCZoH5xXEAUCGoDo59 2v5fSlaG3zcBm8Wmq6K5AzeSBWAgLmZAiFf2MvR/Q0cqxowpPMU2311+56QncZTEesJi i6a0/akkyIY+vOCaiw+rDVPsSJ0HfjPcithS4wZyDNFdKJlZFnmPZyrd//Ip4jzMusag zLZQSsxnT3XvuCBLy9++/tfmg761HWliX986IrbnipdgWvEGdqUuviGZZKHzFwvs9qZ4 s7RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=uHzjr5ujV1e5HRY2u+3bqiCffUkZMSbWoK0OLOu3dko=; b=KgeO64YDVcH+uea8vNtS2GF2w/qvT5vyl24Uaq5WMLq81wn/pc8S6UqTEZhBr1KTiK wjd2IzcUXKKdK/qw9NGt9+x+rR7mkr9hXxlsDGskRVOmm/ZiPPdJBPoJBItz7etAWs2U gMsANYPIiBQW6Szv4VTq3XKN79zIw2se+4Ph40FIkwHYRFX6r6DWGXfswZhjFhYJ4o23 JvA1ldJuVBVdm1DQC4Ua28aV+jAYuHehjqqUUMEbC9TgO7oUnmveng4DqL8abc1QNvTJ ytl2Qo6D8uXDow5rAIGRyTMiEDNIro8macQxnJQJ2GiIXPDOZwAdynRZTPSL4aJKSRyb LKnw== X-Gm-Message-State: APjAAAUolrPXTKFt5dvjLuVp5S6Uaq1GZ1E8rvO454/le6Yx8mV0kUr5 QtajHWA7SPobl73cVc2aQf4+5L6N X-Google-Smtp-Source: APXvYqxrlowuNZmnRx8qre1b220q0FuO1wS/rmckX++Y3znHA9ujNxlpYGrskA2IEm/mCvsoN4L3RA== X-Received: by 2002:a24:dcc5:: with SMTP id q188mr4214915itg.64.1559067683419; Tue, 28 May 2019 11:21:23 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id i25sm4684349ioi.42.2019.05.28.11.21.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:23 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILM5Z014531; Tue, 28 May 2019 18:21:22 GMT Subject: [PATCH RFC 06/12] xprtrdma: Add mechanism to place MRs back on the free list From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:22 -0400 Message-ID: <20190528182122.19012.54699.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A marshal operation can fail if there are not enough MRs or if the sendctx queue is empty. In this case, any MRs that were already set up for that request are recycled. Recycling releases MRs and creates new ones, which is expensive. Since commit f2877623082b ("xprtrdma: Chain Send to FastReg WRs") was merged, recycling FRWRs is unnecessary. This is because before that commit, frwr_map had already posted FAST_REG Work Requests, so the MRs were owned by the NIC and dealing with them had to be delayed. Since that commit, the FAST_REG WRs are posted at the same time as the Send WR. This means that if marshaling fails, we are certain the MRs are safe to simply unmap and place back on the free list because neither the Send nor the FAST_REG WRs have been posted yet. This reduces the total number of MRs that the xprt has to create under heavy workloads and makes the logic a bit less brittle. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 24 ++++++++++++++++++++++++ net/sunrpc/xprtrdma/rpc_rdma.c | 1 + net/sunrpc/xprtrdma/xprt_rdma.h | 1 + 3 files changed, 26 insertions(+) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 99871fbf..ae97119 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -144,6 +144,30 @@ void frwr_release_mr(struct rpcrdma_mr *mr) frwr_release_mr(mr); } +/* frwr_reset - Place MRs back on the free list + * @req: request to reset + * + * Used after a failed marshal. For FRWR, this means the MRs + * don't have to be fully released and recreated. + * + * NB: This is safe only as long as none of @req's MRs are + * involved with an ongoing asynchronous FAST_REG or LOCAL_INV + * Work Request. + */ +void frwr_reset(struct rpcrdma_req *req) +{ + /* If this is a retransmit, discard previously registered + * chunks. Very likely the connection has been replaced, + * so these registrations are invalid and unusable. + */ + while (!list_empty(&req->rl_registered)) { + struct rpcrdma_mr *mr; + + mr = rpcrdma_mr_pop(&req->rl_registered); + rpcrdma_mr_unmap_and_put(mr); + } +} + /** * frwr_init_mr - Initialize one MR * @ia: interface adapter diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 6c049fd..f211f81 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -884,6 +884,7 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, out_err: trace_xprtrdma_marshal_failed(rqst, ret); r_xprt->rx_stats.failed_marshal_count++; + frwr_reset(req); return ret; } diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index a9de116..a396528 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -549,6 +549,7 @@ static inline bool rpcrdma_regbuf_dma_map(struct rpcrdma_xprt *r_xprt, /* Memory registration calls xprtrdma/frwr_ops.c */ bool frwr_is_supported(struct ib_device *device); +void frwr_reset(struct rpcrdma_req *req); int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep); int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr); void frwr_release_mr(struct rpcrdma_mr *mr); From patchwork Tue May 28 18:21:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965639 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E62B813AD for ; Tue, 28 May 2019 18:21:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4F6728640 for ; Tue, 28 May 2019 18:21:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C92A128657; Tue, 28 May 2019 18:21:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D27EF28640 for ; Tue, 28 May 2019 18:21:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727964AbfE1SVa (ORCPT ); Tue, 28 May 2019 14:21:30 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:35546 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVa (ORCPT ); Tue, 28 May 2019 14:21:30 -0400 Received: by mail-it1-f194.google.com with SMTP id u186so5524366ith.0; Tue, 28 May 2019 11:21:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=fDyNjexvvDERAZz/5Qfl5FKJ581gHQMxJdwTpMwFiH0=; b=oiP9QEB3z2NBcYoowIFpqc8tQ14Wjhehwrrqn+fa6gUVqIFQTvRCuOInLvP4XdtxaG jtQBRaub197itVMnC3FPO5l7BN0h/ggMAVlUM2HD4axvFmYEk+8r/sb0qpeAG/OxpMf4 kRYv7rT1tsQifAtnOLdLwu6DlMd42/W9rkWODk0bBKsSDbOguaH3NU4TY1+/wZc+jk6F CgfERt+dkCqVSM6mGAPq4ZpWn/WT9eTmIDtYiaz2J1PEn9DSKrAMhYY1znW8vPoygXrD geDqHxDW3Zjaw09GkmdfoG/dCom+jL4Lb2W1u22gYsqv/cZRkJscs3HUqPBLbcnetp+m 8Qbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=fDyNjexvvDERAZz/5Qfl5FKJ581gHQMxJdwTpMwFiH0=; b=R4gomZineit+i8w49+qWHHALXIp0H5WdpVJ+H/dhbwSy1VqoQSCuXNCDmav57dSCbe 7OwSaJqzgQCZ/91SSO4hXrHz+ICvMohTB8Uh5EmAlKW1JDInVCKdtmlZ3bQEzFJV8jZa I38KKJ0/TfD+tINsgQBmlzNsnowjH9lssmAwMtTHPdO+97OFE2QC6jCh5KWuGkxivPey GXSmjKhqCjgeU4GWFt6KcQhcT+UPayjt5JGg0zJHHlX2QcGAT/WflLslrqAWSFL2PSek IScnWvwKrxk4a+ZPVPMC912HPuSgD9I4faalN4r+Ovh9b4eVjFH4ojgOLBcaJDggKXO/ 7cyQ== X-Gm-Message-State: APjAAAVLf1H1ieJxkd1OmWaHbcystb0W2Ozcg84lP+lUqfOxDCuSyXOY j2YtExUFp5l3J9z0OQB2V64Gnzc/ X-Google-Smtp-Source: APXvYqy04Eqjx/Y2AMPV5ojVpVGKBfUa+ai776RVZ6QBjiSw6sOfK/MSB6bTMvu5XMchgFJWjcV/WA== X-Received: by 2002:a24:c904:: with SMTP id h4mr4149914itg.46.1559067688885; Tue, 28 May 2019 11:21:28 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id k76sm1513842ita.6.2019.05.28.11.21.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:28 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILROI014534; Tue, 28 May 2019 18:21:27 GMT Subject: [PATCH RFC 07/12] xprtrdma: Reduce context switching due to Local Invalidation From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:27 -0400 Message-ID: <20190528182127.19012.45171.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since commit ba69cd122ece ("xprtrdma: Remove support for FMR memory registration"), FRWR is the only supported memory registration mode. We can take advantage of the asynchronous nature of FRWR's LOCAL_INV Work Requests to get rid of the completion wait by having the LOCAL_INV completion handler take care of DMA unmapping MRs and waking the upper layer RPC waiter. This eliminates two context switches when local invalidation is necessary. As a side benefit, we will no longer need the per-xprt deferred completion work queue. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 1 net/sunrpc/xprtrdma/frwr_ops.c | 104 ++++++++++++++++++++++++++++++++++++++- net/sunrpc/xprtrdma/rpc_rdma.c | 61 +++++++++++------------ net/sunrpc/xprtrdma/verbs.c | 17 ------ net/sunrpc/xprtrdma/xprt_rdma.h | 8 ++- 5 files changed, 136 insertions(+), 55 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 7fe21ba..eadcd09 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -710,6 +710,7 @@ DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_fastreg); DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li); DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li_wake); +DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li_done); TRACE_EVENT(xprtrdma_frwr_alloc, TP_PROTO( diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index ae97119..e5500f7 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -546,7 +546,10 @@ static void frwr_wc_localinv_wake(struct ib_cq *cq, struct ib_wc *wc) * @req: rpcrdma_req with a non-empty list of MRs to process * * Sleeps until it is safe for the host CPU to access the previously mapped - * memory regions. + * memory regions. This guarantees that registered MRs are properly fenced + * from the server before the RPC consumer accesses the data in them. It + * also ensures proper Send flow control: waking the next RPC waits until + * this RPC has relinquished all its Send Queue entries. */ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) { @@ -607,8 +610,7 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) if (!rc) return; - /* Unmap and release the MRs in the LOCAL_INV WRs that did not - * get posted. + /* Recycle MRs in the LOCAL_INV chain that did not get posted. */ while (bad_wr) { frwr = container_of(bad_wr, struct rpcrdma_frwr, @@ -620,3 +622,99 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) rpcrdma_mr_recycle(mr); } } + +/** + * frwr_wc_localinv_done - Invoked by RDMA provider for a signaled LOCAL_INV WC + * @cq: completion queue (ignored) + * @wc: completed WR + * + */ +static void frwr_wc_localinv_done(struct ib_cq *cq, struct ib_wc *wc) +{ + struct ib_cqe *cqe = wc->wr_cqe; + struct rpcrdma_frwr *frwr = + container_of(cqe, struct rpcrdma_frwr, fr_cqe); + struct rpcrdma_mr *mr = container_of(frwr, struct rpcrdma_mr, frwr); + + /* WARNING: Only wr_cqe and status are reliable at this point */ + __frwr_release_mr(wc, mr); + trace_xprtrdma_wc_li_done(wc, frwr); + rpcrdma_complete_rqst(frwr->fr_req->rl_reply); +} + +/** + * frwr_unmap_async - invalidate memory regions that were registered for @req + * @r_xprt: controlling transport instance + * @req: rpcrdma_req with a non-empty list of MRs to process + * + * This guarantees that registered MRs are properly fenced from the + * server before the RPC consumer accesses the data in them. It also + * ensures proper Send flow control: waking the next RPC waits until + * this RPC has relinquished all its Send Queue entries. + */ +void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) +{ + struct ib_send_wr *first, *last, **prev; + const struct ib_send_wr *bad_wr; + struct rpcrdma_frwr *frwr; + struct rpcrdma_mr *mr; + int rc; + + /* Chain the LOCAL_INV Work Requests and post them with + * a single ib_post_send() call. + */ + prev = &first; + while (!list_empty(&req->rl_registered)) { + mr = rpcrdma_mr_pop(&req->rl_registered); + + trace_xprtrdma_mr_localinv(mr); + r_xprt->rx_stats.local_inv_needed++; + + frwr = &mr->frwr; + frwr->fr_cqe.done = frwr_wc_localinv; + frwr->fr_req = req; + last = &frwr->fr_invwr; + last->next = NULL; + last->wr_cqe = &frwr->fr_cqe; + last->sg_list = NULL; + last->num_sge = 0; + last->opcode = IB_WR_LOCAL_INV; + last->send_flags = IB_SEND_SIGNALED; + last->ex.invalidate_rkey = mr->mr_handle; + + *prev = last; + prev = &last->next; + } + + /* Strong send queue ordering guarantees that when the + * last WR in the chain completes, all WRs in the chain + * are complete. + */ + frwr->fr_cqe.done = frwr_wc_localinv_done; + + /* Transport disconnect drains the receive CQ before it + * replaces the QP. The RPC reply handler won't call us + * unless ri_id->qp is a valid pointer. + */ + bad_wr = NULL; + rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); + trace_xprtrdma_post_send(req, rc); + if (!rc) + return; + + /* Recycle MRs in the LOCAL_INV chain that did not get posted. + */ + while (bad_wr) { + frwr = container_of(bad_wr, struct rpcrdma_frwr, fr_invwr); + mr = container_of(frwr, struct rpcrdma_mr, frwr); + bad_wr = bad_wr->next; + + rpcrdma_mr_recycle(mr); + } + + /* The final LOCAL_INV WR in the chain is supposed to + * do the wake. If it never gets posted, the wake will + * not happen, so wake here in that case. + */ + rpcrdma_complete_rqst(req->rl_reply); +} diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index f211f81..ea39f74 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1268,24 +1268,15 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep) goto out; } -void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) +/* Ensure that any DMA mapped pages associated with + * the Send of the RPC Call have been unmapped before + * allowing the RPC to complete. This protects argument + * memory not controlled by the RPC client from being + * re-used before we're done with it. + */ +static void rpcrdma_release_tx(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req) { - /* Invalidate and unmap the data payloads before waking - * the waiting application. This guarantees the memory - * regions are properly fenced from the server before the - * application accesses the data. It also ensures proper - * send flow control: waking the next RPC waits until this - * RPC has relinquished all its Send Queue entries. - */ - if (!list_empty(&req->rl_registered)) - frwr_unmap_sync(r_xprt, req); - - /* Ensure that any DMA mapped pages associated with - * the Send of the RPC Call have been unmapped before - * allowing the RPC to complete. This protects argument - * memory not controlled by the RPC client from being - * re-used before we're done with it. - */ if (test_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags)) { r_xprt->rx_stats.reply_waits_for_send++; out_of_line_wait_on_bit(&req->rl_flags, @@ -1295,24 +1286,23 @@ void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) } } -/* Reply handling runs in the poll worker thread. Anything that - * might wait is deferred to a separate workqueue. +/** + * rpcrdma_release_rqst - Release hardware resources + * @r_xprt: controlling transport instance + * @req: request with resources to release + * */ -void rpcrdma_deferred_completion(struct work_struct *work) +void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) { - struct rpcrdma_rep *rep = - container_of(work, struct rpcrdma_rep, rr_work); - struct rpcrdma_req *req = rpcr_to_rdmar(rep->rr_rqst); - struct rpcrdma_xprt *r_xprt = rep->rr_rxprt; + if (!list_empty(&req->rl_registered)) + frwr_unmap_sync(r_xprt, req); - trace_xprtrdma_defer_cmp(rep); - if (rep->rr_wc_flags & IB_WC_WITH_INVALIDATE) - frwr_reminv(rep, &req->rl_registered); - rpcrdma_release_rqst(r_xprt, req); - rpcrdma_complete_rqst(rep); + rpcrdma_release_tx(r_xprt, req); } -/* Process received RPC/RDMA messages. +/** + * rpcrdma_reply_handler - Process received RPC/RDMA messages + * @rep: Incoming rpcrdma_rep object to process * * Errors must result in the RPC task either being awakened, or * allowed to timeout, to discover the errors at that time. @@ -1374,7 +1364,16 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) rep->rr_rqst = rqst; trace_xprtrdma_reply(rqst->rq_task, rep, req, credits); - queue_work(buf->rb_completion_wq, &rep->rr_work); + + if (rep->rr_wc_flags & IB_WC_WITH_INVALIDATE) + frwr_reminv(rep, &req->rl_registered); + if (!list_empty(&req->rl_registered)) { + frwr_unmap_async(r_xprt, req); + /* LocalInv completion will complete the RPC */ + } else { + rpcrdma_release_tx(r_xprt, req); + rpcrdma_complete_rqst(rep); + } return; out_badversion: diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 78be985..0be455b 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -89,14 +89,12 @@ */ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) { - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ia *ia = &r_xprt->rx_ia; /* Flush Receives, then wait for deferred Reply work * to complete. */ ib_drain_rq(ia->ri_id->qp); - drain_workqueue(buf->rb_completion_wq); /* Deferred Reply processing might have scheduled * local invalidations. @@ -1057,7 +1055,6 @@ static bool rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, bool temp) rep->rr_cqe.done = rpcrdma_wc_receive; rep->rr_rxprt = r_xprt; - INIT_WORK(&rep->rr_work, rpcrdma_deferred_completion); rep->rr_recv_wr.next = NULL; rep->rr_recv_wr.wr_cqe = &rep->rr_cqe; rep->rr_recv_wr.sg_list = &rep->rr_rdmabuf->rg_iov; @@ -1118,15 +1115,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) if (rc) goto out; - buf->rb_completion_wq = alloc_workqueue("rpcrdma-%s", - WQ_MEM_RECLAIM | WQ_HIGHPRI, - 0, - r_xprt->rx_xprt.address_strings[RPC_DISPLAY_ADDR]); - if (!buf->rb_completion_wq) { - rc = -ENOMEM; - goto out; - } - return 0; out: rpcrdma_buffer_destroy(buf); @@ -1200,11 +1188,6 @@ static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep) { cancel_delayed_work_sync(&buf->rb_refresh_worker); - if (buf->rb_completion_wq) { - destroy_workqueue(buf->rb_completion_wq); - buf->rb_completion_wq = NULL; - } - rpcrdma_sendctxs_destroy(buf); while (!list_empty(&buf->rb_recv_bufs)) { diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index a396528..e465221 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -202,10 +202,9 @@ struct rpcrdma_rep { bool rr_temp; struct rpcrdma_regbuf *rr_rdmabuf; struct rpcrdma_xprt *rr_rxprt; - struct work_struct rr_work; + struct rpc_rqst *rr_rqst; struct xdr_buf rr_hdrbuf; struct xdr_stream rr_stream; - struct rpc_rqst *rr_rqst; struct list_head rr_list; struct ib_recv_wr rr_recv_wr; }; @@ -240,10 +239,12 @@ struct rpcrdma_sendctx { * An external memory region is any buffer or page that is registered * on the fly (ie, not pre-registered). */ +struct rpcrdma_req; struct rpcrdma_frwr { struct ib_mr *fr_mr; struct ib_cqe fr_cqe; struct completion fr_linv_done; + struct rpcrdma_req *fr_req; union { struct ib_reg_wr fr_regwr; struct ib_send_wr fr_invwr; @@ -388,7 +389,6 @@ struct rpcrdma_buffer { u32 rb_bc_srv_max_requests; u32 rb_bc_max_requests; - struct workqueue_struct *rb_completion_wq; struct delayed_work rb_refresh_worker; }; @@ -561,6 +561,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req); +void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req); /* * RPC/RDMA protocol calls - xprtrdma/rpc_rdma.c @@ -585,7 +586,6 @@ int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, void rpcrdma_reply_handler(struct rpcrdma_rep *rep); void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req); -void rpcrdma_deferred_completion(struct work_struct *work); static inline void rpcrdma_set_xdrlen(struct xdr_buf *xdr, size_t len) { From patchwork Tue May 28 18:21:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965641 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2126C15E6 for ; Tue, 28 May 2019 18:21:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1029728640 for ; Tue, 28 May 2019 18:21:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 04B642865F; Tue, 28 May 2019 18:21:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4627F28640 for ; Tue, 28 May 2019 18:21:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728059AbfE1SVg (ORCPT ); Tue, 28 May 2019 14:21:36 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:56007 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVg (ORCPT ); Tue, 28 May 2019 14:21:36 -0400 Received: by mail-it1-f193.google.com with SMTP id g24so5950881iti.5; Tue, 28 May 2019 11:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=aL7FW6mmLTyhjWDMwg6TWQ+5TITczlqjWNyicyO6aZY=; b=E+nMtPUQdGvT1SqiAO098Vb+8EbjIUpXaiULZ7fFbeHGOxqVDGNShzJfBY+Ik1CZqN lh5Rrjxw8qH+CSor4TnvMAhWQ/QlbuMiS2PraAghp2lqkRbZXSnrUbBMC53rJV+g/UIb MHqLwr2jQVSMBOdvg0PGxu43TNnNo1+qGezUdvKaKEQGTSyxh28LoOCnIhc7eGQsRtZE x/vwdk3TXMx2dQwn2SxxRYCf6zBUpUguWN3dUfcwLyld7GXXmLFjDO1SbqoEhas/k+mt KgbQvh2R8Y+P4oWbG5XPwoKSx6RrHBXpLQ4+6T+TgKSzNQq4NGLD8uLxS+55s2FzS17d HLmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=aL7FW6mmLTyhjWDMwg6TWQ+5TITczlqjWNyicyO6aZY=; b=H1b0Fc51UnVTe5nxk2bIL6TzKuEuJf6J/2ByQuR2QIHMtF0GMG1M7sFYwdeKMFxiof YGTg8frhsz7rSr1pXmupSn2f9XYLEq9bsoeNZ1RQATbRuf+TQ6CVNr6N0dB0ncsaoqFk Q8tj+tRBV58VCmJf4G9N++w7kuPXXUg+YEqhBqogQHkNm3odQEeeLJg6GUqij9DffwFp J9BJ2G1W3sABZYDAANxrg7yz5gXVBXiia5+51YRVo8/TwR1yH8RKwvq0faJS+dIMcrXx QffCYAd9Q9jQkagz46fR96f9A6IQH71TuwJ/WjdLQP60WhiM9QtHNplXao4lFRx2qelZ VT+Q== X-Gm-Message-State: APjAAAW5ib0Yx0dJycybYOSr3Xx4t6+DhjlPWjiuV2UioZtN1tWw363h hLEPDiAa/fx7/Moh+5XY6Dlqny3M X-Google-Smtp-Source: APXvYqy7DgF9ttdF91EXLiOFLCogCwpQsyGdIP7d6m+UuL2sMhOEXeLSuY2ndeY4i7VuQeQ8MDfzQQ== X-Received: by 2002:a02:228f:: with SMTP id o137mr82822449jao.39.1559067695009; Tue, 28 May 2019 11:21:35 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id e2sm1446810ith.39.2019.05.28.11.21.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:33 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILWcc014537; Tue, 28 May 2019 18:21:32 GMT Subject: [PATCH RFC 08/12] xprtrdma: Wake RPCs directly in rpcrdma_wc_send path From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:32 -0400 Message-ID: <20190528182132.19012.55642.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Eliminate a context switch in the path that handles RPC wake-ups when a Receive completion has to wait for a Send completion. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 61 +++++++++++++++------------------------ net/sunrpc/xprtrdma/transport.c | 9 +++++- net/sunrpc/xprtrdma/verbs.c | 3 +- net/sunrpc/xprtrdma/xprt_rdma.h | 12 ++------ 4 files changed, 35 insertions(+), 50 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index ea39f74..6de90d4 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -511,6 +511,16 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, return 0; } +static void rpcrdma_sendctx_done(struct kref *kref) +{ + struct rpcrdma_req *req = + container_of(kref, struct rpcrdma_req, rl_kref); + struct rpcrdma_rep *rep = req->rl_reply; + + rpcrdma_complete_rqst(rep); + rep->rr_rxprt->rx_stats.reply_waits_for_send++; +} + /** * rpcrdma_sendctx_unmap - DMA-unmap Send buffer * @sc: sendctx containing SGEs to unmap @@ -520,6 +530,9 @@ void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc) { struct ib_sge *sge; + if (!sc->sc_unmap_count) + return; + /* The first two SGEs contain the transport header and * the inline buffer. These are always left mapped so * they can be cheaply re-used. @@ -529,9 +542,7 @@ void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc) ib_dma_unmap_page(sc->sc_device, sge->addr, sge->length, DMA_TO_DEVICE); - if (test_and_clear_bit(RPCRDMA_REQ_F_TX_RESOURCES, - &sc->sc_req->rl_flags)) - wake_up_bit(&sc->sc_req->rl_flags, RPCRDMA_REQ_F_TX_RESOURCES); + kref_put(&sc->sc_req->rl_kref, rpcrdma_sendctx_done); } /* Prepare an SGE for the RPC-over-RDMA transport header. @@ -666,7 +677,7 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, out: sc->sc_wr.num_sge += sge_no; if (sc->sc_unmap_count) - __set_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags); + kref_get(&req->rl_kref); return true; out_regbuf: @@ -708,7 +719,7 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, req->rl_sendctx->sc_wr.num_sge = 0; req->rl_sendctx->sc_unmap_count = 0; req->rl_sendctx->sc_req = req; - __clear_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags); + kref_init(&req->rl_kref); ret = -EIO; if (!rpcrdma_prepare_hdr_sge(r_xprt, req, hdrlen)) @@ -1268,36 +1279,12 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep) goto out; } -/* Ensure that any DMA mapped pages associated with - * the Send of the RPC Call have been unmapped before - * allowing the RPC to complete. This protects argument - * memory not controlled by the RPC client from being - * re-used before we're done with it. - */ -static void rpcrdma_release_tx(struct rpcrdma_xprt *r_xprt, - struct rpcrdma_req *req) +static void rpcrdma_reply_done(struct kref *kref) { - if (test_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags)) { - r_xprt->rx_stats.reply_waits_for_send++; - out_of_line_wait_on_bit(&req->rl_flags, - RPCRDMA_REQ_F_TX_RESOURCES, - bit_wait, - TASK_UNINTERRUPTIBLE); - } -} + struct rpcrdma_req *req = + container_of(kref, struct rpcrdma_req, rl_kref); -/** - * rpcrdma_release_rqst - Release hardware resources - * @r_xprt: controlling transport instance - * @req: request with resources to release - * - */ -void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) -{ - if (!list_empty(&req->rl_registered)) - frwr_unmap_sync(r_xprt, req); - - rpcrdma_release_tx(r_xprt, req); + rpcrdma_complete_rqst(req->rl_reply); } /** @@ -1367,13 +1354,11 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) if (rep->rr_wc_flags & IB_WC_WITH_INVALIDATE) frwr_reminv(rep, &req->rl_registered); - if (!list_empty(&req->rl_registered)) { + if (!list_empty(&req->rl_registered)) frwr_unmap_async(r_xprt, req); /* LocalInv completion will complete the RPC */ - } else { - rpcrdma_release_tx(r_xprt, req); - rpcrdma_complete_rqst(rep); - } + else + kref_put(&req->rl_kref, rpcrdma_reply_done); return; out_badversion: diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index f84375d..5b1d5d7 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -618,8 +618,15 @@ static bool rpcrdma_check_regbuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(rqst->rq_xprt); struct rpcrdma_req *req = rpcr_to_rdmar(rqst); - rpcrdma_release_rqst(r_xprt, req); trace_xprtrdma_op_free(task, req); + + if (!list_empty(&req->rl_registered)) + frwr_unmap_sync(r_xprt, req); + + /* XXX: We should wait for the Send completion here, + * although it's very likely it's already fired in + * this case. + */ } /** diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 0be455b..729266e 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1463,8 +1463,7 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) struct ib_send_wr *send_wr = &req->rl_sendctx->sc_wr; int rc; - if (!ep->rep_send_count || - test_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags)) { + if (!ep->rep_send_count || kref_read(&req->rl_kref) > 1) { send_wr->send_flags |= IB_SEND_SIGNALED; ep->rep_send_count = ep->rep_send_batch; } else { diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index e465221..5475f0d 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -44,7 +44,8 @@ #include /* wait_queue_head_t, etc */ #include /* spinlock_t, etc */ -#include /* atomic_t, etc */ +#include /* atomic_t, etc */ +#include /* struct kref */ #include /* struct work_struct */ #include /* RDMA connection api */ @@ -329,17 +330,12 @@ struct rpcrdma_req { struct rpcrdma_regbuf *rl_recvbuf; /* rq_rcv_buf */ struct list_head rl_all; - unsigned long rl_flags; + struct kref rl_kref; struct list_head rl_registered; /* registered segments */ struct rpcrdma_mr_seg rl_segments[RPCRDMA_MAX_SEGS]; }; -/* rl_flags */ -enum { - RPCRDMA_REQ_F_TX_RESOURCES, -}; - static inline struct rpcrdma_req * rpcr_to_rdmar(const struct rpc_rqst *rqst) { @@ -584,8 +580,6 @@ int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, void rpcrdma_set_max_header_sizes(struct rpcrdma_xprt *); void rpcrdma_complete_rqst(struct rpcrdma_rep *rep); void rpcrdma_reply_handler(struct rpcrdma_rep *rep); -void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, - struct rpcrdma_req *req); static inline void rpcrdma_set_xdrlen(struct xdr_buf *xdr, size_t len) { From patchwork Tue May 28 18:21:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965645 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D82E713AD for ; Tue, 28 May 2019 18:21:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C59752864E for ; Tue, 28 May 2019 18:21:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B988328657; Tue, 28 May 2019 18:21:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5551628640 for ; Tue, 28 May 2019 18:21:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727947AbfE1SVk (ORCPT ); Tue, 28 May 2019 14:21:40 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:39986 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727752AbfE1SVk (ORCPT ); Tue, 28 May 2019 14:21:40 -0400 Received: by mail-it1-f193.google.com with SMTP id h11so5491434itf.5; Tue, 28 May 2019 11:21:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=AYDAlu3lshTWu+thI1Eb0xhXztpFpS+AwLcO72xeFgg=; b=QjI4AlsJ+3d6BBMyO5ijZjnwzVuiuJkkoJKp3MjThLosRE1sE1bMq185QiZ2SpUqm/ zq4aDAWNFb0LdkWhB4l/26qF4iAUHuEB+Ltwau2sNe2TREr+q6+949NFeiSsjE8JTx/y KXG50iNxmRgALuTxWWWGsHS5jo/eHQ00MIxT8U2L/vEbQblaW8qDtAVwEKzKTjoiuxby fXJWLSCnz6Dtcd9QYsgCxCOiX9AbfJudEb6VENi0lGYis91jyytmeNsILTFVeN8w4SN3 7YiKBjz9cy94RT7FRJQcDbCjWM/iJ5e8h/I0vS86mbHP/9G+QyhnqS0qH/vpGvJDbxUw /jqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=AYDAlu3lshTWu+thI1Eb0xhXztpFpS+AwLcO72xeFgg=; b=LvLqcPE0wZ0zjIRxJOdswNX+BppSyvwLSFH4GRcUp9nFdm+IulPLNDBbDLcO+MsU5Y M/tmz8x0bOqDB/2kis45yrMy+9CpxxjnhOlz/eHWM+q3AYGWtERT4+u7A/s6Y2jnXJmX J3joJWI1GNhtfcVLVUWuHaXw3A/hE4nbUMDebQyghZlqVqKQ2v8IdUqrxsyvDNT3qzgn Jlgewd9/P5IGBG7Rt2FG+UK6sRzItSVklMD0H2PRUBGOCqZKdbqOjKRqLKZrR7kUG31+ NoH4rjGuDe4KT3f9NW0dmZUArICphVdGhud9BMUpZKPcxLkvQSR9RidatwxrpZlbeL4n zg4g== X-Gm-Message-State: APjAAAUJrd9aDG5llToz98DAt1eRa2GWQRl+hdZHMNRQ128hsUWFmYFw Be2lrjoibhYNE34SvKqD+2jhywnj X-Google-Smtp-Source: APXvYqx7HzZlbX1kjkkdv7IUs4Qlg/KfOU6cXJmL3KUpGQBEBLPSpjSFVaunHCNOVVcBUGOil1iaaA== X-Received: by 2002:a24:3988:: with SMTP id l130mr4048009ita.13.1559067699616; Tue, 28 May 2019 11:21:39 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id n26sm3366771ioc.74.2019.05.28.11.21.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:39 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILcgX014540; Tue, 28 May 2019 18:21:38 GMT Subject: [PATCH RFC 09/12] xprtrdma: Simplify rpcrdma_rep_create From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:38 -0400 Message-ID: <20190528182138.19012.62904.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up. Commit 7c8d9e7c8863 ("xprtrdma: Move Receive posting to Receive handler") reduced the number of rpcrdma_rep_create call sites to one. After that commit, the backchannel code no longer invokes it. Therefore the free list logic added by commit d698c4a02ee0 ("xprtrdma: Fix backchannel allocation of extra rpcrdma_reps") is no longer necessary, and in fact adds some extra overhead that we can do without. Simply post any newly created reps. They will get added back to the rb_recv_bufs list when they subsequently complete. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 729266e..390c3bc 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1037,9 +1037,9 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, return NULL; } -static bool rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, bool temp) +static struct rpcrdma_rep *rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, + bool temp) { - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_rep *rep; rep = kzalloc(sizeof(*rep), GFP_KERNEL); @@ -1050,9 +1050,9 @@ static bool rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, bool temp) DMA_FROM_DEVICE, GFP_KERNEL); if (!rep->rr_rdmabuf) goto out_free; + xdr_buf_init(&rep->rr_hdrbuf, rdmab_data(rep->rr_rdmabuf), rdmab_length(rep->rr_rdmabuf)); - rep->rr_cqe.done = rpcrdma_wc_receive; rep->rr_rxprt = r_xprt; rep->rr_recv_wr.next = NULL; @@ -1060,16 +1060,12 @@ static bool rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, bool temp) rep->rr_recv_wr.sg_list = &rep->rr_rdmabuf->rg_iov; rep->rr_recv_wr.num_sge = 1; rep->rr_temp = temp; - - spin_lock(&buf->rb_lock); - list_add(&rep->rr_list, &buf->rb_recv_bufs); - spin_unlock(&buf->rb_lock); - return true; + return rep; out_free: kfree(rep); out: - return false; + return NULL; } /** @@ -1498,7 +1494,6 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) count = 0; wr = NULL; while (needed) { - struct rpcrdma_regbuf *rb; struct rpcrdma_rep *rep; spin_lock(&buf->rb_lock); @@ -1508,13 +1503,12 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) list_del(&rep->rr_list); spin_unlock(&buf->rb_lock); if (!rep) { - if (!rpcrdma_rep_create(r_xprt, temp)) + rep = rpcrdma_rep_create(r_xprt, temp); + if (!rep) break; - continue; } - rb = rep->rr_rdmabuf; - if (!rpcrdma_regbuf_dma_map(r_xprt, rb)) { + if (!rpcrdma_regbuf_dma_map(r_xprt, rep->rr_rdmabuf)) { rpcrdma_recv_buffer_put(rep); break; } From patchwork Tue May 28 18:21:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38DEF1575 for ; Tue, 28 May 2019 18:21:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 25CF828640 for ; Tue, 28 May 2019 18:21:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1AA7B28657; Tue, 28 May 2019 18:21:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A94F628640 for ; Tue, 28 May 2019 18:21:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727752AbfE1SVq (ORCPT ); Tue, 28 May 2019 14:21:46 -0400 Received: from mail-it1-f196.google.com ([209.85.166.196]:34094 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727428AbfE1SVq (ORCPT ); Tue, 28 May 2019 14:21:46 -0400 Received: by mail-it1-f196.google.com with SMTP id g23so3327378iti.1; Tue, 28 May 2019 11:21:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=VkIQGynJzS6VoI5SV8mWOm9mNwDUXUQEwJkQa0bOUmI=; b=h2Y9b1S+B+fXqtSDQmd1IgwdrY8QdTg6Luytwa7KbZgCgoSlbD2PY5xzsGmfuuEKnG mkX26Hky+CXx1Nej+ZSd9S0340Ec2qxojz5OQmU1UotZto1ZybsNADi1kTNrYepk5q+u qzMPk2b5CPM9u3WGd1psRzrN70O90TA81iFQ1HRWOW7UoMFeOS/MGzJGag8d9Zhr8hxH BAhLmVzk9VGHtkcq6+1M4f9JKn+GuN2ouPDpkVVMA22pj5p63qVHPlMlP9WxeCE3WoJ8 nsChdfOnqhJZaW4sRWuxmZrHwacETB6Fu+nPKocbouS0HOBL1Xd0fyJsPJg/tc5C7CL+ LejQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=VkIQGynJzS6VoI5SV8mWOm9mNwDUXUQEwJkQa0bOUmI=; b=dxfDmbL7yVpiCqqxbSUnzTiikvuV00/tbIM6b8X0nPQpln2rjeMmAWjO7kZ8wS5rfv 7tBEgTO+DYOGDrccMqAEwg4hgQPX5dgH7Mh2gfW7JeMadZlYOPewTLL4h48kxbSJ9PXe uBYfT6gvjEn2G1807OO2gMiaNxnDBxX2BxLLoTLOy9QdvijMYHUA+TeGaorucnLAq1cG boHjDeqAYIQS26TVoUj4kYExShesG4Av+FSm1NhsYnANWstZ8z+7WWEPuohG35hy5Oou PkuLFMriuB9yoV5pvdpWEjdORWYC0iTdszdDd7YDI8Gt1i3PCZBB5sdoRIvfxTIAtGxG I1SA== X-Gm-Message-State: APjAAAXdEcT5Im1S8Yo6T63fIBhwee8CtLYHklmNxTDW8bPxWeOZ5aDp 7zhgKj430DSvRd2V9MeETxRZWsoV X-Google-Smtp-Source: APXvYqxj/otpynpxqc0ItJdAVvNsAovYSUWlMKcimhZahXK4aOVe9szkNTu+IG4thx1LSe2bET9s/w== X-Received: by 2002:a24:688a:: with SMTP id v132mr823471itb.31.1559067704667; Tue, 28 May 2019 11:21:44 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id s2sm456129ioj.8.2019.05.28.11.21.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:44 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILhSF014543; Tue, 28 May 2019 18:21:43 GMT Subject: [PATCH RFC 10/12] xprtrdma: Streamline rpcrdma_post_recvs From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:43 -0400 Message-ID: <20190528182143.19012.75326.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP rb_lock is contended between rpcrdma_buffer_create, rpcrdma_buffer_put, and rpcrdma_post_recvs. Commit e340c2d6ef2a ("xprtrdma: Reduce the doorbell rate (Receive)") causes rpcrdma_post_recvs to take the rb_lock repeatedly when it determines more Receives are needed. Streamline this code path so it takes the lock just once in most cases to build the Receive chain that is about to be posted. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 59 ++++++++++++++++++++++++++++--------------- 1 file changed, 38 insertions(+), 21 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 390c3bc..cc1fdca 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1479,11 +1479,13 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ep *ep = &r_xprt->rx_ep; - struct ib_recv_wr *wr, *bad_wr; + struct ib_recv_wr *i, *wr, *bad_wr; + struct rpcrdma_rep *rep; int needed, count, rc; rc = 0; count = 0; + needed = buf->rb_credits + (buf->rb_bc_srv_max_requests << 1); if (ep->rep_receive_count > needed) goto out; @@ -1491,39 +1493,48 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) if (!temp) needed += RPCRDMA_MAX_RECV_BATCH; - count = 0; + /* fast path: all needed reps can be found on the free list */ wr = NULL; + spin_lock(&buf->rb_lock); while (needed) { - struct rpcrdma_rep *rep; - - spin_lock(&buf->rb_lock); rep = list_first_entry_or_null(&buf->rb_recv_bufs, struct rpcrdma_rep, rr_list); - if (likely(rep)) - list_del(&rep->rr_list); - spin_unlock(&buf->rb_lock); - if (!rep) { - rep = rpcrdma_rep_create(r_xprt, temp); - if (!rep) - break; - } + if (!rep) + break; - if (!rpcrdma_regbuf_dma_map(r_xprt, rep->rr_rdmabuf)) { - rpcrdma_recv_buffer_put(rep); + list_del(&rep->rr_list); + rep->rr_recv_wr.next = wr; + wr = &rep->rr_recv_wr; + --needed; + } + spin_unlock(&buf->rb_lock); + + while (needed) { + rep = rpcrdma_rep_create(r_xprt, temp); + if (!rep) break; - } - trace_xprtrdma_post_recv(rep->rr_recv_wr.wr_cqe); rep->rr_recv_wr.next = wr; wr = &rep->rr_recv_wr; - ++count; --needed; } - if (!count) + if (!wr) goto out; + for (i = wr; i; i = i->next) { + rep = container_of(i, struct rpcrdma_rep, rr_recv_wr); + + if (!rpcrdma_regbuf_dma_map(r_xprt, rep->rr_rdmabuf)) + goto release_wrs; + + trace_xprtrdma_post_recv(rep->rr_recv_wr.wr_cqe); + ++count; + } + rc = ib_post_recv(r_xprt->rx_ia.ri_id->qp, wr, (const struct ib_recv_wr **)&bad_wr); +out: + trace_xprtrdma_post_recvs(r_xprt, count, rc); if (rc) { for (wr = bad_wr; wr;) { struct rpcrdma_rep *rep; @@ -1535,6 +1546,12 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) } } ep->rep_receive_count += count; -out: - trace_xprtrdma_post_recvs(r_xprt, count, rc); + return; + +release_wrs: + for (i = wr; i;) { + rep = container_of(i, struct rpcrdma_rep, rr_recv_wr); + i = i->next; + rpcrdma_recv_buffer_put(rep); + } } From patchwork Tue May 28 18:21:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C265113AD for ; Tue, 28 May 2019 18:22:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B46D528640 for ; Tue, 28 May 2019 18:22:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A8E202864E; Tue, 28 May 2019 18:22:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 51F8428640 for ; Tue, 28 May 2019 18:22:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727961AbfE1SVv (ORCPT ); Tue, 28 May 2019 14:21:51 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:37213 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727428AbfE1SVv (ORCPT ); Tue, 28 May 2019 14:21:51 -0400 Received: by mail-it1-f193.google.com with SMTP id s16so537370ita.2; Tue, 28 May 2019 11:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=m9Pkw4CV2aVA90rqOBAJOqo5SGoCQ+VOLkxHnhH2bO0=; b=F+pKe2tRjOAstateAoDOmO5RRgQxDA7PraZVKTdDusHf98TVJs8r0BnogvdF4D5HbD JlpLJEeU8o6+SI+skUb+TCd0dV4Pue/XaKXKxJxhBpoIXhS4ynik/rKNxgpuSP1+dYOH n4wOne0soQcpFnimDZbwGug+IsDoVLGw0Dp/goErt0InaS6FR/NRBI06fzxnZvncxGZi EXueY6SKvWBEHf2RDlewls3JPgEzvamAwo3kHr0VJ4RTYpkO/vucP1vz0fyoZj5MAIW2 PSSURKUFVUzYvtukEYcZUVMxj+1LrS1KxaVCM5E5Jd6UvkpBzCl4HrlH39oE/jB57wI3 rJ0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=m9Pkw4CV2aVA90rqOBAJOqo5SGoCQ+VOLkxHnhH2bO0=; b=abjdChhMpVhxuI9rWmG32hZFsnWNIY7zVNB0e3lGf642OlPAmGvI8qIHCHwfXSbOv6 /4QaTQFz2hJLpG23QaCvqNwQXVqC0megMvzUQZ3WVafaCeoPw/4M2quezIYNn1Exupq5 wTFQbKLRvIPQ5tHEKzSQX+eVc4Q4Q73MKxtQkzwdk6UuApUFkpVlqIuBwkK+GmEEAmge arWuZnwgPehecnQF7ZasXT8GJ1hfcVJVZ5O9SEQLzYguEsg+UBNtkKiZcgH4xi8WF5Ej lkcJlJ0SxIEHLHepyo5W4CUH8xDhNjLVtlqgf12EokHFgj5o9X90TSfZAenMpANntGT0 ry1A== X-Gm-Message-State: APjAAAVa051XA6gS7pklyvjHzFq2VYu14bz0jsDxjh0XArVYD0j8B5Ua JaknmlcVUnAxikGTpItnD1f9mouJ X-Google-Smtp-Source: APXvYqzqrGkWs3q7V9VB9LDnEm1zFKjOYkH9v4aUA4eO7W+OuysarCbJ0q6L8Lulf5vmMQiOnAjeTg== X-Received: by 2002:a24:4794:: with SMTP id t142mr4239355itb.121.1559067709965; Tue, 28 May 2019 11:21:49 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id 199sm1530028itk.21.2019.05.28.11.21.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:49 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILmaJ014546; Tue, 28 May 2019 18:21:48 GMT Subject: [PATCH RFC 11/12] xprtrdma: Refactor chunk encoding From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:48 -0400 Message-ID: <20190528182148.19012.78763.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up. Move the "not present" case into the individual chunk encoders. This improves code organization and readability. The reason for the original organization was to optimize for the case where there there are no chunks. The optimization turned out to be inconsequential. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 6de90d4..310614f 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -366,6 +366,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, unsigned int pos; int nsegs; + if (rtype == rpcrdma_noch) + goto done; + pos = rqst->rq_snd_buf.head[0].iov_len; if (rtype == rpcrdma_areadch) pos = 0; @@ -389,7 +392,8 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nsegs -= mr->mr_nents; } while (nsegs); - return 0; +done: + return encode_item_not_present(xdr); } /* Register and XDR encode the Write list. Supports encoding a list @@ -417,6 +421,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, int nsegs, nchunks; __be32 *segcount; + if (wtype != rpcrdma_writech) + goto done; + seg = req->rl_segments; nsegs = rpcrdma_convert_iovs(r_xprt, &rqst->rq_rcv_buf, rqst->rq_rcv_buf.head[0].iov_len, @@ -451,7 +458,8 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, /* Update count of segments in this Write chunk */ *segcount = cpu_to_be32(nchunks); - return 0; +done: + return encode_item_not_present(xdr); } /* Register and XDR encode the Reply chunk. Supports encoding an array @@ -476,6 +484,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, int nsegs, nchunks; __be32 *segcount; + if (wtype != rpcrdma_replych) + return encode_item_not_present(xdr); + seg = req->rl_segments; nsegs = rpcrdma_convert_iovs(r_xprt, &rqst->rq_rcv_buf, 0, wtype, seg); if (nsegs < 0) @@ -859,28 +870,13 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, * send a Call message with a Position Zero Read chunk and a * regular Read chunk at the same time. */ - if (rtype != rpcrdma_noch) { - ret = rpcrdma_encode_read_list(r_xprt, req, rqst, rtype); - if (ret) - goto out_err; - } - ret = encode_item_not_present(xdr); + ret = rpcrdma_encode_read_list(r_xprt, req, rqst, rtype); if (ret) goto out_err; - - if (wtype == rpcrdma_writech) { - ret = rpcrdma_encode_write_list(r_xprt, req, rqst, wtype); - if (ret) - goto out_err; - } - ret = encode_item_not_present(xdr); + ret = rpcrdma_encode_write_list(r_xprt, req, rqst, wtype); if (ret) goto out_err; - - if (wtype != rpcrdma_replych) - ret = encode_item_not_present(xdr); - else - ret = rpcrdma_encode_reply_chunk(r_xprt, req, rqst, wtype); + ret = rpcrdma_encode_reply_chunk(r_xprt, req, rqst, wtype); if (ret) goto out_err; From patchwork Tue May 28 18:21:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10965655 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C9F91575 for ; Tue, 28 May 2019 18:21:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7DB5A28640 for ; Tue, 28 May 2019 18:21:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 71B5B28657; Tue, 28 May 2019 18:21:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1283A28640 for ; Tue, 28 May 2019 18:21:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727973AbfE1SV4 (ORCPT ); Tue, 28 May 2019 14:21:56 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:36894 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727811AbfE1SV4 (ORCPT ); Tue, 28 May 2019 14:21:56 -0400 Received: by mail-io1-f67.google.com with SMTP id e5so7573418iok.4; Tue, 28 May 2019 11:21:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=x1+4z7FPCyxG9SE0OvF9MthFwRgmmON5YajNorKmYnI=; b=Rggn0/ctwYlvLQATI8XW2RF8/JofngJ8zeRU1xltM2HkOvciaoTOI5lQZoRb/f7sxz 52qkpYBVx/J7hnR5yWdNbtsBsKiY1UPZACdnzW/5wF60L41cbi0J7zrFgKBogNjowKjY Az2QtDvfnd8bZnVtYgxbXwrWBczGpcpfzZYHzEftFUZyg+976Zjd7FG/0lV2FieSNrYc BDm+p1S1hJac7lRJ+k3su5+uTq/UAjDdOsOTRTdpUvBTeTmjQumgk5MrZb6IvIdFW6AY m92SwPqR31SsDm4ay1Kb2MlUbjlGXP0pNzU4WemchplN5/2k6wZg3t+uUPT7qxt9htuv 8yRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=x1+4z7FPCyxG9SE0OvF9MthFwRgmmON5YajNorKmYnI=; b=Av5yok066HTNdk6pM6svaUKhuThMoCdJOku2yuWG06YumGrZ9UtL8stLDOlGltmMfC uCIC19iTWIq6y2AGQDC/jiZICPBZ9Pjts5xG59hvjdApMnnVmKgl6hW8cSXqAvOcuVIb 380mLPVcnLwQvCQrjjtNonxM3Sus3nmLa2hbDvqptcmqLf+M8x8zXJ9dqPAT3OBaJdYq BZlE20bvn1CtZrs69RmY6ERqTuexzpGHcRgQx1wS0jwOLZxAS9zPtQhjs3clUZlvbcS8 sgVsdGyBiGKpRd1QmOFual6IX0rbvZdBXnNvuSs7IH6T6Zh0rftaOFjqYHTEvNgc9Zww bKxA== X-Gm-Message-State: APjAAAWSxQhToSZ/cKodxrGpLBj3MLY4Fk3knAdLsfyronUeKEnCuo8B KIdr5xkd1fU1dqf/60EKIyhXp4rB X-Google-Smtp-Source: APXvYqyrOuR3p8Ga8DaATcp+xY/zmSBOxj0jeTLbqWfcgSEWWgQkxZOdgF8khX+4WfYWIkVyHmUCxg== X-Received: by 2002:a6b:b642:: with SMTP id g63mr15408439iof.142.1559067715328; Tue, 28 May 2019 11:21:55 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id c1sm1590904ioc.43.2019.05.28.11.21.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 May 2019 11:21:55 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x4SILsBp014549; Tue, 28 May 2019 18:21:54 GMT Subject: [PATCH RFC 12/12] xprtrdma: Remove rpcrdma_req::rl_buffer From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 28 May 2019 14:21:54 -0400 Message-ID: <20190528182154.19012.46781.stgit@manet.1015granger.net> In-Reply-To: <20190528181018.19012.61210.stgit@manet.1015granger.net> References: <20190528181018.19012.61210.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up. There is only one remaining function that uses this field. This caller can supply the correct rpcrdma_buffer, allowing the removal of an 8-byte pointer field. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/transport.c | 5 ++++- net/sunrpc/xprtrdma/verbs.c | 6 ++---- net/sunrpc/xprtrdma/xprt_rdma.h | 4 ++-- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 5b1d5d7..cff090b 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -550,8 +550,11 @@ void xprt_rdma_close(struct rpc_xprt *xprt) static void xprt_rdma_free_slot(struct rpc_xprt *xprt, struct rpc_rqst *rqst) { + struct rpcrdma_xprt *r_xprt = + container_of(xprt, struct rpcrdma_xprt, rx_xprt); + memset(rqst, 0, sizeof(*rqst)); - rpcrdma_buffer_put(rpcr_to_rdmar(rqst)); + rpcrdma_buffer_put(&r_xprt->rx_buf, rpcr_to_rdmar(rqst)); rpc_wake_up_next(&xprt->backlog); } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index cc1fdca..ea97b98 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1020,7 +1020,6 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, if (!req->rl_recvbuf) goto out4; - req->rl_buffer = buffer; INIT_LIST_HEAD(&req->rl_registered); spin_lock(&buffer->rb_lock); list_add(&req->rl_all, &buffer->rb_allreqs); @@ -1300,13 +1299,12 @@ struct rpcrdma_req * /** * rpcrdma_buffer_put - Put request/reply buffers back into pool + * @buffers: buffer pool * @req: object to return * */ -void -rpcrdma_buffer_put(struct rpcrdma_req *req) +void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req) { - struct rpcrdma_buffer *buffers = req->rl_buffer; struct rpcrdma_rep *rep = req->rl_reply; req->rl_reply = NULL; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 5475f0d..117e328 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -320,7 +320,6 @@ enum { struct rpcrdma_req { struct list_head rl_list; struct rpc_rqst rl_slot; - struct rpcrdma_buffer *rl_buffer; struct rpcrdma_rep *rl_reply; struct xdr_stream rl_stream; struct xdr_buf rl_hdrbuf; @@ -499,7 +498,8 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, } struct rpcrdma_req *rpcrdma_buffer_get(struct rpcrdma_buffer *); -void rpcrdma_buffer_put(struct rpcrdma_req *); +void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers, + struct rpcrdma_req *req); void rpcrdma_recv_buffer_put(struct rpcrdma_rep *); bool rpcrdma_regbuf_realloc(struct rpcrdma_regbuf *rb, size_t size,