From patchwork Wed Oct 9 17:07:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11181665 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3254913BD for ; Wed, 9 Oct 2019 17:07:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 113782190F for ; Wed, 9 Oct 2019 17:07:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q953NcmU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731433AbfJIRHY (ORCPT ); Wed, 9 Oct 2019 13:07:24 -0400 Received: from mail-io1-f65.google.com ([209.85.166.65]:36825 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731173AbfJIRHY (ORCPT ); Wed, 9 Oct 2019 13:07:24 -0400 Received: by mail-io1-f65.google.com with SMTP id b136so6697769iof.3; Wed, 09 Oct 2019 10:07:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=Lz9tMIsumw5i1GzZW4l3sduzzds0vLRRwfuvTlZdP90=; b=Q953NcmU0SegOoVdWGIXN43J0tyeRlM+susb+7kbwlBiuILW/PpBNO95m4xqIBw9+k 9gJso4vPvt9aK9FIrCMlS7t68bWbjiv/PtJrsNhw0fSWvN0MlRLUS6sBfdYz7+gIaxwg 6xaL4F9EJgy4Ag+WyBxLCeJNCuy31GJuTjsJwIfE5fg0uH7CNsN8zI6+SKAtJagxxdT1 egb6dAhMFw/fprrKYMXGb1LuUM2NIZLF0Qi4posL1egWv9bvLKlHf89Vokoo0F3JqI3c wuOCrgqEptRrfCdWw57yjvkp26KbiyP//C2h6TPTyW4MyCV5tiugE4OEd1vtW9F2Lzwd +KpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :user-agent:mime-version:content-transfer-encoding; bh=Lz9tMIsumw5i1GzZW4l3sduzzds0vLRRwfuvTlZdP90=; b=MCir1xzBDHZFBqLNXXpqv+Vnn5UAjwTqfEvSplshsb6F4MNcUsN4YD0x+I018BvPM2 Q/hsWQpxTsl5CuEoLoeGHKdKB2cnn7SlMt2nJ5PZxHBNtO1iBXei00ctHI2kt8H/km2H V3NSP20c80rA55zO1Rq0lHR6sXhzPHm1IMC2WjpHRcLXSNesKvhSMQHb0fpELI0Rzk9s RIG/HzpElT0SKbMDAykgLlfOzDCcfNsNArI+2UXzuSNKISF8warjz6mQFbnGiRQAqn9M 2rZtkjOCouiSKZN/2jp/D3nQqmIwxil6oY4zhrNvEU8BeZBvpj/k6eyjXxAZ7McwCLyG ad1g== X-Gm-Message-State: APjAAAWP+iKax86UQXFQ5Ff8e3l0XrsTfmoV+cy5WvNMKgYHMwmrUMS0 icYoofxwJwCkpiJQkodGo16YvPlN X-Google-Smtp-Source: APXvYqxOdmfG9QGtulQdviHiihmGJZT+CItHkjiqq/PFDINYoktC5xAl6KMnLc3vr5Q65KdUR9MYbQ== X-Received: by 2002:a02:6508:: with SMTP id u8mr4367752jab.28.1570640843216; Wed, 09 Oct 2019 10:07:23 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id x12sm1479581ioh.76.2019.10.09.10.07.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Oct 2019 10:07:22 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x99H7LoH001489; Wed, 9 Oct 2019 17:07:21 GMT Subject: [PATCH v1 1/6] xprtrdma: Add unique trace points for posting Local Invalidate WRs From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Oct 2019 13:07:21 -0400 Message-ID: <20191009170721.2978.128.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org When adding frwr_unmap_async way back when, I re-used the existing trace_xprtrdma_post_send() trace point to record the return code of ib_post_send. Unfortunately there are some cases where re-using that trace point causes a crash. Instead, construct a trace point specific to posting Local Invalidate WRs that will always be safe to use in that context, and will act as a trace log eye-catcher for Local Invalidation. Fixes: 847568942f93 ("xprtrdma: Remove fr_state") Fixes: d8099feda483 ("xprtrdma: Reduce context switching due ... ") Signed-off-by: Chuck Lever Tested-by: Bill Baker --- include/trace/events/rpcrdma.h | 25 +++++++++++++++++++++++++ net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 9dd7680..31681cb 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -735,6 +735,31 @@ ) ); +TRACE_EVENT(xprtrdma_post_linv, + TP_PROTO( + const struct rpcrdma_req *req, + int status + ), + + TP_ARGS(req, status), + + TP_STRUCT__entry( + __field(const void *, req) + __field(int, status) + __field(u32, xid) + ), + + TP_fast_assign( + __entry->req = req; + __entry->status = status; + __entry->xid = be32_to_cpu(req->rl_slot.rq_xid); + ), + + TP_printk("req=%p xid=0x%08x status=%d", + __entry->req, __entry->xid, __entry->status + ) +); + /** ** Completion events **/ diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 30065a2..9901a81 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -570,7 +570,6 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) */ bad_wr = NULL; rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); - trace_xprtrdma_post_send(req, rc); /* The final LOCAL_INV WR in the chain is supposed to * do the wake. If it was never posted, the wake will @@ -583,6 +582,7 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) /* Recycle MRs in the LOCAL_INV chain that did not get posted. */ + trace_xprtrdma_post_linv(req, rc); while (bad_wr) { frwr = container_of(bad_wr, struct rpcrdma_frwr, fr_invwr); @@ -673,12 +673,12 @@ void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) */ bad_wr = NULL; rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); - trace_xprtrdma_post_send(req, rc); if (!rc) return; /* Recycle MRs in the LOCAL_INV chain that did not get posted. */ + trace_xprtrdma_post_linv(req, rc); while (bad_wr) { frwr = container_of(bad_wr, struct rpcrdma_frwr, fr_invwr); mr = container_of(frwr, struct rpcrdma_mr, frwr); From patchwork Wed Oct 9 17:07:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11181669 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 08FFC17EE for ; Wed, 9 Oct 2019 17:07:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DCD0B218AC for ; Wed, 9 Oct 2019 17:07:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NFrl/zSZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731584AbfJIRHb (ORCPT ); Wed, 9 Oct 2019 13:07:31 -0400 Received: from mail-io1-f68.google.com ([209.85.166.68]:42236 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731173AbfJIRHb (ORCPT ); Wed, 9 Oct 2019 13:07:31 -0400 Received: by mail-io1-f68.google.com with SMTP id n197so6606570iod.9; Wed, 09 Oct 2019 10:07:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=R+veZFw5uLnQWWdDzkaXSVH6JXa3tUr8mmWEO/GcUJs=; b=NFrl/zSZfSX0RzpfvNZUD9WmTwB1XbeVmOJIieGoFJ7T/7ZCnAwP3gLifhDab5DkVh k9GoI01ynXEk3HDpZkFaL7c8N5A8VWnYyaxWEUvnL8EvZ9AzQ7ZAz11vgaDOMVT/0Scg Uql1v7m/hd2axOZwU8YW23kO3hKWCLt7TqeyrpTN6JWUhOtSyFpvr7HVv0SovEHIi87R PmmeVkZqp+G01Md3qocrEX7qM2CMPJO2uzw3CWle5SE8KPCPyGLOJG10eOfXQQ/AmbMx 4Ik0fJQDlEu5SY6mlDiHmGalVeWvorLsFqBvPIQUKTT63kIGM+UhU3taUQSLAlH6C/oh JXhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=R+veZFw5uLnQWWdDzkaXSVH6JXa3tUr8mmWEO/GcUJs=; b=ARi5NGZD6+EpR1/IkWRdbBnTRN2yVLEXUgtEXlGt/+uetckLXiWvz/buEYC0C9/lHV Uog5kMbFVZgSvr//mOGlLCxiN+qg+bB0ymibnqn8gGxu+EFkgOxDC32Z1Y38GcXo1bEK PY5tZaBwcf3fC5mMo3bzhy9LitjPR1IL0uzm1mqrKG4AsOppqd8QL8rTH52SYjdeebaI zDJL/ori2eEdZXLVQd/qru1GJWIt4IpZS5RhHutp3DUv2TnMaxqbJLJQY8Li0NU2nTUG a8kcqQjmYKq3wPH/TUM6kIbXYhV4KyZFs7JFvY+k6vUOMpC/A8iF4epgtLOBb9jBYz/9 r+6g== X-Gm-Message-State: APjAAAVLrfpr4WG4XEO+GuglrUozKMg3LJZOx5jEgl/tUMMisrHyFAE5 +xwfwcwH2JRr+ni5BqmdbO86H/Lt X-Google-Smtp-Source: APXvYqznnvo18JlVkrZpqe51Mpn5zzZtGWKc5wAoJzwLv0r08d8Yggix7ldhbAGikdrPn0acUt+lqg== X-Received: by 2002:a02:aa0b:: with SMTP id r11mr4349590jam.26.1570640848606; Wed, 09 Oct 2019 10:07:28 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id h4sm1336763iom.17.2019.10.09.10.07.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Oct 2019 10:07:28 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x99H7RhO001492; Wed, 9 Oct 2019 17:07:27 GMT Subject: [PATCH v1 2/6] xprtrdma: Connection becomes unstable after a reconnect From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Oct 2019 13:07:27 -0400 Message-ID: <20191009170727.2978.14774.stgit@manet.1015granger.net> In-Reply-To: <20191009170721.2978.128.stgit@manet.1015granger.net> References: <20191009170721.2978.128.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org This is because xprt_request_get_cong() is allowing more than one RPC Call to be transmitted before the first Receive on the new connection. The first Receive fills the Receive Queue based on the server's credit grant. Before that Receive, there is only a single Receive WR posted because the client doesn't know the server's credit grant. Solution is to clear rq_cong on all outstanding rpc_rqsts when the the cwnd is reset. This is because an RPC/RDMA credit is good for one connection instance only. Fixes: 75891f502f5f ("SUNRPC: Support for congestion control ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/transport.c | 3 +++ net/sunrpc/xprtrdma/verbs.c | 22 ++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 160558b..c67d465 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -428,8 +428,11 @@ void xprt_rdma_close(struct rpc_xprt *xprt) /* Prepare @xprt for the next connection by reinitializing * its credit grant to one (see RFC 8166, Section 3.3.3). */ + spin_lock(&xprt->transport_lock); r_xprt->rx_buf.rb_credits = 1; + xprt->cong = 0; xprt->cwnd = RPC_CWNDSHIFT; + spin_unlock(&xprt->transport_lock); out: xprt->reestablish_timeout = 0; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3a90753..f4b1365 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -75,6 +75,7 @@ * internal functions */ static void rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc); +static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt); static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); @@ -780,6 +781,7 @@ static int rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, trace_xprtrdma_disconnect(r_xprt, rc); rpcrdma_xprt_drain(r_xprt); + rpcrdma_reqs_reset(r_xprt); } /* Fixed-size circular FIFO queue. This implementation is wait-free and @@ -1042,6 +1044,26 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, return NULL; } +/** + * rpcrdma_reqs_reset - Reset all reqs owned by a transport + * @r_xprt: controlling transport instance + * + * ASSUMPTION: the rb_allreqs list is stable for the duration, + * and thus can be walked without holding rb_lock. Eg. the + * caller is holding the transport send lock to exclude + * device removal or disconnection. + */ +static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt) +{ + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_req *req; + + list_for_each_entry(req, &buf->rb_allreqs, rl_all) { + /* Credits are valid only for one connection */ + req->rl_slot.rq_cong = 0; + } +} + static struct rpcrdma_rep *rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, bool temp) { From patchwork Wed Oct 9 17:07:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11181673 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7C4E13BD for ; Wed, 9 Oct 2019 17:07:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A7AEB20B7C for ; Wed, 9 Oct 2019 17:07:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="b4giwvtU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731821AbfJIRHg (ORCPT ); Wed, 9 Oct 2019 13:07:36 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:41887 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731173AbfJIRHg (ORCPT ); Wed, 9 Oct 2019 13:07:36 -0400 Received: by mail-io1-f67.google.com with SMTP id n26so6621550ioj.8; Wed, 09 Oct 2019 10:07:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=+2MlgiRc3Dhzscg9dsXGedOrjDQX3x+rEItIK/b1K2s=; b=b4giwvtU9Mo9btJMUW6pRnhvkcbABjDHiSZGPqPEa04GurvT0dauoRNUdRzS0R1xVn S7/n0zQhYuSoXI6CnF8cl2tiA6ZS4aF4cMOWHgfI7vRc2489PmbKO6oyRm8WvHNfFd+j bpmVYxXkX1vEBQK423BTOhppXPDKMfVjxATfeJdeM0DK8szWO92xnkvZVmX2OVLBfF/p 7WdQYXL0Ad4omFemXCQFRBMAuI2Jpq4w7QQzCI5D3qn3kEYxV/p9jk7w11kyw5VTNjss n4NZet7iD0eotWumR51jfcCN3cAxNDGnbWCo/UC3P5PQqmGPMPX4wIvAwlSewqN9DsZj CppQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=+2MlgiRc3Dhzscg9dsXGedOrjDQX3x+rEItIK/b1K2s=; b=AhnJInq3GR9yr/ivTHF7Ipd+3QwuM0ibtxoYWvYoLo9xY9T7DJxLDWk8RY+IXGl2IM 8trlaYM8RPsfdt1hehzgixmlzOmT8kONrUSxjwNY8FNPSfI7TO4w93GV+9QnsHnkxwCS sZAYPSlIwn6YmOC6gEJZwxF4GmhIolJdtNhs3iXHjTpcTDGe8VpOVx5NEGubYrMFbBhy U6OtKbmXuYfVuKJSWFucHmwyljJ0sJ1yPTNJ6xaVvaG6iBIVPnqTQU3qk43bNQ5X6bIE MP6D27piXKMRke55bQKZ901VIY/JvrAhDB+kheGYyGxGrqnmeIfOtKzivZVQJmbiGDBh 2LOw== X-Gm-Message-State: APjAAAVfp64aogRaeWsV6U19ese1kpmE/Gipeg143AeQPeWEIGX22cia qGIjUvLgJBUs4uFG3DAr3dFziy1z X-Google-Smtp-Source: APXvYqz0kCITGHU0cyilHwZ+C6iFwZQHpHZBOjN60Edtz8uPApSnqa9m+AxINW1qefu+V1bIrBMgjQ== X-Received: by 2002:a6b:b54c:: with SMTP id e73mr4594933iof.259.1570640853790; Wed, 09 Oct 2019 10:07:33 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id a25sm1096357iod.62.2019.10.09.10.07.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Oct 2019 10:07:33 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x99H7WE2001495; Wed, 9 Oct 2019 17:07:32 GMT Subject: [PATCH v1 3/6] xprtrdma: Initialize rb_credits in one place From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Oct 2019 13:07:32 -0400 Message-ID: <20191009170732.2978.21414.stgit@manet.1015granger.net> In-Reply-To: <20191009170721.2978.128.stgit@manet.1015granger.net> References: <20191009170721.2978.128.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up/code de-duplication. Nit: RPC_CWNDSHIFT is incorrect as the initial value for xprt->cwnd. This mistake does not appear to have operational consequences, since the cwnd value is replaced with a valid value upon the first Receive completion. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 42 +++++++++++++++++++++++++++++++++------ net/sunrpc/xprtrdma/transport.c | 9 -------- net/sunrpc/xprtrdma/verbs.c | 2 +- net/sunrpc/xprtrdma/xprt_rdma.h | 1 + 4 files changed, 38 insertions(+), 16 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index b86b5fd..f1e3639 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -916,6 +916,40 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, return ret; } +static void __rpcrdma_update_cwnd_locked(struct rpc_xprt *xprt, + struct rpcrdma_buffer *buf, + u32 grant) +{ + buf->rb_credits = grant; + xprt->cwnd = grant << RPC_CWNDSHIFT; +} + +static void rpcrdma_update_cwnd(struct rpcrdma_xprt *r_xprt, u32 grant) +{ + struct rpc_xprt *xprt = &r_xprt->rx_xprt; + + spin_lock(&xprt->transport_lock); + __rpcrdma_update_cwnd_locked(xprt, &r_xprt->rx_buf, grant); + spin_unlock(&xprt->transport_lock); +} + +/** + * rpcrdma_reset_cwnd - Reset the xprt's congestion window + * @r_xprt: controlling transport instance + * + * Prepare @r_xprt for the next connection by reinitializing + * its credit grant to one (see RFC 8166, Section 3.3.3). + */ +void rpcrdma_reset_cwnd(struct rpcrdma_xprt *r_xprt) +{ + struct rpc_xprt *xprt = &r_xprt->rx_xprt; + + spin_lock(&xprt->transport_lock); + xprt->cong = 0; + __rpcrdma_update_cwnd_locked(xprt, &r_xprt->rx_buf, 1); + spin_unlock(&xprt->transport_lock); +} + /** * rpcrdma_inline_fixup - Scatter inline received data into rqst's iovecs * @rqst: controlling RPC request @@ -1356,12 +1390,8 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) credits = 1; /* don't deadlock */ else if (credits > buf->rb_max_requests) credits = buf->rb_max_requests; - if (buf->rb_credits != credits) { - spin_lock(&xprt->transport_lock); - buf->rb_credits = credits; - xprt->cwnd = credits << RPC_CWNDSHIFT; - spin_unlock(&xprt->transport_lock); - } + if (buf->rb_credits != credits) + rpcrdma_update_cwnd(r_xprt, credits); req = rpcr_to_rdmar(rqst); if (req->rl_reply) { diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index c67d465..0711308 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -425,15 +425,6 @@ void xprt_rdma_close(struct rpc_xprt *xprt) return; rpcrdma_ep_disconnect(ep, ia); - /* Prepare @xprt for the next connection by reinitializing - * its credit grant to one (see RFC 8166, Section 3.3.3). - */ - spin_lock(&xprt->transport_lock); - r_xprt->rx_buf.rb_credits = 1; - xprt->cong = 0; - xprt->cwnd = RPC_CWNDSHIFT; - spin_unlock(&xprt->transport_lock); - out: xprt->reestablish_timeout = 0; ++xprt->connect_cookie; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index f4b1365..97bc15e 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -727,6 +727,7 @@ static int rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, ep->rep_connected = 0; xprt_clear_connected(xprt); + rpcrdma_reset_cwnd(r_xprt); rpcrdma_post_recvs(r_xprt, true); rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma); @@ -1163,7 +1164,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) list_add(&req->rl_list, &buf->rb_send_bufs); } - buf->rb_credits = 1; init_llist_head(&buf->rb_free_reps); rc = rpcrdma_sendctxs_create(r_xprt); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 65e6b0e..b170cf5 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -576,6 +576,7 @@ int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc); int rpcrdma_marshal_req(struct rpcrdma_xprt *r_xprt, struct rpc_rqst *rqst); void rpcrdma_set_max_header_sizes(struct rpcrdma_xprt *); +void rpcrdma_reset_cwnd(struct rpcrdma_xprt *r_xprt); void rpcrdma_complete_rqst(struct rpcrdma_rep *rep); void rpcrdma_reply_handler(struct rpcrdma_rep *rep); From patchwork Wed Oct 9 17:07:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11181677 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D69213BD for ; Wed, 9 Oct 2019 17:07:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7D78120B7C for ; Wed, 9 Oct 2019 17:07:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="otVlT0Wg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731878AbfJIRHl (ORCPT ); Wed, 9 Oct 2019 13:07:41 -0400 Received: from mail-io1-f68.google.com ([209.85.166.68]:44723 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731173AbfJIRHk (ORCPT ); Wed, 9 Oct 2019 13:07:40 -0400 Received: by mail-io1-f68.google.com with SMTP id w12so6585326iol.11; Wed, 09 Oct 2019 10:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=AMNLq7xDZcAcz1zxL24cAbXH24gbHtJzkdtcWX5wkwk=; b=otVlT0Wgqh717GMZ3JzErGI7RZrbUFSC+JDy+PIwvQTz/TZzsxItXFkQPrYGjJtVxe p4WTwbzbOQUj8dz/sAckqukzLCxZmplFvcYDwt1tuXkOfNJODeqyYlGmZW0W8bTQzvTg qO0lmY54WVE0azEdS/g+YJd4UnOb7Bkx3x534Dz9DW2KiOoTxjjAJ/ttk1L5IJZuqCfc mWrQHQu966gDr0lElDipGJPONwtkonr74mZFvtnUAFTjpEyzDKZOUKr5cBjqYkSpTcO7 dKKPFREJf1a1TOPXTf9S40hJED6j3hsuAD7N0aCB/miyv+106ZfnvNYaFK5QRN5ee6dB 9ETQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=AMNLq7xDZcAcz1zxL24cAbXH24gbHtJzkdtcWX5wkwk=; b=ZO+kN/nQuM0K/NGEBuFoVvFGU3J+4DJp9DoSQFTy3gixbvLOgYk0ZHMT1jd8AMzrKy THVhk2o2lneddrV6ZEQRi4DdQxw/DzrWkRBNKAT6fhulJZDJzwBGRlJPEVVKccIdtcxy DYKEF1DGHTboL9B0bWqj2wDUQglYOBNIuwc+H/JtJ/y5i0op81HamuBJs0MknPwUIWLh K+5Y2Ftc3yiUcHjxfaTXOj8ojItC93doi6SU0plzrM6D883JAa5k6TWMCwwONHQH6K3V k3jl4oM+dPOyo/QXqyXKtYmJRLCpUeq82Mkyim7o3ETD8Cq+fy+0cSofxxUiuiaBkMll GuWg== X-Gm-Message-State: APjAAAWfI5PNYOydl/ytxbJbFmKNTjDfMPajoWU8pIYBjPGR1+gSbGPU Qy/2CvUaByIt8PJxFexMjwT1+LHi X-Google-Smtp-Source: APXvYqwKfSl7HszFNYPsHuySKtiitMXcfbVEAasUUYIdpTzaPFOPTvBIqKR+72bgg6dtCP/NmMGcWQ== X-Received: by 2002:a5e:9410:: with SMTP id q16mr59429ioj.302.1570640859216; Wed, 09 Oct 2019 10:07:39 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id q3sm1229051ioi.68.2019.10.09.10.07.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Oct 2019 10:07:38 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x99H7ccM001498; Wed, 9 Oct 2019 17:07:38 GMT Subject: [PATCH v1 4/6] xprtrdma: Close window between waking RPC senders and posting Receives From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Oct 2019 13:07:38 -0400 Message-ID: <20191009170737.2978.4060.stgit@manet.1015granger.net> In-Reply-To: <20191009170721.2978.128.stgit@manet.1015granger.net> References: <20191009170721.2978.128.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org A recent clean up attempted to separate Receive handling and RPC Reply processing, in the name of clean layering. Unfortunately, we can't do this because the Receive Queue has to be refilled _after_ the most recent credit update from the responder is parsed from the transport header, but _before_ we wake up the next RPC sender. That is right in the middle of rpcrdma_reply_handler(). Usually this isn't a problem because current responder implementations don't vary their credit grant. The one exception is when a connection is established: the grant goes from one to a much larger number on the first Receive. The requester MUST post enough Receives right then so that any outstanding requests can be sent without risking RNR and connection loss. Fixes: 6ceea36890a0 ("xprtrdma: Refactor Receive accounting") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 1 + net/sunrpc/xprtrdma/verbs.c | 11 +++++++---- net/sunrpc/xprtrdma/xprt_rdma.h | 1 + 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index f1e3639..7c125e6 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1392,6 +1392,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) credits = buf->rb_max_requests; if (buf->rb_credits != credits) rpcrdma_update_cwnd(r_xprt, credits); + rpcrdma_post_recvs(r_xprt, false); req = rpcr_to_rdmar(rqst); if (req->rl_reply) { diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 97bc15e..3a1a31c 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -85,7 +85,6 @@ gfp_t flags); static void rpcrdma_regbuf_dma_unmap(struct rpcrdma_regbuf *rb); static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb); -static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); /* Wait for outstanding transport work to finish. ib_drain_qp * handles the drains in the wrong order for us, so open code @@ -171,7 +170,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) rdmab_addr(rep->rr_rdmabuf), wc->byte_len, DMA_FROM_DEVICE); - rpcrdma_post_recvs(r_xprt, false); rpcrdma_reply_handler(rep); return; @@ -1477,8 +1475,13 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) return 0; } -static void -rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) +/** + * rpcrdma_post_recvs - Refill the Receive Queue + * @r_xprt: controlling transport instance + * @temp: mark Receive buffers to be deleted after use + * + */ +void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ep *ep = &r_xprt->rx_ep; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index b170cf5..2f89cfc 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -474,6 +474,7 @@ struct rpcrdma_xprt { int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *, struct rpcrdma_req *); +void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); /* * Buffer calls - xprtrdma/verbs.c From patchwork Wed Oct 9 17:07:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11181683 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87DCF1864 for ; Wed, 9 Oct 2019 17:07:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6850F218AC for ; Wed, 9 Oct 2019 17:07:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uj/eT2Qi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731904AbfJIRHp (ORCPT ); Wed, 9 Oct 2019 13:07:45 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:43545 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731173AbfJIRHp (ORCPT ); Wed, 9 Oct 2019 13:07:45 -0400 Received: by mail-io1-f67.google.com with SMTP id v2so6615632iob.10; Wed, 09 Oct 2019 10:07:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=G6n8CniltSB2k/H07ucx+zOK4rEf38iNpHeH0fwJg4Y=; b=uj/eT2QiKDv8uCEK4jTvHhS3mHGzMsw6x8CC3QmC7us0DtF+jTMHjzgWVx8rK4LTCu Ado8494thV4CN5NndddFWifD/8B/eW5p22q8+UH0DQ87mCc6zDR+ITIFQYH7SlWyHjjD U6zXmzaN22X/nOPjrMuPHHeLajlYd/tTOe5a6y+EOYLS3REh9zQdU6ivDtLb3Wv4LCDe HAfRrwS06bumTVtxjYAZo7gXgmA57eitOfGEmFL04QrqzeOiupUCZh5N7pEwK0yCwI3j b7tgrdSAYAILtk82lXy+iCeR+Nepu7YQD/QIlWjqjFNTxD3Z065b/wP2ZnbNmveKTdfD E9NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=G6n8CniltSB2k/H07ucx+zOK4rEf38iNpHeH0fwJg4Y=; b=FZRK8lRAJ7FZETInSjo5/2ykf+0O2KeE6uYsF6q2K+BYcZJBGBR2T1EbP4X5f7SauE t5L57OIEToARXO/0Hg1EaZI44vlvecUNChvtuhIB1OY3Q8bxGUsZfjb13+f5WD2Lz0xX GZVWQnoNQRpsixp/vBDyHfn3NrSNqEs6MF8MpASq40MKjZFDG4XGedZsqRp0SI/PNr9J TQT7o0iG4WV17R6//hggTrFNVOty/9x7ddaCgobto9PlMiWETaNREbQ3VeUzRu7cUhr2 88dhs8dSZu0IWxO4IaffHcB4aHlALCwjGe6cRPhOkHIPq1cDQDMX2qOwQUFECWpfCt5P eWvg== X-Gm-Message-State: APjAAAXgdXqmF0w4aPbS/63j/93Bee2y9PMW2UVrBh2fNrdjp+JolcBQ 9ku2Y6G3YoVXzuWo7rqL9IYZiyc9 X-Google-Smtp-Source: APXvYqzv29ZoVWuAMcca83s4uJM2BEUQ1OgUnYBatNROgnwlLR7h92nZIVywRMKvnG2wq8dhtITGGA== X-Received: by 2002:a6b:7d0b:: with SMTP id c11mr4759547ioq.236.1570640864649; Wed, 09 Oct 2019 10:07:44 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id a26sm1357396iot.46.2019.10.09.10.07.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Oct 2019 10:07:44 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x99H7hRo001501; Wed, 9 Oct 2019 17:07:43 GMT Subject: [PATCH v1 5/6] xprtrdma: Fix MR list handling From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Oct 2019 13:07:43 -0400 Message-ID: <20191009170743.2978.11370.stgit@manet.1015granger.net> In-Reply-To: <20191009170721.2978.128.stgit@manet.1015granger.net> References: <20191009170721.2978.128.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Close some holes introduced by commit 6dc6ec9e04c4 ("xprtrdma: Cache free MRs in each rpcrdma_req") that could result in list corruption. In addition, the result that is tabulated in @count is no longer used, so @count is removed. Fixes: 6dc6ec9e04c4 ("xprtrdma: Cache free MRs in each rpcrdma_req") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3a1a31c..82361e7 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -79,7 +79,6 @@ static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); -static void rpcrdma_mr_free(struct rpcrdma_mr *mr); static struct rpcrdma_regbuf * rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction, gfp_t flags); @@ -966,7 +965,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) mr->mr_xprt = r_xprt; spin_lock(&buf->rb_lock); - list_add(&mr->mr_list, &buf->rb_mrs); + rpcrdma_mr_push(mr, &buf->rb_mrs); list_add(&mr->mr_all, &buf->rb_all_mrs); spin_unlock(&buf->rb_lock); } @@ -1183,10 +1182,19 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) */ void rpcrdma_req_destroy(struct rpcrdma_req *req) { + struct rpcrdma_mr *mr; + list_del(&req->rl_all); - while (!list_empty(&req->rl_free_mrs)) - rpcrdma_mr_free(rpcrdma_mr_pop(&req->rl_free_mrs)); + while ((mr = rpcrdma_mr_pop(&req->rl_free_mrs))) { + struct rpcrdma_buffer *buf = &mr->mr_xprt->rx_buf; + + spin_lock(&buf->rb_lock); + list_del(&mr->mr_all); + spin_unlock(&buf->rb_lock); + + frwr_release_mr(mr); + } rpcrdma_regbuf_free(req->rl_recvbuf); rpcrdma_regbuf_free(req->rl_sendbuf); @@ -1194,24 +1202,28 @@ void rpcrdma_req_destroy(struct rpcrdma_req *req) kfree(req); } -static void -rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) +/** + * rpcrdma_mrs_destroy - Release all of a transport's MRs + * @buf: controlling buffer instance + * + * Relies on caller holding the transport send lock to protect + * removing mr->mr_list from req->rl_free_mrs safely. + */ +static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) { struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt, rx_buf); struct rpcrdma_mr *mr; - unsigned int count; - count = 0; spin_lock(&buf->rb_lock); while ((mr = list_first_entry_or_null(&buf->rb_all_mrs, struct rpcrdma_mr, mr_all)) != NULL) { + list_del(&mr->mr_list); list_del(&mr->mr_all); spin_unlock(&buf->rb_lock); frwr_release_mr(mr); - count++; spin_lock(&buf->rb_lock); } spin_unlock(&buf->rb_lock); @@ -1284,17 +1296,6 @@ void rpcrdma_mr_put(struct rpcrdma_mr *mr) rpcrdma_mr_push(mr, &mr->mr_req->rl_free_mrs); } -static void rpcrdma_mr_free(struct rpcrdma_mr *mr) -{ - struct rpcrdma_xprt *r_xprt = mr->mr_xprt; - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; - - mr->mr_req = NULL; - spin_lock(&buf->rb_lock); - rpcrdma_mr_push(mr, &buf->rb_mrs); - spin_unlock(&buf->rb_lock); -} - /** * rpcrdma_buffer_get - Get a request buffer * @buffers: Buffer pool from which to obtain a buffer From patchwork Wed Oct 9 17:07:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11181685 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B312F13BD for ; Wed, 9 Oct 2019 17:07:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8A3B9218DE for ; Wed, 9 Oct 2019 17:07:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fgH4sIMQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731788AbfJIRHw (ORCPT ); Wed, 9 Oct 2019 13:07:52 -0400 Received: from mail-io1-f68.google.com ([209.85.166.68]:44772 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731173AbfJIRHv (ORCPT ); Wed, 9 Oct 2019 13:07:51 -0400 Received: by mail-io1-f68.google.com with SMTP id w12so6586858iol.11; Wed, 09 Oct 2019 10:07:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=yZ4qkD9pdlXqRniqBictIawhNMxJa12wZcHix94iZ9Y=; b=fgH4sIMQCyYnpOhQfmxmgSIZUNKL9VS184s0VZrIhA2jCpQG4dgcI0DI1pASrJgf3d SY2Ca+R4kDyDGnetnER91VQ8sgEU/xz4+1GKQyuQug3GLiQMYRScV5NnMq9wu/jc8isH PcnE7WgfgUYN4JLkN4uHQUt5t9FhY6Ry2CgopgN1aMzVnXdOejUiTEUOTN4h/Ev5f5oq Jlv3mXcJUSlo58RXflK4BKDpwy/bSqdkVH5EQ5puslu/vFyzVoPOIstBMAwqUDJwk7Ff xQCrD3MbuiWH13eFW+MRI+yn9J1rdRvJ36O3zTaH8yQpM8RhHjkSfZqGxKzAnX1fRTBz hoCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=yZ4qkD9pdlXqRniqBictIawhNMxJa12wZcHix94iZ9Y=; b=NXTH97qysP9amLCG6WhrWslKOGD4lfqnBfYhzwepAMKb7dmd4axrUSYZHGvr3hP5oK WPide1g7LwFdX8/6+uGuRnAMBOKVwyJdUIUNAdVhGbvJdyM3Xx/EztDDZG+sRZIuAEpN N/2lMNsykJlFilV6mUDSGo+3OSSppnFww0u+Zc+rMYw0O81LKYASi7Upaga3ALPiaBHV hzUMkZk79Qr4cfxnnVfA30M5XJ1MsPx6uYXlFSGIEckfeZMK+37mjA/ryqn8wRi+la23 6pRAXQds2IMnZxlTtUxcbtTaDKk1PvEuiauA4+rTU6YlsDVYjEZ6ll6jtdc1LsjY3XfN rukA== X-Gm-Message-State: APjAAAXZIPLqBk0AqHU0dLMFrDAkasABys8BodopfMqv/FygsLyTkh5W 8U8E++VGBsE3bitfh8aJixAfc7JL X-Google-Smtp-Source: APXvYqxgN1Kw+SKUYJUXRgHpQ10Yn3Mz1WqDy/1szHU8RXk54G8/Oa4MhL3murbN2CyNmQvCte0QGw== X-Received: by 2002:a6b:6418:: with SMTP id t24mr4393761iog.185.1570640870345; Wed, 09 Oct 2019 10:07:50 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id 128sm1598065iox.35.2019.10.09.10.07.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Oct 2019 10:07:49 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id x99H7mTA001504; Wed, 9 Oct 2019 17:07:48 GMT Subject: [PATCH v1 6/6] xprtrdma: Manage MRs in context of a single connection From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Oct 2019 13:07:48 -0400 Message-ID: <20191009170748.2978.97255.stgit@manet.1015granger.net> In-Reply-To: <20191009170721.2978.128.stgit@manet.1015granger.net> References: <20191009170721.2978.128.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org MRs are now allocated on demand so we can safely throw them away on disconnect. This way an idle transport can disconnect and it won't pin hardware MR resources. Two additional changes: - Now that all MRs are destroyed on disconnect, there's no need to check during header marshaling if a req has MRs to recycle. Each req is sent only once per connection, and now rl_registered is guaranteed to be empty when rpcrdma_marshal_req is invoked. - Because MRs are now destroyed in a WQ_MEM_RECLAIM context, they also must be allocated in a WQ_MEM_RECLAIM context. This reduces the likelihood that device driver memory allocation will trigger memory reclaim during NFS writeback. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 24 +-------------- net/sunrpc/xprtrdma/rpc_rdma.c | 9 +----- net/sunrpc/xprtrdma/verbs.c | 62 +++++++++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 2 + 4 files changed, 40 insertions(+), 57 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 9901a81..37ba82d 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -36,8 +36,8 @@ * connect worker from running concurrently. * * When the underlying transport disconnects, MRs that are in flight - * are flushed and are likely unusable. Thus all flushed MRs are - * destroyed. New MRs are created on demand. + * are flushed and are likely unusable. Thus all MRs are destroyed. + * New MRs are created on demand. */ #include @@ -119,20 +119,6 @@ static void frwr_mr_recycle(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr) frwr_mr_recycle(mr->mr_xprt, mr); } -/* frwr_recycle - Discard MRs - * @req: request to reset - * - * Used after a reconnect. These MRs could be in flight, we can't - * tell. Safe thing to do is release them. - */ -void frwr_recycle(struct rpcrdma_req *req) -{ - struct rpcrdma_mr *mr; - - while ((mr = rpcrdma_mr_pop(&req->rl_registered))) - frwr_mr_recycle(mr->mr_xprt, mr); -} - /* frwr_reset - Place MRs back on the free list * @req: request to reset * @@ -166,9 +152,6 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) struct ib_mr *frmr; int rc; - /* NB: ib_alloc_mr and device drivers typically allocate - * memory with GFP_KERNEL. - */ frmr = ib_alloc_mr(ia->ri_pd, ia->ri_mrtype, depth); if (IS_ERR(frmr)) goto out_mr_err; @@ -440,9 +423,6 @@ int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) post_wr = &frwr->fr_regwr.wr; } - /* If ib_post_send fails, the next ->send_request for - * @req will queue these MRs for recovery. - */ return ib_post_send(ia->ri_id->qp, post_wr, NULL); } diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 7c125e6..7b13582 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -363,8 +363,7 @@ static struct rpcrdma_mr_seg *rpcrdma_mr_prepare(struct rpcrdma_xprt *r_xprt, out_getmr_err: trace_xprtrdma_nomrs(req); xprt_wait_for_buffer_space(&r_xprt->rx_xprt); - if (r_xprt->rx_ep.rep_connected != -ENODEV) - schedule_work(&r_xprt->rx_buf.rb_refresh_worker); + rpcrdma_mrs_refresh(r_xprt); return ERR_PTR(-EAGAIN); } @@ -863,12 +862,6 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, rtype = rpcrdma_areadch; } - /* If this is a retransmit, discard previously registered - * chunks. Very likely the connection has been replaced, - * so these registrations are invalid and unusable. - */ - frwr_recycle(req); - /* This implementation supports the following combinations * of chunk lists in one RPC-over-RDMA Call message: * diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 82361e7..c79d862 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -78,7 +78,7 @@ static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt); static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); -static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); +static void rpcrdma_mrs_destroy(struct rpcrdma_xprt *r_xprt); static struct rpcrdma_regbuf * rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction, gfp_t flags); @@ -407,8 +407,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_req *req; - cancel_work_sync(&buf->rb_refresh_worker); - /* This is similar to rpcrdma_ep_destroy, but: * - Don't cancel the connect worker. * - Don't call rpcrdma_ep_disconnect, which waits @@ -435,7 +433,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) rpcrdma_regbuf_dma_unmap(req->rl_sendbuf); rpcrdma_regbuf_dma_unmap(req->rl_recvbuf); } - rpcrdma_mrs_destroy(buf); + rpcrdma_mrs_destroy(r_xprt); ib_dealloc_pd(ia->ri_pd); ia->ri_pd = NULL; @@ -628,8 +626,6 @@ static int rpcrdma_ep_recreate_xprt(struct rpcrdma_xprt *r_xprt, pr_err("rpcrdma: rdma_create_qp returned %d\n", err); goto out3; } - - rpcrdma_mrs_create(r_xprt); return 0; out3: @@ -703,7 +699,6 @@ static int rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, memcpy(&qp_init_attr, &ep->rep_attr, sizeof(qp_init_attr)); switch (ep->rep_connected) { case 0: - dprintk("RPC: %s: connecting...\n", __func__); rc = rdma_create_qp(ia->ri_id, ia->ri_pd, &qp_init_attr); if (rc) { rc = -ENETUNREACH; @@ -741,7 +736,7 @@ static int rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, goto out; } - dprintk("RPC: %s: connected\n", __func__); + rpcrdma_mrs_create(r_xprt); out: if (rc) @@ -756,11 +751,8 @@ static int rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, * @ep: endpoint to disconnect * @ia: associated interface adapter * - * This is separate from destroy to facilitate the ability - * to reconnect without recreating the endpoint. - * - * This call is not reentrant, and must not be made in parallel - * on the same endpoint. + * Caller serializes. Either the transport send lock is held, + * or we're being called to destroy the transport. */ void rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) @@ -780,6 +772,7 @@ static int rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, rpcrdma_xprt_drain(r_xprt); rpcrdma_reqs_reset(r_xprt); + rpcrdma_mrs_destroy(r_xprt); } /* Fixed-size circular FIFO queue. This implementation is wait-free and @@ -987,6 +980,28 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) } /** + * rpcrdma_mrs_refresh - Wake the MR refresh worker + * @r_xprt: controlling transport instance + * + */ +void rpcrdma_mrs_refresh(struct rpcrdma_xprt *r_xprt) +{ + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ep *ep = &r_xprt->rx_ep; + + /* If there is no underlying device, it's no use to + * wake the refresh worker. + */ + if (ep->rep_connected != -ENODEV) { + /* The work is scheduled on a WQ_MEM_RECLAIM + * workqueue in order to prevent MR allocation + * from recursing into NFS during direct reclaim. + */ + queue_work(xprtiod_workqueue, &buf->rb_refresh_worker); + } +} + +/** * rpcrdma_req_create - Allocate an rpcrdma_req object * @r_xprt: controlling r_xprt * @size: initial size, in bytes, of send and receive buffers @@ -1145,8 +1160,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) INIT_LIST_HEAD(&buf->rb_all_mrs); INIT_WORK(&buf->rb_refresh_worker, rpcrdma_mr_refresh_worker); - rpcrdma_mrs_create(r_xprt); - INIT_LIST_HEAD(&buf->rb_send_bufs); INIT_LIST_HEAD(&buf->rb_allreqs); @@ -1177,8 +1190,8 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) * rpcrdma_req_destroy - Destroy an rpcrdma_req object * @req: unused object to be destroyed * - * This function assumes that the caller prevents concurrent device - * unload and transport tear-down. + * Relies on caller holding the transport send lock to protect + * removing req->rl_all from buf->rb_all_reqs safely. */ void rpcrdma_req_destroy(struct rpcrdma_req *req) { @@ -1204,17 +1217,18 @@ void rpcrdma_req_destroy(struct rpcrdma_req *req) /** * rpcrdma_mrs_destroy - Release all of a transport's MRs - * @buf: controlling buffer instance + * @r_xprt: controlling transport instance * * Relies on caller holding the transport send lock to protect * removing mr->mr_list from req->rl_free_mrs safely. */ -static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) +static void rpcrdma_mrs_destroy(struct rpcrdma_xprt *r_xprt) { - struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt, - rx_buf); + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_mr *mr; + cancel_work_sync(&buf->rb_refresh_worker); + spin_lock(&buf->rb_lock); while ((mr = list_first_entry_or_null(&buf->rb_all_mrs, struct rpcrdma_mr, @@ -1224,10 +1238,10 @@ static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) spin_unlock(&buf->rb_lock); frwr_release_mr(mr); + spin_lock(&buf->rb_lock); } spin_unlock(&buf->rb_lock); - r_xprt->rx_stats.mrs_allocated = 0; } /** @@ -1241,8 +1255,6 @@ static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) void rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) { - cancel_work_sync(&buf->rb_refresh_worker); - rpcrdma_sendctxs_destroy(buf); rpcrdma_reps_destroy(buf); @@ -1254,8 +1266,6 @@ static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) list_del(&req->rl_list); rpcrdma_req_destroy(req); } - - rpcrdma_mrs_destroy(buf); } /** diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 2f89cfc..a7ef965 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -488,6 +488,7 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, struct rpcrdma_mr *rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt); void rpcrdma_mr_put(struct rpcrdma_mr *mr); +void rpcrdma_mrs_refresh(struct rpcrdma_xprt *r_xprt); static inline void rpcrdma_mr_recycle(struct rpcrdma_mr *mr) @@ -543,7 +544,6 @@ static inline bool rpcrdma_regbuf_dma_map(struct rpcrdma_xprt *r_xprt, /* Memory registration calls xprtrdma/frwr_ops.c */ bool frwr_is_supported(struct ib_device *device); -void frwr_recycle(struct rpcrdma_req *req); void frwr_reset(struct rpcrdma_req *req); int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep); int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr);