From patchwork Wed Dec 19 15:58:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737555 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F063414DE for ; Wed, 19 Dec 2018 15:58:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE86F2B434 for ; Wed, 19 Dec 2018 15:58:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D19172B4BE; Wed, 19 Dec 2018 15:58:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7301B2B434 for ; Wed, 19 Dec 2018 15:58:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728063AbeLSP6Q (ORCPT ); Wed, 19 Dec 2018 10:58:16 -0500 Received: from mail-io1-f68.google.com ([209.85.166.68]:41582 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726652AbeLSP6Q (ORCPT ); Wed, 19 Dec 2018 10:58:16 -0500 Received: by mail-io1-f68.google.com with SMTP id s22so15968078ioc.8; Wed, 19 Dec 2018 07:58:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=YnmL2QcytZ7nDWQP0b2qARfKenfoawrMQNmn+bhvVt8=; b=l8RL7YV+1UhyA0irxqLHI78jGSt5iWdGjis36YA+akI3D24hgHi/jXUws4Bq6Iq+c0 gLB5bVjTUhwB28cA9yXuIvNpedtJT25a07CadSNimSDBl4HUAdE+GlcpEHGHnXxS2W/X k9zIgZxaMR3i4Mr51p/YS7D69luNccVm85KX8Wp9yl8DpcIBzYNbHfkF990trlMoi71s aN1dW9/V7GSdnb49mqsMLqR4zDdSLs+jliLN5kwPxkRAq7P3vV7K/QhVQcrLkMlv9kOW kuj0i0Yz8EmzH7c5vhRiI5SbiYIKHkDVhLHN8DGqJ7B3rhSE9Yp0B1c71j5QZtyW0qi/ gs/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=YnmL2QcytZ7nDWQP0b2qARfKenfoawrMQNmn+bhvVt8=; b=eGER4phQmGmA/yHgD5hW7AmsngDMsCmEOdAKTr8GE8gN4O4hJgxtZbmxztINORPX/F q26Xh2eSv1xdNcMvna/fVIEllDQRmms7GmHsttqR9eF6Hfzbh9wvkel4LUWK5BGSq58k Y43lnpJhBnpj+2P8JbXupwJLGYrDn9fnWgWgZV8cIJP1f4B8/6ipuIIohT53/MdP1Znw HvW4q83wvpHuN/C2LgspKQk4uyTuOWqk+Cf9FlBOnSCOeax+CM7cdWSDZrpcV7Zqb8XL yo8bTOhiTCrfaqv1Tza0LssaMPkfDQ0rM5pBy88Qx1HwA9THVQw1OkW1qnNOCZyfFPYb fpyw== X-Gm-Message-State: AA+aEWa/Nle536fJK9RSn093um4JzMwsdSbfMQzEd8+qEW4ClpzfbCEx cSfjBOo3B8RKnOdNEeV6MqQ= X-Google-Smtp-Source: AFSGD/Woxa1yh1tMUIaDCc58GHFMFqUDs1WLhUgmHGG8nkAvGLUnBRKZ3c4epIVcBpAXtMzG6C3ysg== X-Received: by 2002:a6b:7d42:: with SMTP id d2mr8762309ioq.150.1545235095338; Wed, 19 Dec 2018 07:58:15 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id s15sm8890640ioe.52.2018.12.19.07.58.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:14 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwDcq025759; Wed, 19 Dec 2018 15:58:13 GMT Subject: [PATCH v5 01/30] xprtrdma: Yet another double DMA-unmap From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:13 -0500 Message-ID: <20181219155813.11602.13506.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP While chasing yet another set of DMAR fault reports, I noticed that the frwr recycler conflates whether or not an MR has been DMA unmapped with frwr->fr_state. Actually the two have only an indirect relationship. It's in fact impossible to guess reliably whether the MR has been DMA unmapped based on its fr_state field, especially as the surrounding code and its assumptions have changed over time. A better approach is to track the DMA mapping status explicitly so that the recycler is less brittle to unexpected situations, and attempts to DMA-unmap a second time are prevented. Signed-off-by: Chuck Lever Cc: stable@vger.kernel.org # v4.20 --- net/sunrpc/xprtrdma/frwr_ops.c | 6 ++++-- net/sunrpc/xprtrdma/verbs.c | 9 ++++++--- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index fc6378cc..20ced24 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -117,15 +117,15 @@ frwr_mr_recycle_worker(struct work_struct *work) { struct rpcrdma_mr *mr = container_of(work, struct rpcrdma_mr, mr_recycle); - enum rpcrdma_frwr_state state = mr->frwr.fr_state; struct rpcrdma_xprt *r_xprt = mr->mr_xprt; trace_xprtrdma_mr_recycle(mr); - if (state != FRWR_FLUSHED_LI) { + if (mr->mr_dir != DMA_NONE) { trace_xprtrdma_mr_unmap(mr); ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, mr->mr_sg, mr->mr_nents, mr->mr_dir); + mr->mr_dir = DMA_NONE; } spin_lock(&r_xprt->rx_buf.rb_mrlock); @@ -150,6 +150,8 @@ if (!mr->mr_sg) goto out_list_err; + frwr->fr_state = FRWR_IS_INVALID; + mr->mr_dir = DMA_NONE; INIT_LIST_HEAD(&mr->mr_list); INIT_WORK(&mr->mr_recycle, frwr_mr_recycle_worker); sg_init_table(mr->mr_sg, depth); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3ddba94..b9bc7f9 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1329,9 +1329,12 @@ struct rpcrdma_mr * { struct rpcrdma_xprt *r_xprt = mr->mr_xprt; - trace_xprtrdma_mr_unmap(mr); - ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, - mr->mr_sg, mr->mr_nents, mr->mr_dir); + if (mr->mr_dir != DMA_NONE) { + trace_xprtrdma_mr_unmap(mr); + ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, + mr->mr_sg, mr->mr_nents, mr->mr_dir); + mr->mr_dir = DMA_NONE; + } __rpcrdma_mr_put(&r_xprt->rx_buf, mr); } From patchwork Wed Dec 19 15:58:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737561 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CDE3114E2 for ; Wed, 19 Dec 2018 15:58:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BFE7A2B434 for ; Wed, 19 Dec 2018 15:58:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B49F62B4BE; Wed, 19 Dec 2018 15:58:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7730E2B434 for ; Wed, 19 Dec 2018 15:58:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728450AbeLSP6V (ORCPT ); Wed, 19 Dec 2018 10:58:21 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:50526 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728091AbeLSP6V (ORCPT ); Wed, 19 Dec 2018 10:58:21 -0500 Received: by mail-it1-f193.google.com with SMTP id z7so10396467iti.0; Wed, 19 Dec 2018 07:58:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=nqihSE1hg4o1H0UcWmtuHnYhVxMpkCBQQUarfYAvFW0=; b=PyPzNW84fHCg4eFenASrdsWa/ztAQpVoKkeVezvrB3eIW2m6VRpqOuMPh5it64m4cR VKl7H5hTt0ksfCjzIGAL8+KvZx1nrvSuYFk+hv/yL6EcCYM18BDEFG1Ff8WJK226ccyy wQysBb3Hcx+fQxTh4JE7tTqt+Ry1UIgps1l4eloRYdOHipDHca67W6HLNnlcFzg1/X8b hq8wzd3wama0nFf+ZWTUNxAwuNN4JZ2MF8LO/ITfeEsZ9RBmn6ITCbe3Dq86NalD/+aG lcNGaT0pAW9MTZNYs3pt5m9iwNvszGjQLbwxjxUZUO2pvUOIlH+Bkv2So3kUb6VskmS9 FcDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=nqihSE1hg4o1H0UcWmtuHnYhVxMpkCBQQUarfYAvFW0=; b=kEwOipva5Quz1KPeGsy9fRhs6Um6FxCSrRqzS27+jmsETunFuO9KD/0Q/mQ4HvBNjL OI9sDTt41+UOHm4mioKMRtb47V3wu0hei7jeLz/rynetlvi2Eb/pTfgxZirop6MhPuP+ BsZHRUz7oHp/1i5pDtS/3NCjuWkZnD6sE7+ofZHW1/lIUOFVgd/lK8YQEIhoJtiSAYss 4Dfe6C5YdrIl5hSnuzwwunVnqrxCM6lRXL79hLHQ3Gw7SjJbNnOOaMjvQ8TlRZDwkmGV w2aqaB7eXLiUGVmNlh3l2rJ/XAN+NgZB8U1Bd9mBjtWK6nKHemtJpVUP4fOFycPR5P8F CtaA== X-Gm-Message-State: AA+aEWYnoGVcqtjA5jEtMARYEuKi/+9W3CoEcwL3cDvqGfqJyOPrC+DZ KnuEkozUd2AGXl+DnWwf0tXji5JK X-Google-Smtp-Source: AFSGD/XeqgE6ZRgOiMlg0HuqO8XQ08EJYs0SbzFBxw+W6kDvRMDSCqKQZb1e7PH/VZCtXLq16BhxCw== X-Received: by 2002:a02:6:: with SMTP id 6mr19237213jaa.19.1545235100803; Wed, 19 Dec 2018 07:58:20 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id b91sm3733984itd.26.2018.12.19.07.58.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:20 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwJKx025762; Wed, 19 Dec 2018 15:58:19 GMT Subject: [PATCH v5 02/30] xprtrdma: Ensure MRs are DMA-unmapped when posting LOCAL_INV fails From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:19 -0500 Message-ID: <20181219155819.11602.799.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The recovery case in frwr_op_unmap_sync needs to DMA unmap each MR. frwr_release_mr does not DMA-unmap, but the recycle worker does. Fixes: 61da886bf74e ("xprtrdma: Explicitly resetting MRs is ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 20ced24..27222c0 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -563,8 +563,8 @@ mr = container_of(frwr, struct rpcrdma_mr, frwr); bad_wr = bad_wr->next; - list_del(&mr->mr_list); - frwr_op_release_mr(mr); + list_del_init(&mr->mr_list); + rpcrdma_mr_recycle(mr); } } From patchwork Wed Dec 19 15:58:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737565 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B23914E2 for ; Wed, 19 Dec 2018 15:58:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B8CE2B440 for ; Wed, 19 Dec 2018 15:58:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F3BFF2B4E1; Wed, 19 Dec 2018 15:58:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 66F172B4BE for ; Wed, 19 Dec 2018 15:58:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729230AbeLSP62 (ORCPT ); Wed, 19 Dec 2018 10:58:28 -0500 Received: from mail-it1-f177.google.com ([209.85.166.177]:35076 "EHLO mail-it1-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728095AbeLSP61 (ORCPT ); Wed, 19 Dec 2018 10:58:27 -0500 Received: by mail-it1-f177.google.com with SMTP id p197so10063770itp.0; Wed, 19 Dec 2018 07:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=GYu0UsHQVtaKNGx1din9o11hbp64tcjGdgdTHECvmKg=; b=mYHm5jRfOziEi7n8YykibdSf35NlKB3z+dMTY5uR+/BYL0xpcW/5TVT2JIei5zRyHm 8PSwuMQFf+YCc0QHD48wKEQUOwPWyNrupXgoFWC461U5BLCHcJWvA5nrfdV2uANM6OF3 mdaSG67TeTw7UFYeFwedH40+9N92cNygCfw9dEGxitZgMAN04eFAZlCYD6INdxy8e1sC B7els4AWwHHiZ95mILk+UqdcUM1GMO11RR3PphFYScY32IkKu20Anhdqwvv8/9cc2EYD R9+Y3o6tVExN5fYL8w6CQh1gciHDfGiG0+NOjvNi8bn0TLwiFtuQFB17jllmhArJOc92 SpAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=GYu0UsHQVtaKNGx1din9o11hbp64tcjGdgdTHECvmKg=; b=oOYMEq3BQBTK/mIy/Wy9MxMYJtpUkZTKlrrvZHif8ZgrEtXOA8FYI0uw3Uat6Vl473 ZKbpyrq08f4+L9jV2Uli8wUPBos/NEZ5I/rlO8ugbNpP1GFCcmVFt577k0nPVVKqhyYL 8c+aySqsoz7fBfEtbmZgKE/BHTlt2NRa/swZruC41PvxpWXQPa5/YVbXmxG+l6yWad41 6LfsDZ2rAJ7RilO6+zUl5trggMTPuyQ9gv1GRsi4osGmE3bV1+mHXv3ThQSGKiUt/eJl +S17NRwW+eORVGwRtHJ92hupbDirIQVIbNJaC0MS41BWgp3g53/IZ7DiYl5IcwfdE5SW SRRw== X-Gm-Message-State: AA+aEWY77ITNZ/DBgiGY5vTfmR4V3CiDs3PhnzEMpZPu1CdCIQoI9p8b eEQYSmG8Yzi+hqaxbO9Vjow= X-Google-Smtp-Source: AFSGD/WaZqdX0m/fkcYX2bWiULOlKlA2Hr3s+DWJsN2iqz8B6unJu7AV38dle6GCoRS6u1Go79Inwg== X-Received: by 2002:a02:410e:: with SMTP id x14mr18986173jaa.36.1545235106200; Wed, 19 Dec 2018 07:58:26 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id q16sm7525759ioi.66.2018.12.19.07.58.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:25 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwONH025765; Wed, 19 Dec 2018 15:58:24 GMT Subject: [PATCH v5 03/30] xprtrdma: Refactor Receive accounting From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:24 -0500 Message-ID: <20181219155824.11602.75211.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Divide the work cleanly: - rpcrdma_wc_receive is responsible only for RDMA Receives - rpcrdma_reply_handler is responsible only for RPC Replies - the posted send and receive counts both belong in rpcrdma_ep Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 2 +- net/sunrpc/xprtrdma/backchannel.c | 1 - net/sunrpc/xprtrdma/rpc_rdma.c | 21 +++------------------ net/sunrpc/xprtrdma/verbs.c | 31 ++++++++++++++----------------- net/sunrpc/xprtrdma/xprt_rdma.h | 3 +-- 5 files changed, 19 insertions(+), 39 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index b093058..2efe2d7 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -570,7 +570,7 @@ __entry->r_xprt = r_xprt; __entry->count = count; __entry->status = status; - __entry->posted = r_xprt->rx_buf.rb_posted_receives; + __entry->posted = r_xprt->rx_ep.rep_receive_count; __assign_str(addr, rpcrdma_addrstr(r_xprt)); __assign_str(port, rpcrdma_portstr(r_xprt)); ), diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index e5b367a..2cb07a3 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -207,7 +207,6 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) if (rc < 0) goto failed_marshal; - rpcrdma_post_recvs(r_xprt, true); if (rpcrdma_ep_post(&r_xprt->rx_ia, &r_xprt->rx_ep, req)) goto drop_connection; return 0; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 9f53e02..dc23977 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1312,11 +1312,6 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) u32 credits; __be32 *p; - --buf->rb_posted_receives; - - if (rep->rr_hdrbuf.head[0].iov_len == 0) - goto out_badstatus; - /* Fixed transport header fields */ xdr_init_decode(&rep->rr_stream, &rep->rr_hdrbuf, rep->rr_hdrbuf.head[0].iov_base); @@ -1361,31 +1356,21 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) clear_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags); trace_xprtrdma_reply(rqst->rq_task, rep, req, credits); - - rpcrdma_post_recvs(r_xprt, false); queue_work(rpcrdma_receive_wq, &rep->rr_work); return; out_badversion: trace_xprtrdma_reply_vers(rep); - goto repost; + goto out; -/* The RPC transaction has already been terminated, or the header - * is corrupt. - */ out_norqst: spin_unlock(&xprt->queue_lock); trace_xprtrdma_reply_rqst(rep); - goto repost; + goto out; out_shortreply: trace_xprtrdma_reply_short(rep); -/* If no pending RPC transaction was matched, post a replacement - * receive buffer before returning. - */ -repost: - rpcrdma_post_recvs(r_xprt, false); -out_badstatus: +out: rpcrdma_recv_buffer_put(rep); } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index b9bc7f9..e4461e7 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -78,6 +78,7 @@ static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); static int rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt, bool temp); static void rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb); +static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); struct workqueue_struct *rpcrdma_receive_wq __read_mostly; @@ -189,11 +190,13 @@ struct ib_cqe *cqe = wc->wr_cqe; struct rpcrdma_rep *rep = container_of(cqe, struct rpcrdma_rep, rr_cqe); + struct rpcrdma_xprt *r_xprt = rep->rr_rxprt; - /* WARNING: Only wr_id and status are reliable at this point */ + /* WARNING: Only wr_cqe and status are reliable at this point */ trace_xprtrdma_wc_receive(wc); + --r_xprt->rx_ep.rep_receive_count; if (wc->status != IB_WC_SUCCESS) - goto out_fail; + goto out_flushed; /* status == SUCCESS means all fields in wc are trustworthy */ rpcrdma_set_xdrlen(&rep->rr_hdrbuf, wc->byte_len); @@ -204,17 +207,16 @@ rdmab_addr(rep->rr_rdmabuf), wc->byte_len, DMA_FROM_DEVICE); -out_schedule: + rpcrdma_post_recvs(r_xprt, false); rpcrdma_reply_handler(rep); return; -out_fail: +out_flushed: if (wc->status != IB_WC_WR_FLUSH_ERR) pr_err("rpcrdma: Recv: %s (%u/0x%x)\n", ib_wc_status_msg(wc->status), wc->status, wc->vendor_err); - rpcrdma_set_xdrlen(&rep->rr_hdrbuf, 0); - goto out_schedule; + rpcrdma_recv_buffer_put(rep); } static void @@ -581,6 +583,7 @@ init_waitqueue_head(&ep->rep_connect_wait); INIT_DELAYED_WORK(&ep->rep_disconnect_worker, rpcrdma_disconnect_worker); + ep->rep_receive_count = 0; sendcq = ib_alloc_cq(ia->ri_device, NULL, ep->rep_attr.cap.max_send_wr + 1, @@ -1174,7 +1177,6 @@ struct rpcrdma_req * } buf->rb_credits = 1; - buf->rb_posted_receives = 0; INIT_LIST_HEAD(&buf->rb_recv_bufs); rc = rpcrdma_sendctxs_create(r_xprt); @@ -1511,25 +1513,20 @@ struct rpcrdma_regbuf * return 0; } -/** - * rpcrdma_post_recvs - Maybe post some Receive buffers - * @r_xprt: controlling transport - * @temp: when true, allocate temp rpcrdma_rep objects - * - */ -void +static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ep *ep = &r_xprt->rx_ep; struct ib_recv_wr *wr, *bad_wr; int needed, count, rc; rc = 0; count = 0; needed = buf->rb_credits + (buf->rb_bc_srv_max_requests << 1); - if (buf->rb_posted_receives > needed) + if (ep->rep_receive_count > needed) goto out; - needed -= buf->rb_posted_receives; + needed -= ep->rep_receive_count; count = 0; wr = NULL; @@ -1577,7 +1574,7 @@ struct rpcrdma_regbuf * --count; } } - buf->rb_posted_receives += count; + ep->rep_receive_count += count; out: trace_xprtrdma_post_recvs(r_xprt, count, rc); } diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index a13ccb6..788124c 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -102,6 +102,7 @@ struct rpcrdma_ep { struct rpcrdma_connect_private rep_cm_private; struct rdma_conn_param rep_remote_cma; struct delayed_work rep_disconnect_worker; + int rep_receive_count; }; /* Pre-allocate extra Work Requests for handling backward receives @@ -404,7 +405,6 @@ struct rpcrdma_buffer { unsigned long rb_flags; u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ - int rb_posted_receives; u32 rb_bc_srv_max_requests; spinlock_t rb_reqslock; /* protect rb_allreqs */ @@ -560,7 +560,6 @@ int rpcrdma_ep_create(struct rpcrdma_ep *, struct rpcrdma_ia *, int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *, struct rpcrdma_req *); -void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); /* * Buffer calls - xprtrdma/verbs.c From patchwork Wed Dec 19 15:58:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E87E14E2 for ; Wed, 19 Dec 2018 15:58:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F4312B434 for ; Wed, 19 Dec 2018 15:58:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7400F2B4BE; Wed, 19 Dec 2018 15:58:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E7B0C2B440 for ; Wed, 19 Dec 2018 15:58:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729276AbeLSP6d (ORCPT ); Wed, 19 Dec 2018 10:58:33 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:36337 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728095AbeLSP6c (ORCPT ); Wed, 19 Dec 2018 10:58:32 -0500 Received: by mail-it1-f193.google.com with SMTP id c9so10052217itj.1; Wed, 19 Dec 2018 07:58:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=HmItSEbpQ0nLYBpzJa2hliI2p7Ennn0e5DZHPjfdd58=; b=PN1mwfsbPRqndWvKZ4sT5NnwV7NC7ON46NIWVy3jgF0ihfqROqX5o6G9YmmYLR9lQu TO22Nu0x1r9dM1i3twWrK4Zx/AG/CmWs+TXPEaMYT+Xg/bvT6khuP6Vcit0sl9D3Sj2I ybuJXa8HeFJVrCkRxK1IW35sNAmoGD01ytaU6OiVdD/VIMMVuxUfcYuCUb7X8Uh5GURl 70W6ZVJb7+qflBrU7qOpiextv8EEC1QzaodlOyPE2BdZLFK4swlTC7cJjX6eXwHR8MCU sWVuKMZWPJWSiWPlzxxQiEa6tjXiuKTuqqvMfabwHse48yMAaq1VuYzPvLtqSJKvYWHq OBMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=HmItSEbpQ0nLYBpzJa2hliI2p7Ennn0e5DZHPjfdd58=; b=nNbStV5ENyFIITHqUqYZZ8Peym9h2HmcXmsnJJSpxENDs4mTUUR4yt6tS+L+E3Ig7K cSMcWEKQ/A9IgtdLpD7JkFJxJwDaVdnqd1peIqB0agfTQVxTORqi3JKJvbccJV8jDlPe /sawRf0svVsFr+HA3ZLvdkGsoUdh2c8nsxUP4wl1C0GBMtOpsgErNxjWotZb/E4UtMFR R5YiQJsX+KDocJqkxCu633vG3l+aJ1dfod7ydB+nvc57Hpk7WTEdYPPpH0WUY3P7dzvY /2vdqAq38h+aeHD0nxUQuYYppKJDFT4Nmc+oexYmtJCywpijdspF1FmbuvWwsiK9gtm+ oWCQ== X-Gm-Message-State: AA+aEWZexsS+4bNH1TMv/83he2h9emLLxECBm+2gz+jZTCfEyxgh0az5 yxaE3l5YooMWABDIpbez5c92mSPu X-Google-Smtp-Source: AFSGD/Va0w8e/fyMnWFZDZaywJ/kUW8Jj0X8Kzf0g+De1YvHA7PYz2VUgOb1Vr95JJ0L5gWgZ9GTpw== X-Received: by 2002:a02:8943:: with SMTP id u3mr19440189jaj.92.1545235111440; Wed, 19 Dec 2018 07:58:31 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id r11sm8062214iog.46.2018.12.19.07.58.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:30 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwTfg025768; Wed, 19 Dec 2018 15:58:29 GMT Subject: [PATCH v5 04/30] xprtrdma: Replace rpcrdma_receive_wq with a per-xprt workqueue From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:29 -0500 Message-ID: <20181219155829.11602.21903.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To address a connection-close ordering problem, we need the ability to drain the RPC completions running on rpcrdma_receive_wq for just one transport. Give each transport its own RPC completion workqueue, and drain that workqueue when disconnecting the transport. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 2 + net/sunrpc/xprtrdma/transport.c | 17 +++------- net/sunrpc/xprtrdma/verbs.c | 67 +++++++++++++++++++++------------------ net/sunrpc/xprtrdma/xprt_rdma.h | 6 +-- 4 files changed, 44 insertions(+), 48 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index dc23977..5738c9f 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1356,7 +1356,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) clear_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags); trace_xprtrdma_reply(rqst->rq_task, rep, req, credits); - queue_work(rpcrdma_receive_wq, &rep->rr_work); + queue_work(buf->rb_completion_wq, &rep->rr_work); return; out_badversion: diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index ae2a838..91c476a 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -444,10 +444,14 @@ struct rpcrdma_ep *ep = &r_xprt->rx_ep; struct rpcrdma_ia *ia = &r_xprt->rx_ia; + might_sleep(); + dprintk("RPC: %s: closing xprt %p\n", __func__, xprt); + /* Prevent marshaling and sending of new requests */ + xprt_clear_connected(xprt); + if (test_and_clear_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags)) { - xprt_clear_connected(xprt); rpcrdma_ia_remove(ia); return; } @@ -858,8 +862,6 @@ void xprt_rdma_cleanup(void) dprintk("RPC: %s: xprt_unregister returned %i\n", __func__, rc); - rpcrdma_destroy_wq(); - rc = xprt_unregister_transport(&xprt_rdma_bc); if (rc) dprintk("RPC: %s: xprt_unregister(bc) returned %i\n", @@ -870,20 +872,13 @@ int xprt_rdma_init(void) { int rc; - rc = rpcrdma_alloc_wq(); - if (rc) - return rc; - rc = xprt_register_transport(&xprt_rdma); - if (rc) { - rpcrdma_destroy_wq(); + if (rc) return rc; - } rc = xprt_register_transport(&xprt_rdma_bc); if (rc) { xprt_unregister_transport(&xprt_rdma); - rpcrdma_destroy_wq(); return rc; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index e4461e7..cff3a5d 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -80,33 +80,23 @@ static void rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb); static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); -struct workqueue_struct *rpcrdma_receive_wq __read_mostly; - -int -rpcrdma_alloc_wq(void) +/* Wait for outstanding transport work to finish. + */ +static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) { - struct workqueue_struct *recv_wq; - - recv_wq = alloc_workqueue("xprtrdma_receive", - WQ_MEM_RECLAIM | WQ_HIGHPRI, - 0); - if (!recv_wq) - return -ENOMEM; - - rpcrdma_receive_wq = recv_wq; - return 0; -} + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ia *ia = &r_xprt->rx_ia; -void -rpcrdma_destroy_wq(void) -{ - struct workqueue_struct *wq; + /* Flush Receives, then wait for deferred Reply work + * to complete. + */ + ib_drain_qp(ia->ri_id->qp); + drain_workqueue(buf->rb_completion_wq); - if (rpcrdma_receive_wq) { - wq = rpcrdma_receive_wq; - rpcrdma_receive_wq = NULL; - destroy_workqueue(wq); - } + /* Deferred Reply processing might have scheduled + * local invalidations. + */ + ib_drain_sq(ia->ri_id->qp); } /** @@ -483,7 +473,7 @@ * connection is already gone. */ if (ia->ri_id->qp) { - ib_drain_qp(ia->ri_id->qp); + rpcrdma_xprt_drain(r_xprt); rdma_destroy_qp(ia->ri_id); ia->ri_id->qp = NULL; } @@ -825,8 +815,10 @@ return rc; } -/* - * rpcrdma_ep_disconnect +/** + * rpcrdma_ep_disconnect - Disconnect underlying transport + * @ep: endpoint to disconnect + * @ia: associated interface adapter * * This is separate from destroy to facilitate the ability * to reconnect without recreating the endpoint. @@ -837,19 +829,20 @@ void rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { + struct rpcrdma_xprt *r_xprt = container_of(ep, struct rpcrdma_xprt, + rx_ep); int rc; + /* returns without wait if ID is not connected */ rc = rdma_disconnect(ia->ri_id); if (!rc) - /* returns without wait if not connected */ wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected != 1); else ep->rep_connected = rc; - trace_xprtrdma_disconnect(container_of(ep, struct rpcrdma_xprt, - rx_ep), rc); + trace_xprtrdma_disconnect(r_xprt, rc); - ib_drain_qp(ia->ri_id->qp); + rpcrdma_xprt_drain(r_xprt); } /* Fixed-size circular FIFO queue. This implementation is wait-free and @@ -1183,6 +1176,13 @@ struct rpcrdma_req * if (rc) goto out; + buf->rb_completion_wq = alloc_workqueue("rpcrdma-%s", + WQ_MEM_RECLAIM | WQ_HIGHPRI, + 0, + r_xprt->rx_xprt.address_strings[RPC_DISPLAY_ADDR]); + if (!buf->rb_completion_wq) + goto out; + return 0; out: rpcrdma_buffer_destroy(buf); @@ -1241,6 +1241,11 @@ struct rpcrdma_req * { cancel_delayed_work_sync(&buf->rb_refresh_worker); + if (buf->rb_completion_wq) { + destroy_workqueue(buf->rb_completion_wq); + buf->rb_completion_wq = NULL; + } + rpcrdma_sendctxs_destroy(buf); while (!list_empty(&buf->rb_recv_bufs)) { diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 788124c..3f198cd 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -412,6 +412,7 @@ struct rpcrdma_buffer { u32 rb_bc_max_requests; + struct workqueue_struct *rb_completion_wq; struct delayed_work rb_refresh_worker; }; #define rdmab_to_ia(b) (&container_of((b), struct rpcrdma_xprt, rx_buf)->rx_ia) @@ -547,8 +548,6 @@ struct rpcrdma_xprt { bool frwr_is_supported(struct rpcrdma_ia *); bool fmr_is_supported(struct rpcrdma_ia *); -extern struct workqueue_struct *rpcrdma_receive_wq; - /* * Endpoint calls - xprtrdma/verbs.c */ @@ -603,9 +602,6 @@ struct rpcrdma_regbuf *rpcrdma_alloc_regbuf(size_t, enum dma_data_direction, return __rpcrdma_dma_map_regbuf(ia, rb); } -int rpcrdma_alloc_wq(void); -void rpcrdma_destroy_wq(void); - /* * Wrappers for chunk registration, shared by read/write chunk code. */ From patchwork Wed Dec 19 15:58:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 698D114E2 for ; Wed, 19 Dec 2018 15:58:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E4F929551 for ; Wed, 19 Dec 2018 15:58:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E2282946D; Wed, 19 Dec 2018 15:58:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C03DB2871C for ; Wed, 19 Dec 2018 15:58:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729323AbeLSP6i (ORCPT ); Wed, 19 Dec 2018 10:58:38 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:36365 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728889AbeLSP6h (ORCPT ); Wed, 19 Dec 2018 10:58:37 -0500 Received: by mail-it1-f195.google.com with SMTP id c9so10052767itj.1; Wed, 19 Dec 2018 07:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=JrjAwoYZLatTrSpjWp1zdL2hQdofaBBegzEeSbKdEko=; b=YwEqyzyOglQrkEocV35NPH2bwQ/LmSsAuNvs7n4u43EN/Se+cpe5VRCLfpyZY4VXqa O4PTNr+nXFG//Tl/cE1Ew3WayPk8a3vkdToKqne4oPawmw0ZbzVAcRVkdL7vseZmF19S sbGZS2QqqX8u8jicLPedpvwFAgTOOpueV9JJSYVJ4OM1aCfrdqoK8dkDP8jCT85KKz+1 ngEwfMrgLRGglkvypedoEeU2VNhGZiGGMPURqtv31zk70L3PhRVY72R4L/ET5USJgxa5 O2OmztvGNOkTs2cfht1lxTscw+mnvacs0PeN9fdimxLnCx6yg4USZ3TPNdE0h1iHvDf3 PKGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=JrjAwoYZLatTrSpjWp1zdL2hQdofaBBegzEeSbKdEko=; b=ewBoOQRi8/r2vBHhTwcz+xoyPdslak7Y8d/FakKxKoP0Jbs18mbw+7JnTlqFxgiMfZ ZbXOwde01pn59Oi6gsNLsrXSzmE06hMKSMcLVEit1DO089Y0D/oXqdnY+IQ5oa2pIuXE 16KqI6F0FzONYnJcB58r8zzw+j93WYBKvbz56FLm3IzGxk4PsFr1pe+BI5D27VWmtlvy Xks15s45ODpYk8aBWHpfgGGXG+xonfzN3WiQm3Pm0n+eNec/pTK0GcCsxQkyw6EziOZ5 F7RNowiSMD1MO9n1qXD7YfBRlKHY4Gz5/v8UyqhUuoeAGTaYLy4XBBwXSzkQF3cqZKV3 zuVA== X-Gm-Message-State: AA+aEWaoa/OTlg7ONmtXQHw5DDZHfZz3V06owm2IWL+VcdNLqbi3WiDF X5RvxfapWOTnRLDZSASJIlcTJnCC X-Google-Smtp-Source: AFSGD/WX3QNHw2ceoJIdtJQ7/kBtdaPb1mdT9Yxhc9syyMzmfnccm4Zj6AeZ47vpgLTfCcFrVP0Vww== X-Received: by 2002:a02:4311:: with SMTP id s17mr18296700jab.124.1545235116737; Wed, 19 Dec 2018 07:58:36 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id e16sm9410718iob.43.2018.12.19.07.58.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:36 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwZWd025771; Wed, 19 Dec 2018 15:58:35 GMT Subject: [PATCH v5 05/30] xprtrdma: No qp_event disconnect From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:35 -0500 Message-ID: <20181219155835.11602.87412.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP After thinking about this more, and auditing other kernel ULP imple- mentations, I believe that a DISCONNECT cm_event will occur after a fatal QP event. If that's the case, there's no need for an explicit disconnect in the QP event handler. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 32 -------------------------------- net/sunrpc/xprtrdma/xprt_rdma.h | 1 - 2 files changed, 33 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index cff3a5d..9a0a765 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -100,25 +100,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) } /** - * rpcrdma_disconnect_worker - Force a disconnect - * @work: endpoint to be disconnected - * - * Provider callbacks can possibly run in an IRQ context. This function - * is invoked in a worker thread to guarantee that disconnect wake-up - * calls are always done in process context. - */ -static void -rpcrdma_disconnect_worker(struct work_struct *work) -{ - struct rpcrdma_ep *ep = container_of(work, struct rpcrdma_ep, - rep_disconnect_worker.work); - struct rpcrdma_xprt *r_xprt = - container_of(ep, struct rpcrdma_xprt, rx_ep); - - xprt_force_disconnect(&r_xprt->rx_xprt); -} - -/** * rpcrdma_qp_event_handler - Handle one QP event (error notification) * @event: details of the event * @context: ep that owns QP where event occurred @@ -134,15 +115,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) rx_ep); trace_xprtrdma_qp_event(r_xprt, event); - pr_err("rpcrdma: %s on device %s connected to %s:%s\n", - ib_event_msg(event->event), event->device->name, - rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt)); - - if (ep->rep_connected == 1) { - ep->rep_connected = -EIO; - schedule_delayed_work(&ep->rep_disconnect_worker, 0); - wake_up_all(&ep->rep_connect_wait); - } } /** @@ -571,8 +543,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) cdata->max_requests >> 2); ep->rep_send_count = ep->rep_send_batch; init_waitqueue_head(&ep->rep_connect_wait); - INIT_DELAYED_WORK(&ep->rep_disconnect_worker, - rpcrdma_disconnect_worker); ep->rep_receive_count = 0; sendcq = ib_alloc_cq(ia->ri_device, NULL, @@ -646,8 +616,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) void rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { - cancel_delayed_work_sync(&ep->rep_disconnect_worker); - if (ia->ri_id && ia->ri_id->qp) { rpcrdma_ep_disconnect(ep, ia); rdma_destroy_qp(ia->ri_id); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 3f198cd..7c1b519 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -101,7 +101,6 @@ struct rpcrdma_ep { wait_queue_head_t rep_connect_wait; struct rpcrdma_connect_private rep_cm_private; struct rdma_conn_param rep_remote_cma; - struct delayed_work rep_disconnect_worker; int rep_receive_count; }; From patchwork Wed Dec 19 15:58:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737581 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8902513B5 for ; Wed, 19 Dec 2018 15:58:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A43929551 for ; Wed, 19 Dec 2018 15:58:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6E5E729584; Wed, 19 Dec 2018 15:58:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBB642957A for ; Wed, 19 Dec 2018 15:58:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729370AbeLSP6o (ORCPT ); Wed, 19 Dec 2018 10:58:44 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:37434 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP6o (ORCPT ); Wed, 19 Dec 2018 10:58:44 -0500 Received: by mail-it1-f194.google.com with SMTP id b5so10048822iti.2; Wed, 19 Dec 2018 07:58:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=aGyQWJVJ75JWH0t61EgFa3A/spEXQp3kHqveaGDMMpI=; b=VwUqk9nRg3BN9pPawD4Md46hpE8zlyH7/LSD7rF68+A14Sut9IiCnTrRSL7GqZh5oV rLcCiyzn93iczcvdakO3q15tT+omTUzdT/d7eX3zu2t2z6ior9S5gqSpAYeqbcYMr4ZI tMaydgNffNVVZ1j4CH0DU9O9NXOPnXqRnLYarhu7HZFzAAqRd/U5GEFANiY9oxoEGQM+ rvFFJPKBUEKVhrQf1is5MUXfafDw7zB6a7uwhmbdGPnm8Ybs3qiD6WQFTfvrwZo6A1YE c7ooyVqC49lYaUOOiVM0KN4egj4wWlkNePGd8A1tKpqPWqBu4BWTVmeRGpW0/53cV5iM os4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=aGyQWJVJ75JWH0t61EgFa3A/spEXQp3kHqveaGDMMpI=; b=gL4Nzblr7AR1FyF2R26qO2yHJdIWf93vGdraSWKfeGkbgjQsp3X49XCRajJVU+a5aG ZNxI9r4I+8e+oN5ld39F7zBQl/JvEM0rE1V98NaktjrckHcaUA0oPRvn0o40eI2f6+rB b6Qfh+4KJgiZ8QZm5m6ozhBljbaY9aQmkQDvgdwXTPTHLj6N3DG1Hnk2oZP9S3Q6CVD2 H60JfmSzwGSwQo03ZNjMUbrGjswKzenRi+LoO76PLwOS/M1fJE5syzOy1NFH1wFo8iHd sH5aBamGvCAsL96wiZTgpsbDIVtc7VQdnwhBbWY+p6M3vBDa+HgX3+I+deRDeqqq6il6 vmuQ== X-Gm-Message-State: AA+aEWaQNVpc9KkRYwnUad7IeERFr5cjfvlSk0TqW53FZLapSdvZ3QwI LLhF/wBSjMoT7hlEgNJlhOVpxtOZ X-Google-Smtp-Source: AFSGD/WLyfYU0BEglEY7Zx2Wxjxt553YeVp9PdlqFL8hXN7y3TTbaox3nDXDYYYG2WYyPw6ypCzryg== X-Received: by 2002:a24:d997:: with SMTP id p145mr6701917itg.30.1545235121804; Wed, 19 Dec 2018 07:58:41 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id c194sm4144777itc.3.2018.12.19.07.58.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:41 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFweUf025774; Wed, 19 Dec 2018 15:58:40 GMT Subject: [PATCH v5 06/30] xprtrdma: Don't wake pending tasks until disconnect is done From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:40 -0500 Message-ID: <20181219155840.11602.58228.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Transport disconnect processing does a "wake pending tasks" at various points. Suppose an RPC Reply is being processed. The RPC task that Reply goes with is waiting on the pending queue. If a disconnect wake-up happens before reply processing is done, that reply, even if it is good, is thrown away, and the RPC has to be sent again. This window apparently does not exist for socket transports because there is a lock held while a reply is being received which prevents the wake-up call until after reply processing is done. To resolve this, all RPC replies being processed on an RPC-over-RDMA transport have to complete before pending tasks are awoken due to a transport disconnect. Callers that already hold the transport write lock may invoke ->ops->close directly. Others use a generic helper that schedules a close when the write lock can be taken safely. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 13 +++++++------ net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 8 +++++--- net/sunrpc/xprtrdma/transport.c | 17 ++++++++++------- net/sunrpc/xprtrdma/verbs.c | 1 - net/sunrpc/xprtrdma/xprt_rdma.h | 1 + 5 files changed, 23 insertions(+), 17 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 2cb07a3..79a55fc 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -193,14 +193,15 @@ static int rpcrdma_bc_marshal_reply(struct rpc_rqst *rqst) */ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) { - struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(rqst->rq_xprt); + struct rpc_xprt *xprt = rqst->rq_xprt; + struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); struct rpcrdma_req *req = rpcr_to_rdmar(rqst); int rc; - if (!xprt_connected(rqst->rq_xprt)) - goto drop_connection; + if (!xprt_connected(xprt)) + return -ENOTCONN; - if (!xprt_request_get_cong(rqst->rq_xprt, rqst)) + if (!xprt_request_get_cong(xprt, rqst)) return -EBADSLT; rc = rpcrdma_bc_marshal_reply(rqst); @@ -215,7 +216,7 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) if (rc != -ENOTCONN) return rc; drop_connection: - xprt_disconnect_done(rqst->rq_xprt); + xprt_rdma_close(xprt); return -ENOTCONN; } @@ -338,7 +339,7 @@ void rpcrdma_bc_receive_call(struct rpcrdma_xprt *r_xprt, out_overflow: pr_warn("RPC/RDMA backchannel overflow\n"); - xprt_disconnect_done(xprt); + xprt_force_disconnect(xprt); /* This receive buffer gets reposted automatically * when the connection is re-established. */ diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c index f3c147d..b908f2c 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c +++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c @@ -200,11 +200,10 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, svc_rdma_send_ctxt_put(rdma, ctxt); goto drop_connection; } - return rc; + return 0; drop_connection: dprintk("svcrdma: failed to send bc call\n"); - xprt_disconnect_done(xprt); return -ENOTCONN; } @@ -225,8 +224,11 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, ret = -ENOTCONN; rdma = container_of(sxprt, struct svcxprt_rdma, sc_xprt); - if (!test_bit(XPT_DEAD, &sxprt->xpt_flags)) + if (!test_bit(XPT_DEAD, &sxprt->xpt_flags)) { ret = rpcrdma_bc_send_request(rdma, rqst); + if (ret == -ENOTCONN) + svc_close_xprt(sxprt); + } mutex_unlock(&sxprt->xpt_mutex); diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 91c476a..134aae2 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -437,8 +437,7 @@ * Caller holds @xprt's send lock to prevent activity on this * transport while the connection is torn down. */ -static void -xprt_rdma_close(struct rpc_xprt *xprt) +void xprt_rdma_close(struct rpc_xprt *xprt) { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); struct rpcrdma_ep *ep = &r_xprt->rx_ep; @@ -453,13 +452,13 @@ if (test_and_clear_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags)) { rpcrdma_ia_remove(ia); - return; + goto out; } + if (ep->rep_connected == -ENODEV) return; if (ep->rep_connected > 0) xprt->reestablish_timeout = 0; - xprt_disconnect_done(xprt); rpcrdma_ep_disconnect(ep, ia); /* Prepare @xprt for the next connection by reinitializing @@ -467,6 +466,10 @@ */ r_xprt->rx_buf.rb_credits = 1; xprt->cwnd = RPC_CWNDSHIFT; + +out: + ++xprt->connect_cookie; + xprt_disconnect_done(xprt); } /** @@ -717,7 +720,7 @@ #endif /* CONFIG_SUNRPC_BACKCHANNEL */ if (!xprt_connected(xprt)) - goto drop_connection; + return -ENOTCONN; if (!xprt_request_get_cong(xprt, rqst)) return -EBADSLT; @@ -749,8 +752,8 @@ if (rc != -ENOTCONN) return rc; drop_connection: - xprt_disconnect_done(xprt); - return -ENOTCONN; /* implies disconnect */ + xprt_rdma_close(xprt); + return -ENOTCONN; } void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 9a0a765..29798b6 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -280,7 +280,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) ep->rep_connected = -EAGAIN; goto disconnected; case RDMA_CM_EVENT_DISCONNECTED: - ++xprt->connect_cookie; ep->rep_connected = -ECONNABORTED; disconnected: xprt_force_disconnect(xprt); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 7c1b519..99b7f8e 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -647,6 +647,7 @@ static inline void rpcrdma_set_xdrlen(struct xdr_buf *xdr, size_t len) extern unsigned int xprt_rdma_max_inline_read; void xprt_rdma_format_addresses(struct rpc_xprt *xprt, struct sockaddr *sap); void xprt_rdma_free_addresses(struct rpc_xprt *xprt); +void xprt_rdma_close(struct rpc_xprt *xprt); void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq); int xprt_rdma_init(void); void xprt_rdma_cleanup(void); From patchwork Wed Dec 19 15:58:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737583 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61D7214DE for ; Wed, 19 Dec 2018 15:58:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 502262953A for ; Wed, 19 Dec 2018 15:58:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4447629584; Wed, 19 Dec 2018 15:58:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBC6B2953A for ; Wed, 19 Dec 2018 15:58:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728889AbeLSP6s (ORCPT ); Wed, 19 Dec 2018 10:58:48 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:33679 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP6r (ORCPT ); Wed, 19 Dec 2018 10:58:47 -0500 Received: by mail-io1-f67.google.com with SMTP id t24so15998372ioi.0; Wed, 19 Dec 2018 07:58:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=tYYd+d/3w8thIUzENvVD06hgPLxIXGsKVk7XNtDGXEo=; b=TMSbqieenCunwJTLAuSLJjYvgiSQ+7IZ5pZZry/h61ESI/KK6Xd4mOikQNhJ2IgBS0 j1y0NUiN4gcikZU8ZFpZMK3kyAK9pgGcAGk8q+ic1/5sf1cybpJJKxnELrBZoW5hTz1l VgVrZGGQz4FNd9rZ5dYxkS5FhonQzhcf0HOf8d+Hl8c07gw1sw8nO7T61ItfPOZXAzPz vCgAbF10qDw5WxXTYrMpdsaIa0H3o+eIi/sRT2FAkGz0dViJ+31NyDWNrnhdf8JyADfO VqZai3wSasKDG6UzyZ3XCTF12zGx49SlSaShV5Bx7IfI8+uJvRRWfD6/+dt/UlMFb0e3 V/Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=tYYd+d/3w8thIUzENvVD06hgPLxIXGsKVk7XNtDGXEo=; b=uLJT8P4R0KVuNCGcsJDcRGT+QpWLD6Tgy6pep07KVHFwgRbhd2qUvhYPTYrrBf77hA olntiXS+Zfj+uuMTDw/bS3l4aVzxXL632/aryrClPuvWjfWfjDOaAgxHxt66PBrKwP89 pids9Vh2ULrOtn3Sg51Yyy4XmVtSDDLlkaFzKSY0hwrYbF2E+7JQau85AsqYxucgMxfX Dtjia0q14TDvneg/XAu32IlNtq0bz/2mFbywiE5ZZPXPnQPL8lWdYA7Cqp+5v06CPUhf /hiHPujmy+4EYGlOCxbFmNQQFE5BrwsAY7F8V0BKsWQY5ItxDfFdwbHu20vTPLMAK/gV uvSw== X-Gm-Message-State: AA+aEWZDroPBoBfu0cz9jXu4q8sR8d9xLFAP4sa0qlq9pUxl0jX1hbeQ rDKuVzVvCajzh6idR+bmFYX80ZkD X-Google-Smtp-Source: AFSGD/UTnKew7I481wUMibEL13uW1YYRyAxxU6EIhtzKepYG9MjVJ09qRV+eJB0mFj5Uqpw8taveHw== X-Received: by 2002:a6b:14c6:: with SMTP id 189mr18122972iou.179.1545235127122; Wed, 19 Dec 2018 07:58:47 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id u13sm11204iop.78.2018.12.19.07.58.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:46 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwjCl025777; Wed, 19 Dec 2018 15:58:45 GMT Subject: [PATCH v5 07/30] xprtrdma: Fix ri_max_segs and the result of ro_maxpages From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:45 -0500 Message-ID: <20181219155845.11602.42218.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With certain combinations of krb5i/p, MR size, and r/wsize, I/O can fail with EMSGSIZE. This is because the calculated value of ri_max_segs (the max number of MRs per RPC) exceeded RPCRDMA_MAX_HDR_SEGS, which caused Read or Write list encoding to walk off the end of the transport header. Once that was addressed, the ro_maxpages result has to be corrected to account for the number of MRs needed for Reply chunks, which is 2 MRs smaller than a normal Read or Write chunk. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/fmr_ops.c | 7 +++++-- net/sunrpc/xprtrdma/frwr_ops.c | 7 +++++-- net/sunrpc/xprtrdma/transport.c | 6 ++++-- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/net/sunrpc/xprtrdma/fmr_ops.c b/net/sunrpc/xprtrdma/fmr_ops.c index 7f5632c..dd1e91b 100644 --- a/net/sunrpc/xprtrdma/fmr_ops.c +++ b/net/sunrpc/xprtrdma/fmr_ops.c @@ -176,7 +176,10 @@ enum { ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / RPCRDMA_MAX_FMR_SGES); - ia->ri_max_segs += 2; /* segments for head and tail buffers */ + /* Reply chunks require segments for head and tail buffers */ + ia->ri_max_segs += 2; + if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) + ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS; return 0; } @@ -186,7 +189,7 @@ enum { fmr_op_maxpages(struct rpcrdma_xprt *r_xprt) { return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - RPCRDMA_MAX_HDR_SEGS * RPCRDMA_MAX_FMR_SGES); + (r_xprt->rx_ia.ri_max_segs - 2) * RPCRDMA_MAX_FMR_SGES); } /* Use the ib_map_phys_fmr() verb to register a memory region diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 27222c0..f587e44 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -244,7 +244,10 @@ ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / ia->ri_max_frwr_depth); - ia->ri_max_segs += 2; /* segments for head and tail buffers */ + /* Reply chunks require segments for head and tail buffers */ + ia->ri_max_segs += 2; + if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) + ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS; return 0; } @@ -257,7 +260,7 @@ struct rpcrdma_ia *ia = &r_xprt->rx_ia; return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - RPCRDMA_MAX_HDR_SEGS * ia->ri_max_frwr_depth); + (ia->ri_max_segs - 2) * ia->ri_max_frwr_depth); } static void diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 134aae2..d3f26c5 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -703,8 +703,10 @@ void xprt_rdma_close(struct rpc_xprt *xprt) * %-ENOTCONN if the caller should reconnect and call again * %-EAGAIN if the caller should call again * %-ENOBUFS if the caller should call again after a delay - * %-EIO if a permanent error occurred and the request was not - * sent. Do not try to send this message again. + * %-EMSGSIZE if encoding ran out of buffer space. The request + * was not sent. Do not try to send this message again. + * %-EIO if an I/O error occurred. The request was not sent. + * Do not try to send this message again. */ static int xprt_rdma_send_request(struct rpc_rqst *rqst) From patchwork Wed Dec 19 15:58:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C76EC13B5 for ; Wed, 19 Dec 2018 15:58:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B46572953A for ; Wed, 19 Dec 2018 15:58:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A5A8129581; Wed, 19 Dec 2018 15:58:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4BF402953A for ; Wed, 19 Dec 2018 15:58:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728095AbeLSP6x (ORCPT ); Wed, 19 Dec 2018 10:58:53 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:55781 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP6x (ORCPT ); Wed, 19 Dec 2018 10:58:53 -0500 Received: by mail-it1-f195.google.com with SMTP id m62so10358060ith.5; Wed, 19 Dec 2018 07:58:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=fZt2sJtHBzkz3cxTvQxkd76ytJ2+vEepYSr4HQtV1IE=; b=jgOBexe7mgLWcIJt/O0aNgFAMbn+tCtUCDjCKsDhi8qhkEIZDZ/dt8bJWgdCR+xaVT R3BRsTC2my6LGbbXY25qRKSOe4J68vGNVgL35hz2h2L/O82k9z/IOFPTQNJIN/q/UtN2 2utElo/+70ZLX84uW8lapHQuuB8z/7+A4oZJmGMyfJmYtji1zAvE1U4CB0bpBtUNmT1g HUhgJxMWykYKRIuzIIolWulA6bqoi862h/r6u7S5K2/lvYXOfba0Dpxp8caKokp/Q8vg 9AqJStcSbmBHR1AHGrY21f9T40QXhIlGuSBjpf0oc4cLdulMeDMcDQf9IBNCh4bDoWsQ wdiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=fZt2sJtHBzkz3cxTvQxkd76ytJ2+vEepYSr4HQtV1IE=; b=hrzUPOTyI/v1oTX+TcP/ae9NDTHF/hqWuDlvr8ATYGU/L+RDQtZdEbrcPRcjH3Ycvs bFF4nYUhXx0u/P1Ywqn/gq744xy9aCA+Kt61El0AcFg/wLxrjKZIwWlMfWCv9s0P+VBP ECtNLz+A76s4/JCzmWQSiGyv7Ec37mlwsSeq91Y4e2Chfh5fI77K5lhozyG5Or8rDOyT olbqmnheHaUq1SL4HIzf76cbTeAmiMb520ErygEczQ2wy/2lwJ2w0Ctz3nAstu2IrOq+ ljd+cEP4q6k0HMuGYw+ZanZkS9Ky9Qy4q0UQ9GQiC2cyUiDvaGNOPZkwkR3HRDNTXYW3 X+Ug== X-Gm-Message-State: AA+aEWaTXLbewrszxlpNCWr63hvcctJEUUE6cJn8036EOVzXYBC10L1a VjDJ/NTfUIu4JzKMcrbqaoQ= X-Google-Smtp-Source: AFSGD/Wpe7rgYzKA8XOTXJOB+okFKn2B23jxnBSYJxC0ousTWw7oHYok9ZxBhmRGM/tI+1uHfBPZ5g== X-Received: by 2002:a02:9b26:: with SMTP id o35mr20264818jak.23.1545235132715; Wed, 19 Dec 2018 07:58:52 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id a16sm8486442ioh.48.2018.12.19.07.58.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:52 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwpDj025780; Wed, 19 Dec 2018 15:58:51 GMT Subject: [PATCH v5 08/30] xprtrdma: Reduce max_frwr_depth From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:51 -0500 Message-ID: <20181219155851.11602.24862.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some devices advertise a large max_fast_reg_page_list_len capability, but perform optimally when MRs are significantly smaller than that depth -- probably when the MR itself is no larger than a page. By default, the RDMA R/W core API uses max_sge_rd as the maximum page depth for MRs. For some devices, the value of max_sge_rd is 1, which is also not optimal. Thus, when max_sge_rd is larger than 1, use that value. Otherwise use the value of the max_fast_reg_page_list_len attribute. I've tested this with CX-3 Pro, FastLinq, and CX-5 devices. It reproducibly improves the throughput of large I/Os by several percent. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index f587e44..16976b0 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -193,10 +193,17 @@ if (attrs->device_cap_flags & IB_DEVICE_SG_GAPS_REG) ia->ri_mrtype = IB_MR_TYPE_SG_GAPS; - ia->ri_max_frwr_depth = - min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - attrs->max_fast_reg_page_list_len); - dprintk("RPC: %s: device's max FR page list len = %u\n", + /* Quirk: Some devices advertise a large max_fast_reg_page_list_len + * capability, but perform optimally when the MRs are not larger + * than a page. + */ + if (attrs->max_sge_rd > 1) + ia->ri_max_frwr_depth = attrs->max_sge_rd; + else + ia->ri_max_frwr_depth = attrs->max_fast_reg_page_list_len; + if (ia->ri_max_frwr_depth > RPCRDMA_MAX_DATA_SEGS) + ia->ri_max_frwr_depth = RPCRDMA_MAX_DATA_SEGS; + dprintk("RPC: %s: max FR page list depth = %u\n", __func__, ia->ri_max_frwr_depth); /* Add room for frwr register and invalidate WRs. From patchwork Wed Dec 19 15:58:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737591 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D6D0714DE for ; Wed, 19 Dec 2018 15:59:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C40EC29584 for ; Wed, 19 Dec 2018 15:59:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B7EAF295E0; Wed, 19 Dec 2018 15:59:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 695DB29584 for ; Wed, 19 Dec 2018 15:59:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729338AbeLSP67 (ORCPT ); Wed, 19 Dec 2018 10:58:59 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:39373 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP67 (ORCPT ); Wed, 19 Dec 2018 10:58:59 -0500 Received: by mail-it1-f193.google.com with SMTP id a6so10004956itl.4; Wed, 19 Dec 2018 07:58:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=OORiC2wAnW7FG4R2YzuLb2JoY0Sas2INBajUJHi6Zxw=; b=I34zhbAD8tw122kbRCO58j4ydcvHPNRGkix2ti/FUpQKfmbFuGvWwh8TnBKPzAZjKZ zWeKMSaCHY4wn7FNxigpOuMzEhlpDwMeh0UMpzE8iRwBF487Zilc1K0RR5l14uGvbblF P1S7+YFIq/WRm6mY3OEu0ygFQRERex5+G43oNdWfupex7BdE7czjNs2IdGYvUvTYkRN+ Z57Yt5id7Mn3iS/u2LPEMu7UgGNQur8A5d06CVTAeOnVc0h0tGnugCMyqmYxE6D/MisZ TTvNwuosCFBEnDyfs86/KIF1hVq3QzC9lIRmUZbK3ZS9+o7uzcI0PK2c11pFZ6axkZbo N/3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=OORiC2wAnW7FG4R2YzuLb2JoY0Sas2INBajUJHi6Zxw=; b=FM3oHIfb9cCe6qnkODMaBUFYvvf+dy2SCE5HTNbk3f7oY0uDLAfxiPBPyy+EimwUWN l8y82l46/YP9LtaecQCp3FB6Ni4cINUA2x5l/mq/4qWHDvvnjyMk1eXM20Wv0FnOdsmN pQs/zWp7Nyl79zFX+Mb/2FW/3TzuMPJkL/WWCFGaZwSSYTqquhTOIIRSFhsGjb11kKB4 6o6EYoS1ehpbw8PmZEVz+2oC7frA/0NvMY8HkpxDBmrhl6BQYVAzAT6ADPeyS/gR9Rw5 iNJD4voT72+32gJaBnTzsx5FWfkeuGJHlKyeQsm6MsTmXlBhzScr7UiVS0OfKqVxJENQ wb9w== X-Gm-Message-State: AA+aEWYOKApP6zo872gD226yunxEQpr9vkweVujSQ/E3darhcy3YZPxG drpWdl34JKiyBPM2vRijJ2u30T34 X-Google-Smtp-Source: AFSGD/UsGIrFdKLf0TA5DRnWhXuyY5uuPrehBonZIoSc2HHVMKfsD18kmuDxLuTuruJolLIG5DDQ1w== X-Received: by 2002:a24:5a11:: with SMTP id v17mr7089016ita.114.1545235138003; Wed, 19 Dec 2018 07:58:58 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id i135sm3753477iti.34.2018.12.19.07.58.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:58:57 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFwuEG025783; Wed, 19 Dec 2018 15:58:56 GMT Subject: [PATCH v5 09/30] xprtrdma: Remove support for FMR memory registration From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:58:56 -0500 Message-ID: <20181219155856.11602.30114.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP FMR is not supported on most recent RDMA devices. It is also less secure than FRWR because an FMR memory registration can expose adjacent bytes to remote reading or writing. As discussed during the RDMA BoF at LPC 2018, it is time to remove support for FMR in the NFS/RDMA client stack. Note that NFS/RDMA server-side uses either local memory registration or FRWR. FMR is not used. There are a few Infiniband/RoCE devices in the kernel tree that do not appear to support MEM_MGT_EXTENSIONS (FRWR), and therefore will not support client-side NFS/RDMA after this patch. These are: - mthca - qib - hns (RoCE) Users of these devices can use NFS/TCP on IPoIB instead. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/Makefile | 3 net/sunrpc/xprtrdma/fmr_ops.c | 340 --------------------------------------- net/sunrpc/xprtrdma/verbs.c | 6 - net/sunrpc/xprtrdma/xprt_rdma.h | 12 - 4 files changed, 2 insertions(+), 359 deletions(-) delete mode 100644 net/sunrpc/xprtrdma/fmr_ops.c diff --git a/net/sunrpc/xprtrdma/Makefile b/net/sunrpc/xprtrdma/Makefile index 8bf19e1..8ed0377 100644 --- a/net/sunrpc/xprtrdma/Makefile +++ b/net/sunrpc/xprtrdma/Makefile @@ -1,8 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_SUNRPC_XPRT_RDMA) += rpcrdma.o -rpcrdma-y := transport.o rpc_rdma.o verbs.o \ - fmr_ops.o frwr_ops.o \ +rpcrdma-y := transport.o rpc_rdma.o verbs.o frwr_ops.o \ svc_rdma.o svc_rdma_backchannel.o svc_rdma_transport.o \ svc_rdma_sendto.o svc_rdma_recvfrom.o svc_rdma_rw.o \ module.o diff --git a/net/sunrpc/xprtrdma/fmr_ops.c b/net/sunrpc/xprtrdma/fmr_ops.c deleted file mode 100644 index dd1e91b..0000000 --- a/net/sunrpc/xprtrdma/fmr_ops.c +++ /dev/null @@ -1,340 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Copyright (c) 2015, 2017 Oracle. All rights reserved. - * Copyright (c) 2003-2007 Network Appliance, Inc. All rights reserved. - */ - -/* Lightweight memory registration using Fast Memory Regions (FMR). - * Referred to sometimes as MTHCAFMR mode. - * - * FMR uses synchronous memory registration and deregistration. - * FMR registration is known to be fast, but FMR deregistration - * can take tens of usecs to complete. - */ - -/* Normal operation - * - * A Memory Region is prepared for RDMA READ or WRITE using the - * ib_map_phys_fmr verb (fmr_op_map). When the RDMA operation is - * finished, the Memory Region is unmapped using the ib_unmap_fmr - * verb (fmr_op_unmap). - */ - -#include - -#include "xprt_rdma.h" -#include - -#if IS_ENABLED(CONFIG_SUNRPC_DEBUG) -# define RPCDBG_FACILITY RPCDBG_TRANS -#endif - -/* Maximum scatter/gather per FMR */ -#define RPCRDMA_MAX_FMR_SGES (64) - -/* Access mode of externally registered pages */ -enum { - RPCRDMA_FMR_ACCESS_FLAGS = IB_ACCESS_REMOTE_WRITE | - IB_ACCESS_REMOTE_READ, -}; - -bool -fmr_is_supported(struct rpcrdma_ia *ia) -{ - if (!ia->ri_device->alloc_fmr) { - pr_info("rpcrdma: 'fmr' mode is not supported by device %s\n", - ia->ri_device->name); - return false; - } - return true; -} - -static void -__fmr_unmap(struct rpcrdma_mr *mr) -{ - LIST_HEAD(l); - int rc; - - list_add(&mr->fmr.fm_mr->list, &l); - rc = ib_unmap_fmr(&l); - list_del(&mr->fmr.fm_mr->list); - if (rc) - pr_err("rpcrdma: final ib_unmap_fmr for %p failed %i\n", - mr, rc); -} - -/* Release an MR. - */ -static void -fmr_op_release_mr(struct rpcrdma_mr *mr) -{ - int rc; - - kfree(mr->fmr.fm_physaddrs); - kfree(mr->mr_sg); - - /* In case this one was left mapped, try to unmap it - * to prevent dealloc_fmr from failing with EBUSY - */ - __fmr_unmap(mr); - - rc = ib_dealloc_fmr(mr->fmr.fm_mr); - if (rc) - pr_err("rpcrdma: final ib_dealloc_fmr for %p returned %i\n", - mr, rc); - - kfree(mr); -} - -/* MRs are dynamically allocated, so simply clean up and release the MR. - * A replacement MR will subsequently be allocated on demand. - */ -static void -fmr_mr_recycle_worker(struct work_struct *work) -{ - struct rpcrdma_mr *mr = container_of(work, struct rpcrdma_mr, mr_recycle); - struct rpcrdma_xprt *r_xprt = mr->mr_xprt; - - trace_xprtrdma_mr_recycle(mr); - - trace_xprtrdma_mr_unmap(mr); - ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, - mr->mr_sg, mr->mr_nents, mr->mr_dir); - - spin_lock(&r_xprt->rx_buf.rb_mrlock); - list_del(&mr->mr_all); - r_xprt->rx_stats.mrs_recycled++; - spin_unlock(&r_xprt->rx_buf.rb_mrlock); - fmr_op_release_mr(mr); -} - -static int -fmr_op_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) -{ - static struct ib_fmr_attr fmr_attr = { - .max_pages = RPCRDMA_MAX_FMR_SGES, - .max_maps = 1, - .page_shift = PAGE_SHIFT - }; - - mr->fmr.fm_physaddrs = kcalloc(RPCRDMA_MAX_FMR_SGES, - sizeof(u64), GFP_KERNEL); - if (!mr->fmr.fm_physaddrs) - goto out_free; - - mr->mr_sg = kcalloc(RPCRDMA_MAX_FMR_SGES, - sizeof(*mr->mr_sg), GFP_KERNEL); - if (!mr->mr_sg) - goto out_free; - - sg_init_table(mr->mr_sg, RPCRDMA_MAX_FMR_SGES); - - mr->fmr.fm_mr = ib_alloc_fmr(ia->ri_pd, RPCRDMA_FMR_ACCESS_FLAGS, - &fmr_attr); - if (IS_ERR(mr->fmr.fm_mr)) - goto out_fmr_err; - - INIT_LIST_HEAD(&mr->mr_list); - INIT_WORK(&mr->mr_recycle, fmr_mr_recycle_worker); - return 0; - -out_fmr_err: - dprintk("RPC: %s: ib_alloc_fmr returned %ld\n", __func__, - PTR_ERR(mr->fmr.fm_mr)); - -out_free: - kfree(mr->mr_sg); - kfree(mr->fmr.fm_physaddrs); - return -ENOMEM; -} - -/* On success, sets: - * ep->rep_attr.cap.max_send_wr - * ep->rep_attr.cap.max_recv_wr - * cdata->max_requests - * ia->ri_max_segs - */ -static int -fmr_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, - struct rpcrdma_create_data_internal *cdata) -{ - int max_qp_wr; - - max_qp_wr = ia->ri_device->attrs.max_qp_wr; - max_qp_wr -= RPCRDMA_BACKWARD_WRS; - max_qp_wr -= 1; - if (max_qp_wr < RPCRDMA_MIN_SLOT_TABLE) - return -ENOMEM; - if (cdata->max_requests > max_qp_wr) - cdata->max_requests = max_qp_wr; - ep->rep_attr.cap.max_send_wr = cdata->max_requests; - ep->rep_attr.cap.max_send_wr += RPCRDMA_BACKWARD_WRS; - ep->rep_attr.cap.max_send_wr += 1; /* for ib_drain_sq */ - ep->rep_attr.cap.max_recv_wr = cdata->max_requests; - ep->rep_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS; - ep->rep_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */ - - ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / - RPCRDMA_MAX_FMR_SGES); - /* Reply chunks require segments for head and tail buffers */ - ia->ri_max_segs += 2; - if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) - ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS; - return 0; -} - -/* FMR mode conveys up to 64 pages of payload per chunk segment. - */ -static size_t -fmr_op_maxpages(struct rpcrdma_xprt *r_xprt) -{ - return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - (r_xprt->rx_ia.ri_max_segs - 2) * RPCRDMA_MAX_FMR_SGES); -} - -/* Use the ib_map_phys_fmr() verb to register a memory region - * for remote access via RDMA READ or RDMA WRITE. - */ -static struct rpcrdma_mr_seg * -fmr_op_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, struct rpcrdma_mr **out) -{ - struct rpcrdma_mr_seg *seg1 = seg; - int len, pageoff, i, rc; - struct rpcrdma_mr *mr; - u64 *dma_pages; - - mr = rpcrdma_mr_get(r_xprt); - if (!mr) - return ERR_PTR(-EAGAIN); - - pageoff = offset_in_page(seg1->mr_offset); - seg1->mr_offset -= pageoff; /* start of page */ - seg1->mr_len += pageoff; - len = -pageoff; - if (nsegs > RPCRDMA_MAX_FMR_SGES) - nsegs = RPCRDMA_MAX_FMR_SGES; - for (i = 0; i < nsegs;) { - if (seg->mr_page) - sg_set_page(&mr->mr_sg[i], - seg->mr_page, - seg->mr_len, - offset_in_page(seg->mr_offset)); - else - sg_set_buf(&mr->mr_sg[i], seg->mr_offset, - seg->mr_len); - len += seg->mr_len; - ++seg; - ++i; - /* Check for holes */ - if ((i < nsegs && offset_in_page(seg->mr_offset)) || - offset_in_page((seg-1)->mr_offset + (seg-1)->mr_len)) - break; - } - mr->mr_dir = rpcrdma_data_dir(writing); - - mr->mr_nents = ib_dma_map_sg(r_xprt->rx_ia.ri_device, - mr->mr_sg, i, mr->mr_dir); - if (!mr->mr_nents) - goto out_dmamap_err; - trace_xprtrdma_mr_map(mr); - - for (i = 0, dma_pages = mr->fmr.fm_physaddrs; i < mr->mr_nents; i++) - dma_pages[i] = sg_dma_address(&mr->mr_sg[i]); - rc = ib_map_phys_fmr(mr->fmr.fm_mr, dma_pages, mr->mr_nents, - dma_pages[0]); - if (rc) - goto out_maperr; - - mr->mr_handle = mr->fmr.fm_mr->rkey; - mr->mr_length = len; - mr->mr_offset = dma_pages[0] + pageoff; - - *out = mr; - return seg; - -out_dmamap_err: - pr_err("rpcrdma: failed to DMA map sg %p sg_nents %d\n", - mr->mr_sg, i); - rpcrdma_mr_put(mr); - return ERR_PTR(-EIO); - -out_maperr: - pr_err("rpcrdma: ib_map_phys_fmr %u@0x%llx+%i (%d) status %i\n", - len, (unsigned long long)dma_pages[0], - pageoff, mr->mr_nents, rc); - rpcrdma_mr_unmap_and_put(mr); - return ERR_PTR(-EIO); -} - -/* Post Send WR containing the RPC Call message. - */ -static int -fmr_op_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) -{ - return ib_post_send(ia->ri_id->qp, &req->rl_sendctx->sc_wr, NULL); -} - -/* Invalidate all memory regions that were registered for "req". - * - * Sleeps until it is safe for the host CPU to access the - * previously mapped memory regions. - * - * Caller ensures that @mrs is not empty before the call. This - * function empties the list. - */ -static void -fmr_op_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) -{ - struct rpcrdma_mr *mr; - LIST_HEAD(unmap_list); - int rc; - - /* ORDER: Invalidate all of the req's MRs first - * - * ib_unmap_fmr() is slow, so use a single call instead - * of one call per mapped FMR. - */ - list_for_each_entry(mr, mrs, mr_list) { - dprintk("RPC: %s: unmapping fmr %p\n", - __func__, &mr->fmr); - trace_xprtrdma_mr_localinv(mr); - list_add_tail(&mr->fmr.fm_mr->list, &unmap_list); - } - r_xprt->rx_stats.local_inv_needed++; - rc = ib_unmap_fmr(&unmap_list); - if (rc) - goto out_release; - - /* ORDER: Now DMA unmap all of the req's MRs, and return - * them to the free MW list. - */ - while (!list_empty(mrs)) { - mr = rpcrdma_mr_pop(mrs); - list_del(&mr->fmr.fm_mr->list); - rpcrdma_mr_unmap_and_put(mr); - } - - return; - -out_release: - pr_err("rpcrdma: ib_unmap_fmr failed (%i)\n", rc); - - while (!list_empty(mrs)) { - mr = rpcrdma_mr_pop(mrs); - list_del(&mr->fmr.fm_mr->list); - rpcrdma_mr_recycle(mr); - } -} - -const struct rpcrdma_memreg_ops rpcrdma_fmr_memreg_ops = { - .ro_map = fmr_op_map, - .ro_send = fmr_op_send, - .ro_unmap_sync = fmr_op_unmap_sync, - .ro_open = fmr_op_open, - .ro_maxpages = fmr_op_maxpages, - .ro_init_mr = fmr_op_init_mr, - .ro_release_mr = fmr_op_release_mr, - .ro_displayname = "fmr", - .ro_send_w_inv_ok = 0, -}; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 29798b6..ef17596 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -397,12 +397,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) break; } /*FALLTHROUGH*/ - case RPCRDMA_MTHCAFMR: - if (fmr_is_supported(ia)) { - ia->ri_ops = &rpcrdma_fmr_memreg_ops; - break; - } - /*FALLTHROUGH*/ default: pr_err("rpcrdma: Device %s does not support memreg mode %d\n", ia->ri_device->name, xprt_rdma_memreg_strategy); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 99b7f8e..84f7bbe 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -262,20 +262,12 @@ struct rpcrdma_frwr { }; }; -struct rpcrdma_fmr { - struct ib_fmr *fm_mr; - u64 *fm_physaddrs; -}; - struct rpcrdma_mr { struct list_head mr_list; struct scatterlist *mr_sg; int mr_nents; enum dma_data_direction mr_dir; - union { - struct rpcrdma_fmr fmr; - struct rpcrdma_frwr frwr; - }; + struct rpcrdma_frwr frwr; struct rpcrdma_xprt *mr_xprt; u32 mr_handle; u32 mr_length; @@ -490,7 +482,6 @@ struct rpcrdma_memreg_ops { const int ro_send_w_inv_ok; }; -extern const struct rpcrdma_memreg_ops rpcrdma_fmr_memreg_ops; extern const struct rpcrdma_memreg_ops rpcrdma_frwr_memreg_ops; /* @@ -545,7 +536,6 @@ struct rpcrdma_xprt { void rpcrdma_ia_remove(struct rpcrdma_ia *ia); void rpcrdma_ia_close(struct rpcrdma_ia *); bool frwr_is_supported(struct rpcrdma_ia *); -bool fmr_is_supported(struct rpcrdma_ia *); /* * Endpoint calls - xprtrdma/verbs.c From patchwork Wed Dec 19 15:59:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737595 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94D3E13B5 for ; Wed, 19 Dec 2018 15:59:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8227429584 for ; Wed, 19 Dec 2018 15:59:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7473229595; Wed, 19 Dec 2018 15:59:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4DF482957A for ; Wed, 19 Dec 2018 15:59:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727920AbeLSP7E (ORCPT ); Wed, 19 Dec 2018 10:59:04 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:54631 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP7E (ORCPT ); Wed, 19 Dec 2018 10:59:04 -0500 Received: by mail-it1-f196.google.com with SMTP id i145so10381543ita.4; Wed, 19 Dec 2018 07:59:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=t1SJVy6xliRSaevRIUULty8en2M27Vr2nLDmtsR3UYE=; b=q83qzki6E9n0zBeOegCDAcMaTciEFAUIr/hm3dC98pzp6j/ocfVHYSPAlG31xorU9C g3ydVthFDTJlPelFZ14GM1dcGI9Z6kWhwI7cIx35pT093Vhl7J99Tuna3upp9qsSaDen s14+m5fzX994FoJqfzgeysJ/s1hnKSlOG35VjILZbZFUMXFQ+dUVCjs3j0Fb5Ca/y9A7 RzOpOh3eatFef/VFa6BgWrqQCWPg6jwPSE8xIzfFP3++CkKOLdeK/k2e4T93Xr8DsGSF vy9dtC7gvgG4HWRQALWI/HBKLP9XQ4DYiLacyPgzxluk06rJAMiMwODgxIhwjMnyC03n XefQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=t1SJVy6xliRSaevRIUULty8en2M27Vr2nLDmtsR3UYE=; b=ITlZsLv2QaeSe48LPQJUTbI1G16y8aCVtjzHPmlFlQQNKP72D94E0ua7mOxW8VAko5 OToGTnsQZQAjLF7GWT49w6zIEqza5Yl6Ba7eCbN3xvzFQjfagD2sLcNQVMjl/Zm17N+q +vK1J4xtq03Yp0QHgn+MmVaibLogZ0w9vyUvpzDcYg8qdJXdgOb1SECwdO0vS96MYFmy 7ZUfcg5UCWE6LUfY8FFbf4VByDdFJ+NYXQNe8zqqOE1/i+BWM9WGFlmJ4TNyETqKV0S8 pno/tOVdyqCm3r9xB6GR9Wewzs+nlhnFGZ9JjKjMHv41ldBFUFEnuBS+jSl6CNLzGnlh 0OGA== X-Gm-Message-State: AA+aEWZWnNkMGFXTpZkMH8wi7PQPL5pBftppYpgT2+6Fr2cSTNrDXYc4 rTOEu923pZP+2OEp/5CxiNA= X-Google-Smtp-Source: AFSGD/X3AsOpZjjvYAQecV22PiwOoF87P2D6OAO/6DQw5Z2n+hlyWHWIx+CmQ/0LiEzzM9qzMPlbYQ== X-Received: by 2002:a02:818c:: with SMTP id n12mr19594785jag.108.1545235143205; Wed, 19 Dec 2018 07:59:03 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id 81sm5789059itz.15.2018.12.19.07.59.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:02 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFx1oM025786; Wed, 19 Dec 2018 15:59:01 GMT Subject: [PATCH v5 10/30] xprtrdma: Remove rpcrdma_memreg_ops From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:01 -0500 Message-ID: <20181219155901.11602.15647.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Now that there is only FRWR, there is no need for a memory registration switch. The indirect calls to the memreg operations can be replaced with faster direct calls. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 131 +++++++++++++++++++++++++-------------- net/sunrpc/xprtrdma/rpc_rdma.c | 14 +--- net/sunrpc/xprtrdma/transport.c | 2 - net/sunrpc/xprtrdma/verbs.c | 22 +++---- net/sunrpc/xprtrdma/xprt_rdma.h | 48 +++++--------- 5 files changed, 116 insertions(+), 101 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 16976b0..fb0944d 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -15,21 +15,21 @@ /* Normal operation * * A Memory Region is prepared for RDMA READ or WRITE using a FAST_REG - * Work Request (frwr_op_map). When the RDMA operation is finished, this + * Work Request (frwr_map). When the RDMA operation is finished, this * Memory Region is invalidated using a LOCAL_INV Work Request - * (frwr_op_unmap_sync). + * (frwr_unmap_sync). * * Typically these Work Requests are not signaled, and neither are RDMA * SEND Work Requests (with the exception of signaling occasionally to * prevent provider work queue overflows). This greatly reduces HCA * interrupt workload. * - * As an optimization, frwr_op_unmap marks MRs INVALID before the + * As an optimization, frwr_unmap marks MRs INVALID before the * LOCAL_INV WR is posted. If posting succeeds, the MR is placed on * rb_mrs immediately so that no work (like managing a linked list * under a spinlock) is needed in the completion upcall. * - * But this means that frwr_op_map() can occasionally encounter an MR + * But this means that frwr_map() can occasionally encounter an MR * that is INVALID but the LOCAL_INV WR has not completed. Work Queue * ordering prevents a subsequent FAST_REG WR from executing against * that MR while it is still being invalidated. @@ -57,14 +57,14 @@ * FLUSHED_LI: The MR was being invalidated when the QP entered ERROR * state, and the pending WR was flushed. * - * When frwr_op_map encounters FLUSHED and VALID MRs, they are recovered + * When frwr_map encounters FLUSHED and VALID MRs, they are recovered * with ib_dereg_mr and then are re-initialized. Because MR recovery * allocates fresh resources, it is deferred to a workqueue, and the * recovered MRs are placed back on the rb_mrs list when recovery is - * complete. frwr_op_map allocates another MR for the current RPC while + * complete. frwr_map allocates another MR for the current RPC while * the broken MR is reset. * - * To ensure that frwr_op_map doesn't encounter an MR that is marked + * To ensure that frwr_map doesn't encounter an MR that is marked * INVALID but that is about to be flushed due to a previous transport * disconnect, the transport connect worker attempts to drain all * pending send queue WRs before the transport is reconnected. @@ -80,8 +80,13 @@ # define RPCDBG_FACILITY RPCDBG_TRANS #endif -bool -frwr_is_supported(struct rpcrdma_ia *ia) +/** + * frwr_is_supported - Check if device supports FRWR + * @ia: interface adapter to check + * + * Returns true if device supports FRWR, otherwise false + */ +bool frwr_is_supported(struct rpcrdma_ia *ia) { struct ib_device_attr *attrs = &ia->ri_device->attrs; @@ -97,8 +102,12 @@ return false; } -static void -frwr_op_release_mr(struct rpcrdma_mr *mr) +/** + * frwr_release_mr - Destroy one MR + * @mr: MR allocated by frwr_init_mr + * + */ +void frwr_release_mr(struct rpcrdma_mr *mr) { int rc; @@ -132,11 +141,19 @@ list_del(&mr->mr_all); r_xprt->rx_stats.mrs_recycled++; spin_unlock(&r_xprt->rx_buf.rb_mrlock); - frwr_op_release_mr(mr); + + frwr_release_mr(mr); } -static int -frwr_op_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) +/** + * frwr_init_mr - Initialize one MR + * @ia: interface adapter + * @mr: generic MR to prepare for FRWR + * + * Returns zero if successful. Otherwise a negative errno + * is returned. + */ +int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) { unsigned int depth = ia->ri_max_frwr_depth; struct rpcrdma_frwr *frwr = &mr->frwr; @@ -172,7 +189,13 @@ return rc; } -/* On success, sets: +/** + * frwr_open - Prepare an endpoint for use with FRWR + * @ia: interface adapter this endpoint will use + * @ep: endpoint to prepare + * @cdata: transport parameters + * + * On success, sets: * ep->rep_attr.cap.max_send_wr * ep->rep_attr.cap.max_recv_wr * cdata->max_requests @@ -181,10 +204,11 @@ * And these FRWR-related fields: * ia->ri_max_frwr_depth * ia->ri_mrtype + * + * On failure, a negative errno is returned. */ -static int -frwr_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, - struct rpcrdma_create_data_internal *cdata) +int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, + struct rpcrdma_create_data_internal *cdata) { struct ib_device_attr *attrs = &ia->ri_device->attrs; int max_qp_wr, depth, delta; @@ -258,11 +282,16 @@ return 0; } -/* FRWR mode conveys a list of pages per chunk segment. The +/** + * frwr_maxpages - Compute size of largest payload + * @r_xprt: transport + * + * Returns maximum size of an RPC message, in pages. + * + * FRWR mode conveys a list of pages per chunk segment. The * maximum length of that list is the FRWR page list depth. */ -static size_t -frwr_op_maxpages(struct rpcrdma_xprt *r_xprt) +size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; @@ -344,12 +373,24 @@ trace_xprtrdma_wc_li_wake(wc, frwr); } -/* Post a REG_MR Work Request to register a memory region +/** + * frwr_map - Register a memory region + * @r_xprt: controlling transport + * @seg: memory region co-ordinates + * @nsegs: number of segments remaining + * @writing: true when RDMA Write will be used + * @out: initialized MR + * + * Prepare a REG_MR Work Request to register a memory region * for remote access via RDMA READ or RDMA WRITE. + * + * Returns the next segment or a negative errno pointer. + * On success, the prepared MR is planted in @out. */ -static struct rpcrdma_mr_seg * -frwr_op_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, struct rpcrdma_mr **out) +struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_mr_seg *seg, + int nsegs, bool writing, + struct rpcrdma_mr **out) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; bool holes_ok = ia->ri_mrtype == IB_MR_TYPE_SG_GAPS; @@ -434,14 +475,18 @@ return ERR_PTR(-EIO); } -/* Post Send WR containing the RPC Call message. +/** + * frwr_send - post Send WR containing the RPC Call message + * @ia: interface adapter + * @req: Prepared RPC Call * * For FRMR, chain any FastReg WRs to the Send WR. Only a * single ib_post_send call is needed to register memory * and then post the Send WR. + * + * Returns the result of ib_post_send. */ -static int -frwr_op_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) +int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) { struct ib_send_wr *post_wr; struct rpcrdma_mr *mr; @@ -468,10 +513,13 @@ return ib_post_send(ia->ri_id->qp, post_wr, NULL); } -/* Handle a remotely invalidated mr on the @mrs list +/** + * frwr_reminv - handle a remotely invalidated mr on the @mrs list + * @rep: Received reply + * @mrs: list of MRs to check + * */ -static void -frwr_op_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) +void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) { struct rpcrdma_mr *mr; @@ -485,7 +533,10 @@ } } -/* Invalidate all memory regions that were registered for "req". +/** + * frwr_unmap_sync - invalidate memory regions that were registered for @req + * @r_xprt: controlling transport + * @mrs: list of MRs to process * * Sleeps until it is safe for the host CPU to access the * previously mapped memory regions. @@ -493,8 +544,7 @@ * Caller ensures that @mrs is not empty before the call. This * function empties the list. */ -static void -frwr_op_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) +void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) { struct ib_send_wr *first, **prev, *last; const struct ib_send_wr *bad_wr; @@ -577,16 +627,3 @@ rpcrdma_mr_recycle(mr); } } - -const struct rpcrdma_memreg_ops rpcrdma_frwr_memreg_ops = { - .ro_map = frwr_op_map, - .ro_send = frwr_op_send, - .ro_reminv = frwr_op_reminv, - .ro_unmap_sync = frwr_op_unmap_sync, - .ro_open = frwr_op_open, - .ro_maxpages = frwr_op_maxpages, - .ro_init_mr = frwr_op_init_mr, - .ro_release_mr = frwr_op_release_mr, - .ro_displayname = "frwr", - .ro_send_w_inv_ok = RPCRDMA_CMP_F_SND_W_INV_OK, -}; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 5738c9f..2a2023d 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -356,8 +356,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, return nsegs; do { - seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs, - false, &mr); + seg = frwr_map(r_xprt, seg, nsegs, false, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -414,8 +413,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs, - true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -472,8 +470,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs, - true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -1262,8 +1259,7 @@ void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) * RPC has relinquished all its Send Queue entries. */ if (!list_empty(&req->rl_registered)) - r_xprt->rx_ia.ri_ops->ro_unmap_sync(r_xprt, - &req->rl_registered); + frwr_unmap_sync(r_xprt, &req->rl_registered); /* Ensure that any DMA mapped pages associated with * the Send of the RPC Call have been unmapped before @@ -1292,7 +1288,7 @@ void rpcrdma_deferred_completion(struct work_struct *work) trace_xprtrdma_defer_cmp(rep); if (rep->rr_wc_flags & IB_WC_WITH_INVALIDATE) - r_xprt->rx_ia.ri_ops->ro_reminv(rep, &req->rl_registered); + frwr_reminv(rep, &req->rl_registered); rpcrdma_release_rqst(r_xprt, req); rpcrdma_complete_rqst(rep); } diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index d3f26c5..89e11f9 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -399,7 +399,7 @@ INIT_DELAYED_WORK(&new_xprt->rx_connect_worker, xprt_rdma_connect_worker); - xprt->max_payload = new_xprt->rx_ia.ri_ops->ro_maxpages(new_xprt); + xprt->max_payload = frwr_maxpages(new_xprt); if (xprt->max_payload == 0) goto out4; xprt->max_payload <<= PAGE_SHIFT; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index ef17596..0cce7b2 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -289,10 +289,9 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) break; } - dprintk("RPC: %s: %s:%s on %s/%s: %s\n", __func__, + dprintk("RPC: %s: %s:%s on %s/frwr: %s\n", __func__, rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt), - ia->ri_device->name, ia->ri_ops->ro_displayname, - rdma_event_msg(event->event)); + ia->ri_device->name, rdma_event_msg(event->event)); return 0; } @@ -392,10 +391,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) switch (xprt_rdma_memreg_strategy) { case RPCRDMA_FRWR: - if (frwr_is_supported(ia)) { - ia->ri_ops = &rpcrdma_frwr_memreg_ops; + if (frwr_is_supported(ia)) break; - } /*FALLTHROUGH*/ default: pr_err("rpcrdma: Device %s does not support memreg mode %d\n", @@ -509,7 +506,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) } ia->ri_max_send_sges = max_sge; - rc = ia->ri_ops->ro_open(ia, ep, cdata); + rc = frwr_open(ia, ep, cdata); if (rc) return rc; @@ -567,7 +564,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) /* Prepare RDMA-CM private message */ pmsg->cp_magic = rpcrdma_cmp_magic; pmsg->cp_version = RPCRDMA_CMP_VERSION; - pmsg->cp_flags |= ia->ri_ops->ro_send_w_inv_ok; + pmsg->cp_flags |= RPCRDMA_CMP_F_SND_W_INV_OK; pmsg->cp_send_size = rpcrdma_encode_buffer_size(cdata->inline_wsize); pmsg->cp_recv_size = rpcrdma_encode_buffer_size(cdata->inline_rsize); ep->rep_remote_cma.private_data = pmsg; @@ -991,7 +988,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf) if (!mr) break; - rc = ia->ri_ops->ro_init_mr(ia, mr); + rc = frwr_init_mr(ia, mr); if (rc) { kfree(mr); break; @@ -1171,7 +1168,6 @@ struct rpcrdma_req * { struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt, rx_buf); - struct rpcrdma_ia *ia = rdmab_to_ia(buf); struct rpcrdma_mr *mr; unsigned int count; @@ -1187,7 +1183,7 @@ struct rpcrdma_req * if (!list_empty(&mr->mr_list)) list_del(&mr->mr_list); - ia->ri_ops->ro_release_mr(mr); + frwr_release_mr(mr); count++; spin_lock(&buf->rb_mrlock); } @@ -1381,7 +1377,7 @@ struct rpcrdma_req * * * xprtrdma uses a regbuf for posting an outgoing RDMA SEND, or for * receiving the payload of RDMA RECV operations. During Long Calls - * or Replies they may be registered externally via ro_map. + * or Replies they may be registered externally via frwr_map. */ struct rpcrdma_regbuf * rpcrdma_alloc_regbuf(size_t size, enum dma_data_direction direction, @@ -1472,7 +1468,7 @@ struct rpcrdma_regbuf * --ep->rep_send_count; } - rc = ia->ri_ops->ro_send(ia, req); + rc = frwr_send(ia, req); trace_xprtrdma_post_send(req, rc); if (rc) return -ENOTCONN; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 84f7bbe..c42a003 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -66,7 +66,6 @@ * Interface Adapter -- one per transport instance */ struct rpcrdma_ia { - const struct rpcrdma_memreg_ops *ri_ops; struct ib_device *ri_device; struct rdma_cm_id *ri_id; struct ib_pd *ri_pd; @@ -406,7 +405,6 @@ struct rpcrdma_buffer { struct workqueue_struct *rb_completion_wq; struct delayed_work rb_refresh_worker; }; -#define rdmab_to_ia(b) (&container_of((b), struct rpcrdma_xprt, rx_buf)->rx_ia) /* rb_flags */ enum { @@ -457,34 +455,6 @@ struct rpcrdma_stats { }; /* - * Per-registration mode operations - */ -struct rpcrdma_xprt; -struct rpcrdma_memreg_ops { - struct rpcrdma_mr_seg * - (*ro_map)(struct rpcrdma_xprt *, - struct rpcrdma_mr_seg *, int, bool, - struct rpcrdma_mr **); - int (*ro_send)(struct rpcrdma_ia *ia, - struct rpcrdma_req *req); - void (*ro_reminv)(struct rpcrdma_rep *rep, - struct list_head *mrs); - void (*ro_unmap_sync)(struct rpcrdma_xprt *, - struct list_head *); - int (*ro_open)(struct rpcrdma_ia *, - struct rpcrdma_ep *, - struct rpcrdma_create_data_internal *); - size_t (*ro_maxpages)(struct rpcrdma_xprt *); - int (*ro_init_mr)(struct rpcrdma_ia *, - struct rpcrdma_mr *); - void (*ro_release_mr)(struct rpcrdma_mr *mr); - const char *ro_displayname; - const int ro_send_w_inv_ok; -}; - -extern const struct rpcrdma_memreg_ops rpcrdma_frwr_memreg_ops; - -/* * RPCRDMA transport -- encapsulates the structures above for * integration with RPC. * @@ -535,7 +505,6 @@ struct rpcrdma_xprt { int rpcrdma_ia_open(struct rpcrdma_xprt *xprt); void rpcrdma_ia_remove(struct rpcrdma_ia *ia); void rpcrdma_ia_close(struct rpcrdma_ia *); -bool frwr_is_supported(struct rpcrdma_ia *); /* * Endpoint calls - xprtrdma/verbs.c @@ -601,6 +570,23 @@ struct rpcrdma_regbuf *rpcrdma_alloc_regbuf(size_t, enum dma_data_direction, return writing ? DMA_FROM_DEVICE : DMA_TO_DEVICE; } +/* Memory registration calls xprtrdma/frwr_ops.c + */ +bool frwr_is_supported(struct rpcrdma_ia *); +int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, + struct rpcrdma_create_data_internal *cdata); +int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr); +void frwr_release_mr(struct rpcrdma_mr *mr); +size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt); +struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_mr_seg *seg, + int nsegs, bool writing, + struct rpcrdma_mr **mr); +int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); +void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); +void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, + struct list_head *mrs); + /* * RPC/RDMA protocol calls - xprtrdma/rpc_rdma.c */ From patchwork Wed Dec 19 15:59:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737599 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 253EA13B5 for ; Wed, 19 Dec 2018 15:59:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 13EB02B429 for ; Wed, 19 Dec 2018 15:59:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 08DC62B48D; Wed, 19 Dec 2018 15:59:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0183B2B429 for ; Wed, 19 Dec 2018 15:59:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729385AbeLSP7K (ORCPT ); Wed, 19 Dec 2018 10:59:10 -0500 Received: from mail-io1-f65.google.com ([209.85.166.65]:42903 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP7K (ORCPT ); Wed, 19 Dec 2018 10:59:10 -0500 Received: by mail-io1-f65.google.com with SMTP id x6so15959110ioa.9; Wed, 19 Dec 2018 07:59:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=tRIh6Vlmff0elQPQHQ6Kjc+t1Fp0s6+iXGm0s/cylgY=; b=JMo2c2xxpmS5vgqAzjn1Dm4cLsYfbzq5oSvgkBDHlHY16rcA2vwWLFGu76mUU61KZt PbxZm+1a2tcSw4deCHtiDW6zWMKvhaN9Aaxu5HHZiIz92hLQZGDQyjAS+IMTtqfeEgV9 FaM4HeNUgztGAceFy+YA4iEBJRP+DQgbM43Pn2P5Bl2RTUEm5AqGfFNQdmra9su8GUeh g+0r57jaGrhlDMcQ9YSUAbdaEUoDZEAPet0wppnhqQx3y+cagkpQhBIDWVPdmlZWrlO0 7lG04zcJujyXsDgz5gEcm12t9v/4eQft4cgEZFDXBP+hho91NvWpTT3Z0AB7nUCDVE9r sj4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=tRIh6Vlmff0elQPQHQ6Kjc+t1Fp0s6+iXGm0s/cylgY=; b=RWbAB6Sw1cW6obEbDMkrfHtQB+B0TgJNotNkZsfzR2EypatxnuDvqJwTEQgS0v76yC pED+q2BNP7dut2GUuusRLhALXfo5SCr9thj17TLLlUTCdMbyLYyEH81XZgkDEqVnTaW5 TY2Wb2JYiUmQ+nKtw4YxZfHq7k/WEI2oiOYxeYrzBAHzpT9kr1lGleGQcv8PNneE9wpk yW7DJjiUH6KBvTigKWrNvisP/mYi5HUTWhhH0l+HnfFuzO5LOlzinS7GuSzuP6qRXWMr +50o9mdlhnwcxywafjzeSLu35S2zPDekFdA2qvLmkXqdNxmJ6OhRSUT5VRWQqV1dMwsW AxbQ== X-Gm-Message-State: AA+aEWbJ7zEuYKAoSi2DAn47NIDT8SqJmRKu2M7duT4cpyE148yGFfn2 OvYuQ+GaF2OoJKakgi3DaHQ= X-Google-Smtp-Source: AFSGD/Un9hvIh2GehMpHD3elQyTm1f61Knn+sjQQHLqoNbf+HWS+wmRTOIFNLUW0ikkinz3sm8RJxQ== X-Received: by 2002:a5e:c00c:: with SMTP id u12mr17949413iol.152.1545235148615; Wed, 19 Dec 2018 07:59:08 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id h139sm3455355ith.24.2018.12.19.07.59.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:07 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFx7fl025789; Wed, 19 Dec 2018 15:59:07 GMT Subject: [PATCH v5 11/30] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR) From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:07 -0500 Message-ID: <20181219155906.11602.58244.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Place the associated RPC transaction's XID in the upper 32 bits of each RDMA segment's rdma_offset field. There are two reasons to do this: - The R_key only has 8 bits that are different from registration to registration. The XID adds more uniqueness to each RDMA segment to reduce the likelihood of a software bug on the server reading from or writing into memory it's not supposed to. - On-the-wire RDMA Read and Write requests do not otherwise carry any identifier that matches them up to an RPC. The XID in the upper 32 bits will act as an eye-catcher in network captures. Suggested-by: Tom Talpey Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 5 ++++- net/sunrpc/xprtrdma/rpc_rdma.c | 6 +++--- net/sunrpc/xprtrdma/xprt_rdma.h | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index fb0944d..97f88bb 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -379,6 +379,7 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) * @seg: memory region co-ordinates * @nsegs: number of segments remaining * @writing: true when RDMA Write will be used + * @xid: XID of RPC using the registered memory * @out: initialized MR * * Prepare a REG_MR Work Request to register a memory region @@ -389,7 +390,7 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) */ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, + int nsegs, bool writing, u32 xid, struct rpcrdma_mr **out) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; @@ -444,6 +445,8 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, if (unlikely(n != mr->mr_nents)) goto out_mapmr_err; + ibmr->iova &= 0x00000000ffffffff; + ibmr->iova |= ((u64)cpu_to_be32(xid)) << 32; key = (u8)(ibmr->rkey & 0x000000FF); ib_update_fast_reg_key(ibmr, ++key); diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 2a2023d..3804fb3 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -356,7 +356,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, return nsegs; do { - seg = frwr_map(r_xprt, seg, nsegs, false, &mr); + seg = frwr_map(r_xprt, seg, nsegs, false, rqst->rq_xid, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -413,7 +413,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = frwr_map(r_xprt, seg, nsegs, true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, rqst->rq_xid, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -470,7 +470,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = frwr_map(r_xprt, seg, nsegs, true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, rqst->rq_xid, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index c42a003..ff4eab1 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -580,7 +580,7 @@ int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt); struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, + int nsegs, bool writing, u32 xid, struct rpcrdma_mr **mr); int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); From patchwork Wed Dec 19 15:59:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737605 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39D3614E2 for ; Wed, 19 Dec 2018 15:59:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A7732B434 for ; Wed, 19 Dec 2018 15:59:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1ED682B437; Wed, 19 Dec 2018 15:59:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C927F2B4BE for ; Wed, 19 Dec 2018 15:59:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729408AbeLSP7P (ORCPT ); Wed, 19 Dec 2018 10:59:15 -0500 Received: from mail-it1-f170.google.com ([209.85.166.170]:55496 "EHLO mail-it1-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP7P (ORCPT ); Wed, 19 Dec 2018 10:59:15 -0500 Received: by mail-it1-f170.google.com with SMTP id m62so10359383ith.5; Wed, 19 Dec 2018 07:59:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Tke2KOJ89T4vcbDVIGt35hCRU2JIcQsMOR9LJ34omWk=; b=Y9D1iII7K9ZCQYP8u3f00lqVSzYiJV0UIO6OK3NRq5pTULlG7ALDKudtKKHjMzzn4b USDULsDnMlDV364nksli2oqMYL/7uqWnSRyywychrPiOwr1gA3lHl190/Ytb4isd3Dnw atUPEEqXkIHbSqKJNWRIViaLh2ILhHFwHZO6I62R3gEcaQKf5H6pMG9vZ7bZ7PCi2H28 uVOM2n1xQR2IiOJFQyMhqWHg8D4PoGHsEIr/bkZBR7L3rJ6MZv3To9vGcfWCzlp0YgDC JjTnQdowVBgXnV3DmZu+0KYxpe0/1CkuOyQkulCYNBIGrtyDRa6Ho+bgZKCPSiZ3UQ9c gWFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Tke2KOJ89T4vcbDVIGt35hCRU2JIcQsMOR9LJ34omWk=; b=jken/hNiMtfRCzk8K1/WP7Lwwgp/lmZ8xu3RY1m9krEvHIcP3dZZp3SADfuNSfs3V5 uBsh3plEZRdCJkLQPreLcuG35CkCpl/Kp+9IPclihDJiUms0l1zHtd++8ShPplrQB8yh HIlgqJ5o+0a0wdQPkRSd6Q0Hjd2ZBfjcdNVZlhZhFnUzbIKhiKwgXDZHX13dJk6leCfA pF3wWWTMWq5Y1yZubbdy4IE0nrh/hy/6lN6FCwTN5BvHcj0zSAxO3Rynza22Xd+DP+a3 XFDEF/LmA+Jdc31GDTqDsXp/fQjDz0e3MWBSEViRD0NOtiMS7EZBMNpkaV8duyUksxF0 TlAQ== X-Gm-Message-State: AA+aEWbr1lznSUBwJzb3Rm+PbQA7GQimTSoxT8IlXakIoyZbziQjtda1 1NqhXRpD/Baj4KCSRchLyBcD371T X-Google-Smtp-Source: AFSGD/UMtlVH7RDjEi4niC6BnMIIKpG2u0q4pHswQ7vKRTWFJ3Gu/kxsg9DWBdbjtwWuf+IiYgqO3w== X-Received: by 2002:a05:660c:74b:: with SMTP id a11mr6885696itl.27.1545235153930; Wed, 19 Dec 2018 07:59:13 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id j8sm3320894itb.30.2018.12.19.07.59.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:13 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxCmH025792; Wed, 19 Dec 2018 15:59:12 GMT Subject: [PATCH v5 12/30] NFS: Make "port=" mount option optional for RDMA mounts From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:12 -0500 Message-ID: <20181219155912.11602.87201.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Having to specify "proto=rdma,port=20049" is cumbersome. RFC 8267 Section 6.3 requires NFSv4 clients to use "the alternative well-known port number", which is 20049. Make the use of the well- known port number automatic, just as it is for NFS/TCP and port 2049. For NFSv2/3, Section 4.2 allows clients to simply choose 20049 as the default or use rpcbind. I don't know of an NFS/RDMA server implementation that registers it's NFS/RDMA service with rpcbind, so automatically choosing 20049 seems like the better choice. The other widely-deployed NFS/RDMA client, Solaris, also uses 20049 as the default port. Signed-off-by: Chuck Lever --- fs/nfs/super.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/fs/nfs/super.c b/fs/nfs/super.c index ac4b2f0..22247c2 100644 --- a/fs/nfs/super.c +++ b/fs/nfs/super.c @@ -2168,7 +2168,10 @@ static int nfs_validate_text_mount_data(void *options, if (args->version == 4) { #if IS_ENABLED(CONFIG_NFS_V4) - port = NFS_PORT; + if (args->nfs_server.protocol == XPRT_TRANSPORT_RDMA) + port = NFS_RDMA_PORT; + else + port = NFS_PORT; max_namelen = NFS4_MAXNAMLEN; max_pathlen = NFS4_MAXPATHLEN; nfs_validate_transport_protocol(args); @@ -2178,8 +2181,11 @@ static int nfs_validate_text_mount_data(void *options, #else goto out_v4_not_compiled; #endif /* CONFIG_NFS_V4 */ - } else + } else { nfs_set_mount_transport_protocol(args); + if (args->nfs_server.protocol == XPRT_TRANSPORT_RDMA) + port = NFS_RDMA_PORT; + } nfs_set_port(sap, &args->nfs_server.port, port); From patchwork Wed Dec 19 15:59:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737607 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D74C13B5 for ; Wed, 19 Dec 2018 15:59:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D3C52B437 for ; Wed, 19 Dec 2018 15:59:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 71CB32B440; Wed, 19 Dec 2018 15:59:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AEBB2B430 for ; Wed, 19 Dec 2018 15:59:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728002AbeLSP7U (ORCPT ); Wed, 19 Dec 2018 10:59:20 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:52834 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP7U (ORCPT ); Wed, 19 Dec 2018 10:59:20 -0500 Received: by mail-it1-f195.google.com with SMTP id g76so10381892itg.2; Wed, 19 Dec 2018 07:59:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=2kgtkyFhKXfzrghxvQsBnyUA+KcQv3ItXO8NKS1hJvg=; b=cDtZjQHZVINMuzPdIBxMGWl/aSWf8EfwBydRbW5IE7/qCQ1hO21/LTFq+J2plprgOt l6bpinAIQzF2Xh18YojqtZtUXfBprQZaYvNRvbPHyIHOdx8zp/XQbOJZ6+SPi2Iw3PFv J372n0d1ppXvOia/ilByArmthiYJfVLzNlRj+v2RX+mr7v9kSwEHwWEwmPwEUilzTu2C MrJCxpvYkub9pY420lIz0B9uRynKaDUwLToRf6w9NX4rTobTh6KJJLNp9K2axjplSAFH OVL/w3sepFsCRsHrA9hFw8LN+A6InIfwWq7CmeaYNmFWx9VHQfXq582imPE3yuydamyy 7aVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=2kgtkyFhKXfzrghxvQsBnyUA+KcQv3ItXO8NKS1hJvg=; b=d8nkqQvIWfqk3jnVrHv7fRwBFlieaQtQV2CW1b98l0BGC3g8LEHr34A0AqZhy/CkCA lEb9NxX1NDJ4GCduiokeGOiKTYqqu7Y0alc4i9tg382p9IpXtJbGKKGBE3iascIur9aC 6lJleY+HuwFmlCCyA948Dpc6a0QeI977JxBBvtJ4/3fMcwOyEzgzOEuS63sz/6W76lQu JYuxRQy8HMZCVd3kgZNof4Wsudyb2BBj4TZzFHl4QjCQ5th4EmFMbxHUGvcFOtVB2mz9 fdZ1g6BsX2tQSwC8SUcMaNGVxvPdjtyZEEgqxjL6b2CydSSE6FKruoqHt9wkz1zpnRYY xXeQ== X-Gm-Message-State: AA+aEWZ4c4q/64XPT90YLlh3LRt7JuNfuojMUt6QU0sctYenSMkjj6Yy mkBJeONLKpt5rfuja2oo8wo= X-Google-Smtp-Source: AFSGD/V7meSFOA0INxG4BARwIRAT+nkpWvcFGQjbsYG5SoSa/MArUkQPeaLuRYOITqEw4kPzxkfTBg== X-Received: by 2002:a02:c81a:: with SMTP id p26mr8868869jao.59.1545235159301; Wed, 19 Dec 2018 07:59:19 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id c3sm8493574ioi.2.2018.12.19.07.59.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:18 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxHZx025795; Wed, 19 Dec 2018 15:59:17 GMT Subject: [PATCH v5 13/30] xprtrdma: Recognize XDRBUF_SPARSE_PAGES From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:17 -0500 Message-ID: <20181219155917.11602.66631.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit 431f6eb3570f ("SUNRPC: Add a label for RPC calls that require allocation on receive") didn't update similar logic in rpc_rdma.c. I don't think this is a bug, per-se; the commit just adds more careful checking for broken upper layer behavior. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 3804fb3..939f84a 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -218,11 +218,12 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, ppages = xdrbuf->pages + (xdrbuf->page_base >> PAGE_SHIFT); page_base = offset_in_page(xdrbuf->page_base); while (len) { - if (unlikely(!*ppages)) { - /* XXX: Certain upper layer operations do - * not provide receive buffer pages. - */ - *ppages = alloc_page(GFP_ATOMIC); + /* ACL likes to be lazy in allocating pages - ACLs + * are small by default but can get huge. + */ + if (unlikely(xdrbuf->flags & XDRBUF_SPARSE_PAGES)) { + if (!*ppages) + *ppages = alloc_page(GFP_ATOMIC); if (!*ppages) return -ENOBUFS; } From patchwork Wed Dec 19 15:59:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B3BB14E2 for ; Wed, 19 Dec 2018 15:59:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4BF8E2B434 for ; Wed, 19 Dec 2018 15:59:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 404402B440; Wed, 19 Dec 2018 15:59:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F134A2B434 for ; Wed, 19 Dec 2018 15:59:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729507AbeLSP71 (ORCPT ); Wed, 19 Dec 2018 10:59:27 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:55875 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP70 (ORCPT ); Wed, 19 Dec 2018 10:59:26 -0500 Received: by mail-it1-f195.google.com with SMTP id m62so10360278ith.5; Wed, 19 Dec 2018 07:59:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=ZiDd9RCmQU8CJXpqpGBxGDa6t1ND1vxS8BmAbpd1q9M=; b=LbcmEpST/hd6zhX2obk5SoKlGPnJ9RyopgWoM+m63/HhaaKZkscjfXq36g2jm9pRYo TQYVVDoFC37ZajCof/29FIMT+q9KKExDTJd1gz6R6jdHtT9ZjN0HPjRw51FG2328V/ui z6WIqKPOdesE6JRJQ49HLOgjgMDjR8apjREtYm6xzZy6TZUxwmtdmOvXaOODAdD/+bQV B0bk0AYD39ARRyPD2MmUt8H+iGmX52Sj3+leXLKlv3n+VT8Wkh4iuKuDVR7Gu+jywpSD d1MB3Mlfx8CKAzkLxJX5PLtbz+BdC7VfA9jN/0HFtMkrzfmDDJ+AMdZuzr1sVsTfDyo2 aKEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=ZiDd9RCmQU8CJXpqpGBxGDa6t1ND1vxS8BmAbpd1q9M=; b=KlrXvEnTHfCP4rsulyFM3I0/J6FwBGvqpdynmsI9w6rukqpMvEvK6760rvI57eJPl8 CC42kqGv+ee7L9racYzre70N3sfnmfy0UvkMPEfgcyBmu/tND2uyqocYMocjYdFHC3ro UMlzyLMjaE7dof6OJOajoqoXuBgoeXQmqJBr3g7MT2+nXyuZh2wg0jFMZWWSjfzlcjsA tyKUCWKJWq4omzw0YTzRAxX2chhv/8tB4H4SBEjgQec1KUVCWBDnWgNWi3F1aaXREO1A 35o4Aiq9rJGy7w0aTtf93xPPhOvm+az/5BwukqHaOBPHmQmIPJHLqQfkD5RDv0YwRRmp +y7Q== X-Gm-Message-State: AA+aEWa5hWpatblO72zEsKvPIcSCgj4ELNprbCH86pF/O+Act4wSd6mB i5lqzeChAntPCWYzRye0Y/an5xd5 X-Google-Smtp-Source: AFSGD/XhaF4ZlAhFeflxcXNhYyuq68t0NT7Cwam5Aw4XrvzcDmIL2dwatq9FFxh61uCipZHKg8yLAg== X-Received: by 2002:a02:b015:: with SMTP id p21mr20608939jah.17.1545235164468; Wed, 19 Dec 2018 07:59:24 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id a8sm530915ios.9.2018.12.19.07.59.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:23 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxNOt025798; Wed, 19 Dec 2018 15:59:23 GMT Subject: [PATCH v5 14/30] xprtrdma: Remove request_module from backchannel From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:23 -0500 Message-ID: <20181219155923.11602.64778.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since commit ffe1f0df5862 ("rpcrdma: Merge svcrdma and xprtrdma modules into one"), the forward and backchannel components are part of the same kernel module. A separate request_module() call in the backchannel code is no longer necessary. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 79a55fc..e2704db 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -5,7 +5,6 @@ * Support for backward direction RPCs on RPC/RDMA. */ -#include #include #include #include @@ -101,7 +100,6 @@ int xprt_rdma_bc_setup(struct rpc_xprt *xprt, unsigned int reqs) goto out_free; r_xprt->rx_buf.rb_bc_srv_max_requests = reqs; - request_module("svcrdma"); trace_xprtrdma_cb_setup(r_xprt, reqs); return 0; From patchwork Wed Dec 19 15:59:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737615 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EEA0113B5 for ; Wed, 19 Dec 2018 15:59:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DDF872B434 for ; Wed, 19 Dec 2018 15:59:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D26AF2B440; Wed, 19 Dec 2018 15:59:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7FD312B434 for ; Wed, 19 Dec 2018 15:59:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729139AbeLSP7a (ORCPT ); Wed, 19 Dec 2018 10:59:30 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:37563 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727829AbeLSP7a (ORCPT ); Wed, 19 Dec 2018 10:59:30 -0500 Received: by mail-it1-f196.google.com with SMTP id b5so10052733iti.2; Wed, 19 Dec 2018 07:59:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=PXLIaALci2knYQ1gMtWCij3L4mHSL0l8lqusuVMPb0k=; b=LZcL8/FcSvKaLacf4DyLOT0t+5qtExUBt833v8FgKzezP+caGNLHy1s7jVgZzvTIqr wRbcL7IvNz4hzwPO/9LUj2y/rkzTTueUHC2IUeZVjaWwINWXRiY/KFq+Miaivu1lX/bR /Bfa8I14vJm0BxFm3X88C6OszOEgo/GWJgr3aDbMZS5mPVADnAZmgWuOSZg/RQYWZAyk /E6dgL5QV/XZG1p79URp7fD3GnxMEBxmSeJX8hR0+3d2jW9H+nV/ih65G/KkDIuoKyhb I6u7MA9Acx/e1fsk66gaUpMgIWU17oCpdt95FwIl8oY32H5lqwLkvXvczmPvaAns/kOn uUvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=PXLIaALci2knYQ1gMtWCij3L4mHSL0l8lqusuVMPb0k=; b=RLcTFx8Z9/PMtBZ0uS1Y3vL31yCCfb7Bo0gDl+C2RUSOfWIOlGrD1vJZbDRoLF40eJ 145abzAVq5k6au3+nbuK4RvJMf95RBJmyHEKXVvnOiZHn1AXASxxLbiWo8cgLfDg6W2a VfaNuVMo9lgBbTGTQQqUprUojOpHHj5y3nM0z8kY6d3T743SYjoDGxJW784pr/q7at+0 x75MambuMWZ8k4I70NF83MWb7QswpJ2GTg04hXAP62UfLUYi/60QLerCwvEUWL9BDnMW hPVA1befXj8kk9F2IdWdTAZrjuTXXudfhXg+1DjiCZLGwLBKoH8KThdwuXh7pTe72WnT sS3Q== X-Gm-Message-State: AA+aEWYyPUa4nQ9i7/exASS0UetBrrr+jfPwGwnpzRG2SLIjo/8n8YgC MXNA3JI+KW7EO8ICTriE79/O/RMU X-Google-Smtp-Source: AFSGD/WaT4QvtwP68PG+NDpyakmuKJCsgfqQNSIkvHbkbmxwhaWMkZ8j4QXVccJDmpxKBMbl6vPkGA== X-Received: by 2002:a02:97fd:: with SMTP id v58mr20818481jaj.115.1545235169858; Wed, 19 Dec 2018 07:59:29 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id d128sm9372345iof.37.2018.12.19.07.59.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:29 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxSH7025801; Wed, 19 Dec 2018 15:59:28 GMT Subject: [PATCH v5 15/30] xprtrdma: Expose transport header errors From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:28 -0500 Message-ID: <20181219155928.11602.65722.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For better observability of parsing errors, return the error code generated in the decoders to the upper layer consumer. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 1 - 1 file changed, 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 939f84a..8de0b9f 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1246,7 +1246,6 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep) out_badheader: trace_xprtrdma_reply_hdr(rep); r_xprt->rx_stats.bad_reply_count++; - status = -EIO; goto out; } From patchwork Wed Dec 19 15:59:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737619 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6DE3414DE for ; Wed, 19 Dec 2018 15:59:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5BB342B430 for ; Wed, 19 Dec 2018 15:59:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5002B2B440; Wed, 19 Dec 2018 15:59:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACAE12B430 for ; Wed, 19 Dec 2018 15:59:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729154AbeLSP7h (ORCPT ); Wed, 19 Dec 2018 10:59:37 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:53286 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727975AbeLSP7g (ORCPT ); Wed, 19 Dec 2018 10:59:36 -0500 Received: by mail-it1-f193.google.com with SMTP id g85so10381619ita.3; Wed, 19 Dec 2018 07:59:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=ot8YrCAOClaLZEB8d+4BtlkkZuUxk/YFO4keDzT+mbg=; b=tj60ZpjyGtNrcTTzC8iLjMi+OicBzp+H9ovzv9VC6/BkP1ucyCzMf6uVcwv31mxLdX kNUKyDsp1e/fGyAicko3NNX4nOwTfG6MQzRox7zN6LK0/Arp8tLV/ZpQm8PBnWI+3BUt wxoqHlCMh7XJDA3+G5Hf6sS+cVtMUcbdcqAMZRKaYA01RtywjtsbxN6G1xysL1x7QYzK oMu0XEH3B4EAsrSPmcBxio9a50TuFhcKPQ7Y40JsoPKlj6bDz1i3BzovBLhzPUAhZpGV 1tlueA1CuT81tcLsV9FFeQwrgzeHtDouqtccaSHg9plZcmuRkk+hh63oN7QhXjs9J59Y rGHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=ot8YrCAOClaLZEB8d+4BtlkkZuUxk/YFO4keDzT+mbg=; b=i/v6CYbjn1uSsD9blhiYPBCxY6a/su3+fmG3CYA8Va150SNhNob7YtPpkuxCv/uzH7 QJzh52o1o5OXv5scDJ1be+BJiABOoA/eOYRRqoHtmgtCji02/5il6fasKQBM8Gt+4EPz ySOh32ohv/9SEbdCALjrwJjqCps9VQlCGC07Ar+HeKJy6xbInW5fw1jUBWhkQgIDpICt 9gnYHFTcqeT+gPYonL5Om+lXciadhAiOxkLl5+Ln4PF9ruN4u9DbOQY++9H9eAt/98Yo TucAMlFP16cJHEPLTTtl2UFML1YlLS6wg4r52CPv4UtPiSjfPscjdCLnvIG10F5G969k uiDA== X-Gm-Message-State: AA+aEWZNpPnb4DkY3vBcpT5hjAgHQ6opG4Fkz1HMMB4159/LFpEOMe5b O0dOYEQ9pC6BmFiVxomE5H4= X-Google-Smtp-Source: AFSGD/VRIlkcBK25ROWwWzZ8nzWLnaYSL3KcoO9BYRZoYNVgLMeiWupAD04zM7MOo4iJmqLSKSJlGw== X-Received: by 2002:a24:ed4f:: with SMTP id r76mr6582889ith.17.1545235175484; Wed, 19 Dec 2018 07:59:35 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id b188sm3644690itc.9.2018.12.19.07.59.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:34 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxXLW025804; Wed, 19 Dec 2018 15:59:33 GMT Subject: [PATCH v5 16/30] xprtrdma: Simplify locking that protects the rl_allreqs list From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:33 -0500 Message-ID: <20181219155933.11602.86521.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: There's little chance of contention between the use of rb_lock and rb_reqslock, so merge the two. This avoids having to take both in some (possibly future) cases. Transport tear-down is already serialized, thus there is no need for locking at all when destroying rpcrdma_reqs. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 20 +++----------------- net/sunrpc/xprtrdma/verbs.c | 31 +++++++++++++++++-------------- net/sunrpc/xprtrdma/xprt_rdma.h | 7 +++---- 3 files changed, 23 insertions(+), 35 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index e2704db..aae2eb1 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -19,29 +19,16 @@ #undef RPCRDMA_BACKCHANNEL_DEBUG -static void rpcrdma_bc_free_rqst(struct rpcrdma_xprt *r_xprt, - struct rpc_rqst *rqst) -{ - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; - struct rpcrdma_req *req = rpcr_to_rdmar(rqst); - - spin_lock(&buf->rb_reqslock); - list_del(&req->rl_all); - spin_unlock(&buf->rb_reqslock); - - rpcrdma_destroy_req(req); -} - static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt, unsigned int count) { struct rpc_xprt *xprt = &r_xprt->rx_xprt; + struct rpcrdma_req *req; struct rpc_rqst *rqst; unsigned int i; for (i = 0; i < (count << 1); i++) { struct rpcrdma_regbuf *rb; - struct rpcrdma_req *req; size_t size; req = rpcrdma_create_req(r_xprt); @@ -67,7 +54,7 @@ static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt, return 0; out_fail: - rpcrdma_bc_free_rqst(r_xprt, rqst); + rpcrdma_req_destroy(req); return -ENOMEM; } @@ -225,7 +212,6 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) */ void xprt_rdma_bc_destroy(struct rpc_xprt *xprt, unsigned int reqs) { - struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); struct rpc_rqst *rqst, *tmp; spin_lock(&xprt->bc_pa_lock); @@ -233,7 +219,7 @@ void xprt_rdma_bc_destroy(struct rpc_xprt *xprt, unsigned int reqs) list_del(&rqst->rq_bc_pa_list); spin_unlock(&xprt->bc_pa_lock); - rpcrdma_bc_free_rqst(r_xprt, rqst); + rpcrdma_req_destroy(rpcr_to_rdmar(rqst)); spin_lock(&xprt->bc_pa_lock); } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 0cce7b2..51e09ae1 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1043,9 +1043,9 @@ struct rpcrdma_req * req->rl_buffer = buffer; INIT_LIST_HEAD(&req->rl_registered); - spin_lock(&buffer->rb_reqslock); + spin_lock(&buffer->rb_lock); list_add(&req->rl_all, &buffer->rb_allreqs); - spin_unlock(&buffer->rb_reqslock); + spin_unlock(&buffer->rb_lock); return req; } @@ -1113,7 +1113,6 @@ struct rpcrdma_req * INIT_LIST_HEAD(&buf->rb_send_bufs); INIT_LIST_HEAD(&buf->rb_allreqs); - spin_lock_init(&buf->rb_reqslock); for (i = 0; i < buf->rb_max_requests; i++) { struct rpcrdma_req *req; @@ -1154,9 +1153,18 @@ struct rpcrdma_req * kfree(rep); } +/** + * rpcrdma_req_destroy - Destroy an rpcrdma_req object + * @req: unused object to be destroyed + * + * This function assumes that the caller prevents concurrent device + * unload and transport tear-down. + */ void -rpcrdma_destroy_req(struct rpcrdma_req *req) +rpcrdma_req_destroy(struct rpcrdma_req *req) { + list_del(&req->rl_all); + rpcrdma_free_regbuf(req->rl_recvbuf); rpcrdma_free_regbuf(req->rl_sendbuf); rpcrdma_free_regbuf(req->rl_rdmabuf); @@ -1214,19 +1222,14 @@ struct rpcrdma_req * rpcrdma_destroy_rep(rep); } - spin_lock(&buf->rb_reqslock); - while (!list_empty(&buf->rb_allreqs)) { + while (!list_empty(&buf->rb_send_bufs)) { struct rpcrdma_req *req; - req = list_first_entry(&buf->rb_allreqs, - struct rpcrdma_req, rl_all); - list_del(&req->rl_all); - - spin_unlock(&buf->rb_reqslock); - rpcrdma_destroy_req(req); - spin_lock(&buf->rb_reqslock); + req = list_first_entry(&buf->rb_send_bufs, + struct rpcrdma_req, rl_list); + list_del(&req->rl_list); + rpcrdma_req_destroy(req); } - spin_unlock(&buf->rb_reqslock); rpcrdma_mrs_destroy(buf); } diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index ff4eab1..a1cdc85 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -392,14 +392,13 @@ struct rpcrdma_buffer { spinlock_t rb_lock; /* protect buf lists */ struct list_head rb_send_bufs; struct list_head rb_recv_bufs; + struct list_head rb_allreqs; + unsigned long rb_flags; u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ u32 rb_bc_srv_max_requests; - spinlock_t rb_reqslock; /* protect rb_allreqs */ - struct list_head rb_allreqs; - u32 rb_bc_max_requests; struct workqueue_struct *rb_completion_wq; @@ -522,7 +521,7 @@ int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *, * Buffer calls - xprtrdma/verbs.c */ struct rpcrdma_req *rpcrdma_create_req(struct rpcrdma_xprt *); -void rpcrdma_destroy_req(struct rpcrdma_req *); +void rpcrdma_req_destroy(struct rpcrdma_req *req); int rpcrdma_buffer_create(struct rpcrdma_xprt *); void rpcrdma_buffer_destroy(struct rpcrdma_buffer *); struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf); From patchwork Wed Dec 19 15:59:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737623 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 56A4814DE for ; Wed, 19 Dec 2018 15:59:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 42A552B430 for ; Wed, 19 Dec 2018 15:59:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 373632B440; Wed, 19 Dec 2018 15:59:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E2602B430 for ; Wed, 19 Dec 2018 15:59:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727973AbeLSP7n (ORCPT ); Wed, 19 Dec 2018 10:59:43 -0500 Received: from mail-io1-f66.google.com ([209.85.166.66]:39773 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727847AbeLSP7m (ORCPT ); Wed, 19 Dec 2018 10:59:42 -0500 Received: by mail-io1-f66.google.com with SMTP id k7so15970457iob.6; Wed, 19 Dec 2018 07:59:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=ClmdLukZcnfi8TxL6ggUVllZjN7Ko5YX/IX0FD4XDKk=; b=GmAte6PeqR9/o6Mqs6ssVIFRQ2DtX+MXe/in74RZ54B8eksFG/vceE3c5B6cQEIy8j 3GzNFYQCq82COUtgzRE65KAgYY0R/KPGMRXNRlkvEwMTW0Q5hbWFlpeVfM5Ii/uK6k/R AJtgvxpGnhGwag8XvYCqgMna8vdV7P3/7b5M4lfBqIdTFuxV65af9sn1/eED6qj78XMN NEYDdf617w8u6qe478PTDdU3Z7pORKYXMr49pjM4EJ5SQ8bVoBAjl6LdZyjqugXWsgob ciHHvxwvWIUtICYLp/CjOKOWxpVxrnuw8ytG2+6ENJ1+r8QEQIXsgwNtbzpDWkIfKfPW KDnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=ClmdLukZcnfi8TxL6ggUVllZjN7Ko5YX/IX0FD4XDKk=; b=XnGUE/gUAAIdvstpJxYDW/I5uaDoLrR73AJL8K6ETZpnDkVwKqqPmCwFqxE9VolGv4 3y8DGkHIyg+I96LafmWauTGfcopD1StTHr0/UB9tl++X2qZiY4t4PY3r54lk8LJrbAJU DaK846Kbh+LUGbFepzOzd7tai89Gz4w4YCV5UygR1qA7SRewBb5ywRhZsf4uCGDNjRW7 /k7nfA3NcnzCx6xslI0Tn6pIaxeIkDjQPTTX5Vddk07ifF1XwlDW5fRxipBMI1x2O82Y I4r7pdVy9k70EWwdRAuLXgwAINUv7tyBuLgEoSiK4q9/UZ8H2htwrldVweCle46PTLNi R/7g== X-Gm-Message-State: AJcUukfV6LX7wJVM6Z5DbpmMzOH811t96mVbdo+o43YY0Ar4GRmsMh9r XoxEstP7tcHLlg6azmrprrg= X-Google-Smtp-Source: ALg8bN5aTRFC+B1wN90hZDxKyGo/MTOLP4Tn6pg+GOfJw0x03Zsnu2SRXjdJfN7XPsFIL6go7tRtTQ== X-Received: by 2002:a5d:9652:: with SMTP id d18mr415012ios.257.1545235180519; Wed, 19 Dec 2018 07:59:40 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id k64sm3836482itb.7.2018.12.19.07.59.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:40 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxdlK025807; Wed, 19 Dec 2018 15:59:39 GMT Subject: [PATCH v5 17/30] xprtrdma: Cull dprintk() call sites From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:39 -0500 Message-ID: <20181219155939.11602.99457.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Remove dprintk() call sites that report rare or impossible errors. Leave a few that display high-value low noise status information. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 3 --- net/sunrpc/xprtrdma/rpc_rdma.c | 17 ++++++++++------- net/sunrpc/xprtrdma/transport.c | 33 ++++----------------------------- net/sunrpc/xprtrdma/verbs.c | 34 +++++----------------------------- 4 files changed, 19 insertions(+), 68 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index aae2eb1..dea831e 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -235,9 +235,6 @@ void xprt_rdma_bc_free_rqst(struct rpc_rqst *rqst) struct rpcrdma_req *req = rpcr_to_rdmar(rqst); struct rpc_xprt *xprt = rqst->rq_xprt; - dprintk("RPC: %s: freeing rqst %p (req %p)\n", - __func__, rqst, req); - rpcrdma_recv_buffer_put(req->rl_reply); req->rl_reply = NULL; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 8de0b9f..5a58769 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1186,17 +1186,20 @@ static int decode_reply_chunk(struct xdr_stream *xdr, u32 *length) p = xdr_inline_decode(xdr, 2 * sizeof(*p)); if (!p) break; - dprintk("RPC: %5u: %s: server reports version error (%u-%u)\n", - rqst->rq_task->tk_pid, __func__, - be32_to_cpup(p), be32_to_cpu(*(p + 1))); + dprintk("RPC: %s: server reports " + "version error (%u-%u), xid %08x\n", __func__, + be32_to_cpup(p), be32_to_cpu(*(p + 1)), + be32_to_cpu(rep->rr_xid)); break; case err_chunk: - dprintk("RPC: %5u: %s: server reports header decoding error\n", - rqst->rq_task->tk_pid, __func__); + dprintk("RPC: %s: server reports " + "header decoding error, xid %08x\n", __func__, + be32_to_cpu(rep->rr_xid)); break; default: - dprintk("RPC: %5u: %s: server reports unrecognized error %d\n", - rqst->rq_task->tk_pid, __func__, be32_to_cpup(p)); + dprintk("RPC: %s: server reports " + "unrecognized error %d, xid %08x\n", __func__, + be32_to_cpup(p), be32_to_cpu(rep->rr_xid)); } r_xprt->rx_stats.bad_reply_count++; diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 89e11f9..6a57033 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -318,17 +318,12 @@ struct sockaddr *sap; int rc; - if (args->addrlen > sizeof(xprt->addr)) { - dprintk("RPC: %s: address too large\n", __func__); + if (args->addrlen > sizeof(xprt->addr)) return ERR_PTR(-EBADF); - } xprt = xprt_alloc(args->net, sizeof(struct rpcrdma_xprt), 0, 0); - if (xprt == NULL) { - dprintk("RPC: %s: couldn't allocate rpcrdma_xprt\n", - __func__); + if (!xprt) return ERR_PTR(-ENOMEM); - } /* 60 second timeout, no retries */ xprt->timeout = &xprt_rdma_default_timeout; @@ -445,8 +440,6 @@ void xprt_rdma_close(struct rpc_xprt *xprt) might_sleep(); - dprintk("RPC: %s: closing xprt %p\n", __func__, xprt); - /* Prevent marshaling and sending of new requests */ xprt_clear_connected(xprt); @@ -853,24 +846,15 @@ void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) void xprt_rdma_cleanup(void) { - int rc; - - dprintk("RPCRDMA Module Removed, deregister RPC RDMA transport\n"); #if IS_ENABLED(CONFIG_SUNRPC_DEBUG) if (sunrpc_table_header) { unregister_sysctl_table(sunrpc_table_header); sunrpc_table_header = NULL; } #endif - rc = xprt_unregister_transport(&xprt_rdma); - if (rc) - dprintk("RPC: %s: xprt_unregister returned %i\n", - __func__, rc); - rc = xprt_unregister_transport(&xprt_rdma_bc); - if (rc) - dprintk("RPC: %s: xprt_unregister(bc) returned %i\n", - __func__, rc); + xprt_unregister_transport(&xprt_rdma); + xprt_unregister_transport(&xprt_rdma_bc); } int xprt_rdma_init(void) @@ -887,15 +871,6 @@ int xprt_rdma_init(void) return rc; } - dprintk("RPCRDMA Module Init, register RPC RDMA transport\n"); - - dprintk("Defaults:\n"); - dprintk("\tSlots %d\n" - "\tMaxInlineRead %d\n\tMaxInlineWrite %d\n", - xprt_rdma_slot_table_entries, - xprt_rdma_max_inline_read, xprt_rdma_max_inline_write); - dprintk("\tPadding 0\n\tMemreg %d\n", xprt_rdma_memreg_strategy); - #if IS_ENABLED(CONFIG_SUNRPC_DEBUG) if (!sunrpc_table_header) sunrpc_table_header = register_sysctl_table(sunrpc_table); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 51e09ae1..85c51b8 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -309,22 +309,15 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) id = rdma_create_id(xprt->rx_xprt.xprt_net, rpcrdma_cm_event_handler, xprt, RDMA_PS_TCP, IB_QPT_RC); - if (IS_ERR(id)) { - rc = PTR_ERR(id); - dprintk("RPC: %s: rdma_create_id() failed %i\n", - __func__, rc); + if (IS_ERR(id)) return id; - } ia->ri_async_rc = -ETIMEDOUT; rc = rdma_resolve_addr(id, NULL, (struct sockaddr *)&xprt->rx_xprt.addr, RDMA_RESOLVE_TIMEOUT); - if (rc) { - dprintk("RPC: %s: rdma_resolve_addr() failed %i\n", - __func__, rc); + if (rc) goto out; - } rc = wait_for_completion_interruptible_timeout(&ia->ri_done, wtimeout); if (rc < 0) { trace_xprtrdma_conn_tout(xprt); @@ -337,11 +330,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) ia->ri_async_rc = -ETIMEDOUT; rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT); - if (rc) { - dprintk("RPC: %s: rdma_resolve_route() failed %i\n", - __func__, rc); + if (rc) goto out; - } rc = wait_for_completion_interruptible_timeout(&ia->ri_done, wtimeout); if (rc < 0) { trace_xprtrdma_conn_tout(xprt); @@ -540,8 +530,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) 1, IB_POLL_WORKQUEUE); if (IS_ERR(sendcq)) { rc = PTR_ERR(sendcq); - dprintk("RPC: %s: failed to create send CQ: %i\n", - __func__, rc); goto out1; } @@ -550,8 +538,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) 0, IB_POLL_WORKQUEUE); if (IS_ERR(recvcq)) { rc = PTR_ERR(recvcq); - dprintk("RPC: %s: failed to create recv CQ: %i\n", - __func__, rc); goto out2; } @@ -691,11 +677,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) } err = rdma_create_qp(id, ia->ri_pd, &ep->rep_attr); - if (err) { - dprintk("RPC: %s: rdma_create_qp returned %d\n", - __func__, err); + if (err) goto out_destroy; - } /* Atomically replace the transport's ID and QP. */ rc = 0; @@ -726,8 +709,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) dprintk("RPC: %s: connecting...\n", __func__); rc = rdma_create_qp(ia->ri_id, ia->ri_pd, &ep->rep_attr); if (rc) { - dprintk("RPC: %s: rdma_create_qp failed %i\n", - __func__, rc); rc = -ENETUNREACH; goto out_noupdate; } @@ -749,11 +730,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) rpcrdma_post_recvs(r_xprt, true); rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma); - if (rc) { - dprintk("RPC: %s: rdma_connect() failed with %i\n", - __func__, rc); + if (rc) goto out; - } wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected != 0); if (ep->rep_connected <= 0) { @@ -1088,8 +1066,6 @@ struct rpcrdma_req * out_free: kfree(rep); out: - dprintk("RPC: %s: reply buffer %d alloc failed\n", - __func__, rc); return rc; } From patchwork Wed Dec 19 15:59:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737627 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 423B313B5 for ; Wed, 19 Dec 2018 15:59:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F9E52B430 for ; Wed, 19 Dec 2018 15:59:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2431E2B440; Wed, 19 Dec 2018 15:59:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CABED2B430 for ; Wed, 19 Dec 2018 15:59:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728658AbeLSP7r (ORCPT ); Wed, 19 Dec 2018 10:59:47 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:55936 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727847AbeLSP7q (ORCPT ); Wed, 19 Dec 2018 10:59:46 -0500 Received: by mail-it1-f196.google.com with SMTP id m62so10361852ith.5; Wed, 19 Dec 2018 07:59:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=StkroiHWqjwZDypuPifDbqaXAXLZOjOVYFFxVK/ZTNY=; b=S9S0MSZ8NjbX6GoX/RYgaLD93SpOytgCZujOrkwRtf/2M9nzhlx+6dEcw/4yyX5hT5 yj0ZwSDwmcAKvHwSNMqu3wqQW8vs2cnAqXfYQqCSmpb1nr0yxHKd7jABH5GbjS/9ry0s 2YKy5UV0z2c/voN73FL05dFYCmsSe6XW/ivaUEhh1YF3inMNonZ/H+zDI0+TltHfmS2u ElIcb6PejIFNid0wEXviB5sZePhFMAU98L3M6+SYCUwektKk7nw4frMYQe+7ApNLTjqg tbgsRBvyCMQ7UMPb/iwuGFHA96NjdFwx6umPLpFzuR5W+3k98iiJZebrMFpRi0z0HRJw t8gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=StkroiHWqjwZDypuPifDbqaXAXLZOjOVYFFxVK/ZTNY=; b=UYQxh8mtRqZLlL1+3b6eYThKgIvMmhqcNPbUWqST+tUWJZYih5BzXKSHmDKcijNzLO Gyvdd2X7DTQzy49QiG5zi+NVcSiUOiOsNzOSyy8SuhyAQGrwfsmOpBGsrknTlJswGP8H XJh6bPvcSryf6Awp825n4Scs87SSdOLkbRY6ga8Vqa9gMYEBakt69w7O2AY64kYhPf6M IcGP03WIDXZR3AgSxMQIt1qE8Whw4zVysXQT5LeWcSaVtNacRaHi3tATT66CoEovyvQX mMDoGZdBu1UUgbx1g+Q5Je8y5eO08BqeLOh4oOQJGEu2aMz7+eSacgAx4yqiK3nLq5M+ qq7w== X-Gm-Message-State: AA+aEWYquLcxTX1UgyB+zJb6FLrgbn9hjoJB3N6J4xg/3VAkWvzY8uVx 8iilqo2xzWumsGMHUiu3HLM= X-Google-Smtp-Source: AFSGD/UzrqLl6L0KRY0FC+Q2bJek8876XTHcFb/7Q8M4MJDlaHWgxCRn6LQknrevZgyFATCz2eMaxA== X-Received: by 2002:a24:c8d7:: with SMTP id w206mr7225901itf.56.1545235186132; Wed, 19 Dec 2018 07:59:46 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id y3sm8440919iol.55.2018.12.19.07.59.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:45 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFximE025810; Wed, 19 Dec 2018 15:59:44 GMT Subject: [PATCH v5 18/30] xprtrdma: Remove unused fields from rpcrdma_ia From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:44 -0500 Message-ID: <20181219155944.11602.77732.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up. The last use of these fields was in commit 173b8f49b3af ("xprtrdma: Demote "connect" log messages") . Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/xprt_rdma.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index a1cdc85..90422a6 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -80,8 +80,6 @@ struct rpcrdma_ia { bool ri_implicit_roundup; enum ib_mr_type ri_mrtype; unsigned long ri_flags; - struct ib_qp_attr ri_qp_attr; - struct ib_qp_init_attr ri_qp_init_attr; }; enum { From patchwork Wed Dec 19 15:59:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69FB414DE for ; Wed, 19 Dec 2018 15:59:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A1E32B430 for ; Wed, 19 Dec 2018 15:59:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4E9832B437; Wed, 19 Dec 2018 15:59:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D316D2B434 for ; Wed, 19 Dec 2018 15:59:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728585AbeLSP7x (ORCPT ); Wed, 19 Dec 2018 10:59:53 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:37627 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727847AbeLSP7x (ORCPT ); Wed, 19 Dec 2018 10:59:53 -0500 Received: by mail-it1-f196.google.com with SMTP id b5so10054513iti.2; Wed, 19 Dec 2018 07:59:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=/fajm+Q6Ev3B90pszxLhoKhMSyNLhkbB0DmCyy0qBQc=; b=ADdsmRK+xPYFxtw30URyc9foyYXgcnBVG1KVqXIv91pmn3o/CLmO7/YkRyIM+dmW1S XNwW0XONf/VLjAhGUz7JvO5AvChVgWKMkEPOcPTQc4dngBlTSw4ffsxe+IK+a+XE0RXb SHDOiEA3KbkiULDcnEqrhsDef7gArX4GtTDSg0CLm/NhlqoLXXXPugci7nIz9wKa9zbU HU+3vSniD1trnOrEOWMcevdYhG1I8NHs7JiP9F3A5oLi3jDJiBswq/YZ8Rrh6ubRqJb0 Cy3nv/nyA5nzXJK00mGVFQfPmNo7cfbad1oue79WEjJ08KL36gzwoSRtDurrYX+o/cwP TwoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=/fajm+Q6Ev3B90pszxLhoKhMSyNLhkbB0DmCyy0qBQc=; b=nFOxnLL7Tx/adWNtodd4f//SPZmlj8Ec44yVp/Akb1GDYDzbtXnxE9CEZBbtS83wVR WiiOYcY7Z9wvjf1XRxIjc1PHQriaNVisy0Xa72NHT677kL08pxXtZKS3fH3bhVXBwfTF nOmCA+jtNQR07bYdh37JthK5xEFGNz2fa+DqQa26KhmUxGeSS+LEGxGD2ljsS1TV84e8 jpjA7kWNidWTcBhjnkeoG7MgOp6cMAp7KcrJ1Q5xyRwdfaNwinmoZnf4cEAf5WVe3adi pK1W+wMGNnYe/ry2f/cjX/1hKfnxmieAC0bTFpIqjpDfL/MAAVjj6+zjApaH0Q+ha5HM 9cyg== X-Gm-Message-State: AA+aEWa+hpVCYlnvgHisdw78s5s4tPvGfGvBC0mscF5MLalaquX5uKhy fy4MA/NqM8wDR0MUtabhIE0= X-Google-Smtp-Source: AFSGD/XfRz3LWPuM/xGKCyORqMAr4jo5xQmGqeU7aytTk8lnEJI854xjoTpusqtwNRcVOfXtHr1shQ== X-Received: by 2002:a24:be0e:: with SMTP id i14mr6796186itf.153.1545235191575; Wed, 19 Dec 2018 07:59:51 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id q14sm146149iog.11.2018.12.19.07.59.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:50 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxn1I025813; Wed, 19 Dec 2018 15:59:49 GMT Subject: [PATCH v5 19/30] xprtrdma: Clean up of xprtrdma chunk trace points From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:49 -0500 Message-ID: <20181219155949.11602.19550.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The chunk-related trace points capture nearly the same information as the MR-related trace points. Also, rename them so globbing can be used to enable or disable these trace points more easily. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 42 +++++++++++++++++++++++++--------------- net/sunrpc/xprtrdma/rpc_rdma.c | 6 +++--- 2 files changed, 29 insertions(+), 19 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 2efe2d7..e9fbf7d 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -97,7 +97,6 @@ TP_STRUCT__entry( __field(unsigned int, task_id) __field(unsigned int, client_id) - __field(const void *, mr) __field(unsigned int, pos) __field(int, nents) __field(u32, handle) @@ -109,7 +108,6 @@ TP_fast_assign( __entry->task_id = task->tk_pid; __entry->client_id = task->tk_client->cl_clid; - __entry->mr = mr; __entry->pos = pos; __entry->nents = mr->mr_nents; __entry->handle = mr->mr_handle; @@ -118,8 +116,8 @@ __entry->nsegs = nsegs; ), - TP_printk("task:%u@%u mr=%p pos=%u %u@0x%016llx:0x%08x (%s)", - __entry->task_id, __entry->client_id, __entry->mr, + TP_printk("task:%u@%u pos=%u %u@0x%016llx:0x%08x (%s)", + __entry->task_id, __entry->client_id, __entry->pos, __entry->length, (unsigned long long)__entry->offset, __entry->handle, __entry->nents < __entry->nsegs ? "more" : "last" @@ -127,7 +125,7 @@ ); #define DEFINE_RDCH_EVENT(name) \ - DEFINE_EVENT(xprtrdma_rdch_event, name, \ + DEFINE_EVENT(xprtrdma_rdch_event, xprtrdma_chunk_##name,\ TP_PROTO( \ const struct rpc_task *task, \ unsigned int pos, \ @@ -148,7 +146,6 @@ TP_STRUCT__entry( __field(unsigned int, task_id) __field(unsigned int, client_id) - __field(const void *, mr) __field(int, nents) __field(u32, handle) __field(u32, length) @@ -159,7 +156,6 @@ TP_fast_assign( __entry->task_id = task->tk_pid; __entry->client_id = task->tk_client->cl_clid; - __entry->mr = mr; __entry->nents = mr->mr_nents; __entry->handle = mr->mr_handle; __entry->length = mr->mr_length; @@ -167,8 +163,8 @@ __entry->nsegs = nsegs; ), - TP_printk("task:%u@%u mr=%p %u@0x%016llx:0x%08x (%s)", - __entry->task_id, __entry->client_id, __entry->mr, + TP_printk("task:%u@%u %u@0x%016llx:0x%08x (%s)", + __entry->task_id, __entry->client_id, __entry->length, (unsigned long long)__entry->offset, __entry->handle, __entry->nents < __entry->nsegs ? "more" : "last" @@ -176,7 +172,7 @@ ); #define DEFINE_WRCH_EVENT(name) \ - DEFINE_EVENT(xprtrdma_wrch_event, name, \ + DEFINE_EVENT(xprtrdma_wrch_event, xprtrdma_chunk_##name,\ TP_PROTO( \ const struct rpc_task *task, \ struct rpcrdma_mr *mr, \ @@ -234,6 +230,18 @@ ), \ TP_ARGS(wc, frwr)) +TRACE_DEFINE_ENUM(DMA_BIDIRECTIONAL); +TRACE_DEFINE_ENUM(DMA_TO_DEVICE); +TRACE_DEFINE_ENUM(DMA_FROM_DEVICE); +TRACE_DEFINE_ENUM(DMA_NONE); + +#define xprtrdma_show_direction(x) \ + __print_symbolic(x, \ + { DMA_BIDIRECTIONAL, "BIDIR" }, \ + { DMA_TO_DEVICE, "TO_DEVICE" }, \ + { DMA_FROM_DEVICE, "FROM_DEVICE" }, \ + { DMA_NONE, "NONE" }) + DECLARE_EVENT_CLASS(xprtrdma_mr, TP_PROTO( const struct rpcrdma_mr *mr @@ -246,6 +254,7 @@ __field(u32, handle) __field(u32, length) __field(u64, offset) + __field(u32, dir) ), TP_fast_assign( @@ -253,12 +262,13 @@ __entry->handle = mr->mr_handle; __entry->length = mr->mr_length; __entry->offset = mr->mr_offset; + __entry->dir = mr->mr_dir; ), - TP_printk("mr=%p %u@0x%016llx:0x%08x", + TP_printk("mr=%p %u@0x%016llx:0x%08x (%s)", __entry->mr, __entry->length, - (unsigned long long)__entry->offset, - __entry->handle + (unsigned long long)__entry->offset, __entry->handle, + xprtrdma_show_direction(__entry->dir) ) ); @@ -437,9 +447,9 @@ DEFINE_RXPRT_EVENT(xprtrdma_nomrs); -DEFINE_RDCH_EVENT(xprtrdma_read_chunk); -DEFINE_WRCH_EVENT(xprtrdma_write_chunk); -DEFINE_WRCH_EVENT(xprtrdma_reply_chunk); +DEFINE_RDCH_EVENT(read); +DEFINE_WRCH_EVENT(write); +DEFINE_WRCH_EVENT(reply); TRACE_DEFINE_ENUM(rpcrdma_noch); TRACE_DEFINE_ENUM(rpcrdma_readch); diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 5a58769..54fbd70 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -365,7 +365,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (encode_read_segment(xdr, mr, pos) < 0) return -EMSGSIZE; - trace_xprtrdma_read_chunk(rqst->rq_task, pos, mr, nsegs); + trace_xprtrdma_chunk_read(rqst->rq_task, pos, mr, nsegs); r_xprt->rx_stats.read_chunk_count++; nsegs -= mr->mr_nents; } while (nsegs); @@ -422,7 +422,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (encode_rdma_segment(xdr, mr) < 0) return -EMSGSIZE; - trace_xprtrdma_write_chunk(rqst->rq_task, mr, nsegs); + trace_xprtrdma_chunk_write(rqst->rq_task, mr, nsegs); r_xprt->rx_stats.write_chunk_count++; r_xprt->rx_stats.total_rdma_request += mr->mr_length; nchunks++; @@ -479,7 +479,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (encode_rdma_segment(xdr, mr) < 0) return -EMSGSIZE; - trace_xprtrdma_reply_chunk(rqst->rq_task, mr, nsegs); + trace_xprtrdma_chunk_reply(rqst->rq_task, mr, nsegs); r_xprt->rx_stats.reply_chunk_count++; r_xprt->rx_stats.total_rdma_request += mr->mr_length; nchunks++; From patchwork Wed Dec 19 15:59:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737635 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A8AC14DE for ; Wed, 19 Dec 2018 15:59:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 394512B437 for ; Wed, 19 Dec 2018 15:59:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D6FA2B456; Wed, 19 Dec 2018 15:59:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBA7E2B434 for ; Wed, 19 Dec 2018 15:59:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728067AbeLSP76 (ORCPT ); Wed, 19 Dec 2018 10:59:58 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:51765 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727847AbeLSP76 (ORCPT ); Wed, 19 Dec 2018 10:59:58 -0500 Received: by mail-it1-f194.google.com with SMTP id w18so10060124ite.1; Wed, 19 Dec 2018 07:59:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=dTTFxQBHqNac1U654cQS+n8MCI79WsiCTWna3Rmj/a4=; b=lBM/BI0GMQ8MpGxxKyXkPi3Cvs6acPc3ZV5zxKTG1Pj0jAFgKIyKCD5/97USTUwrLI V56a+y9SYj/GsL+JjzqwZXmD12LzFvKFQLNJd3p3VZRFqEzWt5y7yDKJGPtn+A1wT93b e8o3oYEM03r5Zu5P6fgvjppDXbbOJIfDqucBcQ6F/oxosMJBlRkB1P07nxdZR7Hq3CGe Rx2OBeZXybQcf7mlXni+vB59XgLi0XbBiB7mriX8VlzAEESHm8nWAYkjmuzOuQ60oGzT 6YQKuXEFxVbqEBuYDONxOgGtypSmZsAZNx8fIlk/VyZhgNSdCNcMIw0l5NEX+xicLiDV UWiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=dTTFxQBHqNac1U654cQS+n8MCI79WsiCTWna3Rmj/a4=; b=hYvVICu1zZU1eacDb4y/4uHXEmdNisBvedQExFXK0CnA1CVui1F+/LAuEmoabKzr5z dc/sUjq/pgDPKKx8pDobFJB3rq5bl+rM4HKIlOsTh7cHWtnqLohdSE9vQCD85sNS4Vy6 1g1FpjIlR0JztsvO8xCYHc6bn8SQG8X5aYW622D+sFj8SoF71x6rjBTPdGgCC7rB553a H3JpbTscOj4EBdxCe87PKLLHHkvMF9VZo2ksanxRma7eKwAGx2OqUxgYvAgpYlI3hGTJ 29ydQt+Zj1P9CRsvc/QQTEudb8ivlLWOxWMbKAI3Tjs1q6jyeQmOzAM4UZCCl6CeYBUO k17A== X-Gm-Message-State: AA+aEWaLm0PP5dkyYLwoKPMZxD7etQYi/ggWxfNc2u4+pvYlmhn95gRj eE8cYguj8EoQpM9BwY6XO1o= X-Google-Smtp-Source: AFSGD/WmX03Eb361gaDucfcqfCCfXlhPJyD13b/g3aSTe8rLNU6+Uo4CKRr7dqXQLZpcSgJxEjlJwg== X-Received: by 2002:a24:6115:: with SMTP id s21mr7428968itc.62.1545235196584; Wed, 19 Dec 2018 07:59:56 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id r63sm3791061ita.2.2018.12.19.07.59.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 07:59:56 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJFxt01025816; Wed, 19 Dec 2018 15:59:55 GMT Subject: [PATCH v5 20/30] xprtrdma: Relocate the xprtrdma_mr_map trace points From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 10:59:55 -0500 Message-ID: <20181219155955.11602.19265.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The mr_map trace points were capturing information about the previous use of the MR rather than about the segment that was just mapped. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 97f88bb..1f508f4 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -438,7 +438,6 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, mr->mr_nents = ib_dma_map_sg(ia->ri_device, mr->mr_sg, i, mr->mr_dir); if (!mr->mr_nents) goto out_dmamap_err; - trace_xprtrdma_mr_map(mr); ibmr = frwr->fr_mr; n = ib_map_mr_sg(ibmr, mr->mr_sg, mr->mr_nents, NULL, PAGE_SIZE); @@ -460,6 +459,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, mr->mr_handle = ibmr->rkey; mr->mr_length = ibmr->length; mr->mr_offset = ibmr->iova; + trace_xprtrdma_mr_map(mr); *out = mr; return seg; From patchwork Wed Dec 19 16:00:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737639 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C51726C5 for ; Wed, 19 Dec 2018 16:00:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B19A72B48D for ; Wed, 19 Dec 2018 16:00:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A394C2B494; Wed, 19 Dec 2018 16:00:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31C5D2B430 for ; Wed, 19 Dec 2018 16:00:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729530AbeLSQAD (ORCPT ); Wed, 19 Dec 2018 11:00:03 -0500 Received: from mail-io1-f66.google.com ([209.85.166.66]:43986 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727847AbeLSQAD (ORCPT ); Wed, 19 Dec 2018 11:00:03 -0500 Received: by mail-io1-f66.google.com with SMTP id g17so4151415ioc.10; Wed, 19 Dec 2018 08:00:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=q1UNWX1xN3M3bnDETWUWnK9CwpUhoEmmNy0M9CGMCC8=; b=gHzjvJTBDCP0Pom5WNtL3Pxuu+cPOLUaftFkWtLlHd93N8V6dr1kzakKwxHUNFAYES atzP9o3HVDyCveymYDBQEUdkQH7RGQBVswsaxbG5gqFDQfvJCs2CH1cQFZIYZVybNq2Y FnaKXXwSgBqDD6yCtVoi9NALRx5lC0ypNSLChMPzT5/V0L09tWTyn/FJkvqZjvbsN4Xi K+UUKfWnFnCT18zpe3Iu4sMb7EUKwc1H9XDYWlCoFkc7LM8d7us48WjoH/IrMlrRmWsN d5NjUJTBtGcKakLY7oUohNJTz1QP3mHbYMKgrtZj8G/OPPYnVKY3J5KYy6uR64TFVPku 4exA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=q1UNWX1xN3M3bnDETWUWnK9CwpUhoEmmNy0M9CGMCC8=; b=f6B4efsj0Q0OZB4VqQ5VqnznpTyrXhlrnee1UHOcZM7hslFqQKCv1UriwaTlJLOzAC tkIUUwNjdz8fPocC1CH1yCFZTxh9gKiGgYkxUNs3JBCgsKWWer0GJIdZvRngIG+RD5zY Smi5hFq7O/39px40YqX90V5ehgTMczJgRkqWb31QbSx+C72vGU2E5UY0L7xKsrIaHhEm 9g6XzqbjRze0LV54x0tH0v80t9HQTtZQGoWglXkwoMZ1bm7/76cRJsO+C/i3aM2zvkMi Fuetv1Vl/wE2V535wAabdWtfCzBJTBkviCNtuO17ApRqtdCO0PNb9zAgyUNqCpGceXd5 6r5g== X-Gm-Message-State: AA+aEWYrMuSyjfsrW7Ffa3uwsZ8Z9DELie6VssgN4kaxLerSkbjZcJSV 4YarB3gbv1z+gOA4FdtRpN0= X-Google-Smtp-Source: AFSGD/WlWB5lInCNaoeoz9WksGSV7Mf44zbrWuIAzpSIMOnokdWffzdNGoaU3QT+CZtAOv8WE7MJSg== X-Received: by 2002:a6b:3b47:: with SMTP id i68mr18374855ioa.133.1545235202110; Wed, 19 Dec 2018 08:00:02 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id i201sm9334646ioa.1.2018.12.19.08.00.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:01 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG002A025819; Wed, 19 Dec 2018 16:00:00 GMT Subject: [PATCH v5 21/30] xprtrdma: Add trace points for calls to transport switch methods From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:00 -0500 Message-ID: <20181219160000.11602.77153.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Name them "trace_xprtrdma_op_*" so they can be easily enabled as a group. No trace point is added where the generic layer already has observability. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 10 ++++++---- net/sunrpc/xprtrdma/transport.c | 18 +++++++++++------- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index e9fbf7d..3d068bb 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -381,11 +381,13 @@ DEFINE_RXPRT_EVENT(xprtrdma_conn_start); DEFINE_RXPRT_EVENT(xprtrdma_conn_tout); DEFINE_RXPRT_EVENT(xprtrdma_create); -DEFINE_RXPRT_EVENT(xprtrdma_destroy); +DEFINE_RXPRT_EVENT(xprtrdma_op_destroy); DEFINE_RXPRT_EVENT(xprtrdma_remove); DEFINE_RXPRT_EVENT(xprtrdma_reinsert); DEFINE_RXPRT_EVENT(xprtrdma_reconnect); -DEFINE_RXPRT_EVENT(xprtrdma_inject_dsc); +DEFINE_RXPRT_EVENT(xprtrdma_op_inject_dsc); +DEFINE_RXPRT_EVENT(xprtrdma_op_close); +DEFINE_RXPRT_EVENT(xprtrdma_op_connect); TRACE_EVENT(xprtrdma_qp_event, TP_PROTO( @@ -834,7 +836,7 @@ ** Allocation/release of rpcrdma_reqs and rpcrdma_reps **/ -TRACE_EVENT(xprtrdma_allocate, +TRACE_EVENT(xprtrdma_op_allocate, TP_PROTO( const struct rpc_task *task, const struct rpcrdma_req *req @@ -864,7 +866,7 @@ ) ); -TRACE_EVENT(xprtrdma_rpc_done, +TRACE_EVENT(xprtrdma_op_free, TP_PROTO( const struct rpc_task *task, const struct rpcrdma_req *req diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 6a57033..d94da3c 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -268,7 +268,7 @@ { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - trace_xprtrdma_inject_dsc(r_xprt); + trace_xprtrdma_op_inject_dsc(r_xprt); rdma_disconnect(r_xprt->rx_ia.ri_id); } @@ -284,7 +284,7 @@ { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - trace_xprtrdma_destroy(r_xprt); + trace_xprtrdma_op_destroy(r_xprt); cancel_delayed_work_sync(&r_xprt->rx_connect_worker); @@ -418,7 +418,7 @@ out2: rpcrdma_ia_close(&new_xprt->rx_ia); out1: - trace_xprtrdma_destroy(new_xprt); + trace_xprtrdma_op_destroy(new_xprt); xprt_rdma_free_addresses(xprt); xprt_free(xprt); return ERR_PTR(rc); @@ -428,7 +428,8 @@ * xprt_rdma_close - close a transport connection * @xprt: transport context * - * Called during transport shutdown, reconnect, or device removal. + * Called during autoclose or device removal. + * * Caller holds @xprt's send lock to prevent activity on this * transport while the connection is torn down. */ @@ -440,6 +441,8 @@ void xprt_rdma_close(struct rpc_xprt *xprt) might_sleep(); + trace_xprtrdma_op_close(r_xprt); + /* Prevent marshaling and sending of new requests */ xprt_clear_connected(xprt); @@ -525,6 +528,7 @@ void xprt_rdma_close(struct rpc_xprt *xprt) { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); + trace_xprtrdma_op_connect(r_xprt); if (r_xprt->rx_ep.rep_connected != 0) { /* Reconnect */ schedule_delayed_work(&r_xprt->rx_connect_worker, @@ -659,11 +663,11 @@ void xprt_rdma_close(struct rpc_xprt *xprt) rqst->rq_buffer = req->rl_sendbuf->rg_base; rqst->rq_rbuffer = req->rl_recvbuf->rg_base; - trace_xprtrdma_allocate(task, req); + trace_xprtrdma_op_allocate(task, req); return 0; out_fail: - trace_xprtrdma_allocate(task, NULL); + trace_xprtrdma_op_allocate(task, NULL); return -ENOMEM; } @@ -682,7 +686,7 @@ void xprt_rdma_close(struct rpc_xprt *xprt) if (test_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags)) rpcrdma_release_rqst(r_xprt, req); - trace_xprtrdma_rpc_done(task, req); + trace_xprtrdma_op_free(task, req); } /** From patchwork Wed Dec 19 16:00:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737643 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4702813B5 for ; Wed, 19 Dec 2018 16:00:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 313B52B430 for ; Wed, 19 Dec 2018 16:00:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 243972B5A1; Wed, 19 Dec 2018 16:00:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8ADD82B529 for ; Wed, 19 Dec 2018 16:00:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728726AbeLSQAJ (ORCPT ); Wed, 19 Dec 2018 11:00:09 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:50835 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726652AbeLSQAI (ORCPT ); Wed, 19 Dec 2018 11:00:08 -0500 Received: by mail-it1-f194.google.com with SMTP id z7so10405012iti.0; Wed, 19 Dec 2018 08:00:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=zTVAECQNkrqbJM2MfZaEh7QPrxOcciLsx5hGleKHDGo=; b=NAEJqS4CG0MY7cc8EHG68hvvXl348nKRFQKkRyXtpp4VzpSWpp2wjphFmZ/jxuLcqA m5NdP5/+trBMbrlNkjWWiZ2cCUVw510TpZDenB/9VnnEblmVKwLTOumjRkv70u1m2jOx /XAsxICGp6FPAssbMmQqzVwNL9BxSDxcAV1H68GHQKOBAwEDRSPcsGtm2/rhuuKEP5C/ 7U8NviFKQiTK1DQ/rVtsHOKfzf3WWkyLiD/0YIzKNneNdOEO5H/BooWRRoWyTYaUasgL v351Eh5ot7ZFVGjs7QYs8g+9XhjiyAZXuOIkhX7KOjbRdxjdS184BBpUoe3rTZyn+3Ed xDhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=zTVAECQNkrqbJM2MfZaEh7QPrxOcciLsx5hGleKHDGo=; b=mEmFKlIM8lgiNI3eaXav4PrRRX1xZeOczIq4rW5+6S6yWdelC8D1eT8h2uwF+VH2nX MlJvS6caB3t7aJ0hA+TEmyqe4oMZ9Zr1Z9Jem37Ku5IZUQkG0lXHXZfwX0H/tK0HwMWk rnvy8VDxQhZV5k0QjNRCVhXJTaF/Cl5VuYd+Zo4vIPydUV3L6D/CJX++yCsO6aX7ft9h fQUd++93CUDq+XDmvF5bhAzwDNnIX3n6/ubIu1ONwJRqN3j0+ecOG93nkEzi6qX3l2/p 8KKXYpiqjKGe4LxBaU1NwKougbnd6RkVT8+UC+SRQYdkdK/63OR7WpB8yLotb7oTOuKx UtcA== X-Gm-Message-State: AA+aEWYtlq80h+9ub5H6M0QJAqk+FLdt2VTT80bczrwO1H4NUWvtgpJ8 aNVKKdvqvLHW8cjwypsAp+mNvwO0 X-Google-Smtp-Source: AFSGD/WaDhFep2sOtEbY3mem54ym9p/oSgh6ah2mBZxaXRl5RBlHccIOnso7n+eF1lyiaJDIH+d1Qg== X-Received: by 2002:a24:7349:: with SMTP id y70mr6964286itb.48.1545235207428; Wed, 19 Dec 2018 08:00:07 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id v74sm3962634ita.27.2018.12.19.08.00.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:06 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG06bo025832; Wed, 19 Dec 2018 16:00:06 GMT Subject: [PATCH v5 22/30] xprtrdma: Trace mapping, alloc, and dereg failures From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:06 -0500 Message-ID: <20181219160006.11602.94984.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These are rare, but can be helpful at tracking down DMAR and other problems. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 136 ++++++++++++++++++++++++++++++++++++++++ net/sunrpc/xprtrdma/frwr_ops.c | 12 +--- net/sunrpc/xprtrdma/rpc_rdma.c | 2 - net/sunrpc/xprtrdma/verbs.c | 4 + 4 files changed, 144 insertions(+), 10 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 3d068bb..ce528d5 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -10,6 +10,7 @@ #if !defined(_TRACE_RPCRDMA_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_RPCRDMA_H +#include #include #include @@ -663,12 +664,147 @@ DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li); DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li_wake); +TRACE_EVENT(xprtrdma_frwr_alloc, + TP_PROTO( + const struct rpcrdma_mr *mr, + int rc + ), + + TP_ARGS(mr, rc), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(int, rc) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->rc = rc; + ), + + TP_printk("mr=%p: rc=%d", + __entry->mr, __entry->rc + ) +); + +TRACE_EVENT(xprtrdma_frwr_dereg, + TP_PROTO( + const struct rpcrdma_mr *mr, + int rc + ), + + TP_ARGS(mr, rc), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(u32, handle) + __field(u32, length) + __field(u64, offset) + __field(u32, dir) + __field(int, rc) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->handle = mr->mr_handle; + __entry->length = mr->mr_length; + __entry->offset = mr->mr_offset; + __entry->dir = mr->mr_dir; + __entry->rc = rc; + ), + + TP_printk("mr=%p %u@0x%016llx:0x%08x (%s): rc=%d", + __entry->mr, __entry->length, + (unsigned long long)__entry->offset, __entry->handle, + xprtrdma_show_direction(__entry->dir), + __entry->rc + ) +); + +TRACE_EVENT(xprtrdma_frwr_sgerr, + TP_PROTO( + const struct rpcrdma_mr *mr, + int sg_nents + ), + + TP_ARGS(mr, sg_nents), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(u64, addr) + __field(u32, dir) + __field(int, nents) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->addr = mr->mr_sg->dma_address; + __entry->dir = mr->mr_dir; + __entry->nents = sg_nents; + ), + + TP_printk("mr=%p dma addr=0x%llx (%s) sg_nents=%d", + __entry->mr, __entry->addr, + xprtrdma_show_direction(__entry->dir), + __entry->nents + ) +); + +TRACE_EVENT(xprtrdma_frwr_maperr, + TP_PROTO( + const struct rpcrdma_mr *mr, + int num_mapped + ), + + TP_ARGS(mr, num_mapped), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(u64, addr) + __field(u32, dir) + __field(int, num_mapped) + __field(int, nents) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->addr = mr->mr_sg->dma_address; + __entry->dir = mr->mr_dir; + __entry->num_mapped = num_mapped; + __entry->nents = mr->mr_nents; + ), + + TP_printk("mr=%p dma addr=0x%llx (%s) nents=%d of %d", + __entry->mr, __entry->addr, + xprtrdma_show_direction(__entry->dir), + __entry->num_mapped, __entry->nents + ) +); + DEFINE_MR_EVENT(localinv); DEFINE_MR_EVENT(map); DEFINE_MR_EVENT(unmap); DEFINE_MR_EVENT(remoteinv); DEFINE_MR_EVENT(recycle); +TRACE_EVENT(xprtrdma_dma_maperr, + TP_PROTO( + u64 addr + ), + + TP_ARGS(addr), + + TP_STRUCT__entry( + __field(u64, addr) + ), + + TP_fast_assign( + __entry->addr = addr; + ), + + TP_printk("dma addr=0x%llx\n", __entry->addr) +); + /** ** Reply events **/ diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 1f508f4..8a0f1a6 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -113,8 +113,7 @@ void frwr_release_mr(struct rpcrdma_mr *mr) rc = ib_dereg_mr(mr->frwr.fr_mr); if (rc) - pr_err("rpcrdma: final ib_dereg_mr for %p returned %i\n", - mr, rc); + trace_xprtrdma_frwr_dereg(mr, rc); kfree(mr->mr_sg); kfree(mr); } @@ -177,8 +176,7 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) out_mr_err: rc = PTR_ERR(frwr->fr_mr); - dprintk("RPC: %s: ib_alloc_mr status %i\n", - __func__, rc); + trace_xprtrdma_frwr_alloc(mr, rc); return rc; out_list_err: @@ -465,15 +463,13 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, return seg; out_dmamap_err: - pr_err("rpcrdma: failed to DMA map sg %p sg_nents %d\n", - mr->mr_sg, i); frwr->fr_state = FRWR_IS_INVALID; + trace_xprtrdma_frwr_sgerr(mr, i); rpcrdma_mr_put(mr); return ERR_PTR(-EIO); out_mapmr_err: - pr_err("rpcrdma: failed to map mr %p (%d/%d)\n", - frwr->fr_mr, n, mr->mr_nents); + trace_xprtrdma_frwr_maperr(mr, n); rpcrdma_mr_recycle(mr); return ERR_PTR(-EIO); } diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 54fbd70..062aee9 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -665,7 +665,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, out_mapping_err: rpcrdma_unmap_sendctx(sc); - pr_err("rpcrdma: Send mapping error\n"); + trace_xprtrdma_dma_maperr(sge[sge_no].addr); return false; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 85c51b8..719b69c 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1392,8 +1392,10 @@ struct rpcrdma_regbuf * (void *)rb->rg_base, rdmab_length(rb), rb->rg_direction); - if (ib_dma_mapping_error(device, rdmab_addr(rb))) + if (ib_dma_mapping_error(device, rdmab_addr(rb))) { + trace_xprtrdma_dma_maperr(rdmab_addr(rb)); return false; + } rb->rg_device = device; rb->rg_iov.lkey = ia->ri_pd->local_dma_lkey; From patchwork Wed Dec 19 16:00:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 237B013B5 for ; Wed, 19 Dec 2018 16:00:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E2D42B5A8 for ; Wed, 19 Dec 2018 16:00:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 023472B5A5; Wed, 19 Dec 2018 16:00:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,UPPERCASE_75_100 autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3DE92B5A8 for ; Wed, 19 Dec 2018 16:00:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728248AbeLSQAP (ORCPT ); Wed, 19 Dec 2018 11:00:15 -0500 Received: from mail-io1-f65.google.com ([209.85.166.65]:44003 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726652AbeLSQAP (ORCPT ); Wed, 19 Dec 2018 11:00:15 -0500 Received: by mail-io1-f65.google.com with SMTP id g17so4151968ioc.10; Wed, 19 Dec 2018 08:00:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=2mLx28XWl8xoUfC5kvXH/cRNSXyhzKZxmuWP8HjEhqU=; b=tGMOI4z1AU5QzWYfbQtWjIKoBAHhwLK1YiqoBJYpRgerUyIRAUxS7z0U7QUBLmdEg+ yqVlmK6NJ6l0vDopG26p0ovLrbA/JFQygyeOVrc3bThHgK0YXnxesTQPIfbTOKjln6cV iDCQPA6dRtpYqvswyF/Bv57asvaarXGuhmUASAO2hw9E7LZ47D9XCx7SKotTJtKSthwK 4aXDBwmON1uTrnMf1Byoytvbupjyry8wnc+/6vKxkbt9Nw/bVKjUtakDeRogGxCSp+bq 6QKtCWFX4Zu8AmEsGr+fROrlMpaFU0+Z9OrDN+VVgN+Wmk9fppT29ucS93qvPeM9kCoM /iqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=2mLx28XWl8xoUfC5kvXH/cRNSXyhzKZxmuWP8HjEhqU=; b=as044V8hBxnYDec9RE4weEtchdc7BLELzLCi4HA8Q/9mUpZgoDShLhwpqO6QaYNSV9 RB535OL96oSv5MRtaIe0WX0P1jKoToqnT7PpjK3dC36hdFx9BLkDWTTz88iqxc83V/cY 437S+lmERjgZwOexVDj96tt/OovWw7Q69bvbPWeKLwg+j2wdwCzqeoZQ/H2FA0pTMYjj BdDFoXjLWSW/1mP4BaHNNYaLja3gNWuUDA9Q87HQZ4Dvb6j4JpUeV87KkQBZNodyEKz4 IUQl3geH5qnxyo6nf+zVBADOL+ounPezv8sihDIfs4gMKCWEu3dK00GFxEho2CUAAS0D YncQ== X-Gm-Message-State: AA+aEWas6faDtCESBaRw/MvR4SUlqSWXyv0diAFBA9DRix9E4YA5TKIr yruRsICcYXhgunHVZ6XXzXeYNkug X-Google-Smtp-Source: AFSGD/XXECgsv3R9Ov2p6CUCdT3HN6WVn0DFA9VbtalU+earc6Ga3SD7tTLv6VQEfNhW40xZiZ6kMw== X-Received: by 2002:a6b:1411:: with SMTP id 17mr18692820iou.252.1545235212859; Wed, 19 Dec 2018 08:00:12 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id d133sm3375888itd.1.2018.12.19.08.00.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:12 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0BJF025835; Wed, 19 Dec 2018 16:00:11 GMT Subject: [PATCH v5 23/30] NFS: Fix NFSv4 symbolic trace point output From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:11 -0500 Message-ID: <20181219160011.11602.85528.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These symbolic values were not being displayed in string form. TRACE_DEFINE_ENUM was missing in many cases. It also turns out that __print_symbolic wants an unsigned long in the first field... Signed-off-by: Chuck Lever --- fs/nfs/nfs4trace.h | 456 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 313 insertions(+), 143 deletions(-) diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index b1483b3..b4557cf 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -10,157 +10,302 @@ #include +TRACE_DEFINE_ENUM(EPERM); +TRACE_DEFINE_ENUM(ENOENT); +TRACE_DEFINE_ENUM(EIO); +TRACE_DEFINE_ENUM(ENXIO); +TRACE_DEFINE_ENUM(EACCES); +TRACE_DEFINE_ENUM(EEXIST); +TRACE_DEFINE_ENUM(EXDEV); +TRACE_DEFINE_ENUM(ENOTDIR); +TRACE_DEFINE_ENUM(EISDIR); +TRACE_DEFINE_ENUM(EFBIG); +TRACE_DEFINE_ENUM(ENOSPC); +TRACE_DEFINE_ENUM(EROFS); +TRACE_DEFINE_ENUM(EMLINK); +TRACE_DEFINE_ENUM(ENAMETOOLONG); +TRACE_DEFINE_ENUM(ENOTEMPTY); +TRACE_DEFINE_ENUM(EDQUOT); +TRACE_DEFINE_ENUM(ESTALE); +TRACE_DEFINE_ENUM(EBADHANDLE); +TRACE_DEFINE_ENUM(EBADCOOKIE); +TRACE_DEFINE_ENUM(ENOTSUPP); +TRACE_DEFINE_ENUM(ETOOSMALL); +TRACE_DEFINE_ENUM(EREMOTEIO); +TRACE_DEFINE_ENUM(EBADTYPE); +TRACE_DEFINE_ENUM(EAGAIN); +TRACE_DEFINE_ENUM(ELOOP); +TRACE_DEFINE_ENUM(EOPNOTSUPP); +TRACE_DEFINE_ENUM(EDEADLK); +TRACE_DEFINE_ENUM(ENOMEM); +TRACE_DEFINE_ENUM(EKEYEXPIRED); +TRACE_DEFINE_ENUM(ETIMEDOUT); +TRACE_DEFINE_ENUM(ERESTARTSYS); +TRACE_DEFINE_ENUM(ECONNREFUSED); +TRACE_DEFINE_ENUM(ECONNRESET); +TRACE_DEFINE_ENUM(ENETUNREACH); +TRACE_DEFINE_ENUM(EHOSTUNREACH); +TRACE_DEFINE_ENUM(EHOSTDOWN); +TRACE_DEFINE_ENUM(EPIPE); +TRACE_DEFINE_ENUM(EPFNOSUPPORT); +TRACE_DEFINE_ENUM(EPROTONOSUPPORT); + +TRACE_DEFINE_ENUM(NFS4_OK); +TRACE_DEFINE_ENUM(NFS4ERR_ACCESS); +TRACE_DEFINE_ENUM(NFS4ERR_ATTRNOTSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_ADMIN_REVOKED); +TRACE_DEFINE_ENUM(NFS4ERR_BACK_CHAN_BUSY); +TRACE_DEFINE_ENUM(NFS4ERR_BADCHAR); +TRACE_DEFINE_ENUM(NFS4ERR_BADHANDLE); +TRACE_DEFINE_ENUM(NFS4ERR_BADIOMODE); +TRACE_DEFINE_ENUM(NFS4ERR_BADLAYOUT); +TRACE_DEFINE_ENUM(NFS4ERR_BADLABEL); +TRACE_DEFINE_ENUM(NFS4ERR_BADNAME); +TRACE_DEFINE_ENUM(NFS4ERR_BADOWNER); +TRACE_DEFINE_ENUM(NFS4ERR_BADSESSION); +TRACE_DEFINE_ENUM(NFS4ERR_BADSLOT); +TRACE_DEFINE_ENUM(NFS4ERR_BADTYPE); +TRACE_DEFINE_ENUM(NFS4ERR_BADXDR); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_COOKIE); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_HIGH_SLOT); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_RANGE); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_SEQID); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_SESSION_DIGEST); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_STATEID); +TRACE_DEFINE_ENUM(NFS4ERR_CB_PATH_DOWN); +TRACE_DEFINE_ENUM(NFS4ERR_CLID_INUSE); +TRACE_DEFINE_ENUM(NFS4ERR_CLIENTID_BUSY); +TRACE_DEFINE_ENUM(NFS4ERR_COMPLETE_ALREADY); +TRACE_DEFINE_ENUM(NFS4ERR_CONN_NOT_BOUND_TO_SESSION); +TRACE_DEFINE_ENUM(NFS4ERR_DEADLOCK); +TRACE_DEFINE_ENUM(NFS4ERR_DEADSESSION); +TRACE_DEFINE_ENUM(NFS4ERR_DELAY); +TRACE_DEFINE_ENUM(NFS4ERR_DELEG_ALREADY_WANTED); +TRACE_DEFINE_ENUM(NFS4ERR_DELEG_REVOKED); +TRACE_DEFINE_ENUM(NFS4ERR_DENIED); +TRACE_DEFINE_ENUM(NFS4ERR_DIRDELEG_UNAVAIL); +TRACE_DEFINE_ENUM(NFS4ERR_DQUOT); +TRACE_DEFINE_ENUM(NFS4ERR_ENCR_ALG_UNSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_EXIST); +TRACE_DEFINE_ENUM(NFS4ERR_EXPIRED); +TRACE_DEFINE_ENUM(NFS4ERR_FBIG); +TRACE_DEFINE_ENUM(NFS4ERR_FHEXPIRED); +TRACE_DEFINE_ENUM(NFS4ERR_FILE_OPEN); +TRACE_DEFINE_ENUM(NFS4ERR_GRACE); +TRACE_DEFINE_ENUM(NFS4ERR_HASH_ALG_UNSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_INVAL); +TRACE_DEFINE_ENUM(NFS4ERR_IO); +TRACE_DEFINE_ENUM(NFS4ERR_ISDIR); +TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTTRYLATER); +TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTUNAVAILABLE); +TRACE_DEFINE_ENUM(NFS4ERR_LEASE_MOVED); +TRACE_DEFINE_ENUM(NFS4ERR_LOCKED); +TRACE_DEFINE_ENUM(NFS4ERR_LOCKS_HELD); +TRACE_DEFINE_ENUM(NFS4ERR_LOCK_RANGE); +TRACE_DEFINE_ENUM(NFS4ERR_MINOR_VERS_MISMATCH); +TRACE_DEFINE_ENUM(NFS4ERR_MLINK); +TRACE_DEFINE_ENUM(NFS4ERR_MOVED); +TRACE_DEFINE_ENUM(NFS4ERR_NAMETOOLONG); +TRACE_DEFINE_ENUM(NFS4ERR_NOENT); +TRACE_DEFINE_ENUM(NFS4ERR_NOFILEHANDLE); +TRACE_DEFINE_ENUM(NFS4ERR_NOMATCHING_LAYOUT); +TRACE_DEFINE_ENUM(NFS4ERR_NOSPC); +TRACE_DEFINE_ENUM(NFS4ERR_NOTDIR); +TRACE_DEFINE_ENUM(NFS4ERR_NOTEMPTY); +TRACE_DEFINE_ENUM(NFS4ERR_NOTSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_NOT_ONLY_OP); +TRACE_DEFINE_ENUM(NFS4ERR_NOT_SAME); +TRACE_DEFINE_ENUM(NFS4ERR_NO_GRACE); +TRACE_DEFINE_ENUM(NFS4ERR_NXIO); +TRACE_DEFINE_ENUM(NFS4ERR_OLD_STATEID); +TRACE_DEFINE_ENUM(NFS4ERR_OPENMODE); +TRACE_DEFINE_ENUM(NFS4ERR_OP_ILLEGAL); +TRACE_DEFINE_ENUM(NFS4ERR_OP_NOT_IN_SESSION); +TRACE_DEFINE_ENUM(NFS4ERR_PERM); +TRACE_DEFINE_ENUM(NFS4ERR_PNFS_IO_HOLE); +TRACE_DEFINE_ENUM(NFS4ERR_PNFS_NO_LAYOUT); +TRACE_DEFINE_ENUM(NFS4ERR_RECALLCONFLICT); +TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_BAD); +TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_CONFLICT); +TRACE_DEFINE_ENUM(NFS4ERR_REJECT_DELEG); +TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG); +TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG_TO_CACHE); +TRACE_DEFINE_ENUM(NFS4ERR_REQ_TOO_BIG); +TRACE_DEFINE_ENUM(NFS4ERR_RESOURCE); +TRACE_DEFINE_ENUM(NFS4ERR_RESTOREFH); +TRACE_DEFINE_ENUM(NFS4ERR_RETRY_UNCACHED_REP); +TRACE_DEFINE_ENUM(NFS4ERR_RETURNCONFLICT); +TRACE_DEFINE_ENUM(NFS4ERR_ROFS); +TRACE_DEFINE_ENUM(NFS4ERR_SAME); +TRACE_DEFINE_ENUM(NFS4ERR_SHARE_DENIED); +TRACE_DEFINE_ENUM(NFS4ERR_SEQUENCE_POS); +TRACE_DEFINE_ENUM(NFS4ERR_SEQ_FALSE_RETRY); +TRACE_DEFINE_ENUM(NFS4ERR_SEQ_MISORDERED); +TRACE_DEFINE_ENUM(NFS4ERR_SERVERFAULT); +TRACE_DEFINE_ENUM(NFS4ERR_STALE); +TRACE_DEFINE_ENUM(NFS4ERR_STALE_CLIENTID); +TRACE_DEFINE_ENUM(NFS4ERR_STALE_STATEID); +TRACE_DEFINE_ENUM(NFS4ERR_SYMLINK); +TRACE_DEFINE_ENUM(NFS4ERR_TOOSMALL); +TRACE_DEFINE_ENUM(NFS4ERR_TOO_MANY_OPS); +TRACE_DEFINE_ENUM(NFS4ERR_UNKNOWN_LAYOUTTYPE); +TRACE_DEFINE_ENUM(NFS4ERR_UNSAFE_COMPOUND); +TRACE_DEFINE_ENUM(NFS4ERR_WRONGSEC); +TRACE_DEFINE_ENUM(NFS4ERR_WRONG_CRED); +TRACE_DEFINE_ENUM(NFS4ERR_WRONG_TYPE); +TRACE_DEFINE_ENUM(NFS4ERR_XDEV); + #define show_nfsv4_errors(error) \ - __print_symbolic(error, \ + __print_symbolic(-(error), \ { NFS4_OK, "OK" }, \ /* Mapped by nfs4_stat_to_errno() */ \ - { -EPERM, "EPERM" }, \ - { -ENOENT, "ENOENT" }, \ - { -EIO, "EIO" }, \ - { -ENXIO, "ENXIO" }, \ - { -EACCES, "EACCES" }, \ - { -EEXIST, "EEXIST" }, \ - { -EXDEV, "EXDEV" }, \ - { -ENOTDIR, "ENOTDIR" }, \ - { -EISDIR, "EISDIR" }, \ - { -EFBIG, "EFBIG" }, \ - { -ENOSPC, "ENOSPC" }, \ - { -EROFS, "EROFS" }, \ - { -EMLINK, "EMLINK" }, \ - { -ENAMETOOLONG, "ENAMETOOLONG" }, \ - { -ENOTEMPTY, "ENOTEMPTY" }, \ - { -EDQUOT, "EDQUOT" }, \ - { -ESTALE, "ESTALE" }, \ - { -EBADHANDLE, "EBADHANDLE" }, \ - { -EBADCOOKIE, "EBADCOOKIE" }, \ - { -ENOTSUPP, "ENOTSUPP" }, \ - { -ETOOSMALL, "ETOOSMALL" }, \ - { -EREMOTEIO, "EREMOTEIO" }, \ - { -EBADTYPE, "EBADTYPE" }, \ - { -EAGAIN, "EAGAIN" }, \ - { -ELOOP, "ELOOP" }, \ - { -EOPNOTSUPP, "EOPNOTSUPP" }, \ - { -EDEADLK, "EDEADLK" }, \ + { EPERM, "EPERM" }, \ + { ENOENT, "ENOENT" }, \ + { EIO, "EIO" }, \ + { ENXIO, "ENXIO" }, \ + { EACCES, "EACCES" }, \ + { EEXIST, "EEXIST" }, \ + { EXDEV, "EXDEV" }, \ + { ENOTDIR, "ENOTDIR" }, \ + { EISDIR, "EISDIR" }, \ + { EFBIG, "EFBIG" }, \ + { ENOSPC, "ENOSPC" }, \ + { EROFS, "EROFS" }, \ + { EMLINK, "EMLINK" }, \ + { ENAMETOOLONG, "ENAMETOOLONG" }, \ + { ENOTEMPTY, "ENOTEMPTY" }, \ + { EDQUOT, "EDQUOT" }, \ + { ESTALE, "ESTALE" }, \ + { EBADHANDLE, "EBADHANDLE" }, \ + { EBADCOOKIE, "EBADCOOKIE" }, \ + { ENOTSUPP, "ENOTSUPP" }, \ + { ETOOSMALL, "ETOOSMALL" }, \ + { EREMOTEIO, "EREMOTEIO" }, \ + { EBADTYPE, "EBADTYPE" }, \ + { EAGAIN, "EAGAIN" }, \ + { ELOOP, "ELOOP" }, \ + { EOPNOTSUPP, "EOPNOTSUPP" }, \ + { EDEADLK, "EDEADLK" }, \ /* RPC errors */ \ - { -ENOMEM, "ENOMEM" }, \ - { -EKEYEXPIRED, "EKEYEXPIRED" }, \ - { -ETIMEDOUT, "ETIMEDOUT" }, \ - { -ERESTARTSYS, "ERESTARTSYS" }, \ - { -ECONNREFUSED, "ECONNREFUSED" }, \ - { -ECONNRESET, "ECONNRESET" }, \ - { -ENETUNREACH, "ENETUNREACH" }, \ - { -EHOSTUNREACH, "EHOSTUNREACH" }, \ - { -EHOSTDOWN, "EHOSTDOWN" }, \ - { -EPIPE, "EPIPE" }, \ - { -EPFNOSUPPORT, "EPFNOSUPPORT" }, \ - { -EPROTONOSUPPORT, "EPROTONOSUPPORT" }, \ + { ENOMEM, "ENOMEM" }, \ + { EKEYEXPIRED, "EKEYEXPIRED" }, \ + { ETIMEDOUT, "ETIMEDOUT" }, \ + { ERESTARTSYS, "ERESTARTSYS" }, \ + { ECONNREFUSED, "ECONNREFUSED" }, \ + { ECONNRESET, "ECONNRESET" }, \ + { ENETUNREACH, "ENETUNREACH" }, \ + { EHOSTUNREACH, "EHOSTUNREACH" }, \ + { EHOSTDOWN, "EHOSTDOWN" }, \ + { EPIPE, "EPIPE" }, \ + { EPFNOSUPPORT, "EPFNOSUPPORT" }, \ + { EPROTONOSUPPORT, "EPROTONOSUPPORT" }, \ /* NFSv4 native errors */ \ - { -NFS4ERR_ACCESS, "ACCESS" }, \ - { -NFS4ERR_ATTRNOTSUPP, "ATTRNOTSUPP" }, \ - { -NFS4ERR_ADMIN_REVOKED, "ADMIN_REVOKED" }, \ - { -NFS4ERR_BACK_CHAN_BUSY, "BACK_CHAN_BUSY" }, \ - { -NFS4ERR_BADCHAR, "BADCHAR" }, \ - { -NFS4ERR_BADHANDLE, "BADHANDLE" }, \ - { -NFS4ERR_BADIOMODE, "BADIOMODE" }, \ - { -NFS4ERR_BADLAYOUT, "BADLAYOUT" }, \ - { -NFS4ERR_BADLABEL, "BADLABEL" }, \ - { -NFS4ERR_BADNAME, "BADNAME" }, \ - { -NFS4ERR_BADOWNER, "BADOWNER" }, \ - { -NFS4ERR_BADSESSION, "BADSESSION" }, \ - { -NFS4ERR_BADSLOT, "BADSLOT" }, \ - { -NFS4ERR_BADTYPE, "BADTYPE" }, \ - { -NFS4ERR_BADXDR, "BADXDR" }, \ - { -NFS4ERR_BAD_COOKIE, "BAD_COOKIE" }, \ - { -NFS4ERR_BAD_HIGH_SLOT, "BAD_HIGH_SLOT" }, \ - { -NFS4ERR_BAD_RANGE, "BAD_RANGE" }, \ - { -NFS4ERR_BAD_SEQID, "BAD_SEQID" }, \ - { -NFS4ERR_BAD_SESSION_DIGEST, "BAD_SESSION_DIGEST" }, \ - { -NFS4ERR_BAD_STATEID, "BAD_STATEID" }, \ - { -NFS4ERR_CB_PATH_DOWN, "CB_PATH_DOWN" }, \ - { -NFS4ERR_CLID_INUSE, "CLID_INUSE" }, \ - { -NFS4ERR_CLIENTID_BUSY, "CLIENTID_BUSY" }, \ - { -NFS4ERR_COMPLETE_ALREADY, "COMPLETE_ALREADY" }, \ - { -NFS4ERR_CONN_NOT_BOUND_TO_SESSION, \ + { NFS4ERR_ACCESS, "ACCESS" }, \ + { NFS4ERR_ATTRNOTSUPP, "ATTRNOTSUPP" }, \ + { NFS4ERR_ADMIN_REVOKED, "ADMIN_REVOKED" }, \ + { NFS4ERR_BACK_CHAN_BUSY, "BACK_CHAN_BUSY" }, \ + { NFS4ERR_BADCHAR, "BADCHAR" }, \ + { NFS4ERR_BADHANDLE, "BADHANDLE" }, \ + { NFS4ERR_BADIOMODE, "BADIOMODE" }, \ + { NFS4ERR_BADLAYOUT, "BADLAYOUT" }, \ + { NFS4ERR_BADLABEL, "BADLABEL" }, \ + { NFS4ERR_BADNAME, "BADNAME" }, \ + { NFS4ERR_BADOWNER, "BADOWNER" }, \ + { NFS4ERR_BADSESSION, "BADSESSION" }, \ + { NFS4ERR_BADSLOT, "BADSLOT" }, \ + { NFS4ERR_BADTYPE, "BADTYPE" }, \ + { NFS4ERR_BADXDR, "BADXDR" }, \ + { NFS4ERR_BAD_COOKIE, "BAD_COOKIE" }, \ + { NFS4ERR_BAD_HIGH_SLOT, "BAD_HIGH_SLOT" }, \ + { NFS4ERR_BAD_RANGE, "BAD_RANGE" }, \ + { NFS4ERR_BAD_SEQID, "BAD_SEQID" }, \ + { NFS4ERR_BAD_SESSION_DIGEST, "BAD_SESSION_DIGEST" }, \ + { NFS4ERR_BAD_STATEID, "BAD_STATEID" }, \ + { NFS4ERR_CB_PATH_DOWN, "CB_PATH_DOWN" }, \ + { NFS4ERR_CLID_INUSE, "CLID_INUSE" }, \ + { NFS4ERR_CLIENTID_BUSY, "CLIENTID_BUSY" }, \ + { NFS4ERR_COMPLETE_ALREADY, "COMPLETE_ALREADY" }, \ + { NFS4ERR_CONN_NOT_BOUND_TO_SESSION, \ "CONN_NOT_BOUND_TO_SESSION" }, \ - { -NFS4ERR_DEADLOCK, "DEADLOCK" }, \ - { -NFS4ERR_DEADSESSION, "DEAD_SESSION" }, \ - { -NFS4ERR_DELAY, "DELAY" }, \ - { -NFS4ERR_DELEG_ALREADY_WANTED, \ + { NFS4ERR_DEADLOCK, "DEADLOCK" }, \ + { NFS4ERR_DEADSESSION, "DEAD_SESSION" }, \ + { NFS4ERR_DELAY, "DELAY" }, \ + { NFS4ERR_DELEG_ALREADY_WANTED, \ "DELEG_ALREADY_WANTED" }, \ - { -NFS4ERR_DELEG_REVOKED, "DELEG_REVOKED" }, \ - { -NFS4ERR_DENIED, "DENIED" }, \ - { -NFS4ERR_DIRDELEG_UNAVAIL, "DIRDELEG_UNAVAIL" }, \ - { -NFS4ERR_DQUOT, "DQUOT" }, \ - { -NFS4ERR_ENCR_ALG_UNSUPP, "ENCR_ALG_UNSUPP" }, \ - { -NFS4ERR_EXIST, "EXIST" }, \ - { -NFS4ERR_EXPIRED, "EXPIRED" }, \ - { -NFS4ERR_FBIG, "FBIG" }, \ - { -NFS4ERR_FHEXPIRED, "FHEXPIRED" }, \ - { -NFS4ERR_FILE_OPEN, "FILE_OPEN" }, \ - { -NFS4ERR_GRACE, "GRACE" }, \ - { -NFS4ERR_HASH_ALG_UNSUPP, "HASH_ALG_UNSUPP" }, \ - { -NFS4ERR_INVAL, "INVAL" }, \ - { -NFS4ERR_IO, "IO" }, \ - { -NFS4ERR_ISDIR, "ISDIR" }, \ - { -NFS4ERR_LAYOUTTRYLATER, "LAYOUTTRYLATER" }, \ - { -NFS4ERR_LAYOUTUNAVAILABLE, "LAYOUTUNAVAILABLE" }, \ - { -NFS4ERR_LEASE_MOVED, "LEASE_MOVED" }, \ - { -NFS4ERR_LOCKED, "LOCKED" }, \ - { -NFS4ERR_LOCKS_HELD, "LOCKS_HELD" }, \ - { -NFS4ERR_LOCK_RANGE, "LOCK_RANGE" }, \ - { -NFS4ERR_MINOR_VERS_MISMATCH, "MINOR_VERS_MISMATCH" }, \ - { -NFS4ERR_MLINK, "MLINK" }, \ - { -NFS4ERR_MOVED, "MOVED" }, \ - { -NFS4ERR_NAMETOOLONG, "NAMETOOLONG" }, \ - { -NFS4ERR_NOENT, "NOENT" }, \ - { -NFS4ERR_NOFILEHANDLE, "NOFILEHANDLE" }, \ - { -NFS4ERR_NOMATCHING_LAYOUT, "NOMATCHING_LAYOUT" }, \ - { -NFS4ERR_NOSPC, "NOSPC" }, \ - { -NFS4ERR_NOTDIR, "NOTDIR" }, \ - { -NFS4ERR_NOTEMPTY, "NOTEMPTY" }, \ - { -NFS4ERR_NOTSUPP, "NOTSUPP" }, \ - { -NFS4ERR_NOT_ONLY_OP, "NOT_ONLY_OP" }, \ - { -NFS4ERR_NOT_SAME, "NOT_SAME" }, \ - { -NFS4ERR_NO_GRACE, "NO_GRACE" }, \ - { -NFS4ERR_NXIO, "NXIO" }, \ - { -NFS4ERR_OLD_STATEID, "OLD_STATEID" }, \ - { -NFS4ERR_OPENMODE, "OPENMODE" }, \ - { -NFS4ERR_OP_ILLEGAL, "OP_ILLEGAL" }, \ - { -NFS4ERR_OP_NOT_IN_SESSION, "OP_NOT_IN_SESSION" }, \ - { -NFS4ERR_PERM, "PERM" }, \ - { -NFS4ERR_PNFS_IO_HOLE, "PNFS_IO_HOLE" }, \ - { -NFS4ERR_PNFS_NO_LAYOUT, "PNFS_NO_LAYOUT" }, \ - { -NFS4ERR_RECALLCONFLICT, "RECALLCONFLICT" }, \ - { -NFS4ERR_RECLAIM_BAD, "RECLAIM_BAD" }, \ - { -NFS4ERR_RECLAIM_CONFLICT, "RECLAIM_CONFLICT" }, \ - { -NFS4ERR_REJECT_DELEG, "REJECT_DELEG" }, \ - { -NFS4ERR_REP_TOO_BIG, "REP_TOO_BIG" }, \ - { -NFS4ERR_REP_TOO_BIG_TO_CACHE, \ + { NFS4ERR_DELEG_REVOKED, "DELEG_REVOKED" }, \ + { NFS4ERR_DENIED, "DENIED" }, \ + { NFS4ERR_DIRDELEG_UNAVAIL, "DIRDELEG_UNAVAIL" }, \ + { NFS4ERR_DQUOT, "DQUOT" }, \ + { NFS4ERR_ENCR_ALG_UNSUPP, "ENCR_ALG_UNSUPP" }, \ + { NFS4ERR_EXIST, "EXIST" }, \ + { NFS4ERR_EXPIRED, "EXPIRED" }, \ + { NFS4ERR_FBIG, "FBIG" }, \ + { NFS4ERR_FHEXPIRED, "FHEXPIRED" }, \ + { NFS4ERR_FILE_OPEN, "FILE_OPEN" }, \ + { NFS4ERR_GRACE, "GRACE" }, \ + { NFS4ERR_HASH_ALG_UNSUPP, "HASH_ALG_UNSUPP" }, \ + { NFS4ERR_INVAL, "INVAL" }, \ + { NFS4ERR_IO, "IO" }, \ + { NFS4ERR_ISDIR, "ISDIR" }, \ + { NFS4ERR_LAYOUTTRYLATER, "LAYOUTTRYLATER" }, \ + { NFS4ERR_LAYOUTUNAVAILABLE, "LAYOUTUNAVAILABLE" }, \ + { NFS4ERR_LEASE_MOVED, "LEASE_MOVED" }, \ + { NFS4ERR_LOCKED, "LOCKED" }, \ + { NFS4ERR_LOCKS_HELD, "LOCKS_HELD" }, \ + { NFS4ERR_LOCK_RANGE, "LOCK_RANGE" }, \ + { NFS4ERR_MINOR_VERS_MISMATCH, "MINOR_VERS_MISMATCH" }, \ + { NFS4ERR_MLINK, "MLINK" }, \ + { NFS4ERR_MOVED, "MOVED" }, \ + { NFS4ERR_NAMETOOLONG, "NAMETOOLONG" }, \ + { NFS4ERR_NOENT, "NOENT" }, \ + { NFS4ERR_NOFILEHANDLE, "NOFILEHANDLE" }, \ + { NFS4ERR_NOMATCHING_LAYOUT, "NOMATCHING_LAYOUT" }, \ + { NFS4ERR_NOSPC, "NOSPC" }, \ + { NFS4ERR_NOTDIR, "NOTDIR" }, \ + { NFS4ERR_NOTEMPTY, "NOTEMPTY" }, \ + { NFS4ERR_NOTSUPP, "NOTSUPP" }, \ + { NFS4ERR_NOT_ONLY_OP, "NOT_ONLY_OP" }, \ + { NFS4ERR_NOT_SAME, "NOT_SAME" }, \ + { NFS4ERR_NO_GRACE, "NO_GRACE" }, \ + { NFS4ERR_NXIO, "NXIO" }, \ + { NFS4ERR_OLD_STATEID, "OLD_STATEID" }, \ + { NFS4ERR_OPENMODE, "OPENMODE" }, \ + { NFS4ERR_OP_ILLEGAL, "OP_ILLEGAL" }, \ + { NFS4ERR_OP_NOT_IN_SESSION, "OP_NOT_IN_SESSION" }, \ + { NFS4ERR_PERM, "PERM" }, \ + { NFS4ERR_PNFS_IO_HOLE, "PNFS_IO_HOLE" }, \ + { NFS4ERR_PNFS_NO_LAYOUT, "PNFS_NO_LAYOUT" }, \ + { NFS4ERR_RECALLCONFLICT, "RECALLCONFLICT" }, \ + { NFS4ERR_RECLAIM_BAD, "RECLAIM_BAD" }, \ + { NFS4ERR_RECLAIM_CONFLICT, "RECLAIM_CONFLICT" }, \ + { NFS4ERR_REJECT_DELEG, "REJECT_DELEG" }, \ + { NFS4ERR_REP_TOO_BIG, "REP_TOO_BIG" }, \ + { NFS4ERR_REP_TOO_BIG_TO_CACHE, \ "REP_TOO_BIG_TO_CACHE" }, \ - { -NFS4ERR_REQ_TOO_BIG, "REQ_TOO_BIG" }, \ - { -NFS4ERR_RESOURCE, "RESOURCE" }, \ - { -NFS4ERR_RESTOREFH, "RESTOREFH" }, \ - { -NFS4ERR_RETRY_UNCACHED_REP, "RETRY_UNCACHED_REP" }, \ - { -NFS4ERR_RETURNCONFLICT, "RETURNCONFLICT" }, \ - { -NFS4ERR_ROFS, "ROFS" }, \ - { -NFS4ERR_SAME, "SAME" }, \ - { -NFS4ERR_SHARE_DENIED, "SHARE_DENIED" }, \ - { -NFS4ERR_SEQUENCE_POS, "SEQUENCE_POS" }, \ - { -NFS4ERR_SEQ_FALSE_RETRY, "SEQ_FALSE_RETRY" }, \ - { -NFS4ERR_SEQ_MISORDERED, "SEQ_MISORDERED" }, \ - { -NFS4ERR_SERVERFAULT, "SERVERFAULT" }, \ - { -NFS4ERR_STALE, "STALE" }, \ - { -NFS4ERR_STALE_CLIENTID, "STALE_CLIENTID" }, \ - { -NFS4ERR_STALE_STATEID, "STALE_STATEID" }, \ - { -NFS4ERR_SYMLINK, "SYMLINK" }, \ - { -NFS4ERR_TOOSMALL, "TOOSMALL" }, \ - { -NFS4ERR_TOO_MANY_OPS, "TOO_MANY_OPS" }, \ - { -NFS4ERR_UNKNOWN_LAYOUTTYPE, "UNKNOWN_LAYOUTTYPE" }, \ - { -NFS4ERR_UNSAFE_COMPOUND, "UNSAFE_COMPOUND" }, \ - { -NFS4ERR_WRONGSEC, "WRONGSEC" }, \ - { -NFS4ERR_WRONG_CRED, "WRONG_CRED" }, \ - { -NFS4ERR_WRONG_TYPE, "WRONG_TYPE" }, \ - { -NFS4ERR_XDEV, "XDEV" }) + { NFS4ERR_REQ_TOO_BIG, "REQ_TOO_BIG" }, \ + { NFS4ERR_RESOURCE, "RESOURCE" }, \ + { NFS4ERR_RESTOREFH, "RESTOREFH" }, \ + { NFS4ERR_RETRY_UNCACHED_REP, "RETRY_UNCACHED_REP" }, \ + { NFS4ERR_RETURNCONFLICT, "RETURNCONFLICT" }, \ + { NFS4ERR_ROFS, "ROFS" }, \ + { NFS4ERR_SAME, "SAME" }, \ + { NFS4ERR_SHARE_DENIED, "SHARE_DENIED" }, \ + { NFS4ERR_SEQUENCE_POS, "SEQUENCE_POS" }, \ + { NFS4ERR_SEQ_FALSE_RETRY, "SEQ_FALSE_RETRY" }, \ + { NFS4ERR_SEQ_MISORDERED, "SEQ_MISORDERED" }, \ + { NFS4ERR_SERVERFAULT, "SERVERFAULT" }, \ + { NFS4ERR_STALE, "STALE" }, \ + { NFS4ERR_STALE_CLIENTID, "STALE_CLIENTID" }, \ + { NFS4ERR_STALE_STATEID, "STALE_STATEID" }, \ + { NFS4ERR_SYMLINK, "SYMLINK" }, \ + { NFS4ERR_TOOSMALL, "TOOSMALL" }, \ + { NFS4ERR_TOO_MANY_OPS, "TOO_MANY_OPS" }, \ + { NFS4ERR_UNKNOWN_LAYOUTTYPE, "UNKNOWN_LAYOUTTYPE" }, \ + { NFS4ERR_UNSAFE_COMPOUND, "UNSAFE_COMPOUND" }, \ + { NFS4ERR_WRONGSEC, "WRONGSEC" }, \ + { NFS4ERR_WRONG_CRED, "WRONG_CRED" }, \ + { NFS4ERR_WRONG_TYPE, "WRONG_TYPE" }, \ + { NFS4ERR_XDEV, "XDEV" }) #define show_open_flags(flags) \ __print_flags(flags, "|", \ @@ -558,6 +703,13 @@ ) ); +TRACE_DEFINE_ENUM(F_GETLK); +TRACE_DEFINE_ENUM(F_SETLK); +TRACE_DEFINE_ENUM(F_SETLKW); +TRACE_DEFINE_ENUM(F_RDLCK); +TRACE_DEFINE_ENUM(F_WRLCK); +TRACE_DEFINE_ENUM(F_UNLCK); + #define show_lock_cmd(type) \ __print_symbolic((int)type, \ { F_GETLK, "GETLK" }, \ @@ -1451,6 +1603,10 @@ #ifdef CONFIG_NFS_V4_1 DEFINE_NFS4_COMMIT_EVENT(nfs4_pnfs_commit_ds); +TRACE_DEFINE_ENUM(IOMODE_READ); +TRACE_DEFINE_ENUM(IOMODE_RW); +TRACE_DEFINE_ENUM(IOMODE_ANY); + #define show_pnfs_iomode(iomode) \ __print_symbolic(iomode, \ { IOMODE_READ, "READ" }, \ @@ -1528,6 +1684,20 @@ DEFINE_NFS4_INODE_STATEID_EVENT(nfs4_layoutreturn); DEFINE_NFS4_INODE_EVENT(nfs4_layoutreturn_on_close); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_UNKNOWN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_NO_PNFS); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_RD_ZEROLEN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_MDSTHRESH); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_NOMEM); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_BULK_RECALL); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_IO_TEST_FAIL); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_FOUND_CACHED); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_RETURN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_BLOCKED); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_INVALID_OPEN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_RETRY); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_SEND_LAYOUTGET); + #define show_pnfs_update_layout_reason(reason) \ __print_symbolic(reason, \ { PNFS_UPDATE_LAYOUT_UNKNOWN, "unknown" }, \ From patchwork Wed Dec 19 16:00:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737651 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51D256C5 for ; Wed, 19 Dec 2018 16:00:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 40DB72B574 for ; Wed, 19 Dec 2018 16:00:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 351A92B5A5; Wed, 19 Dec 2018 16:00:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 917E42B574 for ; Wed, 19 Dec 2018 16:00:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727569AbeLSQAU (ORCPT ); Wed, 19 Dec 2018 11:00:20 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:38730 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726652AbeLSQAT (ORCPT ); Wed, 19 Dec 2018 11:00:19 -0500 Received: by mail-it1-f194.google.com with SMTP id h65so10038051ith.3; Wed, 19 Dec 2018 08:00:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=P/q4RnJj2efKP18/p35VQ/p0eIvp2esguJVWydMCWg8=; b=mObUl3YO8qiq7RSN/BF3HrW+gUT21FhXJqCEK/+D7J8YMHMBuRJDBA44G66U+ne2vk QDn6ySMA3NKOLz+qoQkVXyrQRUEXoMRL8/duShMDOB/oqnKCRe7EIn5SKlDmNyH2tmU7 fbrsfqJ1Q1rPvm9cgYDYYlsuWPOPv6Vz1OCaGti5hWgWxKB/CGszlwajIR4yVnN1NrwG zz/Z3V35l7yQpqNcbP6sC7loUBqX9sFV3bHq/BX683PWhbbQboi1w48rJcwjJQTxzJxv I1WE6UnZkl/SPtaGJfHKrnoBDlWAzB95YTAAspPLpcVMKOLr7JauFA86+LrZ5X6LajzV Hr9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=P/q4RnJj2efKP18/p35VQ/p0eIvp2esguJVWydMCWg8=; b=lqQzd7uLvOd8CD7TSZhiA2j8TG7OEHbSSG+fBjdLIcDkLZRzyIBGm0kwQUm+wQ7EtP ve5glnnjH9C4QfnEUriDIzfgzbVOytvDIQPWe2L3MuO6XYQLslatzoUtrm8koVzlL3KF rwB6OCccDJKt0iUuAyuT1dWOY+7y5qZEgSon1+p1Bgy7eYPUzondRCKVmXHl0SbNecRm tYt3izKM5vTktVXHIkjZsBgruutlDDSZG81vmHPympFD3d8hsZEfNH93eYVAWwbNKVFC xtOW3avonvbHZnpmPz6v++Wo/AVWaGneTCjeI/NAFNTJ6bvJxFfZ6Zi0Sbdzttx/uxVw qI7A== X-Gm-Message-State: AA+aEWbDfqw3pyXcPEhwX4VUinuTesWKiI0ERhyvXeptIViph7+ekrlA +IikMera2FSOM8Ii4BjG0gg= X-Google-Smtp-Source: AFSGD/Uc2oKIOoUYP8muALMNUaoiEngkP6CLgyeoS1krWcL4es9kLksVJ1oaX+uAhbp9QnvclKEPKg== X-Received: by 2002:a24:41ea:: with SMTP id b103mr6902507itd.120.1545235218210; Wed, 19 Dec 2018 08:00:18 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id g186sm3389803ite.39.2018.12.19.08.00.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:17 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0G7j025838; Wed, 19 Dec 2018 16:00:16 GMT Subject: [PATCH v5 24/30] SUNRPC: Simplify defining common RPC trace events From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:16 -0500 Message-ID: <20181219160016.11602.19808.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up, no functional change is expected. Signed-off-by: Chuck Lever --- include/trace/events/sunrpc.h | 172 ++++++++++++++++------------------------- 1 file changed, 69 insertions(+), 103 deletions(-) diff --git a/include/trace/events/sunrpc.h b/include/trace/events/sunrpc.h index 28e3841..88bda93 100644 --- a/include/trace/events/sunrpc.h +++ b/include/trace/events/sunrpc.h @@ -16,40 +16,6 @@ DECLARE_EVENT_CLASS(rpc_task_status, - TP_PROTO(struct rpc_task *task), - - TP_ARGS(task), - - TP_STRUCT__entry( - __field(unsigned int, task_id) - __field(unsigned int, client_id) - __field(int, status) - ), - - TP_fast_assign( - __entry->task_id = task->tk_pid; - __entry->client_id = task->tk_client->cl_clid; - __entry->status = task->tk_status; - ), - - TP_printk("task:%u@%u status=%d", - __entry->task_id, __entry->client_id, - __entry->status) -); - -DEFINE_EVENT(rpc_task_status, rpc_call_status, - TP_PROTO(struct rpc_task *task), - - TP_ARGS(task) -); - -DEFINE_EVENT(rpc_task_status, rpc_bind_status, - TP_PROTO(struct rpc_task *task), - - TP_ARGS(task) -); - -TRACE_EVENT(rpc_connect_status, TP_PROTO(const struct rpc_task *task), TP_ARGS(task), @@ -70,6 +36,16 @@ __entry->task_id, __entry->client_id, __entry->status) ); +#define DEFINE_RPC_STATUS_EVENT(name) \ + DEFINE_EVENT(rpc_task_status, rpc_##name##_status, \ + TP_PROTO( \ + const struct rpc_task *task \ + ), \ + TP_ARGS(task)) + +DEFINE_RPC_STATUS_EVENT(call); +DEFINE_RPC_STATUS_EVENT(bind); +DEFINE_RPC_STATUS_EVENT(connect); TRACE_EVENT(rpc_request, TP_PROTO(const struct rpc_task *task), @@ -134,30 +110,17 @@ __entry->action ) ); +#define DEFINE_RPC_RUNNING_EVENT(name) \ + DEFINE_EVENT(rpc_task_running, rpc_task_##name, \ + TP_PROTO( \ + const struct rpc_task *task, \ + const void *action \ + ), \ + TP_ARGS(task, action)) -DEFINE_EVENT(rpc_task_running, rpc_task_begin, - - TP_PROTO(const struct rpc_task *task, const void *action), - - TP_ARGS(task, action) - -); - -DEFINE_EVENT(rpc_task_running, rpc_task_run_action, - - TP_PROTO(const struct rpc_task *task, const void *action), - - TP_ARGS(task, action) - -); - -DEFINE_EVENT(rpc_task_running, rpc_task_complete, - - TP_PROTO(const struct rpc_task *task, const void *action), - - TP_ARGS(task, action) - -); +DEFINE_RPC_RUNNING_EVENT(begin); +DEFINE_RPC_RUNNING_EVENT(run_action); +DEFINE_RPC_RUNNING_EVENT(complete); DECLARE_EVENT_CLASS(rpc_task_queued, @@ -195,22 +158,16 @@ __get_str(q_name) ) ); +#define DEFINE_RPC_QUEUED_EVENT(name) \ + DEFINE_EVENT(rpc_task_queued, rpc_task_##name, \ + TP_PROTO( \ + const struct rpc_task *task, \ + const struct rpc_wait_queue *q \ + ), \ + TP_ARGS(task, q)) -DEFINE_EVENT(rpc_task_queued, rpc_task_sleep, - - TP_PROTO(const struct rpc_task *task, const struct rpc_wait_queue *q), - - TP_ARGS(task, q) - -); - -DEFINE_EVENT(rpc_task_queued, rpc_task_wakeup, - - TP_PROTO(const struct rpc_task *task, const struct rpc_wait_queue *q), - - TP_ARGS(task, q) - -); +DEFINE_RPC_QUEUED_EVENT(sleep); +DEFINE_RPC_QUEUED_EVENT(wakeup); TRACE_EVENT(rpc_stats_latency, @@ -410,7 +367,11 @@ DEFINE_RPC_SOCKET_EVENT(rpc_socket_shutdown); DECLARE_EVENT_CLASS(rpc_xprt_event, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), + TP_PROTO( + const struct rpc_xprt *xprt, + __be32 xid, + int status + ), TP_ARGS(xprt, xid, status), @@ -432,22 +393,19 @@ __get_str(port), __entry->xid, __entry->status) ); +#define DEFINE_RPC_XPRT_EVENT(name) \ + DEFINE_EVENT(rpc_xprt_event, xprt_##name, \ + TP_PROTO( \ + const struct rpc_xprt *xprt, \ + __be32 xid, \ + int status \ + ), \ + TP_ARGS(xprt, xid, status)) -DEFINE_EVENT(rpc_xprt_event, xprt_timer, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); - -DEFINE_EVENT(rpc_xprt_event, xprt_lookup_rqst, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); - -DEFINE_EVENT(rpc_xprt_event, xprt_transmit, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); - -DEFINE_EVENT(rpc_xprt_event, xprt_complete_rqst, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); +DEFINE_RPC_XPRT_EVENT(timer); +DEFINE_RPC_XPRT_EVENT(lookup_rqst); +DEFINE_RPC_XPRT_EVENT(transmit); +DEFINE_RPC_XPRT_EVENT(complete_rqst); TRACE_EVENT(xprt_ping, TP_PROTO(const struct rpc_xprt *xprt, int status), @@ -587,7 +545,9 @@ DECLARE_EVENT_CLASS(svc_rqst_event, - TP_PROTO(struct svc_rqst *rqst), + TP_PROTO( + const struct svc_rqst *rqst + ), TP_ARGS(rqst), @@ -607,14 +567,15 @@ __get_str(addr), __entry->xid, show_rqstp_flags(__entry->flags)) ); +#define DEFINE_SVC_RQST_EVENT(name) \ + DEFINE_EVENT(svc_rqst_event, svc_##name, \ + TP_PROTO( \ + const struct svc_rqst *rqst \ + ), \ + TP_ARGS(rqst)) -DEFINE_EVENT(svc_rqst_event, svc_defer, - TP_PROTO(struct svc_rqst *rqst), - TP_ARGS(rqst)); - -DEFINE_EVENT(svc_rqst_event, svc_drop, - TP_PROTO(struct svc_rqst *rqst), - TP_ARGS(rqst)); +DEFINE_SVC_RQST_EVENT(defer); +DEFINE_SVC_RQST_EVENT(drop); DECLARE_EVENT_CLASS(svc_rqst_status, @@ -801,7 +762,9 @@ ); DECLARE_EVENT_CLASS(svc_deferred_event, - TP_PROTO(struct svc_deferred_req *dr), + TP_PROTO( + const struct svc_deferred_req *dr + ), TP_ARGS(dr), @@ -818,13 +781,16 @@ TP_printk("addr=%s xid=0x%08x", __get_str(addr), __entry->xid) ); +#define DEFINE_SVC_DEFERRED_EVENT(name) \ + DEFINE_EVENT(svc_deferred_event, svc_##name##_deferred, \ + TP_PROTO( \ + const struct svc_deferred_req *dr \ + ), \ + TP_ARGS(dr)) + +DEFINE_SVC_DEFERRED_EVENT(drop); +DEFINE_SVC_DEFERRED_EVENT(revisit); -DEFINE_EVENT(svc_deferred_event, svc_drop_deferred, - TP_PROTO(struct svc_deferred_req *dr), - TP_ARGS(dr)); -DEFINE_EVENT(svc_deferred_event, svc_revisit_deferred, - TP_PROTO(struct svc_deferred_req *dr), - TP_ARGS(dr)); #endif /* _TRACE_SUNRPC_H */ #include From patchwork Wed Dec 19 16:00:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE6B26C5 for ; Wed, 19 Dec 2018 16:00:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CFD332953A for ; Wed, 19 Dec 2018 16:00:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C44AE2B437; Wed, 19 Dec 2018 16:00:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C4382953A for ; Wed, 19 Dec 2018 16:00:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729561AbeLSQAY (ORCPT ); Wed, 19 Dec 2018 11:00:24 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:35467 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726652AbeLSQAY (ORCPT ); Wed, 19 Dec 2018 11:00:24 -0500 Received: by mail-io1-f67.google.com with SMTP id o13so275304ioh.2; Wed, 19 Dec 2018 08:00:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=yZQOIXq8m8T5Efj8bS/bVcfcxBqC9DeCwu7vAOO8MfU=; b=Vx+0MCCkHGc4rRgTVq3WR1EFeyxFbcoS/qhi2T7ZR31nB0O8xqEAG3evxsTDbzx04W CEOAP1lxiVWyOXZQCyungQsk8yENZ8F6kK+ZJ1Opd7Vs0Tnpa2WLYKtVA0mi1Fm+9hsy mJD6X1THFR9wEJAihtMHDySHT00k+KNEFP5FOW+VUrdiN64/qoXjsJHn5slJTlu8GFiU qOAtkJzJYRZ4GgBXsF4gCXAKP/zd/LgzOUzts7bv1wrLSM66SWnK2TuEo+EFcceH4Cke 2orqQrg4TmCgEi8NdZvy2Pxt6k2EzdlLHqtI8rR9qGYy9R/Se0tlQwQUqsTwFDtcqrBt IbIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=yZQOIXq8m8T5Efj8bS/bVcfcxBqC9DeCwu7vAOO8MfU=; b=CJQYoJzUSf2l+XQao5NgStcry4JFXhus0ErFk2tEEJGpSFNvPWDHOytTcjWItrkdQP jsUKya3gLrT6YoqQFUQmqtqw1LQPoIt8b5e29s6E32PSmldiAHuac7SnCKEhY0mzKV6G YmTm1oPAz/T57iQIq8TC0fFWv71ILqUIlAA/RnPTtljxnSDXxE+eoMDIHUPE+bmuvkO9 IT29CxNaofZK3kYfIOa5yF0M/1aoUpeKWEOUT1le9HfdqdvLwsW2XxdJLxYS03YTCFRe fJ8+xf5d0Ay6fTRg6ZLWYglCCoikDXc7BI9ow/UWuO69KlzzpIbZxGeSJy8Ns/zfVzJE 5Opw== X-Gm-Message-State: AA+aEWYB8JH5PVw3SFyQ6FgxzeC6KLuIxasdRy/ygtCTLtVI+Mu3p8YY 9isRje6BAR2u1ob6W+4TO64CIaGi X-Google-Smtp-Source: AFSGD/WhmKV6E1HW1JAVJJkKiXpVCu/CLxJ2ZE3WxcRapqahf9f5Vcvp5DNEvYw2kOxiToewQXOSGQ== X-Received: by 2002:a6b:720f:: with SMTP id n15mr18679088ioc.175.1545235223493; Wed, 19 Dec 2018 08:00:23 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id x79sm3373857ita.34.2018.12.19.08.00.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:22 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0MtO025841; Wed, 19 Dec 2018 16:00:22 GMT Subject: [PATCH v5 25/30] SUNRPC: Fix some kernel doc complaints From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:22 -0500 Message-ID: <20181219160022.11602.84906.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up some warnings observed when building with "make W=1". Signed-off-by: Chuck Lever --- net/sunrpc/auth_gss/gss_mech_switch.c | 2 +- net/sunrpc/backchannel_rqst.c | 2 +- net/sunrpc/xprtmultipath.c | 4 ++-- net/sunrpc/xprtsock.c | 2 ++ 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/auth_gss/gss_mech_switch.c b/net/sunrpc/auth_gss/gss_mech_switch.c index 16ac0f4..379318d 100644 --- a/net/sunrpc/auth_gss/gss_mech_switch.c +++ b/net/sunrpc/auth_gss/gss_mech_switch.c @@ -244,7 +244,7 @@ struct gss_api_mech * /** * gss_mech_list_pseudoflavors - Discover registered GSS pseudoflavors - * @array: array to fill in + * @array_ptr: array to fill in * @size: size of "array" * * Returns the number of array items filled in, or a negative errno. diff --git a/net/sunrpc/backchannel_rqst.c b/net/sunrpc/backchannel_rqst.c index fa5ba6e..ec451b8 100644 --- a/net/sunrpc/backchannel_rqst.c +++ b/net/sunrpc/backchannel_rqst.c @@ -197,7 +197,7 @@ int xprt_setup_bc(struct rpc_xprt *xprt, unsigned int min_reqs) /** * xprt_destroy_backchannel - Destroys the backchannel preallocated structures. * @xprt: the transport holding the preallocated strucures - * @max_reqs the maximum number of preallocated structures to destroy + * @max_reqs: the maximum number of preallocated structures to destroy * * Since these structures may have been allocated by multiple calls * to xprt_setup_backchannel, we only destroy up to the maximum number diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c index e2d64c7..8394124 100644 --- a/net/sunrpc/xprtmultipath.c +++ b/net/sunrpc/xprtmultipath.c @@ -383,7 +383,7 @@ void xprt_iter_init_listall(struct rpc_xprt_iter *xpi, /** * xprt_iter_xchg_switch - Atomically swap out the rpc_xprt_switch * @xpi: pointer to rpc_xprt_iter - * @xps: pointer to a new rpc_xprt_switch or NULL + * @newswitch: pointer to a new rpc_xprt_switch or NULL * * Swaps out the existing xpi->xpi_xpswitch with a new value. */ @@ -401,7 +401,7 @@ struct rpc_xprt_switch *xprt_iter_xchg_switch(struct rpc_xprt_iter *xpi, /** * xprt_iter_destroy - Destroys the xprt iterator - * @xpi pointer to rpc_xprt_iter + * @xpi: pointer to rpc_xprt_iter */ void xprt_iter_destroy(struct rpc_xprt_iter *xpi) { diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index f0b3700..d5ce1a8 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1600,6 +1600,7 @@ static void xs_udp_set_buffer_size(struct rpc_xprt *xprt, size_t sndsize, size_t /** * xs_udp_timer - called when a retransmit timeout occurs on a UDP transport + * @xprt: controlling transport * @task: task that timed out * * Adjust the congestion window after a retransmit timeout has occurred. @@ -2257,6 +2258,7 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) /** * xs_tcp_setup_socket - create a TCP socket and connect to a remote endpoint + * @work: queued work item * * Invoked by a work queue tasklet. */ From patchwork Wed Dec 19 16:00:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 674C06C5 for ; Wed, 19 Dec 2018 16:00:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 574CA2B48D for ; Wed, 19 Dec 2018 16:00:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 559D22B596; Wed, 19 Dec 2018 16:00:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F399F2B5AF for ; Wed, 19 Dec 2018 16:00:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729576AbeLSQAa (ORCPT ); Wed, 19 Dec 2018 11:00:30 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:33941 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729503AbeLSQAa (ORCPT ); Wed, 19 Dec 2018 11:00:30 -0500 Received: by mail-it1-f193.google.com with SMTP id x124so11334319itd.1; Wed, 19 Dec 2018 08:00:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=P/nO2EcoT+jAHQTgWRUXGGXa4Ij30Kh35hLNAzyaj+Q=; b=mBq0crVecqqhjB0DcnoxQ4YmwWC4uhOyWe4K92ZPJMo6lBgRT0Awfuzk6RmdIzGMGc 3hjLBTX95nNg3Q4VrFQd8ni8bIl+BMK/G1Ixypzqd2n4NBCrNp5vjZOhVwwoiaFdmkSG Q+WaQwKBVFMvq9HmxDxQfrLi9KCanyVRtyehM5OxPy1pHNljYRQGUbAaywRg5BdJbhz8 X49tkITuN93YgZ8SlKBzS5MmCqF3gFV/o3OVX9KUE182odlcsGaqS+kiavK1wtHR1hz6 sQrunJad68AIt77Vk1oXWGqf4iH7oQbnvj8eQb+DT/Tn5szFIgwZFM6hgY+5soZ6Z2/r ZrkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=P/nO2EcoT+jAHQTgWRUXGGXa4Ij30Kh35hLNAzyaj+Q=; b=ps02av5FmgzeB9iqRKcqCEDMfCq1BcD86ly/sO3uzNJYqMtrdY5iZ32TAeFqb9E35M rEFCAGWGd7/eVb9pdaEyrr/QNx35NejfuEPnD4OkdOUvAR0Wjy3RpbP7dP+YQy6sHpQJ pjTzAHXo6ISfBRTmcxrxRzxw4cKkMm7wnOkKaGK9C8Zqf7+WcF8oXNW7kWPZ2o7fMYNc Rll3ILjs1q3qPVnjrRjkjxQgKnG5xVQOG53EjaiXV5Ao/8s1Ice5in/xyceTsgsdEb2R gYgmQ9YD2bTAkWYaVTz49pvUz5Q6CY3wcy/pzoPh6630qRN/BLf3gOGpGe9JuHZxL/fV ktTA== X-Gm-Message-State: AA+aEWYv6XzLHnTFnqeirhRu7tqAcbFq6LEeGd1vlDbTbVP6wsivX0kf xeu9pV5NokDjZZueR3UCOXUw0Q0B X-Google-Smtp-Source: AFSGD/WKcjQ4MNqbUUZ5jSjApccQUShAbClu556GpLrHi2QzmifwXlM8VffCtlUErIWv03R7JWTCxA== X-Received: by 2002:a02:7127:: with SMTP id n39mr19966548jac.91.1545235228608; Wed, 19 Dec 2018 08:00:28 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id k64sm3837732itb.7.2018.12.19.08.00.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:27 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0RVG025844; Wed, 19 Dec 2018 16:00:27 GMT Subject: [PATCH v5 26/30] xprtrdma: Update comments in frwr_op_send From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:27 -0500 Message-ID: <20181219160027.11602.66945.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit f2877623082b ("xprtrdma: Chain Send to FastReg WRs") was written before commit ce5b37178283 ("xprtrdma: Replace all usage of "frmr" with "frwr""), but was merged afterwards. Thus it still refers to FRMR and MWs. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 8a0f1a6..35c8f62 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -479,7 +479,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, * @ia: interface adapter * @req: Prepared RPC Call * - * For FRMR, chain any FastReg WRs to the Send WR. Only a + * For FRWR, chain any FastReg WRs to the Send WR. Only a * single ib_post_send call is needed to register memory * and then post the Send WR. * @@ -507,7 +507,7 @@ int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) } /* If ib_post_send fails, the next ->send_request for - * @req will queue these MWs for recovery. + * @req will queue these MRs for recovery. */ return ib_post_send(ia->ri_id->qp, post_wr, NULL); } From patchwork Wed Dec 19 16:00:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737663 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18D2B13B5 for ; Wed, 19 Dec 2018 16:00:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0717D2B5B6 for ; Wed, 19 Dec 2018 16:00:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 03AE42B430; Wed, 19 Dec 2018 16:00:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B0462B5B4 for ; Wed, 19 Dec 2018 16:00:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729577AbeLSQAg (ORCPT ); Wed, 19 Dec 2018 11:00:36 -0500 Received: from mail-io1-f65.google.com ([209.85.166.65]:36258 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729503AbeLSQAf (ORCPT ); Wed, 19 Dec 2018 11:00:35 -0500 Received: by mail-io1-f65.google.com with SMTP id m19so16003378ioh.3; Wed, 19 Dec 2018 08:00:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=6e831iAyEvL7fncpppgV0vuBdc5zA07EQhaFKLVSxrI=; b=kFKe1T7a6u9iSGQfEi8/SKGrJjCzC5aJtFCW8xmpNbwSjCQWvhhy0Mjl7QaeKbT8bT PSsvVIcKOpMlqDAP1BhCbAIG7rhgL8Ii4IjTQCnoS34/H5NkcRLzxcF/gG5B16hEnA5G hJgPjZWoEW7sBYkGN4D32A9H3sJohffrBQsdBfHYlVCkyQzhHP+S3Q1C3G1z7JyxQCBe aWQnWRLRi9soByT89p2aNocyuJtGbCN4lVCF/yj35ggWxY38r6+cg2W5Jth/jzhpOJN7 26XCcGG7Cd7mpqO1uBkPZ9fH7CxzQlce1v9MzOP4m5D3NP921j14jIsd5+YkTa/uKY0i KqgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=6e831iAyEvL7fncpppgV0vuBdc5zA07EQhaFKLVSxrI=; b=SDXsEbttPq4jPVk9ML8dhNmjkv7kC2lxACOMlwPOYzsbGZxunnM0L+UKQYQ/0T6+uP tyKHNo4MQSp3RxgAP9y0U8hRwDwCIx4NNXxOGcgDUBZUIvghh9CFhlLDiq2bsVz+RTkh Z/caxirqrkSAHMArLD9zoSyRR+G+7ap+o7BCMlwWGY93wM1kaX3WSXKnSVI6BlbcSNWw bRvwF7TFUHQu32PTN01L6TDf0YOhmtxomV+uf/4UpZ3+z8eUz7Cr39Dk+ChqJDhk1c8v 5wf5Sl1uNbU2+zlYQossgVFFtDUFAxVYA6TRWtbBPWXQM9n/Be2/YPcP8r6fPSv3IVpO fMig== X-Gm-Message-State: AA+aEWY0/zNWzWgPOoQZnWuBMLDuOhSDOHTFHDBGFNOh8m06KM3bN2iF tUjvEDgPc0LcI88ry+QhPqaCGXMf X-Google-Smtp-Source: AFSGD/WbZjaQO05GRW7TTVhasfkShwhimqvDamce0wbn7fuYoljDkURMqZM1D+SjhnJgGLnx62CLNA== X-Received: by 2002:a5d:8510:: with SMTP id q16mr17472310ion.204.1545235234021; Wed, 19 Dec 2018 08:00:34 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id h84sm8886486ioa.33.2018.12.19.08.00.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:33 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0W4S025847; Wed, 19 Dec 2018 16:00:32 GMT Subject: [PATCH v5 27/30] xprtrdma: Replace outdated comment for rpcrdma_ep_post From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:32 -0500 Message-ID: <20181219160032.11602.28108.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since commit 7c8d9e7c8863 ("xprtrdma: Move Receive posting to Receive handler"), rpcrdma_ep_post is no longer responsible for posting Receive buffers. Update the documenting comment to reflect this change. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 719b69c..11976c3 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1427,10 +1427,14 @@ struct rpcrdma_regbuf * kfree(rb); } -/* - * Prepost any receive buffer, then post send. +/** + * rpcrdma_ep_post - Post WRs to a transport's Send Queue + * @ia: transport's device information + * @ep: transport's RDMA endpoint information + * @req: rpcrdma_req containing the Send WR to post * - * Receive buffer is donated to hardware, reclaimed upon recv completion. + * Returns 0 if the post was successful, otherwise -ENOTCONN + * is returned. */ int rpcrdma_ep_post(struct rpcrdma_ia *ia, From patchwork Wed Dec 19 16:00:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737667 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA10E6C5 for ; Wed, 19 Dec 2018 16:00:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7EC12953A for ; Wed, 19 Dec 2018 16:00:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D61AC2B5B6; Wed, 19 Dec 2018 16:00:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D6D62953A for ; Wed, 19 Dec 2018 16:00:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729503AbeLSQAk (ORCPT ); Wed, 19 Dec 2018 11:00:40 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:54938 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728222AbeLSQAj (ORCPT ); Wed, 19 Dec 2018 11:00:39 -0500 Received: by mail-it1-f195.google.com with SMTP id i145so10389900ita.4; Wed, 19 Dec 2018 08:00:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=ea5IaX7FdfMfgKFJZJOKeKRVfqNhaR9riT4maJPGG2Y=; b=AIvuUtAtyonr+v9DCHN+LPCDJHHMhv09UO+HnYVIXvVlrV8Kh8qNmpXgOlls0eJK1D g5lqo+VG4eoSFEcxXVbF2UttP6lgU4/nItRoSLkCkOSBv6SraV3TTwc72zRa/4FdgksV uNGRUSuuHVKBKRomWL+cO4APJyy71Y8ayNYccU4FB5RVzUBEYdaePZLtdbADdp17CPUl Vp08DLkD7TwmSjttJg+4/myzJ/yOoWx1bocG6V0ZdhrIpAKF7kvyRPGHfPd6FE+IYg3x UkiPRvEZVvaB7lSxxvBfUYEcuU2xvUVm+cF9RKQsJAPma3DN66+dYSAWiQQm1qKPBKbp h+fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=ea5IaX7FdfMfgKFJZJOKeKRVfqNhaR9riT4maJPGG2Y=; b=dFZ+P+Cj0kP+z0D9a+c/HhtzwBXpqqBC+ZWIBpUaM+z+aiR6g4OLzWn4ZqUh+G9wRx YsC7vTShKfCPL2m0y5rSsb7fHvLD2lCFrJFJmqdCpr9a1nbRNSS5lQFviR9SUehrILAA M2kpPNDmNTd55E2xXfifBnHwGOLkFEhp2WgN5naxvjgk4UvywqlE0LJXphg5pfAdVOpn yKqd5ipOMZL79gfXWKseD9UuOUUxvOfbgkx6lqcP+F0rtQ/HXVsYDjm9z4VvhELICESf 5+kzFogCdFnVuJ2rhMlWxQ3gsg8ZyBOoO1wcVrLkUj/1MCDdpWrHFwz1iHe4qw44wzv6 I2bQ== X-Gm-Message-State: AA+aEWZornKvXsnh/mqHITKw3eYarf68NkStdWqY3vHLqoCPzl0VAys5 rMvmxjnk4fhrmynIvmx0Yzs= X-Google-Smtp-Source: AFSGD/UQ0egqMkM6N9mRvZRVviP0WVniQht+osppbgWAKqLY03GaPX/LM/zMmrOYbvJOGA4iHHi9LA== X-Received: by 2002:a24:2c42:: with SMTP id i63mr6989756iti.22.1545235238912; Wed, 19 Dec 2018 08:00:38 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id t129sm3411307ita.4.2018.12.19.08.00.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:38 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0bct025850; Wed, 19 Dec 2018 16:00:37 GMT Subject: [PATCH v5 28/30] xprtrdma: Add documenting comment for rpcrdma_buffer_destroy From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:37 -0500 Message-ID: <20181219160037.11602.90097.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make a note of the function's dependency on an earlier ib_drain_qp. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 11976c3..7749a2b 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1177,6 +1177,14 @@ struct rpcrdma_req * dprintk("RPC: %s: released %u MRs\n", __func__, count); } +/** + * rpcrdma_buffer_destroy - Release all hw resources + * @buf: root control block for resources + * + * ORDERING: relies on a prior ib_drain_qp : + * - No more Send or Receive completions can occur + * - All MRs, reps, and reqs are returned to their free lists + */ void rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) { From patchwork Wed Dec 19 16:00:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737671 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6415A13B5 for ; Wed, 19 Dec 2018 16:00:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5460D2B5B6 for ; Wed, 19 Dec 2018 16:00:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 51C9E2B5BE; Wed, 19 Dec 2018 16:00:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F16912B5C1 for ; Wed, 19 Dec 2018 16:00:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729144AbeLSQAp (ORCPT ); Wed, 19 Dec 2018 11:00:45 -0500 Received: from mail-io1-f66.google.com ([209.85.166.66]:39862 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728222AbeLSQAp (ORCPT ); Wed, 19 Dec 2018 11:00:45 -0500 Received: by mail-io1-f66.google.com with SMTP id k7so15973321iob.6; Wed, 19 Dec 2018 08:00:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=y+F0Uls2LMF2rBl1EmOLKcQm7k0Vfrw8rkFlSfRf4pQ=; b=d16stPPEE6Wp42dj0J1UfUSvS6lgbRZtvYFErA+ndyxDotEi+7KPIbH/B4Hbw7w0lQ ihzm+8xPQtwS3aJ94iPWurVZJ5YmvbnQHUzy8H0U4pcJ8VZ+93N+1vObo8xuTd3uc/O6 zxzzhcih55fDmcEV0zPVJaiYFMab7yOxt8H6NJtGhkMxFnawucrP3+fKV+3kkkn3/FPT N1dE2Rjvx6zf37MAKgk4h1U1MitIKMBRxEDPXuCuHMjnslZ49GFVzplSBB7X6vPs/xOJ jgLtjEd4gQdtjB4br+ek3wpjNf9OaaNJRSglzdl6M77yifAFmXGcNNPU5fXUORMgEpNP bxDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=y+F0Uls2LMF2rBl1EmOLKcQm7k0Vfrw8rkFlSfRf4pQ=; b=ey/9IXVkAmn721JXsNMlq+2ONHiaKIM97ZY395D8UWR6+HMVfrkecYT/KMXjG2nUsg iFQxN8Dhoho2euFJerp0Wrf8LWzN3vjy2TdBBlkGbh3dd8hNPl6LGTwy88YJINKQuSnJ OnD89Ah14p6zo/vgk34obWo7RnaLQPktJwbQ5yIFlmQ/FOxQUJndvV2YmNpcDVC2qagH FZiOE0hgjPpIZ3LW6vWwAnmlxo60TXO26cQvQveokMiHA5aaImhZl5eugvhcjI3iJ9KO wqRuMMc6FmrWoabcUwzjtO1TPvzrmnbDnXBUDMyEqMTSl3vwaKtmd3cG6+E32faX2H1H lb7w== X-Gm-Message-State: AA+aEWYg25PAirBwRQLA0y9k93a36sjSz5llaVeRgabQWpFq471hPw5r DWbapyaxZHX+dhYN0O4O3zw= X-Google-Smtp-Source: AFSGD/VoS6yuft4qQNG8i+/z5k6tXUEwcFgr+Y2NC7hskS5TpTOeEVsfgH7l2J1zMVYlumoEdfwGaw== X-Received: by 2002:a5d:9405:: with SMTP id v5mr19029429ion.250.1545235244214; Wed, 19 Dec 2018 08:00:44 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id r15sm8611557ioh.12.2018.12.19.08.00.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:43 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0gWv025853; Wed, 19 Dec 2018 16:00:42 GMT Subject: [PATCH v5 29/30] xprtrdma: Clarify comments in rpcrdma_ia_remove From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:42 -0500 Message-ID: <20181219160042.11602.91903.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Comments are clarified to note how transport data structures are protected. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 7749a2b..94af069 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -402,8 +402,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) * rpcrdma_ia_remove - Handle device driver unload * @ia: interface adapter being removed * - * Divest transport H/W resources associated with this adapter, - * but allow it to be restored later. + * Callers must serialize calls to this function. */ void rpcrdma_ia_remove(struct rpcrdma_ia *ia) @@ -434,16 +433,23 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) ib_free_cq(ep->rep_attr.send_cq); ep->rep_attr.send_cq = NULL; - /* The ULP is responsible for ensuring all DMA - * mappings and MRs are gone. + /* The ib_drain_qp above guarantees that all posted + * Receives have flushed, which returns the transport's + * rpcrdma_reps to the rb_recv_bufs list. */ list_for_each_entry(rep, &buf->rb_recv_bufs, rr_list) rpcrdma_dma_unmap_regbuf(rep->rr_rdmabuf); + + /* DMA mapping happens in ->send_request with the + * transport send lock held. Our caller is holding + * the transport send lock. + */ list_for_each_entry(req, &buf->rb_allreqs, rl_all) { rpcrdma_dma_unmap_regbuf(req->rl_rdmabuf); rpcrdma_dma_unmap_regbuf(req->rl_sendbuf); rpcrdma_dma_unmap_regbuf(req->rl_recvbuf); } + rpcrdma_mrs_destroy(buf); ib_dealloc_pd(ia->ri_pd); ia->ri_pd = NULL; From patchwork Wed Dec 19 16:00:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10737677 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5DF5B14E2 for ; Wed, 19 Dec 2018 16:00:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 504012B5BC for ; Wed, 19 Dec 2018 16:00:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4EB022B5C4; Wed, 19 Dec 2018 16:00:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0371A2B5C7 for ; Wed, 19 Dec 2018 16:00:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729184AbeLSQAv (ORCPT ); Wed, 19 Dec 2018 11:00:51 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:36769 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728222AbeLSQAu (ORCPT ); Wed, 19 Dec 2018 11:00:50 -0500 Received: by mail-it1-f194.google.com with SMTP id c9so10064191itj.1; Wed, 19 Dec 2018 08:00:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=p6lkr/16ohwqBFo2MRfmrj0EAPREXCOKqFIKpKn96C8=; b=AOJKLJlwwC88ljQZwMp3y3VGhjNCqSSBH3zQMZRzrZ/bZlnhoTIyRPliiVBbLgN5M8 EfljH4JcPwlw1AzxkPYLxZZiZtPda7nu58EoDK1pdxTAenXOufhnPljViGnaTSa0Y2fN 2SfnJoGNNvHgrppiovBLuQWh9jhyqCJiKYbOEHU85YqmvMxtBlLG1maUh+JE/HFwOdYZ KCYA2dU7LcsG5cktIStFxAnrBiCLvinPLfzUoZuFetDosNnlB2AKJ+3ELUegpKjV+PxH L4xlpdSIdPv9uSCcJujLGeKH5iju1mQfXSRgequZjbdVLWd9lUcuRqwnLHVy4l9xYSip DLKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=p6lkr/16ohwqBFo2MRfmrj0EAPREXCOKqFIKpKn96C8=; b=UT+5xOmKqBH00XZxs8hGb5wv2+E57u2SO2Vdi+HqVlqqjJZTbYqb/hjyS1dqltQ2gD eUF8AkyI+Lay0svdvr41GU66tGFfXxLlsvyYUX7MquCFaTvv1elnmzjC3iz5t6vx7eHp 94AGFEU5kmCEJ6wkyAuCHn2/mfjwEebPlDvW0+sUKdMol0wMwl1f/NBDfAVH/FrrHpb9 oHDPWOy2i1Bx7QqreCkJdkQxGO4WcQif1tN3Sph8CyNjqkaB8wxYYLvizDKPjnvl0pxE 8zaPhuLyoqFaubcac3sWn8FVxJIWyy/YeRoRjkwFw8ZAwQd3o+MjJskP47pKJ/b3XUtv 7/MA== X-Gm-Message-State: AA+aEWbOxm4m2XkQRCn64Uk+iG9zSKY3ufs88D9JmCJ0zABaFiIktupE 2NvgL/WdxNx8EyaIT0l4aL7aeMYR X-Google-Smtp-Source: AFSGD/VUUgK/+75VvJD5sgKn5iR1ePaG1v2YGaNYd3yfkYhYMrNwX2cURmBOdNZlXGzKl/YjtgXCKw== X-Received: by 2002:a24:80cc:: with SMTP id g195mr6693737itd.178.1545235249698; Wed, 19 Dec 2018 08:00:49 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id b5sm8707486ioc.21.2018.12.19.08.00.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 08:00:49 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBJG0meB025856; Wed, 19 Dec 2018 16:00:48 GMT Subject: [PATCH v5 30/30] xprtrdma: Don't leak freed MRs From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 19 Dec 2018 11:00:48 -0500 Message-ID: <20181219160048.11602.91563.stgit@manet.1015granger.net> In-Reply-To: <20181219155152.11602.18605.stgit@manet.1015granger.net> References: <20181219155152.11602.18605.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Defensive clean up. Don't set frwr->fr_mr until we know that the scatterlist allocation has succeeded. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 35c8f62..6a56105 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -155,36 +155,39 @@ void frwr_release_mr(struct rpcrdma_mr *mr) int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) { unsigned int depth = ia->ri_max_frwr_depth; - struct rpcrdma_frwr *frwr = &mr->frwr; + struct scatterlist *sg; + struct ib_mr *frmr; int rc; - frwr->fr_mr = ib_alloc_mr(ia->ri_pd, ia->ri_mrtype, depth); - if (IS_ERR(frwr->fr_mr)) + frmr = ib_alloc_mr(ia->ri_pd, ia->ri_mrtype, depth); + if (IS_ERR(frmr)) goto out_mr_err; - mr->mr_sg = kcalloc(depth, sizeof(*mr->mr_sg), GFP_KERNEL); - if (!mr->mr_sg) + sg = kcalloc(depth, sizeof(*sg), GFP_KERNEL); + if (!sg) goto out_list_err; - frwr->fr_state = FRWR_IS_INVALID; + mr->frwr.fr_mr = frmr; + mr->frwr.fr_state = FRWR_IS_INVALID; mr->mr_dir = DMA_NONE; INIT_LIST_HEAD(&mr->mr_list); INIT_WORK(&mr->mr_recycle, frwr_mr_recycle_worker); - sg_init_table(mr->mr_sg, depth); - init_completion(&frwr->fr_linv_done); + init_completion(&mr->frwr.fr_linv_done); + + sg_init_table(sg, depth); + mr->mr_sg = sg; return 0; out_mr_err: - rc = PTR_ERR(frwr->fr_mr); + rc = PTR_ERR(frmr); trace_xprtrdma_frwr_alloc(mr, rc); return rc; out_list_err: - rc = -ENOMEM; dprintk("RPC: %s: sg allocation failure\n", __func__); - ib_dereg_mr(frwr->fr_mr); - return rc; + ib_dereg_mr(frmr); + return -ENOMEM; } /**