From patchwork Mon Dec 17 16:39:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B54F513AD for ; Mon, 17 Dec 2018 16:39:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6E392A1F9 for ; Mon, 17 Dec 2018 16:39:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9AF2C2A205; Mon, 17 Dec 2018 16:39:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B1262A1F9 for ; Mon, 17 Dec 2018 16:39:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387981AbeLQQj3 (ORCPT ); Mon, 17 Dec 2018 11:39:29 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:38295 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387737AbeLQQj3 (ORCPT ); Mon, 17 Dec 2018 11:39:29 -0500 Received: by mail-it1-f194.google.com with SMTP id h65so20898547ith.3; Mon, 17 Dec 2018 08:39:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=YnmL2QcytZ7nDWQP0b2qARfKenfoawrMQNmn+bhvVt8=; b=N3ZkeNBBhiaYv1gQ/+IR2BmE6eYN0wdBNvtzKlaXTr9gWsjLolAj3AMFdcuOxMdX79 SXoIN1K8DVwgGGXoRp4BYXj8D6iNf63OEePvXqzoeaUj9Mu8eSrWFvoF3d+YWGulzO6U 4fAuUuxU6r4KeuThjFrYx+9RMf79SM97jhiiXd6OCjWsQxdWg4HbvWwjLaFwHicpdpC6 p4YDxCy83s+4tJN2heCePs/lFGOMUvJcdQ0hMjCQ72rrR2O0lX4USBZhy9QmU0Y96EgV 9bTDcBXrWCpQsL+HATstiLtBMnyzWCfxNpY4V/GdMemI3Ez5Sx2PTaj/pMNb2C9Aj5oI JzGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=YnmL2QcytZ7nDWQP0b2qARfKenfoawrMQNmn+bhvVt8=; b=EaKEoPhfIQE4i1YFHdrX4FsRX4UoE5Ovaqapvh9EXpnbpiMrZfl/As+WEjBLxLLGJY uIZWr/2yClGxRmhTJ0seXoVIfxSmx0bB7h+H5cinY6eAv5V9ergnKbCBhIGGKlUp0AIp JhVwvRRXR+y9feK25jRe1IQGH4ve7xU2id32uHTA0IgQlMtGuK4AisF/v/xoQPOmLBCJ QLabOC84dVdLByCD/qIjY2i+gH2b2SQ77y/msv+2xY1OtcEpMn77tJ4AISesQPDy3u7c ifkSVu+tc/qtTwZRPau4BBJfyZ36V7o/dFjx133y7dtUUTjQmvbXYQcSFCe2+Hrdt2Kt 3ojQ== X-Gm-Message-State: AA+aEWZzStczlN86DV15J06QtNp/0BaNZGnw0D150izQFd/Gy7Tvz+tI q2DNSylENqSxzUnOs8iCuk6xJlis X-Google-Smtp-Source: AFSGD/WcSY+x9u+0qS5TUlWzaQjykI+ezOYQ74A0MpziArE/Dm9bd/b5hhCQpYzSw9Mm75VIDsRShg== X-Received: by 2002:a24:8a44:: with SMTP id v65mr12430160itd.67.1545064767874; Mon, 17 Dec 2018 08:39:27 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id k64sm9754861itb.7.2018.12.17.08.39.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:27 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdQtA018573; Mon, 17 Dec 2018 16:39:26 GMT Subject: [PATCH v4 01/30] xprtrdma: Yet another double DMA-unmap From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:26 -0500 Message-ID: <20181217163926.24133.67489.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP While chasing yet another set of DMAR fault reports, I noticed that the frwr recycler conflates whether or not an MR has been DMA unmapped with frwr->fr_state. Actually the two have only an indirect relationship. It's in fact impossible to guess reliably whether the MR has been DMA unmapped based on its fr_state field, especially as the surrounding code and its assumptions have changed over time. A better approach is to track the DMA mapping status explicitly so that the recycler is less brittle to unexpected situations, and attempts to DMA-unmap a second time are prevented. Signed-off-by: Chuck Lever Cc: stable@vger.kernel.org # v4.20 --- net/sunrpc/xprtrdma/frwr_ops.c | 6 ++++-- net/sunrpc/xprtrdma/verbs.c | 9 ++++++--- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index fc6378cc..20ced24 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -117,15 +117,15 @@ frwr_mr_recycle_worker(struct work_struct *work) { struct rpcrdma_mr *mr = container_of(work, struct rpcrdma_mr, mr_recycle); - enum rpcrdma_frwr_state state = mr->frwr.fr_state; struct rpcrdma_xprt *r_xprt = mr->mr_xprt; trace_xprtrdma_mr_recycle(mr); - if (state != FRWR_FLUSHED_LI) { + if (mr->mr_dir != DMA_NONE) { trace_xprtrdma_mr_unmap(mr); ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, mr->mr_sg, mr->mr_nents, mr->mr_dir); + mr->mr_dir = DMA_NONE; } spin_lock(&r_xprt->rx_buf.rb_mrlock); @@ -150,6 +150,8 @@ if (!mr->mr_sg) goto out_list_err; + frwr->fr_state = FRWR_IS_INVALID; + mr->mr_dir = DMA_NONE; INIT_LIST_HEAD(&mr->mr_list); INIT_WORK(&mr->mr_recycle, frwr_mr_recycle_worker); sg_init_table(mr->mr_sg, depth); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3ddba94..b9bc7f9 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1329,9 +1329,12 @@ struct rpcrdma_mr * { struct rpcrdma_xprt *r_xprt = mr->mr_xprt; - trace_xprtrdma_mr_unmap(mr); - ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, - mr->mr_sg, mr->mr_nents, mr->mr_dir); + if (mr->mr_dir != DMA_NONE) { + trace_xprtrdma_mr_unmap(mr); + ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, + mr->mr_sg, mr->mr_nents, mr->mr_dir); + mr->mr_dir = DMA_NONE; + } __rpcrdma_mr_put(&r_xprt->rx_buf, mr); } From patchwork Mon Dec 17 16:39:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 652591575 for ; Mon, 17 Dec 2018 16:39:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5112C2A1F8 for ; Mon, 17 Dec 2018 16:39:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 45A5A2A200; Mon, 17 Dec 2018 16:39:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF1102A1F8 for ; Mon, 17 Dec 2018 16:39:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387995AbeLQQje (ORCPT ); Mon, 17 Dec 2018 11:39:34 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:56298 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727771AbeLQQje (ORCPT ); Mon, 17 Dec 2018 11:39:34 -0500 Received: by mail-it1-f196.google.com with SMTP id o19so19433635itg.5; Mon, 17 Dec 2018 08:39:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=nqihSE1hg4o1H0UcWmtuHnYhVxMpkCBQQUarfYAvFW0=; b=dZ1j9Gy+8gVRrAamlgY0DVOq9cQRyUm6b37cshOhNQ0MV6VwuHl9ttUc1DQ/yoVUuQ EBY6W+lVwcK4Mcs28/p1tfekencvxR6Qd1HVP8UvVUr/xnQAd9Fsl3Ipx9wbPlEmstXk 1zQGJBV8xFHEIK5Fcc8iqMhqwykBpTN3WYSLP3PCS/LszTrLqvcC5r1gdYMhLUDFCUuR 6XeyMuAd9euKyIkfxraxBsgciGz7Qb8aFTxJttHCssUHyWFOo9QcC8KATazxDBVE8n7x A/7Z1MMXWc58HKrgkbrpU90PXhXCCka4necFIoGARgIyXckky9A77jbJsLXqL3OPEWvr hYxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=nqihSE1hg4o1H0UcWmtuHnYhVxMpkCBQQUarfYAvFW0=; b=dRCGadXLHZhWp771gkmn3+djF3STxPbdY2o7FF3FHPlAyXsyLb7OTnVQBfbi+31QHn ksCuLJY4FlKPPNKhLrHxpMqYWKYoIZmB2QXk4HTpNpQI2d05GJxNHhitwRjg//7Kp9v8 4IdEjAvJ97IgBY/6DCxpw4cxsIbL3Sw/E121dXn6ejGiihTL0qy4tKt52Ar0fMwD84V+ YW5KbvEy8Rw/012qTb9+BXXqFHPbTbWyH5ZRwva5B36yPAyYa6Gbipdc55r3uf9zAz6b MeFAQonvRA9/oOSml3rq/6go70fF/bBmLcLWPAnsN+LP7Jl6ch/WsS6VbI3xVFe2T0Oc KYHw== X-Gm-Message-State: AA+aEWam9ozPVlaTWGEbpf9mQN9WSzS/elY8b3StxvAYaghgLZbuCgY3 aiqk2Ke2ieul4A2Rpu0yWAaiLGl6 X-Google-Smtp-Source: AFSGD/WzjiNiRlKpkLkl9I7Paii9UQUeW3i0ey9YzUh9MqnNJn3z+UyYnR8Xg3JUYKeXUnHulkd5Ow== X-Received: by 2002:a24:69cd:: with SMTP id e196mr13353012itc.161.1545064773223; Mon, 17 Dec 2018 08:39:33 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id g12sm5637162iok.38.2018.12.17.08.39.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:32 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdV6s018576; Mon, 17 Dec 2018 16:39:31 GMT Subject: [PATCH v4 02/30] xprtrdma: Ensure MRs are DMA-unmapped when posting LOCAL_INV fails From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:31 -0500 Message-ID: <20181217163931.24133.44118.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The recovery case in frwr_op_unmap_sync needs to DMA unmap each MR. frwr_release_mr does not DMA-unmap, but the recycle worker does. Fixes: 61da886bf74e ("xprtrdma: Explicitly resetting MRs is ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 20ced24..27222c0 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -563,8 +563,8 @@ mr = container_of(frwr, struct rpcrdma_mr, frwr); bad_wr = bad_wr->next; - list_del(&mr->mr_list); - frwr_op_release_mr(mr); + list_del_init(&mr->mr_list); + rpcrdma_mr_recycle(mr); } } From patchwork Mon Dec 17 16:39:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C2C113AD for ; Mon, 17 Dec 2018 16:39:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB1A62A1F8 for ; Mon, 17 Dec 2018 16:39:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DF7B52A200; Mon, 17 Dec 2018 16:39:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4496F2A1F8 for ; Mon, 17 Dec 2018 16:39:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388027AbeLQQjk (ORCPT ); Mon, 17 Dec 2018 11:39:40 -0500 Received: from mail-it1-f169.google.com ([209.85.166.169]:53787 "EHLO mail-it1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388034AbeLQQjk (ORCPT ); Mon, 17 Dec 2018 11:39:40 -0500 Received: by mail-it1-f169.google.com with SMTP id g85so19458388ita.3; Mon, 17 Dec 2018 08:39:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=GYu0UsHQVtaKNGx1din9o11hbp64tcjGdgdTHECvmKg=; b=guEVPl+wdZHrHP/iXgCvvmDCyH4Up6hUDCP9Hh/D1Z/qxipFflS148jG7jQqd5hX+/ gaA5dstEyDSwfDYPP7un+v/2BLf77X2xa/qXBw37z5P8fcDRzgGye56AR075NOqSXodF KzbYaa1WOskb7vY+O3gHAZmI7VglZZsJhZj14LztRUxVlQZFsDWopWKITnmAb5YFL9OU 2eZRetAV4KZuCtKeS8fh1BzUapqlem/kWIpXwATzpoQlMfqOoHM8Ao/LJUoIu6LFBqJF e9CUKI6I05000YRuK8YAM2EuByk2JxsYq21SrnfilP/WLWwH0e4lzQfZxQVHPsO65KPO bv3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=GYu0UsHQVtaKNGx1din9o11hbp64tcjGdgdTHECvmKg=; b=IsWY+qhaL6RsiLL/m995IS8gPzWUOxkLzFVxjlHLnt0mGxhGxKPXveq5st30UFve3X Pe2DtJjaA99ZlZA1C02GQl5NJsHtZpo90RQtc/eC8IrAsCWKCCgEALShiebgzp/5aztF sEk64VniZC1Fw388OJ3byzmA640jP3aysWR+6k5uVRSijXzEIr0SWRvh9laZEHzR9QBz 7/wct2ATkzPzmo6CKk/yGJYy9YEepl+2KRoIm0N2iDr16JwzG/3wJ+3N+h/3LVph9Xj3 lXWHuGPyPItrIb0fh2ittoGh9bKUkJM8kCL3rk2MTmJFXN+wsh+URoAo+P1NFPphhpDB SQBw== X-Gm-Message-State: AA+aEWacG7xR++/5n0FN67rNtVcf24O5I/w9eG9H/Sgh+Lrwt7ncJ/Ma lcKGDVdT14RsxIGC9rAm2qMx//Vw X-Google-Smtp-Source: AFSGD/X9+3H4KwBVVkoA1SDGsG5U6M88WYIHEvUSFh85zL/4RIb5x4vilgkv3U01JVe3A7Tsya9olg== X-Received: by 2002:a05:660c:54d:: with SMTP id w13mr12218058itk.50.1545064778823; Mon, 17 Dec 2018 08:39:38 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id r21sm5452443ioc.18.2018.12.17.08.39.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:38 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdb5F018579; Mon, 17 Dec 2018 16:39:37 GMT Subject: [PATCH v4 03/30] xprtrdma: Refactor Receive accounting From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:37 -0500 Message-ID: <20181217163937.24133.75048.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Divide the work cleanly: - rpcrdma_wc_receive is responsible only for RDMA Receives - rpcrdma_reply_handler is responsible only for RPC Replies - the posted send and receive counts both belong in rpcrdma_ep Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 2 +- net/sunrpc/xprtrdma/backchannel.c | 1 - net/sunrpc/xprtrdma/rpc_rdma.c | 21 +++------------------ net/sunrpc/xprtrdma/verbs.c | 31 ++++++++++++++----------------- net/sunrpc/xprtrdma/xprt_rdma.h | 3 +-- 5 files changed, 19 insertions(+), 39 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index b093058..2efe2d7 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -570,7 +570,7 @@ __entry->r_xprt = r_xprt; __entry->count = count; __entry->status = status; - __entry->posted = r_xprt->rx_buf.rb_posted_receives; + __entry->posted = r_xprt->rx_ep.rep_receive_count; __assign_str(addr, rpcrdma_addrstr(r_xprt)); __assign_str(port, rpcrdma_portstr(r_xprt)); ), diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index e5b367a..2cb07a3 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -207,7 +207,6 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) if (rc < 0) goto failed_marshal; - rpcrdma_post_recvs(r_xprt, true); if (rpcrdma_ep_post(&r_xprt->rx_ia, &r_xprt->rx_ep, req)) goto drop_connection; return 0; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 9f53e02..dc23977 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1312,11 +1312,6 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) u32 credits; __be32 *p; - --buf->rb_posted_receives; - - if (rep->rr_hdrbuf.head[0].iov_len == 0) - goto out_badstatus; - /* Fixed transport header fields */ xdr_init_decode(&rep->rr_stream, &rep->rr_hdrbuf, rep->rr_hdrbuf.head[0].iov_base); @@ -1361,31 +1356,21 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) clear_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags); trace_xprtrdma_reply(rqst->rq_task, rep, req, credits); - - rpcrdma_post_recvs(r_xprt, false); queue_work(rpcrdma_receive_wq, &rep->rr_work); return; out_badversion: trace_xprtrdma_reply_vers(rep); - goto repost; + goto out; -/* The RPC transaction has already been terminated, or the header - * is corrupt. - */ out_norqst: spin_unlock(&xprt->queue_lock); trace_xprtrdma_reply_rqst(rep); - goto repost; + goto out; out_shortreply: trace_xprtrdma_reply_short(rep); -/* If no pending RPC transaction was matched, post a replacement - * receive buffer before returning. - */ -repost: - rpcrdma_post_recvs(r_xprt, false); -out_badstatus: +out: rpcrdma_recv_buffer_put(rep); } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index b9bc7f9..e4461e7 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -78,6 +78,7 @@ static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); static int rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt, bool temp); static void rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb); +static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); struct workqueue_struct *rpcrdma_receive_wq __read_mostly; @@ -189,11 +190,13 @@ struct ib_cqe *cqe = wc->wr_cqe; struct rpcrdma_rep *rep = container_of(cqe, struct rpcrdma_rep, rr_cqe); + struct rpcrdma_xprt *r_xprt = rep->rr_rxprt; - /* WARNING: Only wr_id and status are reliable at this point */ + /* WARNING: Only wr_cqe and status are reliable at this point */ trace_xprtrdma_wc_receive(wc); + --r_xprt->rx_ep.rep_receive_count; if (wc->status != IB_WC_SUCCESS) - goto out_fail; + goto out_flushed; /* status == SUCCESS means all fields in wc are trustworthy */ rpcrdma_set_xdrlen(&rep->rr_hdrbuf, wc->byte_len); @@ -204,17 +207,16 @@ rdmab_addr(rep->rr_rdmabuf), wc->byte_len, DMA_FROM_DEVICE); -out_schedule: + rpcrdma_post_recvs(r_xprt, false); rpcrdma_reply_handler(rep); return; -out_fail: +out_flushed: if (wc->status != IB_WC_WR_FLUSH_ERR) pr_err("rpcrdma: Recv: %s (%u/0x%x)\n", ib_wc_status_msg(wc->status), wc->status, wc->vendor_err); - rpcrdma_set_xdrlen(&rep->rr_hdrbuf, 0); - goto out_schedule; + rpcrdma_recv_buffer_put(rep); } static void @@ -581,6 +583,7 @@ init_waitqueue_head(&ep->rep_connect_wait); INIT_DELAYED_WORK(&ep->rep_disconnect_worker, rpcrdma_disconnect_worker); + ep->rep_receive_count = 0; sendcq = ib_alloc_cq(ia->ri_device, NULL, ep->rep_attr.cap.max_send_wr + 1, @@ -1174,7 +1177,6 @@ struct rpcrdma_req * } buf->rb_credits = 1; - buf->rb_posted_receives = 0; INIT_LIST_HEAD(&buf->rb_recv_bufs); rc = rpcrdma_sendctxs_create(r_xprt); @@ -1511,25 +1513,20 @@ struct rpcrdma_regbuf * return 0; } -/** - * rpcrdma_post_recvs - Maybe post some Receive buffers - * @r_xprt: controlling transport - * @temp: when true, allocate temp rpcrdma_rep objects - * - */ -void +static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ep *ep = &r_xprt->rx_ep; struct ib_recv_wr *wr, *bad_wr; int needed, count, rc; rc = 0; count = 0; needed = buf->rb_credits + (buf->rb_bc_srv_max_requests << 1); - if (buf->rb_posted_receives > needed) + if (ep->rep_receive_count > needed) goto out; - needed -= buf->rb_posted_receives; + needed -= ep->rep_receive_count; count = 0; wr = NULL; @@ -1577,7 +1574,7 @@ struct rpcrdma_regbuf * --count; } } - buf->rb_posted_receives += count; + ep->rep_receive_count += count; out: trace_xprtrdma_post_recvs(r_xprt, count, rc); } diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index a13ccb6..788124c 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -102,6 +102,7 @@ struct rpcrdma_ep { struct rpcrdma_connect_private rep_cm_private; struct rdma_conn_param rep_remote_cma; struct delayed_work rep_disconnect_worker; + int rep_receive_count; }; /* Pre-allocate extra Work Requests for handling backward receives @@ -404,7 +405,6 @@ struct rpcrdma_buffer { unsigned long rb_flags; u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ - int rb_posted_receives; u32 rb_bc_srv_max_requests; spinlock_t rb_reqslock; /* protect rb_allreqs */ @@ -560,7 +560,6 @@ int rpcrdma_ep_create(struct rpcrdma_ep *, struct rpcrdma_ia *, int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *, struct rpcrdma_req *); -void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); /* * Buffer calls - xprtrdma/verbs.c From patchwork Mon Dec 17 16:39:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2A4AD13AD for ; Mon, 17 Dec 2018 16:39:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16BD72A1FC for ; Mon, 17 Dec 2018 16:39:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0AF972A200; Mon, 17 Dec 2018 16:39:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BC812A1F8 for ; Mon, 17 Dec 2018 16:39:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388128AbeLQQjq (ORCPT ); Mon, 17 Dec 2018 11:39:46 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:34671 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727771AbeLQQjp (ORCPT ); Mon, 17 Dec 2018 11:39:45 -0500 Received: by mail-it1-f193.google.com with SMTP id x124so5915383itd.1; Mon, 17 Dec 2018 08:39:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=HmItSEbpQ0nLYBpzJa2hliI2p7Ennn0e5DZHPjfdd58=; b=rR78ME9HGAIO4LSkN7R/rMtay+59VdVJOigXaIkxgH7m1BU3THH8HgeZykpxPZV3eJ ZUo0TDEWnuuzDYClMTJj9dBg13lbBXXY9BwZ7JPJnxhG+EicGopcuo+gGuylbBx2H6yd X1OdgtsYR34VmGaCCmFkm+WIRz4jYIHRiA1m6AyTFi09Y3G+3HiQuTdlaJsYm1aBMLSy JIl88okQz7c6aJsrwo+ip7cq/Enh20lBPsigSvmZYAWov/VHjAFAnmFv5fPl5P99TBXO ffb2QXJpUOtmQoSSbMwQzrvxqPT4Sy7UQXXlgQ8q6Axhwd9o1ZquAhYMIFjAD4lHSXcl 7aWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=HmItSEbpQ0nLYBpzJa2hliI2p7Ennn0e5DZHPjfdd58=; b=nY4FasTt5FcHj/lZIydG65RYm6iB188gGA35PkUoRydMGzDvuzvHCvfaXf7DqWrcFy ZVBRqfKgk02aFsYaoVlvTpER+TBYE3KikLE0rtpJcofcvtqlsm99/NXyTCEGGYBdzMkC 1tQJ5PzmM0Vt8HzUCpBk3U65dABjtoMXnWOjKqOaUTPbwO6Ijsd4TQs34Q/eWaWXzZU9 c8JBLJkLJCI/c/N8HGm43yfEPby4+D6nkfwLgmjvAfa2fCPQ7O54pkRbtcBmIvd8AlT4 mHxaTD+0renWlw7VvotH24H4Fca+PAiZt5oNsjk4O74/SJPL4Zagq3EQ+DnqbgQxJbRQ N8BA== X-Gm-Message-State: AA+aEWanTD8GydiTus5he5jT92sCJJK+2xhMsMgp+17V7E4sY8g9eTaA rqvNgzbUzg4MW86llyrcGz+n/Y8k X-Google-Smtp-Source: AFSGD/WYanoJznCxMQjE/LG9wVmGnLv+o/WZqU3o5iFvlzjSwh3FnGqckUWwGnyX1YQyrKhbpaGkZw== X-Received: by 2002:a24:2e94:: with SMTP id i142mr13747871ita.157.1545064784047; Mon, 17 Dec 2018 08:39:44 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id p19sm8754291itp.13.2018.12.17.08.39.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:43 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdge4018582; Mon, 17 Dec 2018 16:39:42 GMT Subject: [PATCH v4 04/30] xprtrdma: Replace rpcrdma_receive_wq with a per-xprt workqueue From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:42 -0500 Message-ID: <20181217163942.24133.66960.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To address a connection-close ordering problem, we need the ability to drain the RPC completions running on rpcrdma_receive_wq for just one transport. Give each transport its own RPC completion workqueue, and drain that workqueue when disconnecting the transport. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 2 + net/sunrpc/xprtrdma/transport.c | 17 +++------- net/sunrpc/xprtrdma/verbs.c | 67 +++++++++++++++++++++------------------ net/sunrpc/xprtrdma/xprt_rdma.h | 6 +-- 4 files changed, 44 insertions(+), 48 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index dc23977..5738c9f 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1356,7 +1356,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) clear_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags); trace_xprtrdma_reply(rqst->rq_task, rep, req, credits); - queue_work(rpcrdma_receive_wq, &rep->rr_work); + queue_work(buf->rb_completion_wq, &rep->rr_work); return; out_badversion: diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index ae2a838..91c476a 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -444,10 +444,14 @@ struct rpcrdma_ep *ep = &r_xprt->rx_ep; struct rpcrdma_ia *ia = &r_xprt->rx_ia; + might_sleep(); + dprintk("RPC: %s: closing xprt %p\n", __func__, xprt); + /* Prevent marshaling and sending of new requests */ + xprt_clear_connected(xprt); + if (test_and_clear_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags)) { - xprt_clear_connected(xprt); rpcrdma_ia_remove(ia); return; } @@ -858,8 +862,6 @@ void xprt_rdma_cleanup(void) dprintk("RPC: %s: xprt_unregister returned %i\n", __func__, rc); - rpcrdma_destroy_wq(); - rc = xprt_unregister_transport(&xprt_rdma_bc); if (rc) dprintk("RPC: %s: xprt_unregister(bc) returned %i\n", @@ -870,20 +872,13 @@ int xprt_rdma_init(void) { int rc; - rc = rpcrdma_alloc_wq(); - if (rc) - return rc; - rc = xprt_register_transport(&xprt_rdma); - if (rc) { - rpcrdma_destroy_wq(); + if (rc) return rc; - } rc = xprt_register_transport(&xprt_rdma_bc); if (rc) { xprt_unregister_transport(&xprt_rdma); - rpcrdma_destroy_wq(); return rc; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index e4461e7..cff3a5d 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -80,33 +80,23 @@ static void rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb); static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp); -struct workqueue_struct *rpcrdma_receive_wq __read_mostly; - -int -rpcrdma_alloc_wq(void) +/* Wait for outstanding transport work to finish. + */ +static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) { - struct workqueue_struct *recv_wq; - - recv_wq = alloc_workqueue("xprtrdma_receive", - WQ_MEM_RECLAIM | WQ_HIGHPRI, - 0); - if (!recv_wq) - return -ENOMEM; - - rpcrdma_receive_wq = recv_wq; - return 0; -} + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ia *ia = &r_xprt->rx_ia; -void -rpcrdma_destroy_wq(void) -{ - struct workqueue_struct *wq; + /* Flush Receives, then wait for deferred Reply work + * to complete. + */ + ib_drain_qp(ia->ri_id->qp); + drain_workqueue(buf->rb_completion_wq); - if (rpcrdma_receive_wq) { - wq = rpcrdma_receive_wq; - rpcrdma_receive_wq = NULL; - destroy_workqueue(wq); - } + /* Deferred Reply processing might have scheduled + * local invalidations. + */ + ib_drain_sq(ia->ri_id->qp); } /** @@ -483,7 +473,7 @@ * connection is already gone. */ if (ia->ri_id->qp) { - ib_drain_qp(ia->ri_id->qp); + rpcrdma_xprt_drain(r_xprt); rdma_destroy_qp(ia->ri_id); ia->ri_id->qp = NULL; } @@ -825,8 +815,10 @@ return rc; } -/* - * rpcrdma_ep_disconnect +/** + * rpcrdma_ep_disconnect - Disconnect underlying transport + * @ep: endpoint to disconnect + * @ia: associated interface adapter * * This is separate from destroy to facilitate the ability * to reconnect without recreating the endpoint. @@ -837,19 +829,20 @@ void rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { + struct rpcrdma_xprt *r_xprt = container_of(ep, struct rpcrdma_xprt, + rx_ep); int rc; + /* returns without wait if ID is not connected */ rc = rdma_disconnect(ia->ri_id); if (!rc) - /* returns without wait if not connected */ wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected != 1); else ep->rep_connected = rc; - trace_xprtrdma_disconnect(container_of(ep, struct rpcrdma_xprt, - rx_ep), rc); + trace_xprtrdma_disconnect(r_xprt, rc); - ib_drain_qp(ia->ri_id->qp); + rpcrdma_xprt_drain(r_xprt); } /* Fixed-size circular FIFO queue. This implementation is wait-free and @@ -1183,6 +1176,13 @@ struct rpcrdma_req * if (rc) goto out; + buf->rb_completion_wq = alloc_workqueue("rpcrdma-%s", + WQ_MEM_RECLAIM | WQ_HIGHPRI, + 0, + r_xprt->rx_xprt.address_strings[RPC_DISPLAY_ADDR]); + if (!buf->rb_completion_wq) + goto out; + return 0; out: rpcrdma_buffer_destroy(buf); @@ -1241,6 +1241,11 @@ struct rpcrdma_req * { cancel_delayed_work_sync(&buf->rb_refresh_worker); + if (buf->rb_completion_wq) { + destroy_workqueue(buf->rb_completion_wq); + buf->rb_completion_wq = NULL; + } + rpcrdma_sendctxs_destroy(buf); while (!list_empty(&buf->rb_recv_bufs)) { diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 788124c..3f198cd 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -412,6 +412,7 @@ struct rpcrdma_buffer { u32 rb_bc_max_requests; + struct workqueue_struct *rb_completion_wq; struct delayed_work rb_refresh_worker; }; #define rdmab_to_ia(b) (&container_of((b), struct rpcrdma_xprt, rx_buf)->rx_ia) @@ -547,8 +548,6 @@ struct rpcrdma_xprt { bool frwr_is_supported(struct rpcrdma_ia *); bool fmr_is_supported(struct rpcrdma_ia *); -extern struct workqueue_struct *rpcrdma_receive_wq; - /* * Endpoint calls - xprtrdma/verbs.c */ @@ -603,9 +602,6 @@ struct rpcrdma_regbuf *rpcrdma_alloc_regbuf(size_t, enum dma_data_direction, return __rpcrdma_dma_map_regbuf(ia, rb); } -int rpcrdma_alloc_wq(void); -void rpcrdma_destroy_wq(void); - /* * Wrappers for chunk registration, shared by read/write chunk code. */ From patchwork Mon Dec 17 16:39:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733915 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE4561575 for ; Mon, 17 Dec 2018 16:39:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF3B82A1F8 for ; Mon, 17 Dec 2018 16:39:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C39C42A206; Mon, 17 Dec 2018 16:39:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB3902A1F8 for ; Mon, 17 Dec 2018 16:39:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388184AbeLQQjv (ORCPT ); Mon, 17 Dec 2018 11:39:51 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:54384 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727771AbeLQQju (ORCPT ); Mon, 17 Dec 2018 11:39:50 -0500 Received: by mail-it1-f196.google.com with SMTP id i145so19459113ita.4; Mon, 17 Dec 2018 08:39:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=JrjAwoYZLatTrSpjWp1zdL2hQdofaBBegzEeSbKdEko=; b=O3JDD/6xlWimSEYIhZwyhswxm0oPgfkS56OJNAZ2cGmMv/t2Wj6LjRn3EFABiHjRpJ Xp3tM/6UiNdRbfsWvzLVd0U03SSsdk413bRFBE/xVy/jx6EO/WjAKdSXNtHk9TXZYrfp 5sm5wE7HHwWiIbv63A01prsSH+QBPoUru7Yd8cl1Ma60hva60oyyClwrcQ8G5qPaEKCq a5fv31X6//a0kHtXNQD80G9eXSlBfQMjLZHQ1ZCfTDfkaAmhPIU/pgqNuonr3i4ASiAL W0qo+7EaMKWDnTEXyDT7JhzwyShZ9TDSbPOgy6SKRAsFlh3qSr03Ptus70qjZWocJydc lXIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=JrjAwoYZLatTrSpjWp1zdL2hQdofaBBegzEeSbKdEko=; b=PQkR3YKW8JmYIbDCeENQU+SrF/lHVRyvCfZGraCYMpdShmEv4ws4S6bkq7aKnMlONZ QXpvrE7sMw8hSZx0eX1bGgY40cv+kQ56P+bPRSyKPgZkSl7/kmXATFWryBLtGRdWTP95 MTK0bz3V09YMJDCT5zyF3mxlrWRjvip4y5uHdtbv2c8eBtGK+le8sf/GefPK3W6g/mXt OnJF2KmU6WM2tNavTy4inqsQsZW9l7VSkqKRJt13cwleX6C209QCPkccm/PJXpil/0vi wan9G0h+t7pZUI/OHQy9ffEH765KdRT3vPFlD8rm4rmxt34bQTr7eMTc8dp+K117fwL7 h++g== X-Gm-Message-State: AA+aEWZ/x90vbiX8wXKRNp9Z/QDsVOsGdtIEarDLoPDAftjzSFGFVtvf 6mMWeqHofbhrD7c0GryNPNbrxY73 X-Google-Smtp-Source: AFSGD/VVhhJCaiT1uNGfozFftcn27cZ33d1FPDPLZtBgFtDjqxlSezq2YBDWv0zTzkJBDiT9CFaPpw== X-Received: by 2002:a02:85ae:: with SMTP id d43mr613221jai.70.1545064789555; Mon, 17 Dec 2018 08:39:49 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id w8sm4193680iom.26.2018.12.17.08.39.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:49 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdmnf018585; Mon, 17 Dec 2018 16:39:48 GMT Subject: [PATCH v4 05/30] xprtrdma: No qp_event disconnect From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:48 -0500 Message-ID: <20181217163948.24133.60342.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP After thinking about this more, and auditing other kernel ULP imple- mentations, I believe that a DISCONNECT cm_event will occur after a fatal QP event. If that's the case, there's no need for an explicit disconnect in the QP event handler. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 32 -------------------------------- net/sunrpc/xprtrdma/xprt_rdma.h | 1 - 2 files changed, 33 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index cff3a5d..9a0a765 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -100,25 +100,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) } /** - * rpcrdma_disconnect_worker - Force a disconnect - * @work: endpoint to be disconnected - * - * Provider callbacks can possibly run in an IRQ context. This function - * is invoked in a worker thread to guarantee that disconnect wake-up - * calls are always done in process context. - */ -static void -rpcrdma_disconnect_worker(struct work_struct *work) -{ - struct rpcrdma_ep *ep = container_of(work, struct rpcrdma_ep, - rep_disconnect_worker.work); - struct rpcrdma_xprt *r_xprt = - container_of(ep, struct rpcrdma_xprt, rx_ep); - - xprt_force_disconnect(&r_xprt->rx_xprt); -} - -/** * rpcrdma_qp_event_handler - Handle one QP event (error notification) * @event: details of the event * @context: ep that owns QP where event occurred @@ -134,15 +115,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) rx_ep); trace_xprtrdma_qp_event(r_xprt, event); - pr_err("rpcrdma: %s on device %s connected to %s:%s\n", - ib_event_msg(event->event), event->device->name, - rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt)); - - if (ep->rep_connected == 1) { - ep->rep_connected = -EIO; - schedule_delayed_work(&ep->rep_disconnect_worker, 0); - wake_up_all(&ep->rep_connect_wait); - } } /** @@ -571,8 +543,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) cdata->max_requests >> 2); ep->rep_send_count = ep->rep_send_batch; init_waitqueue_head(&ep->rep_connect_wait); - INIT_DELAYED_WORK(&ep->rep_disconnect_worker, - rpcrdma_disconnect_worker); ep->rep_receive_count = 0; sendcq = ib_alloc_cq(ia->ri_device, NULL, @@ -646,8 +616,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) void rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { - cancel_delayed_work_sync(&ep->rep_disconnect_worker); - if (ia->ri_id && ia->ri_id->qp) { rpcrdma_ep_disconnect(ep, ia); rdma_destroy_qp(ia->ri_id); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 3f198cd..7c1b519 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -101,7 +101,6 @@ struct rpcrdma_ep { wait_queue_head_t rep_connect_wait; struct rpcrdma_connect_private rep_cm_private; struct rdma_conn_param rep_remote_cma; - struct delayed_work rep_disconnect_worker; int rep_receive_count; }; From patchwork Mon Dec 17 16:39:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733921 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A67C41575 for ; Mon, 17 Dec 2018 16:39:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A0B72A202 for ; Mon, 17 Dec 2018 16:39:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8E4912A1FC; Mon, 17 Dec 2018 16:39:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F4742A1F9 for ; Mon, 17 Dec 2018 16:39:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388255AbeLQQj5 (ORCPT ); Mon, 17 Dec 2018 11:39:57 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:55345 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388108AbeLQQjz (ORCPT ); Mon, 17 Dec 2018 11:39:55 -0500 Received: by mail-it1-f193.google.com with SMTP id o19so19435612itg.5; Mon, 17 Dec 2018 08:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=gewXrx17EYSQ+INXkilt/dP9W8sTmtrpJiumbs2rzgg=; b=o8E3oGfOTZ8PmgZbBSUugucLJ/PSWwcEJrKzSvZWjwZa15Nh6bUGezyICUm+nS/mtV 4BZTw5LAfSgQ4c+5s/aiofOYIT99zA+TPTot/wKbbf4jSs5A+urOjwx70vw/GGSGcWe3 vaTmBIJZluHdJrM7619hcGfa9zjhkt7hYwb8hjU7KDbltZ3dCJA1OD0pVxg+nLU52Psv T4zuB9EwZaYt64zUqTFmImQTRSgSAWscs561TZVJaqFzFhUgrLr2+yb+wJf9ntiP0C9e BINCaAkEVojpPevPYkluLrV71aAWddQH23EPsh4hmdBRjXd7fhVAcYlWWzt8/KKntxiq tYYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=gewXrx17EYSQ+INXkilt/dP9W8sTmtrpJiumbs2rzgg=; b=rospSii5SFQZ7HTTaZPc0nMvg/UCzdta/7iYs1IHGE9rqB02lK/RukMeUczFYeDb9U C2JYTjCLORWuu7J9nhgPwWD7YcHq6eij4S24nINV/DMDDQPKrKtBMiMU7IRF5PoHidZG fTUi1LaRtG9WdxWd+qXkctWzzJKx8XXsSP+0S331R4FEc/EKCrsjyVRur/6FMEWoUN90 QT1vqjWRdJSAVWHSlPtcTB/djftONbTE9YOEXx82AOQ8PGE60fKuCDwbpMAyVNpNalkD dZ83BkrbjAZhuil+8f0H1YsqGl/jXJFB0LspwAzQErhJ9xQMwI0gf9/4kJV7nPKgdtjD 25AQ== X-Gm-Message-State: AA+aEWYHej7l+6dS2J4GOv99Klyl1hLtFuPqB4ulRk36Zc0Q7IFlXu/Y IIGTsn1vsoiDNHGe7v8jG39+wKT9 X-Google-Smtp-Source: AFSGD/V99uOr/sFBvtBLLRX95wA0FOMKRa++nbIc4Kzf3K98NoPca9MBCfgm6uAidbZs87eDmj5cCQ== X-Received: by 2002:a24:f14d:: with SMTP id q13mr11849470iti.166.1545064794631; Mon, 17 Dec 2018 08:39:54 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id s15sm6091131ioe.52.2018.12.17.08.39.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:54 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdrr8018588; Mon, 17 Dec 2018 16:39:53 GMT Subject: [PATCH v4 06/30] xprtrdma: Don't wake pending tasks until disconnect is done From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:53 -0500 Message-ID: <20181217163953.24133.29214.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Transport disconnect processing does a "wake pending tasks" at various points. Suppose an RPC Reply is being processed. The RPC task that Reply goes with is waiting on the pending queue. If a disconnect wake-up happens before reply processing is done, that reply, even if it is good, is thrown away, and the RPC has to be sent again. This window apparently does not exist for socket transports because there is a lock held while a reply is being received which prevents the wake-up call until after reply processing is done. To resolve this, all RPC replies being processed on an RPC-over-RDMA transport have to complete before pending tasks are awoken due to a transport disconnect. Callers that already hold the transport write lock may invoke ->ops->close directly. Others use a generic helper that schedules a close when the write lock can be taken safely. Signed-off-by: Chuck Lever --- include/linux/sunrpc/xprt.h | 1 + net/sunrpc/xprt.c | 19 +++++++++++++++++++ net/sunrpc/xprtrdma/backchannel.c | 13 +++++++------ net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 8 +++++--- net/sunrpc/xprtrdma/transport.c | 16 ++++++++++------ net/sunrpc/xprtrdma/verbs.c | 5 ++--- 6 files changed, 44 insertions(+), 18 deletions(-) diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h index a4ab4f8..ee94ed0 100644 --- a/include/linux/sunrpc/xprt.h +++ b/include/linux/sunrpc/xprt.h @@ -401,6 +401,7 @@ static inline __be32 *xprt_skip_transport_header(struct rpc_xprt *xprt, __be32 * bool xprt_request_get_cong(struct rpc_xprt *xprt, struct rpc_rqst *req); void xprt_disconnect_done(struct rpc_xprt *xprt); void xprt_force_disconnect(struct rpc_xprt *xprt); +void xprt_disconnect_nowake(struct rpc_xprt *xprt); void xprt_conditional_disconnect(struct rpc_xprt *xprt, unsigned int cookie); bool xprt_lock_connect(struct rpc_xprt *, struct rpc_task *, void *); diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c index ce92700..afe412e 100644 --- a/net/sunrpc/xprt.c +++ b/net/sunrpc/xprt.c @@ -685,6 +685,25 @@ void xprt_force_disconnect(struct rpc_xprt *xprt) } EXPORT_SYMBOL_GPL(xprt_force_disconnect); +/** + * xprt_disconnect_nowake - force a call to xprt->ops->close + * @xprt: transport to disconnect + * + * The caller must ensure that xprt_wake_pending_tasks() is + * called later. + */ +void xprt_disconnect_nowake(struct rpc_xprt *xprt) +{ + /* Don't race with the test_bit() in xprt_clear_locked() */ + spin_lock_bh(&xprt->transport_lock); + set_bit(XPRT_CLOSE_WAIT, &xprt->state); + /* Try to schedule an autoclose RPC call */ + if (test_and_set_bit(XPRT_LOCKED, &xprt->state) == 0) + queue_work(xprtiod_workqueue, &xprt->task_cleanup); + spin_unlock_bh(&xprt->transport_lock); +} +EXPORT_SYMBOL_GPL(xprt_disconnect_nowake); + static unsigned int xprt_connect_cookie(struct rpc_xprt *xprt) { diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 2cb07a3..5d462e8 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -193,14 +193,15 @@ static int rpcrdma_bc_marshal_reply(struct rpc_rqst *rqst) */ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) { - struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(rqst->rq_xprt); + struct rpc_xprt *xprt = rqst->rq_xprt; + struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); struct rpcrdma_req *req = rpcr_to_rdmar(rqst); int rc; - if (!xprt_connected(rqst->rq_xprt)) - goto drop_connection; + if (!xprt_connected(xprt)) + return -ENOTCONN; - if (!xprt_request_get_cong(rqst->rq_xprt, rqst)) + if (!xprt_request_get_cong(xprt, rqst)) return -EBADSLT; rc = rpcrdma_bc_marshal_reply(rqst); @@ -215,7 +216,7 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) if (rc != -ENOTCONN) return rc; drop_connection: - xprt_disconnect_done(rqst->rq_xprt); + xprt->ops->close(xprt); return -ENOTCONN; } @@ -338,7 +339,7 @@ void rpcrdma_bc_receive_call(struct rpcrdma_xprt *r_xprt, out_overflow: pr_warn("RPC/RDMA backchannel overflow\n"); - xprt_disconnect_done(xprt); + xprt_disconnect_nowake(xprt); /* This receive buffer gets reposted automatically * when the connection is re-established. */ diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c index f3c147d..b908f2c 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c +++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c @@ -200,11 +200,10 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, svc_rdma_send_ctxt_put(rdma, ctxt); goto drop_connection; } - return rc; + return 0; drop_connection: dprintk("svcrdma: failed to send bc call\n"); - xprt_disconnect_done(xprt); return -ENOTCONN; } @@ -225,8 +224,11 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, ret = -ENOTCONN; rdma = container_of(sxprt, struct svcxprt_rdma, sc_xprt); - if (!test_bit(XPT_DEAD, &sxprt->xpt_flags)) + if (!test_bit(XPT_DEAD, &sxprt->xpt_flags)) { ret = rpcrdma_bc_send_request(rdma, rqst); + if (ret == -ENOTCONN) + svc_close_xprt(sxprt); + } mutex_unlock(&sxprt->xpt_mutex); diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 91c476a..a16296b 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -453,13 +453,13 @@ if (test_and_clear_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags)) { rpcrdma_ia_remove(ia); - return; + goto out; } + if (ep->rep_connected == -ENODEV) return; if (ep->rep_connected > 0) xprt->reestablish_timeout = 0; - xprt_disconnect_done(xprt); rpcrdma_ep_disconnect(ep, ia); /* Prepare @xprt for the next connection by reinitializing @@ -467,6 +467,10 @@ */ r_xprt->rx_buf.rb_credits = 1; xprt->cwnd = RPC_CWNDSHIFT; + +out: + ++xprt->connect_cookie; + xprt_disconnect_done(xprt); } /** @@ -515,7 +519,7 @@ static void xprt_rdma_timer(struct rpc_xprt *xprt, struct rpc_task *task) { - xprt_force_disconnect(xprt); + xprt_disconnect_nowake(xprt); } /** @@ -717,7 +721,7 @@ #endif /* CONFIG_SUNRPC_BACKCHANNEL */ if (!xprt_connected(xprt)) - goto drop_connection; + return -ENOTCONN; if (!xprt_request_get_cong(xprt, rqst)) return -EBADSLT; @@ -749,8 +753,8 @@ if (rc != -ENOTCONN) return rc; drop_connection: - xprt_disconnect_done(xprt); - return -ENOTCONN; /* implies disconnect */ + xprt_rdma_close(xprt); + return -ENOTCONN; } void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 9a0a765..38a757c 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -252,7 +252,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) #endif set_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags); ep->rep_connected = -ENODEV; - xprt_force_disconnect(xprt); + xprt_disconnect_nowake(xprt); wait_for_completion(&ia->ri_remove_done); ia->ri_id = NULL; @@ -280,10 +280,9 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) ep->rep_connected = -EAGAIN; goto disconnected; case RDMA_CM_EVENT_DISCONNECTED: - ++xprt->connect_cookie; ep->rep_connected = -ECONNABORTED; disconnected: - xprt_force_disconnect(xprt); + xprt_disconnect_nowake(xprt); wake_up_all(&ep->rep_connect_wait); break; default: From patchwork Mon Dec 17 16:39:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C007B1575 for ; Mon, 17 Dec 2018 16:40:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEBE62A1F8 for ; Mon, 17 Dec 2018 16:40:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A3ABA2A1F9; Mon, 17 Dec 2018 16:40:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BEC12A1F8 for ; Mon, 17 Dec 2018 16:40:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388146AbeLQQkB (ORCPT ); Mon, 17 Dec 2018 11:40:01 -0500 Received: from mail-io1-f65.google.com ([209.85.166.65]:39072 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388108AbeLQQkB (ORCPT ); Mon, 17 Dec 2018 11:40:01 -0500 Received: by mail-io1-f65.google.com with SMTP id k7so10454159iob.6; Mon, 17 Dec 2018 08:40:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=zFE3Q/AHFqEM9K7QikRDZ+AdCIpLEYyC9xdEuneC6j0=; b=XQZVUo/EN+Minx7F0vYtjmrixoM5N3G8N6fsc69UDgK+jKFyw3UU8wibAbWcsDW6wg amoA8h4Vj7akv/UHajrNXkUOb1U46CCHJ2F8HbmnN2pe0aqSgGheKWTB/B9IrKCRU9Ro N4AYfOi12JXB+xz6lolp6tRVgxdys69c9Ckbktn/6aXC0/KPSM5EDVhQqtheLTYP9uFk 5SlS0kk50C/J6Y7YXgI/KGlbxFRp5IhZ+5Hv1qQ5gd4uHXwFHwTsmehcU0xISBOJ5kle aeKyuAVArUS2UN06mWdH7Pge/+Ybv49SZQDe2d7DbeXSZTRXU9yGVCT7CkhBhW3sLYIg lKXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=zFE3Q/AHFqEM9K7QikRDZ+AdCIpLEYyC9xdEuneC6j0=; b=ESVPKqvZ48yW1gjIzQCyEbSZXhjuggRoMDdq72YvA6EHcsCn5tayV+YgykQSy5AFMk 6TRxGv8PPIFuCK3qThrsirzLYmxiXDJvCMIGYUYbDmv9CW/HaxiBeUpI66vQAN4WjbqV H9JNz6M94nu9eNU76reT+5evhUG4pefDASyZuxSEheriptldgLE52xxySmyNghN24Kv0 B8SYWdHu8mDPBSIgKBOEj38COoHMhMMjFEAtpUlQmAJDrZilY+pYUvfZ2CDtPaUWBdtM 9eXrJiFFpG7lPpA2ZrTBKZ8Bke2GtfqGRXwrWvOJzDkFn3DQXR5MGAiuXEbMW8sJsCpv IdBg== X-Gm-Message-State: AA+aEWbUQR62KXqACEr3nXfHod2dv4BjEXwOIJkE5yTsY8U59x59ar1g Ya6ehNRLeOaVjeJ0e4JrxcWFNsE4 X-Google-Smtp-Source: AFSGD/U5VMQExksmYMOO+S5W5M4uSArPVNTriwLXODgjuo6WQAiR6GPB8Vp66T0KYrNdD8ipk+OFSw== X-Received: by 2002:a6b:8d8a:: with SMTP id p132mr12166619iod.290.1545064800159; Mon, 17 Dec 2018 08:40:00 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id w3sm4173836ior.28.2018.12.17.08.39.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:39:59 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGdwSU018591; Mon, 17 Dec 2018 16:39:58 GMT Subject: [PATCH v4 07/30] xprtrdma: Fix ri_max_segs and the result of ro_maxpages From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:39:58 -0500 Message-ID: <20181217163958.24133.16923.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With certain combinations of krb5i/p, MR size, and r/wsize, I/O can fail with EMSGSIZE. This is because the calculated value of ri_max_segs (the max number of MRs per RPC) exceeded RPCRDMA_MAX_HDR_SEGS, which caused Read or Write list encoding to walk off the end of the transport header. Once that was addressed, the ro_maxpages result has to be corrected to account for the number of MRs needed for Reply chunks, which is 2 MRs smaller than a normal Read or Write chunk. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/fmr_ops.c | 7 +++++-- net/sunrpc/xprtrdma/frwr_ops.c | 7 +++++-- net/sunrpc/xprtrdma/transport.c | 6 ++++-- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/net/sunrpc/xprtrdma/fmr_ops.c b/net/sunrpc/xprtrdma/fmr_ops.c index 7f5632c..78a0224 100644 --- a/net/sunrpc/xprtrdma/fmr_ops.c +++ b/net/sunrpc/xprtrdma/fmr_ops.c @@ -176,7 +176,10 @@ enum { ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / RPCRDMA_MAX_FMR_SGES); - ia->ri_max_segs += 2; /* segments for head and tail buffers */ + /* Reply chunks require segments for head and tail buffers */ + ia->ri_max_segs += 2; + if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) + ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS; return 0; } @@ -186,7 +189,7 @@ enum { fmr_op_maxpages(struct rpcrdma_xprt *r_xprt) { return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - RPCRDMA_MAX_HDR_SEGS * RPCRDMA_MAX_FMR_SGES); + (ia->ri_max_segs - 2) * RPCRDMA_MAX_FMR_SGES); } /* Use the ib_map_phys_fmr() verb to register a memory region diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 27222c0..f587e44 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -244,7 +244,10 @@ ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / ia->ri_max_frwr_depth); - ia->ri_max_segs += 2; /* segments for head and tail buffers */ + /* Reply chunks require segments for head and tail buffers */ + ia->ri_max_segs += 2; + if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) + ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS; return 0; } @@ -257,7 +260,7 @@ struct rpcrdma_ia *ia = &r_xprt->rx_ia; return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - RPCRDMA_MAX_HDR_SEGS * ia->ri_max_frwr_depth); + (ia->ri_max_segs - 2) * ia->ri_max_frwr_depth); } static void diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index a16296b..fbb14bf 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -704,8 +704,10 @@ * %-ENOTCONN if the caller should reconnect and call again * %-EAGAIN if the caller should call again * %-ENOBUFS if the caller should call again after a delay - * %-EIO if a permanent error occurred and the request was not - * sent. Do not try to send this message again. + * %-EMSGSIZE if encoding ran out of buffer space. The request + * was not sent. Do not try to send this message again. + * %-EIO if an I/O error occurred. The request was not sent. + * Do not try to send this message again. */ static int xprt_rdma_send_request(struct rpc_rqst *rqst) From patchwork Mon Dec 17 16:40:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733929 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D9E5E161F for ; Mon, 17 Dec 2018 16:40:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBE302A1F6 for ; Mon, 17 Dec 2018 16:40:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C02F42A20B; Mon, 17 Dec 2018 16:40:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7301D2A1F6 for ; Mon, 17 Dec 2018 16:40:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388263AbeLQQkH (ORCPT ); Mon, 17 Dec 2018 11:40:07 -0500 Received: from mail-io1-f68.google.com ([209.85.166.68]:42439 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388108AbeLQQkG (ORCPT ); Mon, 17 Dec 2018 11:40:06 -0500 Received: by mail-io1-f68.google.com with SMTP id x6so10450074ioa.9; Mon, 17 Dec 2018 08:40:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=fZt2sJtHBzkz3cxTvQxkd76ytJ2+vEepYSr4HQtV1IE=; b=EPyE7XPVhbv4ZEcl02tSgznq7tnu2gPrag8Dx0BDasZXbprzCoUkldrSKYv0wkyAaL WW3fVlkVoIYGdA1c/CjcO76S89DXnJUQbsfMnwnLeL6VL+tyyV5W3N7BykGyM72iglcv mgSpzBIIOm00DUBfNdsDJtc1NChtEAJFR5SsPXnNSVq3L77J85MO6sp3YCn80PVj9xDf 9ZxNJtpvUSOyd4oit1HWPFPbWoNsvua1euWxgVg4tOgkRcFQbIUxaqgT2qwH2mJ1ZsB+ HI6seIanCU3/zwwGkPHCF2zSnd0OHmBJQQYVGc9u7/wLukzosxMvmNtuMGOk6/qwsnh0 2phQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=fZt2sJtHBzkz3cxTvQxkd76ytJ2+vEepYSr4HQtV1IE=; b=SQbvKyyXAH0M14elpP9TEHKAVznscKwVUGCb86S9W4ZyuSupg396FR2wYA7exatwK4 PIWYmF+CdEiRhEUPcdHCethNcr4UTfA0MQJEGz1rSv3NdioUvdxeHQbzI5cSpwXdWWj+ fxsT+f0YYh9IH4KBHr0QqaHILPCqYlNjZNRNTxEi2iswibbsJEcLxtw3C7+yS+Kg7dyx HxpIXuVoIJMe8LJWYuRQQVHhaxOQ/DH/5dSh12+/KJ78rr77YEy3HQWCUK/hRYky98TR GHOQnvvq3pQ0WpcvCw4nxKLYGZuEcCKn5dXkyyabFibEgfv0CfQqB2MonpkVSxEhWxpi jzvg== X-Gm-Message-State: AA+aEWbp+eTdlUSclLTY6uojzlTV2LP1zkCkS6V4i9WrkIVksfubVkfe fM/ZVWs7Xj+3bZZXh0LVzsxRFU32 X-Google-Smtp-Source: AFSGD/X+zmFU3r6RDu9TjmyTfjZmzr31UrqtDd4ZrQbLnaVqxLw2TMchvMhSQxQgd5a055IXyKJtMg== X-Received: by 2002:a6b:b8c3:: with SMTP id i186mr11006380iof.8.1545064805328; Mon, 17 Dec 2018 08:40:05 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id j1sm4953119iok.79.2018.12.17.08.40.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:04 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGe4bA018602; Mon, 17 Dec 2018 16:40:04 GMT Subject: [PATCH v4 08/30] xprtrdma: Reduce max_frwr_depth From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:04 -0500 Message-ID: <20181217164003.24133.23440.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some devices advertise a large max_fast_reg_page_list_len capability, but perform optimally when MRs are significantly smaller than that depth -- probably when the MR itself is no larger than a page. By default, the RDMA R/W core API uses max_sge_rd as the maximum page depth for MRs. For some devices, the value of max_sge_rd is 1, which is also not optimal. Thus, when max_sge_rd is larger than 1, use that value. Otherwise use the value of the max_fast_reg_page_list_len attribute. I've tested this with CX-3 Pro, FastLinq, and CX-5 devices. It reproducibly improves the throughput of large I/Os by several percent. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index f587e44..16976b0 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -193,10 +193,17 @@ if (attrs->device_cap_flags & IB_DEVICE_SG_GAPS_REG) ia->ri_mrtype = IB_MR_TYPE_SG_GAPS; - ia->ri_max_frwr_depth = - min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - attrs->max_fast_reg_page_list_len); - dprintk("RPC: %s: device's max FR page list len = %u\n", + /* Quirk: Some devices advertise a large max_fast_reg_page_list_len + * capability, but perform optimally when the MRs are not larger + * than a page. + */ + if (attrs->max_sge_rd > 1) + ia->ri_max_frwr_depth = attrs->max_sge_rd; + else + ia->ri_max_frwr_depth = attrs->max_fast_reg_page_list_len; + if (ia->ri_max_frwr_depth > RPCRDMA_MAX_DATA_SEGS) + ia->ri_max_frwr_depth = RPCRDMA_MAX_DATA_SEGS; + dprintk("RPC: %s: max FR page list depth = %u\n", __func__, ia->ri_max_frwr_depth); /* Add room for frwr register and invalidate WRs. From patchwork Mon Dec 17 16:40:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733931 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ABE2613AD for ; Mon, 17 Dec 2018 16:40:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 962762A225 for ; Mon, 17 Dec 2018 16:40:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 885982A226; Mon, 17 Dec 2018 16:40:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 94F712A1F9 for ; Mon, 17 Dec 2018 16:40:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388269AbeLQQkN (ORCPT ); Mon, 17 Dec 2018 11:40:13 -0500 Received: from mail-io1-f52.google.com ([209.85.166.52]:43622 "EHLO mail-io1-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388108AbeLQQkM (ORCPT ); Mon, 17 Dec 2018 11:40:12 -0500 Received: by mail-io1-f52.google.com with SMTP id l3so10437258ioc.10; Mon, 17 Dec 2018 08:40:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Dg+whjZAs2Icbyk5ZQI3ii1G3YobxUwf7YswtT07HvI=; b=OUUQfBsw5IKvl/z95eAFAOs3ipArRiJlM6kOrXlw/xh1GvopcaAiP9t0xRFEQBb+2o pcYLMGUOs93saNSORRMCDKkzMqlyPXaXq+usvl3HzKgnpLUG6NEYnmKh8RrEyTLc7gcz TS0kWExdbqA/X1bY+AS19hgS9D8oW1ggnIUhT1P07dxcdemCXSNN3dH9X6JayxA0rOKr fghttavlMegGdw7e8k7fTZulyc5P9zP5Jg5rCOgWGIXDFfTClbET0XZWIZysOUcZm1j9 eWGnGyUVVm7yxkgTWaHxcE2Otmg3ODT6y4PgSPYoN0jrY8k59quXVZ9QgfVvv4GX7MRv C3pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Dg+whjZAs2Icbyk5ZQI3ii1G3YobxUwf7YswtT07HvI=; b=InjgN/+DBS/2H1iwTP2pUyAdjgpytIZQrm8FCAFuGEAekzch+GGLGL1xnTtOnABlei Vng8QREbKn1LTGb6DIb3xFRymX3lmsdQqrN6jqXst8br6zGXY3RIBNQjHClC7B9tJSA5 gkDyTavs0E6owDln6p83EMgoGoynEa2NL3dOcBx1M70nPWw/XklPziEm6txZSmeuA/zP 7VDPumie+PoI7KMa57xkPicSVN1/rtH0kLlg7yKwDQ0A4NRfT5EQEM6SkhyFc2AlB5Q3 jaE6E8KFPJEL2ddL5S79TFBx6uoUh2BmpgBE8TgE9w0GQ/ZvVRC+wC8eqT17WC4RgKCQ d+uw== X-Gm-Message-State: AA+aEWadjer3FejB2spfIrKEL/Oh3TWCsH/VAAwo6KfRHkB8RKA6GX4N aCWnyswZPX1+gh6XAQUyvRu/yHuP X-Google-Smtp-Source: AFSGD/Vc1ZC5xAurywRKc2Z8GrELMBBq+44SFtbM3qqjSTZkdEQ5mmO7gU+wUm5FdNSbYY1zUoI1Mw== X-Received: by 2002:a6b:f404:: with SMTP id i4mr10513839iog.281.1545064810814; Mon, 17 Dec 2018 08:40:10 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id u25sm5552536iob.23.2018.12.17.08.40.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:10 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGe9V8018605; Mon, 17 Dec 2018 16:40:09 GMT Subject: [PATCH v4 09/30] xprtrdma: Remove support for FMR memory registration From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:09 -0500 Message-ID: <20181217164009.24133.51523.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP FMR is not supported on most recent RDMA devices. It is also less secure than FRWR because an FMR memory registration can expose adjacent bytes to remote reading or writing. As discussed during the RDMA BoF at LPC 2018, it is time to remove support for FMR in the NFS/RDMA client stack. Note that NFS/RDMA server-side uses either local memory registration or FRWR. FMR is not used. There are a few Infiniband/RoCE devices in the kernel tree that do not appear to support MEM_MGT_EXTENSIONS (FRWR), and therefore will not support client-side NFS/RDMA after this patch. These are: - mthca - qib - hns (RoCE) Users of these devices can use NFS/TCP on IPoIB instead. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/Makefile | 3 net/sunrpc/xprtrdma/fmr_ops.c | 340 --------------------------------------- net/sunrpc/xprtrdma/verbs.c | 6 - net/sunrpc/xprtrdma/xprt_rdma.h | 12 - 4 files changed, 2 insertions(+), 359 deletions(-) delete mode 100644 net/sunrpc/xprtrdma/fmr_ops.c diff --git a/net/sunrpc/xprtrdma/Makefile b/net/sunrpc/xprtrdma/Makefile index 8bf19e1..8ed0377 100644 --- a/net/sunrpc/xprtrdma/Makefile +++ b/net/sunrpc/xprtrdma/Makefile @@ -1,8 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_SUNRPC_XPRT_RDMA) += rpcrdma.o -rpcrdma-y := transport.o rpc_rdma.o verbs.o \ - fmr_ops.o frwr_ops.o \ +rpcrdma-y := transport.o rpc_rdma.o verbs.o frwr_ops.o \ svc_rdma.o svc_rdma_backchannel.o svc_rdma_transport.o \ svc_rdma_sendto.o svc_rdma_recvfrom.o svc_rdma_rw.o \ module.o diff --git a/net/sunrpc/xprtrdma/fmr_ops.c b/net/sunrpc/xprtrdma/fmr_ops.c deleted file mode 100644 index 78a0224..0000000 --- a/net/sunrpc/xprtrdma/fmr_ops.c +++ /dev/null @@ -1,340 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Copyright (c) 2015, 2017 Oracle. All rights reserved. - * Copyright (c) 2003-2007 Network Appliance, Inc. All rights reserved. - */ - -/* Lightweight memory registration using Fast Memory Regions (FMR). - * Referred to sometimes as MTHCAFMR mode. - * - * FMR uses synchronous memory registration and deregistration. - * FMR registration is known to be fast, but FMR deregistration - * can take tens of usecs to complete. - */ - -/* Normal operation - * - * A Memory Region is prepared for RDMA READ or WRITE using the - * ib_map_phys_fmr verb (fmr_op_map). When the RDMA operation is - * finished, the Memory Region is unmapped using the ib_unmap_fmr - * verb (fmr_op_unmap). - */ - -#include - -#include "xprt_rdma.h" -#include - -#if IS_ENABLED(CONFIG_SUNRPC_DEBUG) -# define RPCDBG_FACILITY RPCDBG_TRANS -#endif - -/* Maximum scatter/gather per FMR */ -#define RPCRDMA_MAX_FMR_SGES (64) - -/* Access mode of externally registered pages */ -enum { - RPCRDMA_FMR_ACCESS_FLAGS = IB_ACCESS_REMOTE_WRITE | - IB_ACCESS_REMOTE_READ, -}; - -bool -fmr_is_supported(struct rpcrdma_ia *ia) -{ - if (!ia->ri_device->alloc_fmr) { - pr_info("rpcrdma: 'fmr' mode is not supported by device %s\n", - ia->ri_device->name); - return false; - } - return true; -} - -static void -__fmr_unmap(struct rpcrdma_mr *mr) -{ - LIST_HEAD(l); - int rc; - - list_add(&mr->fmr.fm_mr->list, &l); - rc = ib_unmap_fmr(&l); - list_del(&mr->fmr.fm_mr->list); - if (rc) - pr_err("rpcrdma: final ib_unmap_fmr for %p failed %i\n", - mr, rc); -} - -/* Release an MR. - */ -static void -fmr_op_release_mr(struct rpcrdma_mr *mr) -{ - int rc; - - kfree(mr->fmr.fm_physaddrs); - kfree(mr->mr_sg); - - /* In case this one was left mapped, try to unmap it - * to prevent dealloc_fmr from failing with EBUSY - */ - __fmr_unmap(mr); - - rc = ib_dealloc_fmr(mr->fmr.fm_mr); - if (rc) - pr_err("rpcrdma: final ib_dealloc_fmr for %p returned %i\n", - mr, rc); - - kfree(mr); -} - -/* MRs are dynamically allocated, so simply clean up and release the MR. - * A replacement MR will subsequently be allocated on demand. - */ -static void -fmr_mr_recycle_worker(struct work_struct *work) -{ - struct rpcrdma_mr *mr = container_of(work, struct rpcrdma_mr, mr_recycle); - struct rpcrdma_xprt *r_xprt = mr->mr_xprt; - - trace_xprtrdma_mr_recycle(mr); - - trace_xprtrdma_mr_unmap(mr); - ib_dma_unmap_sg(r_xprt->rx_ia.ri_device, - mr->mr_sg, mr->mr_nents, mr->mr_dir); - - spin_lock(&r_xprt->rx_buf.rb_mrlock); - list_del(&mr->mr_all); - r_xprt->rx_stats.mrs_recycled++; - spin_unlock(&r_xprt->rx_buf.rb_mrlock); - fmr_op_release_mr(mr); -} - -static int -fmr_op_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) -{ - static struct ib_fmr_attr fmr_attr = { - .max_pages = RPCRDMA_MAX_FMR_SGES, - .max_maps = 1, - .page_shift = PAGE_SHIFT - }; - - mr->fmr.fm_physaddrs = kcalloc(RPCRDMA_MAX_FMR_SGES, - sizeof(u64), GFP_KERNEL); - if (!mr->fmr.fm_physaddrs) - goto out_free; - - mr->mr_sg = kcalloc(RPCRDMA_MAX_FMR_SGES, - sizeof(*mr->mr_sg), GFP_KERNEL); - if (!mr->mr_sg) - goto out_free; - - sg_init_table(mr->mr_sg, RPCRDMA_MAX_FMR_SGES); - - mr->fmr.fm_mr = ib_alloc_fmr(ia->ri_pd, RPCRDMA_FMR_ACCESS_FLAGS, - &fmr_attr); - if (IS_ERR(mr->fmr.fm_mr)) - goto out_fmr_err; - - INIT_LIST_HEAD(&mr->mr_list); - INIT_WORK(&mr->mr_recycle, fmr_mr_recycle_worker); - return 0; - -out_fmr_err: - dprintk("RPC: %s: ib_alloc_fmr returned %ld\n", __func__, - PTR_ERR(mr->fmr.fm_mr)); - -out_free: - kfree(mr->mr_sg); - kfree(mr->fmr.fm_physaddrs); - return -ENOMEM; -} - -/* On success, sets: - * ep->rep_attr.cap.max_send_wr - * ep->rep_attr.cap.max_recv_wr - * cdata->max_requests - * ia->ri_max_segs - */ -static int -fmr_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, - struct rpcrdma_create_data_internal *cdata) -{ - int max_qp_wr; - - max_qp_wr = ia->ri_device->attrs.max_qp_wr; - max_qp_wr -= RPCRDMA_BACKWARD_WRS; - max_qp_wr -= 1; - if (max_qp_wr < RPCRDMA_MIN_SLOT_TABLE) - return -ENOMEM; - if (cdata->max_requests > max_qp_wr) - cdata->max_requests = max_qp_wr; - ep->rep_attr.cap.max_send_wr = cdata->max_requests; - ep->rep_attr.cap.max_send_wr += RPCRDMA_BACKWARD_WRS; - ep->rep_attr.cap.max_send_wr += 1; /* for ib_drain_sq */ - ep->rep_attr.cap.max_recv_wr = cdata->max_requests; - ep->rep_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS; - ep->rep_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */ - - ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / - RPCRDMA_MAX_FMR_SGES); - /* Reply chunks require segments for head and tail buffers */ - ia->ri_max_segs += 2; - if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) - ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS; - return 0; -} - -/* FMR mode conveys up to 64 pages of payload per chunk segment. - */ -static size_t -fmr_op_maxpages(struct rpcrdma_xprt *r_xprt) -{ - return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS, - (ia->ri_max_segs - 2) * RPCRDMA_MAX_FMR_SGES); -} - -/* Use the ib_map_phys_fmr() verb to register a memory region - * for remote access via RDMA READ or RDMA WRITE. - */ -static struct rpcrdma_mr_seg * -fmr_op_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, struct rpcrdma_mr **out) -{ - struct rpcrdma_mr_seg *seg1 = seg; - int len, pageoff, i, rc; - struct rpcrdma_mr *mr; - u64 *dma_pages; - - mr = rpcrdma_mr_get(r_xprt); - if (!mr) - return ERR_PTR(-EAGAIN); - - pageoff = offset_in_page(seg1->mr_offset); - seg1->mr_offset -= pageoff; /* start of page */ - seg1->mr_len += pageoff; - len = -pageoff; - if (nsegs > RPCRDMA_MAX_FMR_SGES) - nsegs = RPCRDMA_MAX_FMR_SGES; - for (i = 0; i < nsegs;) { - if (seg->mr_page) - sg_set_page(&mr->mr_sg[i], - seg->mr_page, - seg->mr_len, - offset_in_page(seg->mr_offset)); - else - sg_set_buf(&mr->mr_sg[i], seg->mr_offset, - seg->mr_len); - len += seg->mr_len; - ++seg; - ++i; - /* Check for holes */ - if ((i < nsegs && offset_in_page(seg->mr_offset)) || - offset_in_page((seg-1)->mr_offset + (seg-1)->mr_len)) - break; - } - mr->mr_dir = rpcrdma_data_dir(writing); - - mr->mr_nents = ib_dma_map_sg(r_xprt->rx_ia.ri_device, - mr->mr_sg, i, mr->mr_dir); - if (!mr->mr_nents) - goto out_dmamap_err; - trace_xprtrdma_mr_map(mr); - - for (i = 0, dma_pages = mr->fmr.fm_physaddrs; i < mr->mr_nents; i++) - dma_pages[i] = sg_dma_address(&mr->mr_sg[i]); - rc = ib_map_phys_fmr(mr->fmr.fm_mr, dma_pages, mr->mr_nents, - dma_pages[0]); - if (rc) - goto out_maperr; - - mr->mr_handle = mr->fmr.fm_mr->rkey; - mr->mr_length = len; - mr->mr_offset = dma_pages[0] + pageoff; - - *out = mr; - return seg; - -out_dmamap_err: - pr_err("rpcrdma: failed to DMA map sg %p sg_nents %d\n", - mr->mr_sg, i); - rpcrdma_mr_put(mr); - return ERR_PTR(-EIO); - -out_maperr: - pr_err("rpcrdma: ib_map_phys_fmr %u@0x%llx+%i (%d) status %i\n", - len, (unsigned long long)dma_pages[0], - pageoff, mr->mr_nents, rc); - rpcrdma_mr_unmap_and_put(mr); - return ERR_PTR(-EIO); -} - -/* Post Send WR containing the RPC Call message. - */ -static int -fmr_op_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) -{ - return ib_post_send(ia->ri_id->qp, &req->rl_sendctx->sc_wr, NULL); -} - -/* Invalidate all memory regions that were registered for "req". - * - * Sleeps until it is safe for the host CPU to access the - * previously mapped memory regions. - * - * Caller ensures that @mrs is not empty before the call. This - * function empties the list. - */ -static void -fmr_op_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) -{ - struct rpcrdma_mr *mr; - LIST_HEAD(unmap_list); - int rc; - - /* ORDER: Invalidate all of the req's MRs first - * - * ib_unmap_fmr() is slow, so use a single call instead - * of one call per mapped FMR. - */ - list_for_each_entry(mr, mrs, mr_list) { - dprintk("RPC: %s: unmapping fmr %p\n", - __func__, &mr->fmr); - trace_xprtrdma_mr_localinv(mr); - list_add_tail(&mr->fmr.fm_mr->list, &unmap_list); - } - r_xprt->rx_stats.local_inv_needed++; - rc = ib_unmap_fmr(&unmap_list); - if (rc) - goto out_release; - - /* ORDER: Now DMA unmap all of the req's MRs, and return - * them to the free MW list. - */ - while (!list_empty(mrs)) { - mr = rpcrdma_mr_pop(mrs); - list_del(&mr->fmr.fm_mr->list); - rpcrdma_mr_unmap_and_put(mr); - } - - return; - -out_release: - pr_err("rpcrdma: ib_unmap_fmr failed (%i)\n", rc); - - while (!list_empty(mrs)) { - mr = rpcrdma_mr_pop(mrs); - list_del(&mr->fmr.fm_mr->list); - rpcrdma_mr_recycle(mr); - } -} - -const struct rpcrdma_memreg_ops rpcrdma_fmr_memreg_ops = { - .ro_map = fmr_op_map, - .ro_send = fmr_op_send, - .ro_unmap_sync = fmr_op_unmap_sync, - .ro_open = fmr_op_open, - .ro_maxpages = fmr_op_maxpages, - .ro_init_mr = fmr_op_init_mr, - .ro_release_mr = fmr_op_release_mr, - .ro_displayname = "fmr", - .ro_send_w_inv_ok = 0, -}; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 38a757c..389b617 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -397,12 +397,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) break; } /*FALLTHROUGH*/ - case RPCRDMA_MTHCAFMR: - if (fmr_is_supported(ia)) { - ia->ri_ops = &rpcrdma_fmr_memreg_ops; - break; - } - /*FALLTHROUGH*/ default: pr_err("rpcrdma: Device %s does not support memreg mode %d\n", ia->ri_device->name, xprt_rdma_memreg_strategy); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 7c1b519..dc8e178 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -262,20 +262,12 @@ struct rpcrdma_frwr { }; }; -struct rpcrdma_fmr { - struct ib_fmr *fm_mr; - u64 *fm_physaddrs; -}; - struct rpcrdma_mr { struct list_head mr_list; struct scatterlist *mr_sg; int mr_nents; enum dma_data_direction mr_dir; - union { - struct rpcrdma_fmr fmr; - struct rpcrdma_frwr frwr; - }; + struct rpcrdma_frwr frwr; struct rpcrdma_xprt *mr_xprt; u32 mr_handle; u32 mr_length; @@ -490,7 +482,6 @@ struct rpcrdma_memreg_ops { const int ro_send_w_inv_ok; }; -extern const struct rpcrdma_memreg_ops rpcrdma_fmr_memreg_ops; extern const struct rpcrdma_memreg_ops rpcrdma_frwr_memreg_ops; /* @@ -545,7 +536,6 @@ struct rpcrdma_xprt { void rpcrdma_ia_remove(struct rpcrdma_ia *ia); void rpcrdma_ia_close(struct rpcrdma_ia *); bool frwr_is_supported(struct rpcrdma_ia *); -bool fmr_is_supported(struct rpcrdma_ia *); /* * Endpoint calls - xprtrdma/verbs.c From patchwork Mon Dec 17 16:40:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733935 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5312E13AD for ; Mon, 17 Dec 2018 16:40:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 414822A225 for ; Mon, 17 Dec 2018 16:40:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 32E102A226; Mon, 17 Dec 2018 16:40:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D8E62A20B for ; Mon, 17 Dec 2018 16:40:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388277AbeLQQkS (ORCPT ); Mon, 17 Dec 2018 11:40:18 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:55414 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388108AbeLQQkS (ORCPT ); Mon, 17 Dec 2018 11:40:18 -0500 Received: by mail-it1-f194.google.com with SMTP id o19so19437740itg.5; Mon, 17 Dec 2018 08:40:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=EgjYXQW789dA3aQwEwDuGYFts0WsdSOlQI51eCifPqw=; b=eQGMvFWYIHlfyMK7VfBImKhTVQFemwfshQCS6vnHEVUnCW1HzI+XxHrX+Fzk3PHfZ0 T+zvxc2oNBbNW4G9D6YiEzQwBW+NGGfb4GGpVfxKNf4CilCHmOfLTInkgKS0aq5S5JWP 8DG9DfgWahbiAiZcuKSZWpBUdinsPUhlRG2aMVcX29TvLiESMlpQ1CD1y8ENOwqQokHX RJ+hD38Hlfghc50k6tE0ssIDp4tTz5mnbsOk5nkv7OSqKt8yfQ9YpQJZAU3lQ4KOpqkp gMYSt7ACSeRcSCbL6q42Mg+3eUwmOU/qRkGwk7x1s0pl6mvMRRWdNrgAdRJyT6NM/Uj4 W3Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=EgjYXQW789dA3aQwEwDuGYFts0WsdSOlQI51eCifPqw=; b=MNmwnOEEbfFfq/aEwQPu6z0UPHxP4tAOdquxdybGgg60R9zwHw7KqSpjf5bv2Qv3Ip uLhfaYtk0tCSfs3Bc9X21VkEnJKTfmYEybqAzuavMDI7YqTClvMQ5RmCKrLf0T3G+tST f9s5XSEyWFEgGFp44awd3PAXhJ+iSxr9fBVqNH+7Z82Y+2DFY3MzovnW+Ih73bNWae7K ZY/+QEqJiOZO8+iJANvCLVM7SRXw4s1ekMoGc8qIqw6yaXNEnDQy7ISVVREMvpBf5piy qiklTfXoluF5+z7ZrTMjFlmETxukW9Lq+mNscqzWOErPYba9+k7wsHFy4G+59caLOVoZ G5ew== X-Gm-Message-State: AA+aEWb51qynouaBxD4B6hriyYEk3ash6Po/eWqwsD385NsufkUVLvyu f76qsdQuEthP1uCNiRdeykoE+Iej X-Google-Smtp-Source: AFSGD/XiuVnKkIbU5f0GvPANNo+z2ODqpTtMWCbzS8TFk4uyCMcleJlO6LH22S7Jg3fJzw99/1HACQ== X-Received: by 2002:a24:3512:: with SMTP id k18mr12257768ita.83.1545064816177; Mon, 17 Dec 2018 08:40:16 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id g12sm5638020iok.38.2018.12.17.08.40.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:15 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGeEBG018608; Mon, 17 Dec 2018 16:40:14 GMT Subject: [PATCH v4 10/30] xprtrdma: Remove rpcrdma_memreg_ops From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:14 -0500 Message-ID: <20181217164014.24133.7373.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Now that there is only FRWR, there is no need for a memory registration switch. The indirect calls to the memreg operations can be replaced with faster direct calls. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 131 +++++++++++++++++++++++++-------------- net/sunrpc/xprtrdma/rpc_rdma.c | 14 +--- net/sunrpc/xprtrdma/transport.c | 2 - net/sunrpc/xprtrdma/verbs.c | 22 +++---- net/sunrpc/xprtrdma/xprt_rdma.h | 48 +++++--------- 5 files changed, 116 insertions(+), 101 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 16976b0..fb0944d 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -15,21 +15,21 @@ /* Normal operation * * A Memory Region is prepared for RDMA READ or WRITE using a FAST_REG - * Work Request (frwr_op_map). When the RDMA operation is finished, this + * Work Request (frwr_map). When the RDMA operation is finished, this * Memory Region is invalidated using a LOCAL_INV Work Request - * (frwr_op_unmap_sync). + * (frwr_unmap_sync). * * Typically these Work Requests are not signaled, and neither are RDMA * SEND Work Requests (with the exception of signaling occasionally to * prevent provider work queue overflows). This greatly reduces HCA * interrupt workload. * - * As an optimization, frwr_op_unmap marks MRs INVALID before the + * As an optimization, frwr_unmap marks MRs INVALID before the * LOCAL_INV WR is posted. If posting succeeds, the MR is placed on * rb_mrs immediately so that no work (like managing a linked list * under a spinlock) is needed in the completion upcall. * - * But this means that frwr_op_map() can occasionally encounter an MR + * But this means that frwr_map() can occasionally encounter an MR * that is INVALID but the LOCAL_INV WR has not completed. Work Queue * ordering prevents a subsequent FAST_REG WR from executing against * that MR while it is still being invalidated. @@ -57,14 +57,14 @@ * FLUSHED_LI: The MR was being invalidated when the QP entered ERROR * state, and the pending WR was flushed. * - * When frwr_op_map encounters FLUSHED and VALID MRs, they are recovered + * When frwr_map encounters FLUSHED and VALID MRs, they are recovered * with ib_dereg_mr and then are re-initialized. Because MR recovery * allocates fresh resources, it is deferred to a workqueue, and the * recovered MRs are placed back on the rb_mrs list when recovery is - * complete. frwr_op_map allocates another MR for the current RPC while + * complete. frwr_map allocates another MR for the current RPC while * the broken MR is reset. * - * To ensure that frwr_op_map doesn't encounter an MR that is marked + * To ensure that frwr_map doesn't encounter an MR that is marked * INVALID but that is about to be flushed due to a previous transport * disconnect, the transport connect worker attempts to drain all * pending send queue WRs before the transport is reconnected. @@ -80,8 +80,13 @@ # define RPCDBG_FACILITY RPCDBG_TRANS #endif -bool -frwr_is_supported(struct rpcrdma_ia *ia) +/** + * frwr_is_supported - Check if device supports FRWR + * @ia: interface adapter to check + * + * Returns true if device supports FRWR, otherwise false + */ +bool frwr_is_supported(struct rpcrdma_ia *ia) { struct ib_device_attr *attrs = &ia->ri_device->attrs; @@ -97,8 +102,12 @@ return false; } -static void -frwr_op_release_mr(struct rpcrdma_mr *mr) +/** + * frwr_release_mr - Destroy one MR + * @mr: MR allocated by frwr_init_mr + * + */ +void frwr_release_mr(struct rpcrdma_mr *mr) { int rc; @@ -132,11 +141,19 @@ list_del(&mr->mr_all); r_xprt->rx_stats.mrs_recycled++; spin_unlock(&r_xprt->rx_buf.rb_mrlock); - frwr_op_release_mr(mr); + + frwr_release_mr(mr); } -static int -frwr_op_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) +/** + * frwr_init_mr - Initialize one MR + * @ia: interface adapter + * @mr: generic MR to prepare for FRWR + * + * Returns zero if successful. Otherwise a negative errno + * is returned. + */ +int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) { unsigned int depth = ia->ri_max_frwr_depth; struct rpcrdma_frwr *frwr = &mr->frwr; @@ -172,7 +189,13 @@ return rc; } -/* On success, sets: +/** + * frwr_open - Prepare an endpoint for use with FRWR + * @ia: interface adapter this endpoint will use + * @ep: endpoint to prepare + * @cdata: transport parameters + * + * On success, sets: * ep->rep_attr.cap.max_send_wr * ep->rep_attr.cap.max_recv_wr * cdata->max_requests @@ -181,10 +204,11 @@ * And these FRWR-related fields: * ia->ri_max_frwr_depth * ia->ri_mrtype + * + * On failure, a negative errno is returned. */ -static int -frwr_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, - struct rpcrdma_create_data_internal *cdata) +int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, + struct rpcrdma_create_data_internal *cdata) { struct ib_device_attr *attrs = &ia->ri_device->attrs; int max_qp_wr, depth, delta; @@ -258,11 +282,16 @@ return 0; } -/* FRWR mode conveys a list of pages per chunk segment. The +/** + * frwr_maxpages - Compute size of largest payload + * @r_xprt: transport + * + * Returns maximum size of an RPC message, in pages. + * + * FRWR mode conveys a list of pages per chunk segment. The * maximum length of that list is the FRWR page list depth. */ -static size_t -frwr_op_maxpages(struct rpcrdma_xprt *r_xprt) +size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; @@ -344,12 +373,24 @@ trace_xprtrdma_wc_li_wake(wc, frwr); } -/* Post a REG_MR Work Request to register a memory region +/** + * frwr_map - Register a memory region + * @r_xprt: controlling transport + * @seg: memory region co-ordinates + * @nsegs: number of segments remaining + * @writing: true when RDMA Write will be used + * @out: initialized MR + * + * Prepare a REG_MR Work Request to register a memory region * for remote access via RDMA READ or RDMA WRITE. + * + * Returns the next segment or a negative errno pointer. + * On success, the prepared MR is planted in @out. */ -static struct rpcrdma_mr_seg * -frwr_op_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, struct rpcrdma_mr **out) +struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_mr_seg *seg, + int nsegs, bool writing, + struct rpcrdma_mr **out) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; bool holes_ok = ia->ri_mrtype == IB_MR_TYPE_SG_GAPS; @@ -434,14 +475,18 @@ return ERR_PTR(-EIO); } -/* Post Send WR containing the RPC Call message. +/** + * frwr_send - post Send WR containing the RPC Call message + * @ia: interface adapter + * @req: Prepared RPC Call * * For FRMR, chain any FastReg WRs to the Send WR. Only a * single ib_post_send call is needed to register memory * and then post the Send WR. + * + * Returns the result of ib_post_send. */ -static int -frwr_op_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) +int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) { struct ib_send_wr *post_wr; struct rpcrdma_mr *mr; @@ -468,10 +513,13 @@ return ib_post_send(ia->ri_id->qp, post_wr, NULL); } -/* Handle a remotely invalidated mr on the @mrs list +/** + * frwr_reminv - handle a remotely invalidated mr on the @mrs list + * @rep: Received reply + * @mrs: list of MRs to check + * */ -static void -frwr_op_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) +void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) { struct rpcrdma_mr *mr; @@ -485,7 +533,10 @@ } } -/* Invalidate all memory regions that were registered for "req". +/** + * frwr_unmap_sync - invalidate memory regions that were registered for @req + * @r_xprt: controlling transport + * @mrs: list of MRs to process * * Sleeps until it is safe for the host CPU to access the * previously mapped memory regions. @@ -493,8 +544,7 @@ * Caller ensures that @mrs is not empty before the call. This * function empties the list. */ -static void -frwr_op_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) +void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct list_head *mrs) { struct ib_send_wr *first, **prev, *last; const struct ib_send_wr *bad_wr; @@ -577,16 +627,3 @@ rpcrdma_mr_recycle(mr); } } - -const struct rpcrdma_memreg_ops rpcrdma_frwr_memreg_ops = { - .ro_map = frwr_op_map, - .ro_send = frwr_op_send, - .ro_reminv = frwr_op_reminv, - .ro_unmap_sync = frwr_op_unmap_sync, - .ro_open = frwr_op_open, - .ro_maxpages = frwr_op_maxpages, - .ro_init_mr = frwr_op_init_mr, - .ro_release_mr = frwr_op_release_mr, - .ro_displayname = "frwr", - .ro_send_w_inv_ok = RPCRDMA_CMP_F_SND_W_INV_OK, -}; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 5738c9f..2a2023d 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -356,8 +356,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, return nsegs; do { - seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs, - false, &mr); + seg = frwr_map(r_xprt, seg, nsegs, false, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -414,8 +413,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs, - true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -472,8 +470,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs, - true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -1262,8 +1259,7 @@ void rpcrdma_release_rqst(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) * RPC has relinquished all its Send Queue entries. */ if (!list_empty(&req->rl_registered)) - r_xprt->rx_ia.ri_ops->ro_unmap_sync(r_xprt, - &req->rl_registered); + frwr_unmap_sync(r_xprt, &req->rl_registered); /* Ensure that any DMA mapped pages associated with * the Send of the RPC Call have been unmapped before @@ -1292,7 +1288,7 @@ void rpcrdma_deferred_completion(struct work_struct *work) trace_xprtrdma_defer_cmp(rep); if (rep->rr_wc_flags & IB_WC_WITH_INVALIDATE) - r_xprt->rx_ia.ri_ops->ro_reminv(rep, &req->rl_registered); + frwr_reminv(rep, &req->rl_registered); rpcrdma_release_rqst(r_xprt, req); rpcrdma_complete_rqst(rep); } diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index fbb14bf..f824892 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -399,7 +399,7 @@ INIT_DELAYED_WORK(&new_xprt->rx_connect_worker, xprt_rdma_connect_worker); - xprt->max_payload = new_xprt->rx_ia.ri_ops->ro_maxpages(new_xprt); + xprt->max_payload = frwr_maxpages(new_xprt); if (xprt->max_payload == 0) goto out4; xprt->max_payload <<= PAGE_SHIFT; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 389b617..d68efaf 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -289,10 +289,9 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) break; } - dprintk("RPC: %s: %s:%s on %s/%s: %s\n", __func__, + dprintk("RPC: %s: %s:%s on %s/frwr: %s\n", __func__, rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt), - ia->ri_device->name, ia->ri_ops->ro_displayname, - rdma_event_msg(event->event)); + ia->ri_device->name, rdma_event_msg(event->event)); return 0; } @@ -392,10 +391,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) switch (xprt_rdma_memreg_strategy) { case RPCRDMA_FRWR: - if (frwr_is_supported(ia)) { - ia->ri_ops = &rpcrdma_frwr_memreg_ops; + if (frwr_is_supported(ia)) break; - } /*FALLTHROUGH*/ default: pr_err("rpcrdma: Device %s does not support memreg mode %d\n", @@ -509,7 +506,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) } ia->ri_max_send_sges = max_sge; - rc = ia->ri_ops->ro_open(ia, ep, cdata); + rc = frwr_open(ia, ep, cdata); if (rc) return rc; @@ -567,7 +564,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) /* Prepare RDMA-CM private message */ pmsg->cp_magic = rpcrdma_cmp_magic; pmsg->cp_version = RPCRDMA_CMP_VERSION; - pmsg->cp_flags |= ia->ri_ops->ro_send_w_inv_ok; + pmsg->cp_flags |= RPCRDMA_CMP_F_SND_W_INV_OK; pmsg->cp_send_size = rpcrdma_encode_buffer_size(cdata->inline_wsize); pmsg->cp_recv_size = rpcrdma_encode_buffer_size(cdata->inline_rsize); ep->rep_remote_cma.private_data = pmsg; @@ -991,7 +988,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf) if (!mr) break; - rc = ia->ri_ops->ro_init_mr(ia, mr); + rc = frwr_init_mr(ia, mr); if (rc) { kfree(mr); break; @@ -1171,7 +1168,6 @@ struct rpcrdma_req * { struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt, rx_buf); - struct rpcrdma_ia *ia = rdmab_to_ia(buf); struct rpcrdma_mr *mr; unsigned int count; @@ -1187,7 +1183,7 @@ struct rpcrdma_req * if (!list_empty(&mr->mr_list)) list_del(&mr->mr_list); - ia->ri_ops->ro_release_mr(mr); + frwr_release_mr(mr); count++; spin_lock(&buf->rb_mrlock); } @@ -1381,7 +1377,7 @@ struct rpcrdma_req * * * xprtrdma uses a regbuf for posting an outgoing RDMA SEND, or for * receiving the payload of RDMA RECV operations. During Long Calls - * or Replies they may be registered externally via ro_map. + * or Replies they may be registered externally via frwr_map. */ struct rpcrdma_regbuf * rpcrdma_alloc_regbuf(size_t size, enum dma_data_direction direction, @@ -1472,7 +1468,7 @@ struct rpcrdma_regbuf * --ep->rep_send_count; } - rc = ia->ri_ops->ro_send(ia, req); + rc = frwr_send(ia, req); trace_xprtrdma_post_send(req, rc); if (rc) return -ENOTCONN; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index dc8e178..eccb930 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -66,7 +66,6 @@ * Interface Adapter -- one per transport instance */ struct rpcrdma_ia { - const struct rpcrdma_memreg_ops *ri_ops; struct ib_device *ri_device; struct rdma_cm_id *ri_id; struct ib_pd *ri_pd; @@ -406,7 +405,6 @@ struct rpcrdma_buffer { struct workqueue_struct *rb_completion_wq; struct delayed_work rb_refresh_worker; }; -#define rdmab_to_ia(b) (&container_of((b), struct rpcrdma_xprt, rx_buf)->rx_ia) /* rb_flags */ enum { @@ -457,34 +455,6 @@ struct rpcrdma_stats { }; /* - * Per-registration mode operations - */ -struct rpcrdma_xprt; -struct rpcrdma_memreg_ops { - struct rpcrdma_mr_seg * - (*ro_map)(struct rpcrdma_xprt *, - struct rpcrdma_mr_seg *, int, bool, - struct rpcrdma_mr **); - int (*ro_send)(struct rpcrdma_ia *ia, - struct rpcrdma_req *req); - void (*ro_reminv)(struct rpcrdma_rep *rep, - struct list_head *mrs); - void (*ro_unmap_sync)(struct rpcrdma_xprt *, - struct list_head *); - int (*ro_open)(struct rpcrdma_ia *, - struct rpcrdma_ep *, - struct rpcrdma_create_data_internal *); - size_t (*ro_maxpages)(struct rpcrdma_xprt *); - int (*ro_init_mr)(struct rpcrdma_ia *, - struct rpcrdma_mr *); - void (*ro_release_mr)(struct rpcrdma_mr *mr); - const char *ro_displayname; - const int ro_send_w_inv_ok; -}; - -extern const struct rpcrdma_memreg_ops rpcrdma_frwr_memreg_ops; - -/* * RPCRDMA transport -- encapsulates the structures above for * integration with RPC. * @@ -535,7 +505,6 @@ struct rpcrdma_xprt { int rpcrdma_ia_open(struct rpcrdma_xprt *xprt); void rpcrdma_ia_remove(struct rpcrdma_ia *ia); void rpcrdma_ia_close(struct rpcrdma_ia *); -bool frwr_is_supported(struct rpcrdma_ia *); /* * Endpoint calls - xprtrdma/verbs.c @@ -601,6 +570,23 @@ struct rpcrdma_regbuf *rpcrdma_alloc_regbuf(size_t, enum dma_data_direction, return writing ? DMA_FROM_DEVICE : DMA_TO_DEVICE; } +/* Memory registration calls xprtrdma/frwr_ops.c + */ +bool frwr_is_supported(struct rpcrdma_ia *); +int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, + struct rpcrdma_create_data_internal *cdata); +int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr); +void frwr_release_mr(struct rpcrdma_mr *mr); +size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt); +struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_mr_seg *seg, + int nsegs, bool writing, + struct rpcrdma_mr **mr); +int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); +void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); +void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, + struct list_head *mrs); + /* * RPC/RDMA protocol calls - xprtrdma/rpc_rdma.c */ From patchwork Mon Dec 17 16:40:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733939 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83B3F1575 for ; Mon, 17 Dec 2018 16:40:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71BEE2A20B for ; Mon, 17 Dec 2018 16:40:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 669962A226; Mon, 17 Dec 2018 16:40:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ECA092A20B for ; Mon, 17 Dec 2018 16:40:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388262AbeLQQkX (ORCPT ); Mon, 17 Dec 2018 11:40:23 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:55428 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388257AbeLQQkX (ORCPT ); Mon, 17 Dec 2018 11:40:23 -0500 Received: by mail-it1-f196.google.com with SMTP id o19so19438193itg.5; Mon, 17 Dec 2018 08:40:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=ty1oGOtNt9biJZaf/kjTjU9tLsvdxKC6SZs1D4odKpI=; b=PskoHHGbN8GgPWk/kMfPyO6sWlOkONEYpw5lYvVP9bshPM2/gUgIXsoKXNb/0HcKbq bCEXywwfeRt3cGohvkp7q0Kaex2pF2dQWzNiF69WigVQxt2DPX19J1NaBS/R/5wmAXXp B1icXQc5cFYh1O65Df/07WWRgp8Us6MuPFGuJ1DX+FXBNIH2SujBIEPqfNMRMe0P36pG pFRJo0QSStisASs+cBa9qVuFzzgsyHC0TXSsq8X1Hms+IOPxco0A9jl05tA/sjW15jhY mUud2yE6m7neEdIOz9ux/vxbWgycp+YBs1FcQxyDFu74eOuPgf2dOzXMCZQTxcuVXCK9 70Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=ty1oGOtNt9biJZaf/kjTjU9tLsvdxKC6SZs1D4odKpI=; b=Cs3rYe9IyZ95yA1EkMVGAdci8RMa8Wx0/fdz61uOSu4EMdwhdbONgY3dbQlq54gpnM QdnoqTvNjmE4niAL6C5wV5GS+WW/MTW0Zj5/tFz1mcNGH9GuW3V+wscDEZ4mL8hvqFfH tmgcYpbgoHts59ZY2+XfxbSaKzsjXwY0UvJfoil1UXhx9AMGa3+zaJh42mE40TZr7VVz +1od/O0FWwLNVo0tYmyO2JlEEJHJCKoEvumE7qS9XhIle9DjoZjP8gq/bXFk7SRQeGr2 AQ5InI2hK8ECfAghnd/NDeLeFou5uRsllqQVJbySFFoLafcXb24B03Ig/JGwaMyVOHaa sq4A== X-Gm-Message-State: AA+aEWb0a8Q8w8QrJFxNXE7uPztRvCx+2mpHg/FZ0pj2MFvqbvvviKHP 5GGmdZl/r3kjL5tpjZB3Qu6ThZOX X-Google-Smtp-Source: AFSGD/WlZTD/xBckq3tBhokjP14UGxMyhxOcERY4OWRFwnNtcJ+4+Qiae7DaLmMkKFzCWD/0oMinXQ== X-Received: by 2002:a24:7609:: with SMTP id z9mr13860403itb.66.1545064821873; Mon, 17 Dec 2018 08:40:21 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id y76sm7827351ita.18.2018.12.17.08.40.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:21 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGeKGw018611; Mon, 17 Dec 2018 16:40:20 GMT Subject: [PATCH v4 11/30] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR) From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:20 -0500 Message-ID: <20181217164020.24133.66234.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Place the associated RPC transaction's XID in the upper 32 bits of each RDMA segment's rdma_offset field. There are two reasons to do this: - The R_key only has 8 bits that are different from registration to registration. The XID adds more uniqueness to each RDMA segment to reduce the likelihood of a software bug on the server reading from or writing into memory it's not supposed to. - On-the-wire RDMA Read and Write requests do not otherwise carry any identifier that matches them up to an RPC. The XID in the upper 32 bits will act as an eye-catcher in network captures. Suggested-by: Tom Talpey Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 5 ++++- net/sunrpc/xprtrdma/rpc_rdma.c | 6 +++--- net/sunrpc/xprtrdma/xprt_rdma.h | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index fb0944d..97f88bb 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -379,6 +379,7 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) * @seg: memory region co-ordinates * @nsegs: number of segments remaining * @writing: true when RDMA Write will be used + * @xid: XID of RPC using the registered memory * @out: initialized MR * * Prepare a REG_MR Work Request to register a memory region @@ -389,7 +390,7 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) */ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, + int nsegs, bool writing, u32 xid, struct rpcrdma_mr **out) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; @@ -444,6 +445,8 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, if (unlikely(n != mr->mr_nents)) goto out_mapmr_err; + ibmr->iova &= 0x00000000ffffffff; + ibmr->iova |= ((u64)cpu_to_be32(xid)) << 32; key = (u8)(ibmr->rkey & 0x000000FF); ib_update_fast_reg_key(ibmr, ++key); diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 2a2023d..3804fb3 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -356,7 +356,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, return nsegs; do { - seg = frwr_map(r_xprt, seg, nsegs, false, &mr); + seg = frwr_map(r_xprt, seg, nsegs, false, rqst->rq_xid, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -413,7 +413,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = frwr_map(r_xprt, seg, nsegs, true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, rqst->rq_xid, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); @@ -470,7 +470,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nchunks = 0; do { - seg = frwr_map(r_xprt, seg, nsegs, true, &mr); + seg = frwr_map(r_xprt, seg, nsegs, true, rqst->rq_xid, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); rpcrdma_mr_push(mr, &req->rl_registered); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index eccb930..56b299f 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -580,7 +580,7 @@ int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep, size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt); struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, - int nsegs, bool writing, + int nsegs, bool writing, u32 xid, struct rpcrdma_mr **mr); int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); From patchwork Mon Dec 17 16:40:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 801A413AD for ; Mon, 17 Dec 2018 16:40:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 70CAF2A20B for ; Mon, 17 Dec 2018 16:40:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 659FC2A226; Mon, 17 Dec 2018 16:40:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 11AE92A20B for ; Mon, 17 Dec 2018 16:40:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388257AbeLQQk2 (ORCPT ); Mon, 17 Dec 2018 11:40:28 -0500 Received: from mail-it1-f172.google.com ([209.85.166.172]:37941 "EHLO mail-it1-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388108AbeLQQk2 (ORCPT ); Mon, 17 Dec 2018 11:40:28 -0500 Received: by mail-it1-f172.google.com with SMTP id h65so20903471ith.3; Mon, 17 Dec 2018 08:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Tke2KOJ89T4vcbDVIGt35hCRU2JIcQsMOR9LJ34omWk=; b=mrYyQNqvPVMg1z9imePsy3XTU6ZWwoj09cMaJVnX0pH50Q92e+bLTMUwh1K5Uwm81f d+yK0DRfv6bM89ZzU9winzaoIFhKVZIADXKFJ/BSV4L+9WzZJqNe62C5CcSJxAKUQJGK V6x+opFOQUre0+ewBE4XM7wH5oxgjHj3eQkCyCg8aR9nMoOsI0A8ISEnmw+qlycfVi5S jaKKZAK8ya/rIx9z0FsNMRSno1bJnq28bT+kdq+2NpLnwGWYm6A0VuQKKuEyPwwvSBOZ ltx7UBvMEKtPBaoIuOzRTy5+zoRZZik53BVk0BFiTPAQTs/TVyBYnZCXzkPnwV4xO3y0 H+kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Tke2KOJ89T4vcbDVIGt35hCRU2JIcQsMOR9LJ34omWk=; b=qTkLQVpW9a4xsah8q4EKrumtK6CemxTt2IzcjmMJ/6gieFlK0ZiRIbiekAH6nKXhBL Dd299NQ2+tt224EDbGAuEjV2Nerf0RmPDlUDT9FlkIyjXHqlKul3FEtMuLjzprwiGw+S X5iPgjsNJarzQZWoo9poqfIXAJ1hDCWNAVtTyfJZJB9aTvw7w0uyQrhZU5auQ2j/tqxp ZfSY0uCsy1UQGmgTZsAWbM3zMERkdyzZVM/xpKHpC7yCiPnFhaXysLBtadcAhcPLWGRO vyU2+/CJqNmiXFwm21zEqu6Zq0I6IjJ7e4VbKAvnMR1F4qR/JbALY7ZelNnHahNS1I9z 1SuA== X-Gm-Message-State: AA+aEWbIwAZKt6zWb6HextW8GwYfgnhThRQGE56Z1MP+Ny6AVMTbnjpp u3/k0/JsjGnLgxBrG5LiZOI/Hb5F X-Google-Smtp-Source: AFSGD/W9d2gYcPBPH+I29Xs6YcI+jdWKmEumP7+aRo2S8v9KgaEpGz0G7krZMNEuVlh4xfYB1jVkNw== X-Received: by 2002:a02:98d2:: with SMTP id c18mr11908250jak.11.1545064826893; Mon, 17 Dec 2018 08:40:26 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id m81sm7214014itb.43.2018.12.17.08.40.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:26 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGePPm018614; Mon, 17 Dec 2018 16:40:25 GMT Subject: [PATCH v4 12/30] NFS: Make "port=" mount option optional for RDMA mounts From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:25 -0500 Message-ID: <20181217164025.24133.45942.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Having to specify "proto=rdma,port=20049" is cumbersome. RFC 8267 Section 6.3 requires NFSv4 clients to use "the alternative well-known port number", which is 20049. Make the use of the well- known port number automatic, just as it is for NFS/TCP and port 2049. For NFSv2/3, Section 4.2 allows clients to simply choose 20049 as the default or use rpcbind. I don't know of an NFS/RDMA server implementation that registers it's NFS/RDMA service with rpcbind, so automatically choosing 20049 seems like the better choice. The other widely-deployed NFS/RDMA client, Solaris, also uses 20049 as the default port. Signed-off-by: Chuck Lever --- fs/nfs/super.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/fs/nfs/super.c b/fs/nfs/super.c index ac4b2f0..22247c2 100644 --- a/fs/nfs/super.c +++ b/fs/nfs/super.c @@ -2168,7 +2168,10 @@ static int nfs_validate_text_mount_data(void *options, if (args->version == 4) { #if IS_ENABLED(CONFIG_NFS_V4) - port = NFS_PORT; + if (args->nfs_server.protocol == XPRT_TRANSPORT_RDMA) + port = NFS_RDMA_PORT; + else + port = NFS_PORT; max_namelen = NFS4_MAXNAMLEN; max_pathlen = NFS4_MAXPATHLEN; nfs_validate_transport_protocol(args); @@ -2178,8 +2181,11 @@ static int nfs_validate_text_mount_data(void *options, #else goto out_v4_not_compiled; #endif /* CONFIG_NFS_V4 */ - } else + } else { nfs_set_mount_transport_protocol(args); + if (args->nfs_server.protocol == XPRT_TRANSPORT_RDMA) + port = NFS_RDMA_PORT; + } nfs_set_port(sap, &args->nfs_server.port, port); From patchwork Mon Dec 17 16:40:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE50A1575 for ; Mon, 17 Dec 2018 16:40:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BCCA02A20B for ; Mon, 17 Dec 2018 16:40:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B16972A226; Mon, 17 Dec 2018 16:40:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 576E82A20B for ; Mon, 17 Dec 2018 16:40:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388276AbeLQQkd (ORCPT ); Mon, 17 Dec 2018 11:40:33 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:53920 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387905AbeLQQkd (ORCPT ); Mon, 17 Dec 2018 11:40:33 -0500 Received: by mail-it1-f196.google.com with SMTP id g85so19463209ita.3; Mon, 17 Dec 2018 08:40:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=2kgtkyFhKXfzrghxvQsBnyUA+KcQv3ItXO8NKS1hJvg=; b=VimDbzV9Cp0kOTyzi/o5wWfCqdc6nyj3gE43W7aOYYYevitLiQSEUwCehrVm+BnWqT MfAGGo6hg9l5BBP0xQym0DWwrork7tKh5O/bOPNXEyb9C/aJ6wGvCoO9T1ww1/kocVlO 4BxvlUsAWxm3K241VQhXM8oxzoY9rdak8OIF7iurjreLWsv0ZhM/4GMTOMK0HW5AK608 xKMH2kS+iqLp/ZyDbWLdMQsQIcnjYp48rOnsNZjCOjDJEBtmFBoQ+d4RY/5esmJ6CRKr K0ei6pVtCXgMV1rUOxI4s/b8kHOp7QjemcrRG7PBjbDQAma52OG8eN9HrIqR5rRUBUpM GIkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=2kgtkyFhKXfzrghxvQsBnyUA+KcQv3ItXO8NKS1hJvg=; b=sag0fW+YOt1lBy4gf5hB/AcYCtaLR12TYyJc7CA4OIWo5qV7sSUwD2aqOyyg6Dx7+T gVwmUsDvgMtEbDVGZSSrfS4uUsjmEAyxoALME6oiXKZPrJnk2e+Hu77f5hxjsLafIxgK m0TDAGpj4vZjyOCk6B8jPxgT94qV2t+iaMYMdj+qDLzM2Iidd52oInkMeXZnrRfcG6co GGpSnHJZRcLACzRSYKCoBmgTHACiby1+LEunyMPGYEu8Jhz/SwnH/KhKzguSYS+1w1w1 ZHchT+qtSgChQIFGQDZcPTGJ2TV6HqpNhWa2pMHBXQ9H3q51VNvtOd+pWDgx1eMCxdzc RGBw== X-Gm-Message-State: AA+aEWaN7MCe08g5HaAF8TB722SQzsKIXamFco81dMecQNMxAdcN7ymw xBsYbQcCuIHP8BtYPH3WzM3ts55v X-Google-Smtp-Source: AFSGD/XyZbdwNTq8fwlogtW2rFQQfC9+GFOS4lRwOsvJBrASoA/uVI29W9oz+nY6KUW3y3MrFvmikA== X-Received: by 2002:a24:3c43:: with SMTP id m64mr12146546ita.22.1545064832166; Mon, 17 Dec 2018 08:40:32 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id v21sm5340430ios.42.2018.12.17.08.40.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:31 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGeULW018617; Mon, 17 Dec 2018 16:40:30 GMT Subject: [PATCH v4 13/30] xprtrdma: Recognize XDRBUF_SPARSE_PAGES From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:30 -0500 Message-ID: <20181217164030.24133.8079.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit 431f6eb3570f ("SUNRPC: Add a label for RPC calls that require allocation on receive") didn't update similar logic in rpc_rdma.c. I don't think this is a bug, per-se; the commit just adds more careful checking for broken upper layer behavior. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 3804fb3..939f84a 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -218,11 +218,12 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, ppages = xdrbuf->pages + (xdrbuf->page_base >> PAGE_SHIFT); page_base = offset_in_page(xdrbuf->page_base); while (len) { - if (unlikely(!*ppages)) { - /* XXX: Certain upper layer operations do - * not provide receive buffer pages. - */ - *ppages = alloc_page(GFP_ATOMIC); + /* ACL likes to be lazy in allocating pages - ACLs + * are small by default but can get huge. + */ + if (unlikely(xdrbuf->flags & XDRBUF_SPARSE_PAGES)) { + if (!*ppages) + *ppages = alloc_page(GFP_ATOMIC); if (!*ppages) return -ENOBUFS; } From patchwork Mon Dec 17 16:40:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB58E161F for ; Mon, 17 Dec 2018 16:40:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEA7D2A20B for ; Mon, 17 Dec 2018 16:40:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B34B42A22D; Mon, 17 Dec 2018 16:40:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 716622A215 for ; Mon, 17 Dec 2018 16:40:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727771AbeLQQkl (ORCPT ); Mon, 17 Dec 2018 11:40:41 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:38471 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387905AbeLQQkk (ORCPT ); Mon, 17 Dec 2018 11:40:40 -0500 Received: by mail-io1-f67.google.com with SMTP id l14so10467769ioj.5; Mon, 17 Dec 2018 08:40:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=z+FAU7x0pu2cR4+7sT5Y8flHVTgOYIrNWNeq22LHUsc=; b=W0Q4OpgBomIWa8/yt+XPFn+k18aeQAjSnHBDgNZRoeL9q34XgokBa2PsLfpDiZPkqF k6QSWBM96P4mxHMoKG84CZKKbwuQkt2LE3LMVHkSv1TaYaEpzqESETYlWmy4lg3ak6so qrMHDSUF/6LwIEKv1VP3vx73LvwxStUV+ZEv8AxT1OE7Jklw9VACGH4st3kcqPT4gY53 E5jBJ51kXbSFscW79EWXXKY2wyYakDO5+//yXk83HYwokBmGaQjim24PzFbxUc/D3580 62vnQoaRA3geEwK9yjqQM9qkjit+LUWtc4paiFO2u4XV7URuWmRrDwdseWr+t2o/6H5q rEAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=z+FAU7x0pu2cR4+7sT5Y8flHVTgOYIrNWNeq22LHUsc=; b=f0JaUUsOc6jzSdUFTV45r1H17rDNbzq+FUCVJnHFML6mad2Ue18tWfIMeR1t1khfaR eqPL+Z/izrB8bVOnPT0U2+duSG9VVch7dWHRbLGcoJAc4L4yumpr5n3MVmsN6bhJatlC JLMMeRGTloMEbgsKRTv9rlVLsUZzz3Tyuo9+mvZ2mmTh464+tz5HIPnrlWfTzKI4YKrv 5EF7NOT4CPEq4pq3WRAfOVVY1FNj9E/iZnf3u71ymbtlzfEkgddImq/g7quBNVMIo549 TpfneyluE4sJbEhoyLHZIZR6qy4WXToB2bf1p6vVlYqFu2Fwk6+Q7sTB6VKBNIKPXYex xmtg== X-Gm-Message-State: AA+aEWYIzc03fS1ZIbpqmTN4GuYf/3Jeg8gKBgvzTFE/ZpbY14G6NMql CuGftMmPhRY7BGs+2qvnB968akGz X-Google-Smtp-Source: AFSGD/WifFqsJA64UpgjvLHBgd4uU2B73LEVIcNkqZKFXzz5G2PIFm3a0CXBg5wrofBfLcyrj6n9DQ== X-Received: by 2002:a5d:9341:: with SMTP id i1mr11375086ioo.162.1545064838279; Mon, 17 Dec 2018 08:40:38 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id o63sm7894475ita.8.2018.12.17.08.40.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:37 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGeajD018620; Mon, 17 Dec 2018 16:40:36 GMT Subject: [PATCH v4 14/30] xprtrdma: Remove request_module from backchannel From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:36 -0500 Message-ID: <20181217164036.24133.28066.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since commit ffe1f0df5862 ("rpcrdma: Merge svcrdma and xprtrdma modules into one"), the forward and backchannel components are part of the same kernel module. A separate request_module() call in the backchannel code is no longer necessary. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 5d462e8..9cb96a5 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -5,7 +5,6 @@ * Support for backward direction RPCs on RPC/RDMA. */ -#include #include #include #include @@ -101,7 +100,6 @@ int xprt_rdma_bc_setup(struct rpc_xprt *xprt, unsigned int reqs) goto out_free; r_xprt->rx_buf.rb_bc_srv_max_requests = reqs; - request_module("svcrdma"); trace_xprtrdma_cb_setup(r_xprt, reqs); return 0; From patchwork Mon Dec 17 16:40:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733957 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D1BC1575 for ; Mon, 17 Dec 2018 16:40:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7FB152A20B for ; Mon, 17 Dec 2018 16:40:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7473C2A225; Mon, 17 Dec 2018 16:40:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37BD72A20B for ; Mon, 17 Dec 2018 16:40:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388108AbeLQQko (ORCPT ); Mon, 17 Dec 2018 11:40:44 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:53962 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387905AbeLQQkn (ORCPT ); Mon, 17 Dec 2018 11:40:43 -0500 Received: by mail-it1-f194.google.com with SMTP id g85so19464174ita.3; Mon, 17 Dec 2018 08:40:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=PXLIaALci2knYQ1gMtWCij3L4mHSL0l8lqusuVMPb0k=; b=OJuXHt7YnmhJdGdCIqGcbzpCJNjAYpPGcIJwtKd5bdAjDQxCvkt6jDAjXkowFA2gqL ldC6MNUWnEltASIb/MkIVv4aVIXtXGV034GCultyjJPqHNdZ3Eo5JcQHqdlBPSIhcBgs XLXGzTxMHZBbBxNuSPkHi8XFdz9AFJjo4pdJXn24cyvqr7tjROD2G99pnXQWgiqEIaFL CgsYaE7uku+4sbWjMPJJ2dMPq8Za/W/ArGwnYA+iUklSH4s2/WNqpzFB79Fq+BOzBZMv 1Wwma1GlZeJRwQOeHZKzJ91Vt422Yhegm9UDov244/8z4gUD+ewQouwVc4l2hkuUSANw AuPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=PXLIaALci2knYQ1gMtWCij3L4mHSL0l8lqusuVMPb0k=; b=jaJ+Aj17PoTOd7X9ntfaJdZxKSmmhGPCuxIwdVmSgcJCqO/rKePOV/yBFCy2D5ECNU 5w0rF0EIe5+8JoYuD+DNEpADHcdZzBqKHqmL1MVnsam3KydejcQnpP/h1J6VyuBfueIs aUYDDMVY2jf5qUMhQNLpFo6IccPNH3jghLueg0om7KVffwZOhL9M68K+Q7A/huZ0bZ/2 2Wvmq89RIOc6fwj9BExGsfBB+d5FNTmGN5cOGaN+VFZw9JZfwWaj2BLMZ0hqIATahmh7 VPK7sy2FozxR++Ltl8NYhCs3hToh2QC6wQQ5btYcAxtwcUn1StLRPN92wf8huMoCZGhy W2Sw== X-Gm-Message-State: AA+aEWYmpvrT9mM1O61Bv5yJQhlXPIZlEugNcL+njLnWMgJUgFkd80wm dZtrV/XnyWZP0V3+Kdz1zbvp49Al X-Google-Smtp-Source: AFSGD/XHCFrM7ETvENr7cIroH1sACs3Kx+RsdgCM8OitrcjtUvmMH3oCzwhwbpwsJZkbyrWgRWb0dQ== X-Received: by 2002:a02:2b2d:: with SMTP id h45mr11068165jaa.75.1545064843028; Mon, 17 Dec 2018 08:40:43 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id t194sm6572955iof.3.2018.12.17.08.40.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:42 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGeff6018623; Mon, 17 Dec 2018 16:40:41 GMT Subject: [PATCH v4 15/30] xprtrdma: Expose transport header errors From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:41 -0500 Message-ID: <20181217164041.24133.94561.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For better observability of parsing errors, return the error code generated in the decoders to the upper layer consumer. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 1 - 1 file changed, 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 939f84a..8de0b9f 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1246,7 +1246,6 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep) out_badheader: trace_xprtrdma_reply_hdr(rep); r_xprt->rx_stats.bad_reply_count++; - status = -EIO; goto out; } From patchwork Mon Dec 17 16:40:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733959 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1BD5E1575 for ; Mon, 17 Dec 2018 16:40:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A13E2A225 for ; Mon, 17 Dec 2018 16:40:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F27B32A20B; Mon, 17 Dec 2018 16:40:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 738102A20B for ; Mon, 17 Dec 2018 16:40:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388304AbeLQQkt (ORCPT ); Mon, 17 Dec 2018 11:40:49 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:39123 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388281AbeLQQkt (ORCPT ); Mon, 17 Dec 2018 11:40:49 -0500 Received: by mail-it1-f194.google.com with SMTP id a6so20902249itl.4; Mon, 17 Dec 2018 08:40:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=M1mWnOW+fmqs0BSG3OSh1UH3OsQGfVQFafkL8QUXdiM=; b=JxZMAmv1dqBUioP2OqjdhuS5yenJUSosGsiVSm0d4CiKz6rx7rXtFNZioJc2kKRXIV GX2PJ5/obI2RDE4UyUFDm6EMKXip3noX4ai9Q74jSx6iyj+HhYFmDqaDiwMZ2+bU4kMb o7DqEErmsv8kHxLaD00zR1O4GDFYCnWNbHlukgpnW+gzqTFcw/+3VSytisxfFz/0NkjJ bKOvCrIxiIieWv+wil616MY9DgKmsbzyrJBykR8krpc3k+mAvaD4g3ZuP5oWiRQmLP1c hUuLmMJHiaVIzwFCe4o+FUHe3FdhvxfzzYj4NvOacW9z657ulYVdifc69d3ldRpBl3xo 9f4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=M1mWnOW+fmqs0BSG3OSh1UH3OsQGfVQFafkL8QUXdiM=; b=ByD8aS8syDUGeltdCikzoVJSavT9XFtuS11gyYUHsvXw1CbfQtKSm036uXXMAHmEe4 uzW1lwYLF9aQ0HuGDHBzZ92x6TI4Lb5ypWUzeqLtrXc6uJg1MXwjFIzNq9Ufi0PRyT2n uCbF97XtMf3pgY49/4jCZnv/VUD7a2K9d//mNyCNDKEvA4s++d1nG2HMcakfXW13dTi8 TATswBKktLeoYWMdn+j8Q6c3vEQcn6GUF9UTJf416xDxL/kzqfIbo7cccYQ0Rot/XaKi SzaoZe3nxzhaxKR6RnbNp6oSntN9eMNdnLp/UFwAJRqOoKJfiv6K2w6wZuoy8b9SyJCN Q0lA== X-Gm-Message-State: AA+aEWbILE2oF2xlhQio1bdjvovuohYyxZkN3RRXImRXEmFVyr0lgOzP S1DF109pDY+yF9VIXZMa7TRpU8w1 X-Google-Smtp-Source: AFSGD/XAN6O+VaouR+WaaSVJnD+pidO35m+xVfEjRl+Om6Pm9SGkJK1Oei3CvJ9hWbV83YgyUqZPFg== X-Received: by 2002:a24:1490:: with SMTP id 138mr14813325itg.101.1545064848459; Mon, 17 Dec 2018 08:40:48 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id w129sm77140ita.0.2018.12.17.08.40.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:47 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGekvh018626; Mon, 17 Dec 2018 16:40:46 GMT Subject: [PATCH v4 16/30] xprtrdma: Simplify locking that protects the rl_allreqs list From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:46 -0500 Message-ID: <20181217164046.24133.73626.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: There's little chance of contention between the use of rb_lock and rb_reqslock, so merge the two. This avoids having to take both in some (possibly future) cases. Transport tear-down is already serialized, thus there is no need for locking at all when destroying rpcrdma_reqs. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 20 +++----------------- net/sunrpc/xprtrdma/verbs.c | 31 +++++++++++++++++-------------- net/sunrpc/xprtrdma/xprt_rdma.h | 7 +++---- 3 files changed, 23 insertions(+), 35 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 9cb96a5..af8249b 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -19,29 +19,16 @@ #undef RPCRDMA_BACKCHANNEL_DEBUG -static void rpcrdma_bc_free_rqst(struct rpcrdma_xprt *r_xprt, - struct rpc_rqst *rqst) -{ - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; - struct rpcrdma_req *req = rpcr_to_rdmar(rqst); - - spin_lock(&buf->rb_reqslock); - list_del(&req->rl_all); - spin_unlock(&buf->rb_reqslock); - - rpcrdma_destroy_req(req); -} - static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt, unsigned int count) { struct rpc_xprt *xprt = &r_xprt->rx_xprt; + struct rpcrdma_req *req; struct rpc_rqst *rqst; unsigned int i; for (i = 0; i < (count << 1); i++) { struct rpcrdma_regbuf *rb; - struct rpcrdma_req *req; size_t size; req = rpcrdma_create_req(r_xprt); @@ -67,7 +54,7 @@ static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt, return 0; out_fail: - rpcrdma_bc_free_rqst(r_xprt, rqst); + rpcrdma_req_destroy(req); return -ENOMEM; } @@ -225,7 +212,6 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst) */ void xprt_rdma_bc_destroy(struct rpc_xprt *xprt, unsigned int reqs) { - struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); struct rpc_rqst *rqst, *tmp; spin_lock(&xprt->bc_pa_lock); @@ -233,7 +219,7 @@ void xprt_rdma_bc_destroy(struct rpc_xprt *xprt, unsigned int reqs) list_del(&rqst->rq_bc_pa_list); spin_unlock(&xprt->bc_pa_lock); - rpcrdma_bc_free_rqst(r_xprt, rqst); + rpcrdma_req_destroy(rpcr_to_rdmar(rqst)); spin_lock(&xprt->bc_pa_lock); } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index d68efaf..a6ab216 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1043,9 +1043,9 @@ struct rpcrdma_req * req->rl_buffer = buffer; INIT_LIST_HEAD(&req->rl_registered); - spin_lock(&buffer->rb_reqslock); + spin_lock(&buffer->rb_lock); list_add(&req->rl_all, &buffer->rb_allreqs); - spin_unlock(&buffer->rb_reqslock); + spin_unlock(&buffer->rb_lock); return req; } @@ -1113,7 +1113,6 @@ struct rpcrdma_req * INIT_LIST_HEAD(&buf->rb_send_bufs); INIT_LIST_HEAD(&buf->rb_allreqs); - spin_lock_init(&buf->rb_reqslock); for (i = 0; i < buf->rb_max_requests; i++) { struct rpcrdma_req *req; @@ -1154,9 +1153,18 @@ struct rpcrdma_req * kfree(rep); } +/** + * rpcrdma_req_destroy - Destroy an rpcrdma_req object + * @req: unused object to be destroyed + * + * This function assumes that the caller prevents concurrent device + * unload and transport tear-down. + */ void -rpcrdma_destroy_req(struct rpcrdma_req *req) +rpcrdma_req_destroy(struct rpcrdma_req *req) { + list_del(&req->rl_all); + rpcrdma_free_regbuf(req->rl_recvbuf); rpcrdma_free_regbuf(req->rl_sendbuf); rpcrdma_free_regbuf(req->rl_rdmabuf); @@ -1214,19 +1222,14 @@ struct rpcrdma_req * rpcrdma_destroy_rep(rep); } - spin_lock(&buf->rb_reqslock); - while (!list_empty(&buf->rb_allreqs)) { + while (!list_empty(&buf->rb_send_bufs)) { struct rpcrdma_req *req; - req = list_first_entry(&buf->rb_allreqs, - struct rpcrdma_req, rl_all); - list_del(&req->rl_all); - - spin_unlock(&buf->rb_reqslock); - rpcrdma_destroy_req(req); - spin_lock(&buf->rb_reqslock); + req = list_first_entry(&buf->rb_send_bufs, + struct rpcrdma_req, rl_list); + list_del(&req->rl_list); + rpcrdma_req_destroy(req); } - spin_unlock(&buf->rb_reqslock); rpcrdma_mrs_destroy(buf); } diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 56b299f..6e104cd 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -392,14 +392,13 @@ struct rpcrdma_buffer { spinlock_t rb_lock; /* protect buf lists */ struct list_head rb_send_bufs; struct list_head rb_recv_bufs; + struct list_head rb_allreqs; + unsigned long rb_flags; u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ u32 rb_bc_srv_max_requests; - spinlock_t rb_reqslock; /* protect rb_allreqs */ - struct list_head rb_allreqs; - u32 rb_bc_max_requests; struct workqueue_struct *rb_completion_wq; @@ -522,7 +521,7 @@ int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *, * Buffer calls - xprtrdma/verbs.c */ struct rpcrdma_req *rpcrdma_create_req(struct rpcrdma_xprt *); -void rpcrdma_destroy_req(struct rpcrdma_req *); +void rpcrdma_req_destroy(struct rpcrdma_req *req); int rpcrdma_buffer_create(struct rpcrdma_xprt *); void rpcrdma_buffer_destroy(struct rpcrdma_buffer *); struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf); From patchwork Mon Dec 17 16:40:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733963 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D8BE13AD for ; Mon, 17 Dec 2018 16:40:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8C6382A20B for ; Mon, 17 Dec 2018 16:40:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8124A2A226; Mon, 17 Dec 2018 16:40:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CEA602A20B for ; Mon, 17 Dec 2018 16:40:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388307AbeLQQk4 (ORCPT ); Mon, 17 Dec 2018 11:40:56 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:55527 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388281AbeLQQk4 (ORCPT ); Mon, 17 Dec 2018 11:40:56 -0500 Received: by mail-it1-f196.google.com with SMTP id o19so19440924itg.5; Mon, 17 Dec 2018 08:40:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=6klOv+StCAdLA/KPQDlr1jm2BnrTiFkKbZbPVE6T44Y=; b=ZSapJvtWQ2lUiNfBytYwsNM58FgVXVGJFHXjSvHa7pXk59iclWI8TA1GM1Qe4eUqmz ZCgzCD0jfYQMQf3NU82YzYa3/JMyvo3AVv3Rir9HjUYzhdl42L75VV8xKKVrC7N4GqSb Y2kQTqZbJjvOuZVCQJdPoiL67H9XeeBJlwl8V8CcwCDpS0Z2FHyGzbRgh9goZobaGtI0 EAIn/dekQbEfW4rDxmECjumnSBcdvovov23e7e5w7c0wbqJ2F/D7OHEm+x3H6cwilTMx S305ahhWg4bk741q6VyFP7lAO5v2DW91OJOVa6olZ1W1j6Y77iQOy4LTz0bWLFk6wesF cGbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=6klOv+StCAdLA/KPQDlr1jm2BnrTiFkKbZbPVE6T44Y=; b=nZarN8KsCCSp+I3ki2Y4bUktWNdWKYoHFd7GVM66AHom5ea9nRBnGxJgcqUsRw6qRQ IkQWEHfNK/f19VWYnG7nxlk5Bgrvcu/i817jyFOO550ZQdVmtRZw71XcPFmQESNl511x C4ZK0fCySmwrPDMdKKcdEtNG3y0AyWB7aDsLrpEyH+4p+Fqyxyjr90kd4A447kJHYc/X UkUNkcQd4Fb98rONSpk1+2tHGDQyCdTjPDzCpZLhsio1zvQ/uZWiimqCbQ8kz/Gj88a4 sGFVFU82Aarod19yP0ZTSJRHpEQjuRPZZpSu1uM+FuN7V0srK3Gra354BSyuyNSNDlov 0b/w== X-Gm-Message-State: AA+aEWbib20MoAqrFECxfZxWuwZPMYb/t7F2poI7JBvA6kgmrkAPkimv Z8+EfNeOe1ydYZiRrOr8h/+B+C4p X-Google-Smtp-Source: AFSGD/VUcyVRNTF1o0lbOdjfRajLSKag2ecJ/LjBvn96srEnrmCNpsph4E3yer2/ltm0n32pErLQjg== X-Received: by 2002:a02:b4d1:: with SMTP id a17mr13106259jak.27.1545064853607; Mon, 17 Dec 2018 08:40:53 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id u20sm3683028ioc.44.2018.12.17.08.40.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:53 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGeqbZ018629; Mon, 17 Dec 2018 16:40:52 GMT Subject: [PATCH v4 17/30] xprtrdma: Cull dprintk() call sites From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:52 -0500 Message-ID: <20181217164052.24133.20895.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Remove dprintk() call sites that report rare or impossible errors. Leave a few that display high-value low noise status information. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 3 --- net/sunrpc/xprtrdma/rpc_rdma.c | 17 ++++++++++------- net/sunrpc/xprtrdma/transport.c | 33 ++++----------------------------- net/sunrpc/xprtrdma/verbs.c | 34 +++++----------------------------- 4 files changed, 19 insertions(+), 68 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index af8249b..64e1708 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -235,9 +235,6 @@ void xprt_rdma_bc_free_rqst(struct rpc_rqst *rqst) struct rpcrdma_req *req = rpcr_to_rdmar(rqst); struct rpc_xprt *xprt = rqst->rq_xprt; - dprintk("RPC: %s: freeing rqst %p (req %p)\n", - __func__, rqst, req); - rpcrdma_recv_buffer_put(req->rl_reply); req->rl_reply = NULL; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 8de0b9f..5a58769 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1186,17 +1186,20 @@ static int decode_reply_chunk(struct xdr_stream *xdr, u32 *length) p = xdr_inline_decode(xdr, 2 * sizeof(*p)); if (!p) break; - dprintk("RPC: %5u: %s: server reports version error (%u-%u)\n", - rqst->rq_task->tk_pid, __func__, - be32_to_cpup(p), be32_to_cpu(*(p + 1))); + dprintk("RPC: %s: server reports " + "version error (%u-%u), xid %08x\n", __func__, + be32_to_cpup(p), be32_to_cpu(*(p + 1)), + be32_to_cpu(rep->rr_xid)); break; case err_chunk: - dprintk("RPC: %5u: %s: server reports header decoding error\n", - rqst->rq_task->tk_pid, __func__); + dprintk("RPC: %s: server reports " + "header decoding error, xid %08x\n", __func__, + be32_to_cpu(rep->rr_xid)); break; default: - dprintk("RPC: %5u: %s: server reports unrecognized error %d\n", - rqst->rq_task->tk_pid, __func__, be32_to_cpup(p)); + dprintk("RPC: %s: server reports " + "unrecognized error %d, xid %08x\n", __func__, + be32_to_cpup(p), be32_to_cpu(rep->rr_xid)); } r_xprt->rx_stats.bad_reply_count++; diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index f824892..41e4347 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -318,17 +318,12 @@ struct sockaddr *sap; int rc; - if (args->addrlen > sizeof(xprt->addr)) { - dprintk("RPC: %s: address too large\n", __func__); + if (args->addrlen > sizeof(xprt->addr)) return ERR_PTR(-EBADF); - } xprt = xprt_alloc(args->net, sizeof(struct rpcrdma_xprt), 0, 0); - if (xprt == NULL) { - dprintk("RPC: %s: couldn't allocate rpcrdma_xprt\n", - __func__); + if (!xprt) return ERR_PTR(-ENOMEM); - } /* 60 second timeout, no retries */ xprt->timeout = &xprt_rdma_default_timeout; @@ -446,8 +441,6 @@ might_sleep(); - dprintk("RPC: %s: closing xprt %p\n", __func__, xprt); - /* Prevent marshaling and sending of new requests */ xprt_clear_connected(xprt); @@ -854,24 +847,15 @@ void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) void xprt_rdma_cleanup(void) { - int rc; - - dprintk("RPCRDMA Module Removed, deregister RPC RDMA transport\n"); #if IS_ENABLED(CONFIG_SUNRPC_DEBUG) if (sunrpc_table_header) { unregister_sysctl_table(sunrpc_table_header); sunrpc_table_header = NULL; } #endif - rc = xprt_unregister_transport(&xprt_rdma); - if (rc) - dprintk("RPC: %s: xprt_unregister returned %i\n", - __func__, rc); - rc = xprt_unregister_transport(&xprt_rdma_bc); - if (rc) - dprintk("RPC: %s: xprt_unregister(bc) returned %i\n", - __func__, rc); + xprt_unregister_transport(&xprt_rdma); + xprt_unregister_transport(&xprt_rdma_bc); } int xprt_rdma_init(void) @@ -888,15 +872,6 @@ int xprt_rdma_init(void) return rc; } - dprintk("RPCRDMA Module Init, register RPC RDMA transport\n"); - - dprintk("Defaults:\n"); - dprintk("\tSlots %d\n" - "\tMaxInlineRead %d\n\tMaxInlineWrite %d\n", - xprt_rdma_slot_table_entries, - xprt_rdma_max_inline_read, xprt_rdma_max_inline_write); - dprintk("\tPadding 0\n\tMemreg %d\n", xprt_rdma_memreg_strategy); - #if IS_ENABLED(CONFIG_SUNRPC_DEBUG) if (!sunrpc_table_header) sunrpc_table_header = register_sysctl_table(sunrpc_table); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index a6ab216..64672d0 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -309,22 +309,15 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) id = rdma_create_id(xprt->rx_xprt.xprt_net, rpcrdma_cm_event_handler, xprt, RDMA_PS_TCP, IB_QPT_RC); - if (IS_ERR(id)) { - rc = PTR_ERR(id); - dprintk("RPC: %s: rdma_create_id() failed %i\n", - __func__, rc); + if (IS_ERR(id)) return id; - } ia->ri_async_rc = -ETIMEDOUT; rc = rdma_resolve_addr(id, NULL, (struct sockaddr *)&xprt->rx_xprt.addr, RDMA_RESOLVE_TIMEOUT); - if (rc) { - dprintk("RPC: %s: rdma_resolve_addr() failed %i\n", - __func__, rc); + if (rc) goto out; - } rc = wait_for_completion_interruptible_timeout(&ia->ri_done, wtimeout); if (rc < 0) { trace_xprtrdma_conn_tout(xprt); @@ -337,11 +330,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) ia->ri_async_rc = -ETIMEDOUT; rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT); - if (rc) { - dprintk("RPC: %s: rdma_resolve_route() failed %i\n", - __func__, rc); + if (rc) goto out; - } rc = wait_for_completion_interruptible_timeout(&ia->ri_done, wtimeout); if (rc < 0) { trace_xprtrdma_conn_tout(xprt); @@ -540,8 +530,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) 1, IB_POLL_WORKQUEUE); if (IS_ERR(sendcq)) { rc = PTR_ERR(sendcq); - dprintk("RPC: %s: failed to create send CQ: %i\n", - __func__, rc); goto out1; } @@ -550,8 +538,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) 0, IB_POLL_WORKQUEUE); if (IS_ERR(recvcq)) { rc = PTR_ERR(recvcq); - dprintk("RPC: %s: failed to create recv CQ: %i\n", - __func__, rc); goto out2; } @@ -691,11 +677,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) } err = rdma_create_qp(id, ia->ri_pd, &ep->rep_attr); - if (err) { - dprintk("RPC: %s: rdma_create_qp returned %d\n", - __func__, err); + if (err) goto out_destroy; - } /* Atomically replace the transport's ID and QP. */ rc = 0; @@ -726,8 +709,6 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) dprintk("RPC: %s: connecting...\n", __func__); rc = rdma_create_qp(ia->ri_id, ia->ri_pd, &ep->rep_attr); if (rc) { - dprintk("RPC: %s: rdma_create_qp failed %i\n", - __func__, rc); rc = -ENETUNREACH; goto out_noupdate; } @@ -749,11 +730,8 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) rpcrdma_post_recvs(r_xprt, true); rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma); - if (rc) { - dprintk("RPC: %s: rdma_connect() failed with %i\n", - __func__, rc); + if (rc) goto out; - } wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected != 0); if (ep->rep_connected <= 0) { @@ -1088,8 +1066,6 @@ struct rpcrdma_req * out_free: kfree(rep); out: - dprintk("RPC: %s: reply buffer %d alloc failed\n", - __func__, rc); return rc; } From patchwork Mon Dec 17 16:40:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733967 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EEA2D1575 for ; Mon, 17 Dec 2018 16:41:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0684287A1 for ; Mon, 17 Dec 2018 16:41:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D4E112A228; Mon, 17 Dec 2018 16:41:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 929E22A20B for ; Mon, 17 Dec 2018 16:41:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388281AbeLQQlA (ORCPT ); Mon, 17 Dec 2018 11:41:00 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:38564 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388240AbeLQQk7 (ORCPT ); Mon, 17 Dec 2018 11:40:59 -0500 Received: by mail-it1-f193.google.com with SMTP id h65so20906374ith.3; Mon, 17 Dec 2018 08:40:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=H0aZ9IKNvgEE1Hf9G52xNkiw9LXd9hc7toVK3qV8SUQ=; b=ODHGSyqi8ggRhFvypHusH3tmEo+bU/Pga5RVFK019zK5MoMFmajxoL9NLPCRmrYLc/ SXn6H33l2ZIVuP8QN4k6ffW4Y5FzRpTUGTfJTbCF8kQri7MNfbrktIytCU90i/DIXJjL K8HzZ5UBxG4f1C+jFfl1oTt7thN3B/Pk6/y2GvgI4YVUM949z6qNiOt7k/Ls46I43pIh ihbM1NfBAMliFAVK7KoiMsxLaRWjQ0x24JdaNTakuzME+zI56GffeCInDNc6EPqBaXVb P4rzQ9wc7ZJ8rYJBTeNamks8RmXv+IjheFSLpPV1TvWt7mNblW7ivBxtGtabTiMReJ5e 8VVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=H0aZ9IKNvgEE1Hf9G52xNkiw9LXd9hc7toVK3qV8SUQ=; b=T1gjiZdwZo1ouN3tVMvmd/jV8HcIlPlxoVRQv2x8EpoQIRX0OwI6wI5PmkxsfHyiyW 58gjfZ2+xtpEMDDN+FHD71NxCj6cARnAk5mbSNWR8okTaYFGh/SthFzrBB8oXPK1c9Sa ug45Ynjgx3syE7YkEl9rMbpyiEhwlC3P+/eRRjNt88937x1k4c5pv+NFXewdU3nabZdp hpjTkuZqrxLWI3RDPSPoWP4wVkfZ0r7SnQwGOau7cTOsOyDE8ry5Wkc1CmRrJU0B9jND alXjajZfNYXRaPe1y/WZDGgFBLXjP1FQswKK1IYhAr53f7et7pqEGBAZaG88lWo6y0me PZUA== X-Gm-Message-State: AA+aEWbukbxSUTuq8t1qr61n08CvUidX/kG5B4h4ulv4W7gNfzPNaZQu ju8wwPPKfbl91bUa9330l1d6jytg X-Google-Smtp-Source: AFSGD/XvPc2mjGNDqO5mUGpW4SYu2xL57wW/da906F6SRoOiivcbY3NvOTCi6LS1csURjBVz+gV5sg== X-Received: by 2002:a24:76d0:: with SMTP id z199mr11916542itb.165.1545064858753; Mon, 17 Dec 2018 08:40:58 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id o63sm7894960ita.8.2018.12.17.08.40.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:40:58 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGevg9018633; Mon, 17 Dec 2018 16:40:57 GMT Subject: [PATCH v4 18/30] xprtrdma: Remove unused fields from rpcrdma_ia From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:40:57 -0500 Message-ID: <20181217164057.24133.75935.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up. The last use of these fields was in commit 173b8f49b3af ("xprtrdma: Demote "connect" log messages") . Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/xprt_rdma.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 6e104cd..375c7be 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -80,8 +80,6 @@ struct rpcrdma_ia { bool ri_implicit_roundup; enum ib_mr_type ri_mrtype; unsigned long ri_flags; - struct ib_qp_attr ri_qp_attr; - struct ib_qp_init_attr ri_qp_init_attr; }; enum { From patchwork Mon Dec 17 16:41:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733973 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A899A161F for ; Mon, 17 Dec 2018 16:41:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B0B7287A1 for ; Mon, 17 Dec 2018 16:41:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F1F72A215; Mon, 17 Dec 2018 16:41:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5C87287A1 for ; Mon, 17 Dec 2018 16:41:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388240AbeLQQlH (ORCPT ); Mon, 17 Dec 2018 11:41:07 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:54028 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlH (ORCPT ); Mon, 17 Dec 2018 11:41:07 -0500 Received: by mail-it1-f195.google.com with SMTP id g85so19466044ita.3; Mon, 17 Dec 2018 08:41:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=/fajm+Q6Ev3B90pszxLhoKhMSyNLhkbB0DmCyy0qBQc=; b=tC3c6RwAWhhhoMnMOh95TdEYVdjwKyr3mDiDho3/GiKISaEGJ0sXKMVrtYAH/iRUZ0 TXcSV1yoJBBULpqhg2KKCYGNem9wvjls1suQ+gKmgot/hLPqHCKW+0YYbltSjMveiUw+ IjsQS+v4aOJB4M5S7n1dmGoh94PD7kKLLgJoyN7OnkW4jszkdxWt3wxkJZLW6CDBSUWS au7jkZSM2bJerMfxeAypOgYg2FKtmXkmifPa+S2vhRMTgGAKLR6FOLQKkyvMslGgf+uk DrATQ+6lvj8uLnFJieGmbvYYzNn4FaKI3fveuDfjUf+k4DHabHTdsSscdpKtdKnnZa6o ofmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=/fajm+Q6Ev3B90pszxLhoKhMSyNLhkbB0DmCyy0qBQc=; b=Om6EBTrG6EIsMB4y37p9xiZxqU3Kpf8Eg+S8bnWPUhMX0V+9RuPLChk98i7gCSa1A1 wdpB9Ra29kKrPfap+lZkIEBNQbnUmby/43T63kqiD7qHsmbj2fzqJLTe8qoKISgeBqtc p36Rkfvx0vU/99/Soi1I35sksa3dobnlRPLis3vSqMc+2bZ3WOkweTf71pGeb4KYRcFg koUUVskPzRhdDLKkd+4Wz4syyUTKhgGVKntz5FuiwLzRGdcSCbNcIxJPlPgtvZW6pmXM lFr0Cdc8eiqwsm1L2Kc3hwks7rDgJxKkba/DczYFiaW+IG3qkFaOXwK3NCG64KRFkOA1 f8ow== X-Gm-Message-State: AA+aEWZJ7H71pVhWrmUc8jDghc8sBXsL4Qq3rj4ciQ5lYLBZXJPOuAkA QH9BzypFmqibDsytgla9Q/Zym6fh X-Google-Smtp-Source: AFSGD/XZWqGlSStXNgTHM71Hy8yWU5IAa5Bcc3VkKuOfVe8+wrq9KiXKvG/KKF8W73sAtlYwTfQegw== X-Received: by 2002:a24:65c8:: with SMTP id u191mr13061612itb.7.1545064865315; Mon, 17 Dec 2018 08:41:05 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id x99sm7369113ita.23.2018.12.17.08.41.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:03 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGf2Y1018636; Mon, 17 Dec 2018 16:41:02 GMT Subject: [PATCH v4 19/30] xprtrdma: Clean up of xprtrdma chunk trace points From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:02 -0500 Message-ID: <20181217164102.24133.76850.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The chunk-related trace points capture nearly the same information as the MR-related trace points. Also, rename them so globbing can be used to enable or disable these trace points more easily. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 42 +++++++++++++++++++++++++--------------- net/sunrpc/xprtrdma/rpc_rdma.c | 6 +++--- 2 files changed, 29 insertions(+), 19 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 2efe2d7..e9fbf7d 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -97,7 +97,6 @@ TP_STRUCT__entry( __field(unsigned int, task_id) __field(unsigned int, client_id) - __field(const void *, mr) __field(unsigned int, pos) __field(int, nents) __field(u32, handle) @@ -109,7 +108,6 @@ TP_fast_assign( __entry->task_id = task->tk_pid; __entry->client_id = task->tk_client->cl_clid; - __entry->mr = mr; __entry->pos = pos; __entry->nents = mr->mr_nents; __entry->handle = mr->mr_handle; @@ -118,8 +116,8 @@ __entry->nsegs = nsegs; ), - TP_printk("task:%u@%u mr=%p pos=%u %u@0x%016llx:0x%08x (%s)", - __entry->task_id, __entry->client_id, __entry->mr, + TP_printk("task:%u@%u pos=%u %u@0x%016llx:0x%08x (%s)", + __entry->task_id, __entry->client_id, __entry->pos, __entry->length, (unsigned long long)__entry->offset, __entry->handle, __entry->nents < __entry->nsegs ? "more" : "last" @@ -127,7 +125,7 @@ ); #define DEFINE_RDCH_EVENT(name) \ - DEFINE_EVENT(xprtrdma_rdch_event, name, \ + DEFINE_EVENT(xprtrdma_rdch_event, xprtrdma_chunk_##name,\ TP_PROTO( \ const struct rpc_task *task, \ unsigned int pos, \ @@ -148,7 +146,6 @@ TP_STRUCT__entry( __field(unsigned int, task_id) __field(unsigned int, client_id) - __field(const void *, mr) __field(int, nents) __field(u32, handle) __field(u32, length) @@ -159,7 +156,6 @@ TP_fast_assign( __entry->task_id = task->tk_pid; __entry->client_id = task->tk_client->cl_clid; - __entry->mr = mr; __entry->nents = mr->mr_nents; __entry->handle = mr->mr_handle; __entry->length = mr->mr_length; @@ -167,8 +163,8 @@ __entry->nsegs = nsegs; ), - TP_printk("task:%u@%u mr=%p %u@0x%016llx:0x%08x (%s)", - __entry->task_id, __entry->client_id, __entry->mr, + TP_printk("task:%u@%u %u@0x%016llx:0x%08x (%s)", + __entry->task_id, __entry->client_id, __entry->length, (unsigned long long)__entry->offset, __entry->handle, __entry->nents < __entry->nsegs ? "more" : "last" @@ -176,7 +172,7 @@ ); #define DEFINE_WRCH_EVENT(name) \ - DEFINE_EVENT(xprtrdma_wrch_event, name, \ + DEFINE_EVENT(xprtrdma_wrch_event, xprtrdma_chunk_##name,\ TP_PROTO( \ const struct rpc_task *task, \ struct rpcrdma_mr *mr, \ @@ -234,6 +230,18 @@ ), \ TP_ARGS(wc, frwr)) +TRACE_DEFINE_ENUM(DMA_BIDIRECTIONAL); +TRACE_DEFINE_ENUM(DMA_TO_DEVICE); +TRACE_DEFINE_ENUM(DMA_FROM_DEVICE); +TRACE_DEFINE_ENUM(DMA_NONE); + +#define xprtrdma_show_direction(x) \ + __print_symbolic(x, \ + { DMA_BIDIRECTIONAL, "BIDIR" }, \ + { DMA_TO_DEVICE, "TO_DEVICE" }, \ + { DMA_FROM_DEVICE, "FROM_DEVICE" }, \ + { DMA_NONE, "NONE" }) + DECLARE_EVENT_CLASS(xprtrdma_mr, TP_PROTO( const struct rpcrdma_mr *mr @@ -246,6 +254,7 @@ __field(u32, handle) __field(u32, length) __field(u64, offset) + __field(u32, dir) ), TP_fast_assign( @@ -253,12 +262,13 @@ __entry->handle = mr->mr_handle; __entry->length = mr->mr_length; __entry->offset = mr->mr_offset; + __entry->dir = mr->mr_dir; ), - TP_printk("mr=%p %u@0x%016llx:0x%08x", + TP_printk("mr=%p %u@0x%016llx:0x%08x (%s)", __entry->mr, __entry->length, - (unsigned long long)__entry->offset, - __entry->handle + (unsigned long long)__entry->offset, __entry->handle, + xprtrdma_show_direction(__entry->dir) ) ); @@ -437,9 +447,9 @@ DEFINE_RXPRT_EVENT(xprtrdma_nomrs); -DEFINE_RDCH_EVENT(xprtrdma_read_chunk); -DEFINE_WRCH_EVENT(xprtrdma_write_chunk); -DEFINE_WRCH_EVENT(xprtrdma_reply_chunk); +DEFINE_RDCH_EVENT(read); +DEFINE_WRCH_EVENT(write); +DEFINE_WRCH_EVENT(reply); TRACE_DEFINE_ENUM(rpcrdma_noch); TRACE_DEFINE_ENUM(rpcrdma_readch); diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 5a58769..54fbd70 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -365,7 +365,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (encode_read_segment(xdr, mr, pos) < 0) return -EMSGSIZE; - trace_xprtrdma_read_chunk(rqst->rq_task, pos, mr, nsegs); + trace_xprtrdma_chunk_read(rqst->rq_task, pos, mr, nsegs); r_xprt->rx_stats.read_chunk_count++; nsegs -= mr->mr_nents; } while (nsegs); @@ -422,7 +422,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (encode_rdma_segment(xdr, mr) < 0) return -EMSGSIZE; - trace_xprtrdma_write_chunk(rqst->rq_task, mr, nsegs); + trace_xprtrdma_chunk_write(rqst->rq_task, mr, nsegs); r_xprt->rx_stats.write_chunk_count++; r_xprt->rx_stats.total_rdma_request += mr->mr_length; nchunks++; @@ -479,7 +479,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, if (encode_rdma_segment(xdr, mr) < 0) return -EMSGSIZE; - trace_xprtrdma_reply_chunk(rqst->rq_task, mr, nsegs); + trace_xprtrdma_chunk_reply(rqst->rq_task, mr, nsegs); r_xprt->rx_stats.reply_chunk_count++; r_xprt->rx_stats.total_rdma_request += mr->mr_length; nchunks++; From patchwork Mon Dec 17 16:41:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B4ABC161F for ; Mon, 17 Dec 2018 16:41:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A7044287A1 for ; Mon, 17 Dec 2018 16:41:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B8392A215; Mon, 17 Dec 2018 16:41:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A3E6287A1 for ; Mon, 17 Dec 2018 16:41:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388309AbeLQQlL (ORCPT ); Mon, 17 Dec 2018 11:41:11 -0500 Received: from mail-io1-f66.google.com ([209.85.166.66]:45515 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlL (ORCPT ); Mon, 17 Dec 2018 11:41:11 -0500 Received: by mail-io1-f66.google.com with SMTP id p7so2844910iog.12; Mon, 17 Dec 2018 08:41:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=dTTFxQBHqNac1U654cQS+n8MCI79WsiCTWna3Rmj/a4=; b=HgSPVF3LHf1Tw3DLhUzfIWtz+XSknqmnvQcGFHnL8kGyzzPP3dsZoPoY5X/ETcV+qk yCEZ7SQ1Hf6RKDe8QxWWjjKeKx/DHbzwbY0m2f1ap7PbEbMKJGTjEo9u+SWLYmHSjbwh BFr3D2u8DPOxxp3Kzqg1NrkpENcwCntX2OpqKM2vNkmvZ1dk4R/poUZp1qs3DUeZ7+1F 3lN76airxu1u5D9bHRDsxTrcgkKSQfAnkYEGOnMRQkvGJuIyjEE0BKHGyxiCK+L/NrtR b3HswpMwTWcy4ghMLvBq7EJmAKX36gYbTXaXtewwAh9GEsI9Q81w4Jg9uuQbcCB//Pbw gZFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=dTTFxQBHqNac1U654cQS+n8MCI79WsiCTWna3Rmj/a4=; b=kKLqD4BC5x9GWsVkaEWZuiqh7PMOj9LdUAovsrcW3Lb0ytHT1ldfZ/3dHUgj+Dq86J Eciuhu/FGHtc/DCARyA4EV0qDl6qKLYDWH5ejv8qkZRz4CGY43fFE+SN83ZhQ1iqcNYA tkmoFY7niSOUBTJskHQrMbgO4F5HlLdr4nzH1+vT9znaOjumEcHlUtgG9jrsyetaMjyg xAcp8C8ix2jwiiab9FniaokVue5+l/m41adz8umjQWsEsj+wUKVzywZJEyiIpzWkcCxc wbVY85+OLAC4DAQW3qVIJg7z5TSj+X+z9r8qMR5I2hpIFvev5zqXYye4RIpU/Vnz4mxU gqZg== X-Gm-Message-State: AA+aEWaBrYNPwN/FXGK1uLu63UVq/Y26QFdixCHiIgOv+w6TarkffkVw JCBpGXB0RYn2XZ61YfcnPYgBgoWU X-Google-Smtp-Source: AFSGD/USqtZI/g3f88Q4MvhCwmJZRrc/IZunthxLM4nRklUr5w3jdIAGwhxig2uQsjgGR/Ze8V6l1g== X-Received: by 2002:a5e:a513:: with SMTP id 19mr11971392iog.151.1545064869409; Mon, 17 Dec 2018 08:41:09 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id s15sm6093300ioe.52.2018.12.17.08.41.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:08 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGf8xj018639; Mon, 17 Dec 2018 16:41:08 GMT Subject: [PATCH v4 20/30] xprtrdma: Relocate the xprtrdma_mr_map trace points From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:08 -0500 Message-ID: <20181217164108.24133.31649.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The mr_map trace points were capturing information about the previous use of the MR rather than about the segment that was just mapped. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 97f88bb..1f508f4 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -438,7 +438,6 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, mr->mr_nents = ib_dma_map_sg(ia->ri_device, mr->mr_sg, i, mr->mr_dir); if (!mr->mr_nents) goto out_dmamap_err; - trace_xprtrdma_mr_map(mr); ibmr = frwr->fr_mr; n = ib_map_mr_sg(ibmr, mr->mr_sg, mr->mr_nents, NULL, PAGE_SIZE); @@ -460,6 +459,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, mr->mr_handle = ibmr->rkey; mr->mr_length = ibmr->length; mr->mr_offset = ibmr->iova; + trace_xprtrdma_mr_map(mr); *out = mr; return seg; From patchwork Mon Dec 17 16:41:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733979 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F42713AD for ; Mon, 17 Dec 2018 16:41:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3962E287A1 for ; Mon, 17 Dec 2018 16:41:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2E2DE2A225; Mon, 17 Dec 2018 16:41:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE283287A1 for ; Mon, 17 Dec 2018 16:41:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388310AbeLQQlQ (ORCPT ); Mon, 17 Dec 2018 11:41:16 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:50278 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlQ (ORCPT ); Mon, 17 Dec 2018 11:41:16 -0500 Received: by mail-it1-f196.google.com with SMTP id z7so19482548iti.0; Mon, 17 Dec 2018 08:41:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=AuCrVQqqfFuSRgtyALKDS0UafgRojodNvuiKzeqp4Jc=; b=RKIcWbnqEGxQM+SZc+jafbjClCV7gLIeBrrxKukh7taTgeMu+3x1Nm7oXW36iPGKe4 LGvTF1sAw8eGTLsUFXT5TqjA7OFcQmLT/NdY9qp845qgQJLvExrZ77KH4AS8jeo7LmIL qU0tML+NNBoCmyPnbqDq5kWJXsCAqz2HnZ0qYqgsX4RyUKRI1N0fV03jv1gzhwGjg0xV 4FAHzcw3pEfwc/hZ0wz91L0p8XnjJgdCXdQzWeLKMGvVgLX4owcj1qwoNMpQhzeJJRio 5rUzGLNthSmTVdHtQamc0uOm4KRxon2h2abhHnOBlEFa5ZFVca16U5X6vpJMXpiMvGHr SyZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=AuCrVQqqfFuSRgtyALKDS0UafgRojodNvuiKzeqp4Jc=; b=Fi864LZy6njnlfPRM8vyWv2H+aqNVrCd1x01ie703b3OVb0bsVj6lHMSf08RrZP5RA XY0NMsR0nigATpyqHnpFOuXrdQjwl0WlZjp7szZ4WMmTeRREpz94JwZ9Aow+ZhbqYxd8 QUGvfVaErIlMJJy7BqlkloHSuXjKyPKDTiAi+w18nIIHSoFA0huuECEWQB+S+QLdfy9M jXlsptEG6Gxc5jVDFL/KVRF2tD9ymxdmWcM5DO2mvxFhaG3AvcBRo2rT0YpAzFekWQPN RwTn5OyVzKoImtYIpw8sds/avaxQKwbahzxfZWVjfAjn2QuGvvfUy6+UfDcHQXOVGZTj rTKg== X-Gm-Message-State: AA+aEWY9OTtEWFBfCJe3wQT3Qvs3g74493uQKUKkZRaHY+MS1QgrYU/p r0N28XMKkJXnXpy1FacttLY+NQj+ X-Google-Smtp-Source: AFSGD/UdvU+fdl0eyYZBXpvnFR0vNSO4HpZCCnz8a9fJ324H9LbGfOPvanWsyAD1UzMnR1NXrPILbg== X-Received: by 2002:a24:b951:: with SMTP id k17mr11756175iti.147.1545064874671; Mon, 17 Dec 2018 08:41:14 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id j5sm552235iop.17.2018.12.17.08.41.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:14 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfDvG018642; Mon, 17 Dec 2018 16:41:13 GMT Subject: [PATCH v4 21/30] xprtrdma: Add trace points for calls to transport switch methods From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:13 -0500 Message-ID: <20181217164113.24133.16198.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Name them "trace_xprtrdma_op_*" so they can be easily enabled as a group. No trace point is added where the generic layer already has observability. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 10 ++++++---- net/sunrpc/xprtrdma/transport.c | 18 +++++++++++------- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index e9fbf7d..3d068bb 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -381,11 +381,13 @@ DEFINE_RXPRT_EVENT(xprtrdma_conn_start); DEFINE_RXPRT_EVENT(xprtrdma_conn_tout); DEFINE_RXPRT_EVENT(xprtrdma_create); -DEFINE_RXPRT_EVENT(xprtrdma_destroy); +DEFINE_RXPRT_EVENT(xprtrdma_op_destroy); DEFINE_RXPRT_EVENT(xprtrdma_remove); DEFINE_RXPRT_EVENT(xprtrdma_reinsert); DEFINE_RXPRT_EVENT(xprtrdma_reconnect); -DEFINE_RXPRT_EVENT(xprtrdma_inject_dsc); +DEFINE_RXPRT_EVENT(xprtrdma_op_inject_dsc); +DEFINE_RXPRT_EVENT(xprtrdma_op_close); +DEFINE_RXPRT_EVENT(xprtrdma_op_connect); TRACE_EVENT(xprtrdma_qp_event, TP_PROTO( @@ -834,7 +836,7 @@ ** Allocation/release of rpcrdma_reqs and rpcrdma_reps **/ -TRACE_EVENT(xprtrdma_allocate, +TRACE_EVENT(xprtrdma_op_allocate, TP_PROTO( const struct rpc_task *task, const struct rpcrdma_req *req @@ -864,7 +866,7 @@ ) ); -TRACE_EVENT(xprtrdma_rpc_done, +TRACE_EVENT(xprtrdma_op_free, TP_PROTO( const struct rpc_task *task, const struct rpcrdma_req *req diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 41e4347..cb855b2 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -268,7 +268,7 @@ { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - trace_xprtrdma_inject_dsc(r_xprt); + trace_xprtrdma_op_inject_dsc(r_xprt); rdma_disconnect(r_xprt->rx_ia.ri_id); } @@ -284,7 +284,7 @@ { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - trace_xprtrdma_destroy(r_xprt); + trace_xprtrdma_op_destroy(r_xprt); cancel_delayed_work_sync(&r_xprt->rx_connect_worker); @@ -418,7 +418,7 @@ out2: rpcrdma_ia_close(&new_xprt->rx_ia); out1: - trace_xprtrdma_destroy(new_xprt); + trace_xprtrdma_op_destroy(new_xprt); xprt_rdma_free_addresses(xprt); xprt_free(xprt); return ERR_PTR(rc); @@ -428,7 +428,8 @@ * xprt_rdma_close - close a transport connection * @xprt: transport context * - * Called during transport shutdown, reconnect, or device removal. + * Called during autoclose or device removal. + * * Caller holds @xprt's send lock to prevent activity on this * transport while the connection is torn down. */ @@ -441,6 +442,8 @@ might_sleep(); + trace_xprtrdma_op_close(r_xprt); + /* Prevent marshaling and sending of new requests */ xprt_clear_connected(xprt); @@ -526,6 +529,7 @@ { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); + trace_xprtrdma_op_connect(r_xprt); if (r_xprt->rx_ep.rep_connected != 0) { /* Reconnect */ schedule_delayed_work(&r_xprt->rx_connect_worker, @@ -660,11 +664,11 @@ rqst->rq_buffer = req->rl_sendbuf->rg_base; rqst->rq_rbuffer = req->rl_recvbuf->rg_base; - trace_xprtrdma_allocate(task, req); + trace_xprtrdma_op_allocate(task, req); return 0; out_fail: - trace_xprtrdma_allocate(task, NULL); + trace_xprtrdma_op_allocate(task, NULL); return -ENOMEM; } @@ -683,7 +687,7 @@ if (test_bit(RPCRDMA_REQ_F_PENDING, &req->rl_flags)) rpcrdma_release_rqst(r_xprt, req); - trace_xprtrdma_rpc_done(task, req); + trace_xprtrdma_op_free(task, req); } /** From patchwork Mon Dec 17 16:41:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733983 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34AD913AD for ; Mon, 17 Dec 2018 16:41:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22F78287A1 for ; Mon, 17 Dec 2018 16:41:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 16C532A225; Mon, 17 Dec 2018 16:41:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DBD2287A1 for ; Mon, 17 Dec 2018 16:41:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388311AbeLQQlW (ORCPT ); Mon, 17 Dec 2018 11:41:22 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:35468 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388223AbeLQQlW (ORCPT ); Mon, 17 Dec 2018 11:41:22 -0500 Received: by mail-it1-f195.google.com with SMTP id p197so20943325itp.0; Mon, 17 Dec 2018 08:41:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=mMDbUnOSo+R5e+BESpDss6CE0mI2T44DAW3eI1MfElE=; b=lQHo/D1HJaF6na26B/seOWYSW9bGAX208Ngm1MCXo1vUsMHI81uJkUE2rDWMGyDAzj IFJLFy76AG6QRTDy5yIhu5r/Z8x5EtpxjzwnD058YZxsQuC2LzDx09v60wvfsMVH68H7 TKtj3bybJm8JE/bxc+dtRtLSJtiYNm0SEpkKMPsfzA/lPsKBMdWulBV2uigc/zOybWts B+92opa6Rs6Ah9oc05RhQf8AH/5fIeTpbbWq4javyhFKVs12NJtrRCrRevt3/WqXW4gi OI650TucKvuAhu8WgY7fXuvgZCg2GYu7zks+g7+Hbzz/MBrCFfN4jbMIcLK/G2zGWsB2 se3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=mMDbUnOSo+R5e+BESpDss6CE0mI2T44DAW3eI1MfElE=; b=fZPYlL7GIqnu1hV5f+d6Lbk9EkqOTP+OAOvED7mhHHR1x3/Er0ZNXVdO9ooFXlfu0V 6zHV/1o20qhuLU4VlWTEx4zlaLA+p1Www5vbBWKdCGz6v6PaQb47YcvSHGlB9+R28Cxj 9PiZve/YFLB4/0i5CoUI/WTfC+JpGxPiQLbcPJ6rctxSlbj9a5Q23NfNOUTBT9ixDT6h Z7ThHZIrM4KavE/n0fKrRyeYXDE4/2wZmkGzJq8t5aimhaGYsCyLKSbRaj0VFK+ISYcG kRBAsX13EjM6dggJZwzsOKDUzNd/PJ7GhvKHR41bZAv9IeKWtIAK6H32EyX6fuDLIl7V IHBQ== X-Gm-Message-State: AA+aEWbfK3uKkYtwQFgpTtEhgxx8CUTrHk+Uu8eTjivDMUXwkJP/mQPj 1UwVKwuQ0Hqjnq2LTrEdSlt42Zyu X-Google-Smtp-Source: AFSGD/Whui8Z4VVuXf2ieO0v7nF28ZPmzxsqM8eWqRpp+Sx8EUlq6DEJ7+K/hu/2RcvBGqbov27KDA== X-Received: by 2002:a24:e44:: with SMTP id 65mr11386214ite.154.1545064880048; Mon, 17 Dec 2018 08:41:20 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id u9sm5421111iol.77.2018.12.17.08.41.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:19 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfIgw018645; Mon, 17 Dec 2018 16:41:18 GMT Subject: [PATCH v4 22/30] xprtrdma: Trace mapping, alloc, and dereg failures From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:18 -0500 Message-ID: <20181217164118.24133.40400.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These are rare, but can be helpful at tracking down DMAR and other problems. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 136 ++++++++++++++++++++++++++++++++++++++++ net/sunrpc/xprtrdma/frwr_ops.c | 12 +--- net/sunrpc/xprtrdma/rpc_rdma.c | 2 - net/sunrpc/xprtrdma/verbs.c | 4 + 4 files changed, 144 insertions(+), 10 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 3d068bb..ce528d5 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -10,6 +10,7 @@ #if !defined(_TRACE_RPCRDMA_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_RPCRDMA_H +#include #include #include @@ -663,12 +664,147 @@ DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li); DEFINE_FRWR_DONE_EVENT(xprtrdma_wc_li_wake); +TRACE_EVENT(xprtrdma_frwr_alloc, + TP_PROTO( + const struct rpcrdma_mr *mr, + int rc + ), + + TP_ARGS(mr, rc), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(int, rc) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->rc = rc; + ), + + TP_printk("mr=%p: rc=%d", + __entry->mr, __entry->rc + ) +); + +TRACE_EVENT(xprtrdma_frwr_dereg, + TP_PROTO( + const struct rpcrdma_mr *mr, + int rc + ), + + TP_ARGS(mr, rc), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(u32, handle) + __field(u32, length) + __field(u64, offset) + __field(u32, dir) + __field(int, rc) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->handle = mr->mr_handle; + __entry->length = mr->mr_length; + __entry->offset = mr->mr_offset; + __entry->dir = mr->mr_dir; + __entry->rc = rc; + ), + + TP_printk("mr=%p %u@0x%016llx:0x%08x (%s): rc=%d", + __entry->mr, __entry->length, + (unsigned long long)__entry->offset, __entry->handle, + xprtrdma_show_direction(__entry->dir), + __entry->rc + ) +); + +TRACE_EVENT(xprtrdma_frwr_sgerr, + TP_PROTO( + const struct rpcrdma_mr *mr, + int sg_nents + ), + + TP_ARGS(mr, sg_nents), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(u64, addr) + __field(u32, dir) + __field(int, nents) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->addr = mr->mr_sg->dma_address; + __entry->dir = mr->mr_dir; + __entry->nents = sg_nents; + ), + + TP_printk("mr=%p dma addr=0x%llx (%s) sg_nents=%d", + __entry->mr, __entry->addr, + xprtrdma_show_direction(__entry->dir), + __entry->nents + ) +); + +TRACE_EVENT(xprtrdma_frwr_maperr, + TP_PROTO( + const struct rpcrdma_mr *mr, + int num_mapped + ), + + TP_ARGS(mr, num_mapped), + + TP_STRUCT__entry( + __field(const void *, mr) + __field(u64, addr) + __field(u32, dir) + __field(int, num_mapped) + __field(int, nents) + ), + + TP_fast_assign( + __entry->mr = mr; + __entry->addr = mr->mr_sg->dma_address; + __entry->dir = mr->mr_dir; + __entry->num_mapped = num_mapped; + __entry->nents = mr->mr_nents; + ), + + TP_printk("mr=%p dma addr=0x%llx (%s) nents=%d of %d", + __entry->mr, __entry->addr, + xprtrdma_show_direction(__entry->dir), + __entry->num_mapped, __entry->nents + ) +); + DEFINE_MR_EVENT(localinv); DEFINE_MR_EVENT(map); DEFINE_MR_EVENT(unmap); DEFINE_MR_EVENT(remoteinv); DEFINE_MR_EVENT(recycle); +TRACE_EVENT(xprtrdma_dma_maperr, + TP_PROTO( + u64 addr + ), + + TP_ARGS(addr), + + TP_STRUCT__entry( + __field(u64, addr) + ), + + TP_fast_assign( + __entry->addr = addr; + ), + + TP_printk("dma addr=0x%llx\n", __entry->addr) +); + /** ** Reply events **/ diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 1f508f4..8a0f1a6 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -113,8 +113,7 @@ void frwr_release_mr(struct rpcrdma_mr *mr) rc = ib_dereg_mr(mr->frwr.fr_mr); if (rc) - pr_err("rpcrdma: final ib_dereg_mr for %p returned %i\n", - mr, rc); + trace_xprtrdma_frwr_dereg(mr, rc); kfree(mr->mr_sg); kfree(mr); } @@ -177,8 +176,7 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) out_mr_err: rc = PTR_ERR(frwr->fr_mr); - dprintk("RPC: %s: ib_alloc_mr status %i\n", - __func__, rc); + trace_xprtrdma_frwr_alloc(mr, rc); return rc; out_list_err: @@ -465,15 +463,13 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, return seg; out_dmamap_err: - pr_err("rpcrdma: failed to DMA map sg %p sg_nents %d\n", - mr->mr_sg, i); frwr->fr_state = FRWR_IS_INVALID; + trace_xprtrdma_frwr_sgerr(mr, i); rpcrdma_mr_put(mr); return ERR_PTR(-EIO); out_mapmr_err: - pr_err("rpcrdma: failed to map mr %p (%d/%d)\n", - frwr->fr_mr, n, mr->mr_nents); + trace_xprtrdma_frwr_maperr(mr, n); rpcrdma_mr_recycle(mr); return ERR_PTR(-EIO); } diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 54fbd70..062aee9 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -665,7 +665,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, out_mapping_err: rpcrdma_unmap_sendctx(sc); - pr_err("rpcrdma: Send mapping error\n"); + trace_xprtrdma_dma_maperr(sge[sge_no].addr); return false; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 64672d0..b46e2f9 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1392,8 +1392,10 @@ struct rpcrdma_regbuf * (void *)rb->rg_base, rdmab_length(rb), rb->rg_direction); - if (ib_dma_mapping_error(device, rdmab_addr(rb))) + if (ib_dma_mapping_error(device, rdmab_addr(rb))) { + trace_xprtrdma_dma_maperr(rdmab_addr(rb)); return false; + } rb->rg_device = device; rb->rg_iov.lkey = ia->ri_pd->local_dma_lkey; From patchwork Mon Dec 17 16:41:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733989 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8050E161F for ; Mon, 17 Dec 2018 16:41:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7203F2A215 for ; Mon, 17 Dec 2018 16:41:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 661422A225; Mon, 17 Dec 2018 16:41:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,UPPERCASE_75_100 autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 651992A228 for ; Mon, 17 Dec 2018 16:41:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388325AbeLQQl2 (ORCPT ); Mon, 17 Dec 2018 11:41:28 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:52919 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQl1 (ORCPT ); Mon, 17 Dec 2018 11:41:27 -0500 Received: by mail-it1-f193.google.com with SMTP id g76so19472252itg.2; Mon, 17 Dec 2018 08:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=2mLx28XWl8xoUfC5kvXH/cRNSXyhzKZxmuWP8HjEhqU=; b=kxomp/O43/cbz4/iRRAYzo4/yUmgiL66PQ5p5qUqSdMRiv4uLxmI5dX2uUxi4LV2fC tVSmDUHzljMaKaR4Sa5i5GBkLiCGVyVD9VCTokLzDVTgwCym/rvdThLLH0h7OcTaGBP4 eG7D4cuo93r5FNUwyKhQs5WDleZQhFB9P7/AT04lpfC5jLizZg1meB4h/oK0NqR+RP+D OZNUG9aZf0tXNaNXb/zLvRu3TEh7NFLKy917ciy7wqzTyQfOLbGd93NeKMXuJhNW/0jw LaG73P6yl+tmFIclEePLz2vs5zmRpdt/ZCqPtz/FoTEnNA7stAdoPk7trjlOk6Y422wT RZjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=2mLx28XWl8xoUfC5kvXH/cRNSXyhzKZxmuWP8HjEhqU=; b=tIpsDVnQBWiJE/jy38UYIWK77Tv3/UADfNRmpAgqsP1epatbq7i/eP4wv7IdMEF2yS mJEWMmaVsO8BS4Sa9OHruGP1gdM8PaGCAa2+6rx2YNtiNVkV+P6wt6E8Y4nRRjoN65kF nPDzugkCBIhNx6Bd5cQ7jtsBk5vlbTu4CPhlAusjH8uX2pVToiiFOc5zD3R/Eo8JQ3Ed F0LlNmtL+ma/7P7vrZyT3xxfiiUpkqDiAzb5qZ45trejjcPHrsdz9tu7wFXz0LcYagUY uiKCK1FHXGp8vNQvReQ3od57lF1Z61jPi2tK04xwACHTecmm/+TDGE7V4DgHKR/wiEzx P40A== X-Gm-Message-State: AA+aEWa7or6Is6bt4n+1jsJFGbi+T0trC9YO55YkRagf+/h0Dam9Eff+ xTldelXjC2esqi57HOrLOc0DNrYj X-Google-Smtp-Source: AFSGD/Uu6S0AV7E18mW90MZOWm8FfLy9SpOm+jqoqLjzATm7XTZ/h2QlzFd57BZjyRdDmff7Sd3IUg== X-Received: by 2002:a24:f14d:: with SMTP id q13mr11853631iti.166.1545064885741; Mon, 17 Dec 2018 08:41:25 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id v5sm5163954iob.78.2018.12.17.08.41.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:25 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfOjW018648; Mon, 17 Dec 2018 16:41:24 GMT Subject: [PATCH v4 23/30] NFS: Fix NFSv4 symbolic trace point output From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:24 -0500 Message-ID: <20181217164124.24133.23868.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These symbolic values were not being displayed in string form. TRACE_DEFINE_ENUM was missing in many cases. It also turns out that __print_symbolic wants an unsigned long in the first field... Signed-off-by: Chuck Lever --- fs/nfs/nfs4trace.h | 456 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 313 insertions(+), 143 deletions(-) diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index b1483b3..b4557cf 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -10,157 +10,302 @@ #include +TRACE_DEFINE_ENUM(EPERM); +TRACE_DEFINE_ENUM(ENOENT); +TRACE_DEFINE_ENUM(EIO); +TRACE_DEFINE_ENUM(ENXIO); +TRACE_DEFINE_ENUM(EACCES); +TRACE_DEFINE_ENUM(EEXIST); +TRACE_DEFINE_ENUM(EXDEV); +TRACE_DEFINE_ENUM(ENOTDIR); +TRACE_DEFINE_ENUM(EISDIR); +TRACE_DEFINE_ENUM(EFBIG); +TRACE_DEFINE_ENUM(ENOSPC); +TRACE_DEFINE_ENUM(EROFS); +TRACE_DEFINE_ENUM(EMLINK); +TRACE_DEFINE_ENUM(ENAMETOOLONG); +TRACE_DEFINE_ENUM(ENOTEMPTY); +TRACE_DEFINE_ENUM(EDQUOT); +TRACE_DEFINE_ENUM(ESTALE); +TRACE_DEFINE_ENUM(EBADHANDLE); +TRACE_DEFINE_ENUM(EBADCOOKIE); +TRACE_DEFINE_ENUM(ENOTSUPP); +TRACE_DEFINE_ENUM(ETOOSMALL); +TRACE_DEFINE_ENUM(EREMOTEIO); +TRACE_DEFINE_ENUM(EBADTYPE); +TRACE_DEFINE_ENUM(EAGAIN); +TRACE_DEFINE_ENUM(ELOOP); +TRACE_DEFINE_ENUM(EOPNOTSUPP); +TRACE_DEFINE_ENUM(EDEADLK); +TRACE_DEFINE_ENUM(ENOMEM); +TRACE_DEFINE_ENUM(EKEYEXPIRED); +TRACE_DEFINE_ENUM(ETIMEDOUT); +TRACE_DEFINE_ENUM(ERESTARTSYS); +TRACE_DEFINE_ENUM(ECONNREFUSED); +TRACE_DEFINE_ENUM(ECONNRESET); +TRACE_DEFINE_ENUM(ENETUNREACH); +TRACE_DEFINE_ENUM(EHOSTUNREACH); +TRACE_DEFINE_ENUM(EHOSTDOWN); +TRACE_DEFINE_ENUM(EPIPE); +TRACE_DEFINE_ENUM(EPFNOSUPPORT); +TRACE_DEFINE_ENUM(EPROTONOSUPPORT); + +TRACE_DEFINE_ENUM(NFS4_OK); +TRACE_DEFINE_ENUM(NFS4ERR_ACCESS); +TRACE_DEFINE_ENUM(NFS4ERR_ATTRNOTSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_ADMIN_REVOKED); +TRACE_DEFINE_ENUM(NFS4ERR_BACK_CHAN_BUSY); +TRACE_DEFINE_ENUM(NFS4ERR_BADCHAR); +TRACE_DEFINE_ENUM(NFS4ERR_BADHANDLE); +TRACE_DEFINE_ENUM(NFS4ERR_BADIOMODE); +TRACE_DEFINE_ENUM(NFS4ERR_BADLAYOUT); +TRACE_DEFINE_ENUM(NFS4ERR_BADLABEL); +TRACE_DEFINE_ENUM(NFS4ERR_BADNAME); +TRACE_DEFINE_ENUM(NFS4ERR_BADOWNER); +TRACE_DEFINE_ENUM(NFS4ERR_BADSESSION); +TRACE_DEFINE_ENUM(NFS4ERR_BADSLOT); +TRACE_DEFINE_ENUM(NFS4ERR_BADTYPE); +TRACE_DEFINE_ENUM(NFS4ERR_BADXDR); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_COOKIE); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_HIGH_SLOT); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_RANGE); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_SEQID); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_SESSION_DIGEST); +TRACE_DEFINE_ENUM(NFS4ERR_BAD_STATEID); +TRACE_DEFINE_ENUM(NFS4ERR_CB_PATH_DOWN); +TRACE_DEFINE_ENUM(NFS4ERR_CLID_INUSE); +TRACE_DEFINE_ENUM(NFS4ERR_CLIENTID_BUSY); +TRACE_DEFINE_ENUM(NFS4ERR_COMPLETE_ALREADY); +TRACE_DEFINE_ENUM(NFS4ERR_CONN_NOT_BOUND_TO_SESSION); +TRACE_DEFINE_ENUM(NFS4ERR_DEADLOCK); +TRACE_DEFINE_ENUM(NFS4ERR_DEADSESSION); +TRACE_DEFINE_ENUM(NFS4ERR_DELAY); +TRACE_DEFINE_ENUM(NFS4ERR_DELEG_ALREADY_WANTED); +TRACE_DEFINE_ENUM(NFS4ERR_DELEG_REVOKED); +TRACE_DEFINE_ENUM(NFS4ERR_DENIED); +TRACE_DEFINE_ENUM(NFS4ERR_DIRDELEG_UNAVAIL); +TRACE_DEFINE_ENUM(NFS4ERR_DQUOT); +TRACE_DEFINE_ENUM(NFS4ERR_ENCR_ALG_UNSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_EXIST); +TRACE_DEFINE_ENUM(NFS4ERR_EXPIRED); +TRACE_DEFINE_ENUM(NFS4ERR_FBIG); +TRACE_DEFINE_ENUM(NFS4ERR_FHEXPIRED); +TRACE_DEFINE_ENUM(NFS4ERR_FILE_OPEN); +TRACE_DEFINE_ENUM(NFS4ERR_GRACE); +TRACE_DEFINE_ENUM(NFS4ERR_HASH_ALG_UNSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_INVAL); +TRACE_DEFINE_ENUM(NFS4ERR_IO); +TRACE_DEFINE_ENUM(NFS4ERR_ISDIR); +TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTTRYLATER); +TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTUNAVAILABLE); +TRACE_DEFINE_ENUM(NFS4ERR_LEASE_MOVED); +TRACE_DEFINE_ENUM(NFS4ERR_LOCKED); +TRACE_DEFINE_ENUM(NFS4ERR_LOCKS_HELD); +TRACE_DEFINE_ENUM(NFS4ERR_LOCK_RANGE); +TRACE_DEFINE_ENUM(NFS4ERR_MINOR_VERS_MISMATCH); +TRACE_DEFINE_ENUM(NFS4ERR_MLINK); +TRACE_DEFINE_ENUM(NFS4ERR_MOVED); +TRACE_DEFINE_ENUM(NFS4ERR_NAMETOOLONG); +TRACE_DEFINE_ENUM(NFS4ERR_NOENT); +TRACE_DEFINE_ENUM(NFS4ERR_NOFILEHANDLE); +TRACE_DEFINE_ENUM(NFS4ERR_NOMATCHING_LAYOUT); +TRACE_DEFINE_ENUM(NFS4ERR_NOSPC); +TRACE_DEFINE_ENUM(NFS4ERR_NOTDIR); +TRACE_DEFINE_ENUM(NFS4ERR_NOTEMPTY); +TRACE_DEFINE_ENUM(NFS4ERR_NOTSUPP); +TRACE_DEFINE_ENUM(NFS4ERR_NOT_ONLY_OP); +TRACE_DEFINE_ENUM(NFS4ERR_NOT_SAME); +TRACE_DEFINE_ENUM(NFS4ERR_NO_GRACE); +TRACE_DEFINE_ENUM(NFS4ERR_NXIO); +TRACE_DEFINE_ENUM(NFS4ERR_OLD_STATEID); +TRACE_DEFINE_ENUM(NFS4ERR_OPENMODE); +TRACE_DEFINE_ENUM(NFS4ERR_OP_ILLEGAL); +TRACE_DEFINE_ENUM(NFS4ERR_OP_NOT_IN_SESSION); +TRACE_DEFINE_ENUM(NFS4ERR_PERM); +TRACE_DEFINE_ENUM(NFS4ERR_PNFS_IO_HOLE); +TRACE_DEFINE_ENUM(NFS4ERR_PNFS_NO_LAYOUT); +TRACE_DEFINE_ENUM(NFS4ERR_RECALLCONFLICT); +TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_BAD); +TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_CONFLICT); +TRACE_DEFINE_ENUM(NFS4ERR_REJECT_DELEG); +TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG); +TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG_TO_CACHE); +TRACE_DEFINE_ENUM(NFS4ERR_REQ_TOO_BIG); +TRACE_DEFINE_ENUM(NFS4ERR_RESOURCE); +TRACE_DEFINE_ENUM(NFS4ERR_RESTOREFH); +TRACE_DEFINE_ENUM(NFS4ERR_RETRY_UNCACHED_REP); +TRACE_DEFINE_ENUM(NFS4ERR_RETURNCONFLICT); +TRACE_DEFINE_ENUM(NFS4ERR_ROFS); +TRACE_DEFINE_ENUM(NFS4ERR_SAME); +TRACE_DEFINE_ENUM(NFS4ERR_SHARE_DENIED); +TRACE_DEFINE_ENUM(NFS4ERR_SEQUENCE_POS); +TRACE_DEFINE_ENUM(NFS4ERR_SEQ_FALSE_RETRY); +TRACE_DEFINE_ENUM(NFS4ERR_SEQ_MISORDERED); +TRACE_DEFINE_ENUM(NFS4ERR_SERVERFAULT); +TRACE_DEFINE_ENUM(NFS4ERR_STALE); +TRACE_DEFINE_ENUM(NFS4ERR_STALE_CLIENTID); +TRACE_DEFINE_ENUM(NFS4ERR_STALE_STATEID); +TRACE_DEFINE_ENUM(NFS4ERR_SYMLINK); +TRACE_DEFINE_ENUM(NFS4ERR_TOOSMALL); +TRACE_DEFINE_ENUM(NFS4ERR_TOO_MANY_OPS); +TRACE_DEFINE_ENUM(NFS4ERR_UNKNOWN_LAYOUTTYPE); +TRACE_DEFINE_ENUM(NFS4ERR_UNSAFE_COMPOUND); +TRACE_DEFINE_ENUM(NFS4ERR_WRONGSEC); +TRACE_DEFINE_ENUM(NFS4ERR_WRONG_CRED); +TRACE_DEFINE_ENUM(NFS4ERR_WRONG_TYPE); +TRACE_DEFINE_ENUM(NFS4ERR_XDEV); + #define show_nfsv4_errors(error) \ - __print_symbolic(error, \ + __print_symbolic(-(error), \ { NFS4_OK, "OK" }, \ /* Mapped by nfs4_stat_to_errno() */ \ - { -EPERM, "EPERM" }, \ - { -ENOENT, "ENOENT" }, \ - { -EIO, "EIO" }, \ - { -ENXIO, "ENXIO" }, \ - { -EACCES, "EACCES" }, \ - { -EEXIST, "EEXIST" }, \ - { -EXDEV, "EXDEV" }, \ - { -ENOTDIR, "ENOTDIR" }, \ - { -EISDIR, "EISDIR" }, \ - { -EFBIG, "EFBIG" }, \ - { -ENOSPC, "ENOSPC" }, \ - { -EROFS, "EROFS" }, \ - { -EMLINK, "EMLINK" }, \ - { -ENAMETOOLONG, "ENAMETOOLONG" }, \ - { -ENOTEMPTY, "ENOTEMPTY" }, \ - { -EDQUOT, "EDQUOT" }, \ - { -ESTALE, "ESTALE" }, \ - { -EBADHANDLE, "EBADHANDLE" }, \ - { -EBADCOOKIE, "EBADCOOKIE" }, \ - { -ENOTSUPP, "ENOTSUPP" }, \ - { -ETOOSMALL, "ETOOSMALL" }, \ - { -EREMOTEIO, "EREMOTEIO" }, \ - { -EBADTYPE, "EBADTYPE" }, \ - { -EAGAIN, "EAGAIN" }, \ - { -ELOOP, "ELOOP" }, \ - { -EOPNOTSUPP, "EOPNOTSUPP" }, \ - { -EDEADLK, "EDEADLK" }, \ + { EPERM, "EPERM" }, \ + { ENOENT, "ENOENT" }, \ + { EIO, "EIO" }, \ + { ENXIO, "ENXIO" }, \ + { EACCES, "EACCES" }, \ + { EEXIST, "EEXIST" }, \ + { EXDEV, "EXDEV" }, \ + { ENOTDIR, "ENOTDIR" }, \ + { EISDIR, "EISDIR" }, \ + { EFBIG, "EFBIG" }, \ + { ENOSPC, "ENOSPC" }, \ + { EROFS, "EROFS" }, \ + { EMLINK, "EMLINK" }, \ + { ENAMETOOLONG, "ENAMETOOLONG" }, \ + { ENOTEMPTY, "ENOTEMPTY" }, \ + { EDQUOT, "EDQUOT" }, \ + { ESTALE, "ESTALE" }, \ + { EBADHANDLE, "EBADHANDLE" }, \ + { EBADCOOKIE, "EBADCOOKIE" }, \ + { ENOTSUPP, "ENOTSUPP" }, \ + { ETOOSMALL, "ETOOSMALL" }, \ + { EREMOTEIO, "EREMOTEIO" }, \ + { EBADTYPE, "EBADTYPE" }, \ + { EAGAIN, "EAGAIN" }, \ + { ELOOP, "ELOOP" }, \ + { EOPNOTSUPP, "EOPNOTSUPP" }, \ + { EDEADLK, "EDEADLK" }, \ /* RPC errors */ \ - { -ENOMEM, "ENOMEM" }, \ - { -EKEYEXPIRED, "EKEYEXPIRED" }, \ - { -ETIMEDOUT, "ETIMEDOUT" }, \ - { -ERESTARTSYS, "ERESTARTSYS" }, \ - { -ECONNREFUSED, "ECONNREFUSED" }, \ - { -ECONNRESET, "ECONNRESET" }, \ - { -ENETUNREACH, "ENETUNREACH" }, \ - { -EHOSTUNREACH, "EHOSTUNREACH" }, \ - { -EHOSTDOWN, "EHOSTDOWN" }, \ - { -EPIPE, "EPIPE" }, \ - { -EPFNOSUPPORT, "EPFNOSUPPORT" }, \ - { -EPROTONOSUPPORT, "EPROTONOSUPPORT" }, \ + { ENOMEM, "ENOMEM" }, \ + { EKEYEXPIRED, "EKEYEXPIRED" }, \ + { ETIMEDOUT, "ETIMEDOUT" }, \ + { ERESTARTSYS, "ERESTARTSYS" }, \ + { ECONNREFUSED, "ECONNREFUSED" }, \ + { ECONNRESET, "ECONNRESET" }, \ + { ENETUNREACH, "ENETUNREACH" }, \ + { EHOSTUNREACH, "EHOSTUNREACH" }, \ + { EHOSTDOWN, "EHOSTDOWN" }, \ + { EPIPE, "EPIPE" }, \ + { EPFNOSUPPORT, "EPFNOSUPPORT" }, \ + { EPROTONOSUPPORT, "EPROTONOSUPPORT" }, \ /* NFSv4 native errors */ \ - { -NFS4ERR_ACCESS, "ACCESS" }, \ - { -NFS4ERR_ATTRNOTSUPP, "ATTRNOTSUPP" }, \ - { -NFS4ERR_ADMIN_REVOKED, "ADMIN_REVOKED" }, \ - { -NFS4ERR_BACK_CHAN_BUSY, "BACK_CHAN_BUSY" }, \ - { -NFS4ERR_BADCHAR, "BADCHAR" }, \ - { -NFS4ERR_BADHANDLE, "BADHANDLE" }, \ - { -NFS4ERR_BADIOMODE, "BADIOMODE" }, \ - { -NFS4ERR_BADLAYOUT, "BADLAYOUT" }, \ - { -NFS4ERR_BADLABEL, "BADLABEL" }, \ - { -NFS4ERR_BADNAME, "BADNAME" }, \ - { -NFS4ERR_BADOWNER, "BADOWNER" }, \ - { -NFS4ERR_BADSESSION, "BADSESSION" }, \ - { -NFS4ERR_BADSLOT, "BADSLOT" }, \ - { -NFS4ERR_BADTYPE, "BADTYPE" }, \ - { -NFS4ERR_BADXDR, "BADXDR" }, \ - { -NFS4ERR_BAD_COOKIE, "BAD_COOKIE" }, \ - { -NFS4ERR_BAD_HIGH_SLOT, "BAD_HIGH_SLOT" }, \ - { -NFS4ERR_BAD_RANGE, "BAD_RANGE" }, \ - { -NFS4ERR_BAD_SEQID, "BAD_SEQID" }, \ - { -NFS4ERR_BAD_SESSION_DIGEST, "BAD_SESSION_DIGEST" }, \ - { -NFS4ERR_BAD_STATEID, "BAD_STATEID" }, \ - { -NFS4ERR_CB_PATH_DOWN, "CB_PATH_DOWN" }, \ - { -NFS4ERR_CLID_INUSE, "CLID_INUSE" }, \ - { -NFS4ERR_CLIENTID_BUSY, "CLIENTID_BUSY" }, \ - { -NFS4ERR_COMPLETE_ALREADY, "COMPLETE_ALREADY" }, \ - { -NFS4ERR_CONN_NOT_BOUND_TO_SESSION, \ + { NFS4ERR_ACCESS, "ACCESS" }, \ + { NFS4ERR_ATTRNOTSUPP, "ATTRNOTSUPP" }, \ + { NFS4ERR_ADMIN_REVOKED, "ADMIN_REVOKED" }, \ + { NFS4ERR_BACK_CHAN_BUSY, "BACK_CHAN_BUSY" }, \ + { NFS4ERR_BADCHAR, "BADCHAR" }, \ + { NFS4ERR_BADHANDLE, "BADHANDLE" }, \ + { NFS4ERR_BADIOMODE, "BADIOMODE" }, \ + { NFS4ERR_BADLAYOUT, "BADLAYOUT" }, \ + { NFS4ERR_BADLABEL, "BADLABEL" }, \ + { NFS4ERR_BADNAME, "BADNAME" }, \ + { NFS4ERR_BADOWNER, "BADOWNER" }, \ + { NFS4ERR_BADSESSION, "BADSESSION" }, \ + { NFS4ERR_BADSLOT, "BADSLOT" }, \ + { NFS4ERR_BADTYPE, "BADTYPE" }, \ + { NFS4ERR_BADXDR, "BADXDR" }, \ + { NFS4ERR_BAD_COOKIE, "BAD_COOKIE" }, \ + { NFS4ERR_BAD_HIGH_SLOT, "BAD_HIGH_SLOT" }, \ + { NFS4ERR_BAD_RANGE, "BAD_RANGE" }, \ + { NFS4ERR_BAD_SEQID, "BAD_SEQID" }, \ + { NFS4ERR_BAD_SESSION_DIGEST, "BAD_SESSION_DIGEST" }, \ + { NFS4ERR_BAD_STATEID, "BAD_STATEID" }, \ + { NFS4ERR_CB_PATH_DOWN, "CB_PATH_DOWN" }, \ + { NFS4ERR_CLID_INUSE, "CLID_INUSE" }, \ + { NFS4ERR_CLIENTID_BUSY, "CLIENTID_BUSY" }, \ + { NFS4ERR_COMPLETE_ALREADY, "COMPLETE_ALREADY" }, \ + { NFS4ERR_CONN_NOT_BOUND_TO_SESSION, \ "CONN_NOT_BOUND_TO_SESSION" }, \ - { -NFS4ERR_DEADLOCK, "DEADLOCK" }, \ - { -NFS4ERR_DEADSESSION, "DEAD_SESSION" }, \ - { -NFS4ERR_DELAY, "DELAY" }, \ - { -NFS4ERR_DELEG_ALREADY_WANTED, \ + { NFS4ERR_DEADLOCK, "DEADLOCK" }, \ + { NFS4ERR_DEADSESSION, "DEAD_SESSION" }, \ + { NFS4ERR_DELAY, "DELAY" }, \ + { NFS4ERR_DELEG_ALREADY_WANTED, \ "DELEG_ALREADY_WANTED" }, \ - { -NFS4ERR_DELEG_REVOKED, "DELEG_REVOKED" }, \ - { -NFS4ERR_DENIED, "DENIED" }, \ - { -NFS4ERR_DIRDELEG_UNAVAIL, "DIRDELEG_UNAVAIL" }, \ - { -NFS4ERR_DQUOT, "DQUOT" }, \ - { -NFS4ERR_ENCR_ALG_UNSUPP, "ENCR_ALG_UNSUPP" }, \ - { -NFS4ERR_EXIST, "EXIST" }, \ - { -NFS4ERR_EXPIRED, "EXPIRED" }, \ - { -NFS4ERR_FBIG, "FBIG" }, \ - { -NFS4ERR_FHEXPIRED, "FHEXPIRED" }, \ - { -NFS4ERR_FILE_OPEN, "FILE_OPEN" }, \ - { -NFS4ERR_GRACE, "GRACE" }, \ - { -NFS4ERR_HASH_ALG_UNSUPP, "HASH_ALG_UNSUPP" }, \ - { -NFS4ERR_INVAL, "INVAL" }, \ - { -NFS4ERR_IO, "IO" }, \ - { -NFS4ERR_ISDIR, "ISDIR" }, \ - { -NFS4ERR_LAYOUTTRYLATER, "LAYOUTTRYLATER" }, \ - { -NFS4ERR_LAYOUTUNAVAILABLE, "LAYOUTUNAVAILABLE" }, \ - { -NFS4ERR_LEASE_MOVED, "LEASE_MOVED" }, \ - { -NFS4ERR_LOCKED, "LOCKED" }, \ - { -NFS4ERR_LOCKS_HELD, "LOCKS_HELD" }, \ - { -NFS4ERR_LOCK_RANGE, "LOCK_RANGE" }, \ - { -NFS4ERR_MINOR_VERS_MISMATCH, "MINOR_VERS_MISMATCH" }, \ - { -NFS4ERR_MLINK, "MLINK" }, \ - { -NFS4ERR_MOVED, "MOVED" }, \ - { -NFS4ERR_NAMETOOLONG, "NAMETOOLONG" }, \ - { -NFS4ERR_NOENT, "NOENT" }, \ - { -NFS4ERR_NOFILEHANDLE, "NOFILEHANDLE" }, \ - { -NFS4ERR_NOMATCHING_LAYOUT, "NOMATCHING_LAYOUT" }, \ - { -NFS4ERR_NOSPC, "NOSPC" }, \ - { -NFS4ERR_NOTDIR, "NOTDIR" }, \ - { -NFS4ERR_NOTEMPTY, "NOTEMPTY" }, \ - { -NFS4ERR_NOTSUPP, "NOTSUPP" }, \ - { -NFS4ERR_NOT_ONLY_OP, "NOT_ONLY_OP" }, \ - { -NFS4ERR_NOT_SAME, "NOT_SAME" }, \ - { -NFS4ERR_NO_GRACE, "NO_GRACE" }, \ - { -NFS4ERR_NXIO, "NXIO" }, \ - { -NFS4ERR_OLD_STATEID, "OLD_STATEID" }, \ - { -NFS4ERR_OPENMODE, "OPENMODE" }, \ - { -NFS4ERR_OP_ILLEGAL, "OP_ILLEGAL" }, \ - { -NFS4ERR_OP_NOT_IN_SESSION, "OP_NOT_IN_SESSION" }, \ - { -NFS4ERR_PERM, "PERM" }, \ - { -NFS4ERR_PNFS_IO_HOLE, "PNFS_IO_HOLE" }, \ - { -NFS4ERR_PNFS_NO_LAYOUT, "PNFS_NO_LAYOUT" }, \ - { -NFS4ERR_RECALLCONFLICT, "RECALLCONFLICT" }, \ - { -NFS4ERR_RECLAIM_BAD, "RECLAIM_BAD" }, \ - { -NFS4ERR_RECLAIM_CONFLICT, "RECLAIM_CONFLICT" }, \ - { -NFS4ERR_REJECT_DELEG, "REJECT_DELEG" }, \ - { -NFS4ERR_REP_TOO_BIG, "REP_TOO_BIG" }, \ - { -NFS4ERR_REP_TOO_BIG_TO_CACHE, \ + { NFS4ERR_DELEG_REVOKED, "DELEG_REVOKED" }, \ + { NFS4ERR_DENIED, "DENIED" }, \ + { NFS4ERR_DIRDELEG_UNAVAIL, "DIRDELEG_UNAVAIL" }, \ + { NFS4ERR_DQUOT, "DQUOT" }, \ + { NFS4ERR_ENCR_ALG_UNSUPP, "ENCR_ALG_UNSUPP" }, \ + { NFS4ERR_EXIST, "EXIST" }, \ + { NFS4ERR_EXPIRED, "EXPIRED" }, \ + { NFS4ERR_FBIG, "FBIG" }, \ + { NFS4ERR_FHEXPIRED, "FHEXPIRED" }, \ + { NFS4ERR_FILE_OPEN, "FILE_OPEN" }, \ + { NFS4ERR_GRACE, "GRACE" }, \ + { NFS4ERR_HASH_ALG_UNSUPP, "HASH_ALG_UNSUPP" }, \ + { NFS4ERR_INVAL, "INVAL" }, \ + { NFS4ERR_IO, "IO" }, \ + { NFS4ERR_ISDIR, "ISDIR" }, \ + { NFS4ERR_LAYOUTTRYLATER, "LAYOUTTRYLATER" }, \ + { NFS4ERR_LAYOUTUNAVAILABLE, "LAYOUTUNAVAILABLE" }, \ + { NFS4ERR_LEASE_MOVED, "LEASE_MOVED" }, \ + { NFS4ERR_LOCKED, "LOCKED" }, \ + { NFS4ERR_LOCKS_HELD, "LOCKS_HELD" }, \ + { NFS4ERR_LOCK_RANGE, "LOCK_RANGE" }, \ + { NFS4ERR_MINOR_VERS_MISMATCH, "MINOR_VERS_MISMATCH" }, \ + { NFS4ERR_MLINK, "MLINK" }, \ + { NFS4ERR_MOVED, "MOVED" }, \ + { NFS4ERR_NAMETOOLONG, "NAMETOOLONG" }, \ + { NFS4ERR_NOENT, "NOENT" }, \ + { NFS4ERR_NOFILEHANDLE, "NOFILEHANDLE" }, \ + { NFS4ERR_NOMATCHING_LAYOUT, "NOMATCHING_LAYOUT" }, \ + { NFS4ERR_NOSPC, "NOSPC" }, \ + { NFS4ERR_NOTDIR, "NOTDIR" }, \ + { NFS4ERR_NOTEMPTY, "NOTEMPTY" }, \ + { NFS4ERR_NOTSUPP, "NOTSUPP" }, \ + { NFS4ERR_NOT_ONLY_OP, "NOT_ONLY_OP" }, \ + { NFS4ERR_NOT_SAME, "NOT_SAME" }, \ + { NFS4ERR_NO_GRACE, "NO_GRACE" }, \ + { NFS4ERR_NXIO, "NXIO" }, \ + { NFS4ERR_OLD_STATEID, "OLD_STATEID" }, \ + { NFS4ERR_OPENMODE, "OPENMODE" }, \ + { NFS4ERR_OP_ILLEGAL, "OP_ILLEGAL" }, \ + { NFS4ERR_OP_NOT_IN_SESSION, "OP_NOT_IN_SESSION" }, \ + { NFS4ERR_PERM, "PERM" }, \ + { NFS4ERR_PNFS_IO_HOLE, "PNFS_IO_HOLE" }, \ + { NFS4ERR_PNFS_NO_LAYOUT, "PNFS_NO_LAYOUT" }, \ + { NFS4ERR_RECALLCONFLICT, "RECALLCONFLICT" }, \ + { NFS4ERR_RECLAIM_BAD, "RECLAIM_BAD" }, \ + { NFS4ERR_RECLAIM_CONFLICT, "RECLAIM_CONFLICT" }, \ + { NFS4ERR_REJECT_DELEG, "REJECT_DELEG" }, \ + { NFS4ERR_REP_TOO_BIG, "REP_TOO_BIG" }, \ + { NFS4ERR_REP_TOO_BIG_TO_CACHE, \ "REP_TOO_BIG_TO_CACHE" }, \ - { -NFS4ERR_REQ_TOO_BIG, "REQ_TOO_BIG" }, \ - { -NFS4ERR_RESOURCE, "RESOURCE" }, \ - { -NFS4ERR_RESTOREFH, "RESTOREFH" }, \ - { -NFS4ERR_RETRY_UNCACHED_REP, "RETRY_UNCACHED_REP" }, \ - { -NFS4ERR_RETURNCONFLICT, "RETURNCONFLICT" }, \ - { -NFS4ERR_ROFS, "ROFS" }, \ - { -NFS4ERR_SAME, "SAME" }, \ - { -NFS4ERR_SHARE_DENIED, "SHARE_DENIED" }, \ - { -NFS4ERR_SEQUENCE_POS, "SEQUENCE_POS" }, \ - { -NFS4ERR_SEQ_FALSE_RETRY, "SEQ_FALSE_RETRY" }, \ - { -NFS4ERR_SEQ_MISORDERED, "SEQ_MISORDERED" }, \ - { -NFS4ERR_SERVERFAULT, "SERVERFAULT" }, \ - { -NFS4ERR_STALE, "STALE" }, \ - { -NFS4ERR_STALE_CLIENTID, "STALE_CLIENTID" }, \ - { -NFS4ERR_STALE_STATEID, "STALE_STATEID" }, \ - { -NFS4ERR_SYMLINK, "SYMLINK" }, \ - { -NFS4ERR_TOOSMALL, "TOOSMALL" }, \ - { -NFS4ERR_TOO_MANY_OPS, "TOO_MANY_OPS" }, \ - { -NFS4ERR_UNKNOWN_LAYOUTTYPE, "UNKNOWN_LAYOUTTYPE" }, \ - { -NFS4ERR_UNSAFE_COMPOUND, "UNSAFE_COMPOUND" }, \ - { -NFS4ERR_WRONGSEC, "WRONGSEC" }, \ - { -NFS4ERR_WRONG_CRED, "WRONG_CRED" }, \ - { -NFS4ERR_WRONG_TYPE, "WRONG_TYPE" }, \ - { -NFS4ERR_XDEV, "XDEV" }) + { NFS4ERR_REQ_TOO_BIG, "REQ_TOO_BIG" }, \ + { NFS4ERR_RESOURCE, "RESOURCE" }, \ + { NFS4ERR_RESTOREFH, "RESTOREFH" }, \ + { NFS4ERR_RETRY_UNCACHED_REP, "RETRY_UNCACHED_REP" }, \ + { NFS4ERR_RETURNCONFLICT, "RETURNCONFLICT" }, \ + { NFS4ERR_ROFS, "ROFS" }, \ + { NFS4ERR_SAME, "SAME" }, \ + { NFS4ERR_SHARE_DENIED, "SHARE_DENIED" }, \ + { NFS4ERR_SEQUENCE_POS, "SEQUENCE_POS" }, \ + { NFS4ERR_SEQ_FALSE_RETRY, "SEQ_FALSE_RETRY" }, \ + { NFS4ERR_SEQ_MISORDERED, "SEQ_MISORDERED" }, \ + { NFS4ERR_SERVERFAULT, "SERVERFAULT" }, \ + { NFS4ERR_STALE, "STALE" }, \ + { NFS4ERR_STALE_CLIENTID, "STALE_CLIENTID" }, \ + { NFS4ERR_STALE_STATEID, "STALE_STATEID" }, \ + { NFS4ERR_SYMLINK, "SYMLINK" }, \ + { NFS4ERR_TOOSMALL, "TOOSMALL" }, \ + { NFS4ERR_TOO_MANY_OPS, "TOO_MANY_OPS" }, \ + { NFS4ERR_UNKNOWN_LAYOUTTYPE, "UNKNOWN_LAYOUTTYPE" }, \ + { NFS4ERR_UNSAFE_COMPOUND, "UNSAFE_COMPOUND" }, \ + { NFS4ERR_WRONGSEC, "WRONGSEC" }, \ + { NFS4ERR_WRONG_CRED, "WRONG_CRED" }, \ + { NFS4ERR_WRONG_TYPE, "WRONG_TYPE" }, \ + { NFS4ERR_XDEV, "XDEV" }) #define show_open_flags(flags) \ __print_flags(flags, "|", \ @@ -558,6 +703,13 @@ ) ); +TRACE_DEFINE_ENUM(F_GETLK); +TRACE_DEFINE_ENUM(F_SETLK); +TRACE_DEFINE_ENUM(F_SETLKW); +TRACE_DEFINE_ENUM(F_RDLCK); +TRACE_DEFINE_ENUM(F_WRLCK); +TRACE_DEFINE_ENUM(F_UNLCK); + #define show_lock_cmd(type) \ __print_symbolic((int)type, \ { F_GETLK, "GETLK" }, \ @@ -1451,6 +1603,10 @@ #ifdef CONFIG_NFS_V4_1 DEFINE_NFS4_COMMIT_EVENT(nfs4_pnfs_commit_ds); +TRACE_DEFINE_ENUM(IOMODE_READ); +TRACE_DEFINE_ENUM(IOMODE_RW); +TRACE_DEFINE_ENUM(IOMODE_ANY); + #define show_pnfs_iomode(iomode) \ __print_symbolic(iomode, \ { IOMODE_READ, "READ" }, \ @@ -1528,6 +1684,20 @@ DEFINE_NFS4_INODE_STATEID_EVENT(nfs4_layoutreturn); DEFINE_NFS4_INODE_EVENT(nfs4_layoutreturn_on_close); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_UNKNOWN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_NO_PNFS); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_RD_ZEROLEN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_MDSTHRESH); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_NOMEM); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_BULK_RECALL); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_IO_TEST_FAIL); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_FOUND_CACHED); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_RETURN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_BLOCKED); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_INVALID_OPEN); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_RETRY); +TRACE_DEFINE_ENUM(PNFS_UPDATE_LAYOUT_SEND_LAYOUTGET); + #define show_pnfs_update_layout_reason(reason) \ __print_symbolic(reason, \ { PNFS_UPDATE_LAYOUT_UNKNOWN, "unknown" }, \ From patchwork Mon Dec 17 16:41:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733993 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8553161F for ; Mon, 17 Dec 2018 16:41:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA8E72A215 for ; Mon, 17 Dec 2018 16:41:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AF0EF2A228; Mon, 17 Dec 2018 16:41:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3156E2A225 for ; Mon, 17 Dec 2018 16:41:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388284AbeLQQld (ORCPT ); Mon, 17 Dec 2018 11:41:33 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:55654 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlc (ORCPT ); Mon, 17 Dec 2018 11:41:32 -0500 Received: by mail-it1-f196.google.com with SMTP id o19so19443813itg.5; Mon, 17 Dec 2018 08:41:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=P/q4RnJj2efKP18/p35VQ/p0eIvp2esguJVWydMCWg8=; b=DBQc6kPR1/LIY97opEJsiaBmbxcI2fT+saYpHXDhomoQKm/TlmRJxIoB2tSn129fXb qyuqZNxMWnyexvHZux+tuCp/aovwZDl44WpVxct6dHBdclcrm4KFgsc6n5Ka2V+bura5 cDOJyS1sQnfPVCZED070K56HyE1QpLgdfo6eOWEYcQs4JvLRHKXeHZFCHBVBJyxossZM h7kOtNwVsZ1dHvTcYoKMzv4UTfSakjNHMZO7vJDjUn5eib5aRZnBVLBoupupsM3D5sF3 z4U1gxZhh0apHXLcTmwMMj8iU6jW/yJmKvY/0OT9QYu91j3vmaK4QsLC/q2TweP9CkyA QQnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=P/q4RnJj2efKP18/p35VQ/p0eIvp2esguJVWydMCWg8=; b=fAnaah3s3YflDY8MCwBV9Kx99yZdpT8yWoUCu+tkNyE0KXoibL2KRNjqUcS77WzeRG XmVZPVUqOY9rfwSbnnTwFVMUWGruAXYeU2v9GlEAal78mDWVO9KtbykCumSzH6x76vOL vv5SE9NNDEvLA9bv2Mu4rvg0bksuJPye2y5TJ1GTrJwruNfnDl37bEoQfa8S7b4dV9ci 6lyrYi3AwbdP/fS87+YA2OqkmfdSwjnLIi6QrmsHaYVAk+vgT8ieS/InKCd5eR/R1fpF lqjaIFBM4k8iGrYuD1rIsnf0ZDDp6Qa0cnzqb7HYYm7TOPU6T0Asz6S8xEHEutwAPn2R v7/g== X-Gm-Message-State: AA+aEWYByZrGFjaIHtZbbjMq7ioIIcNiag+bRCnnlL11e5OSC+JvEPxl iSuQXHUsT1h2Nc3VBjwd+cq3M3fB X-Google-Smtp-Source: AFSGD/Vbr+BrlWZ6Tfu564j+0fXmwPnlDaW3QFak1+qcus7aeJyqm5CE05ccHD9ABGvXcIVdEVhi0A== X-Received: by 2002:a05:660c:10:: with SMTP id q16mr11496740itj.51.1545064891064; Mon, 17 Dec 2018 08:41:31 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id l3sm7174992itb.43.2018.12.17.08.41.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:30 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfTMO018651; Mon, 17 Dec 2018 16:41:29 GMT Subject: [PATCH v4 24/30] SUNRPC: Simplify defining common RPC trace events From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:29 -0500 Message-ID: <20181217164129.24133.88082.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up, no functional change is expected. Signed-off-by: Chuck Lever --- include/trace/events/sunrpc.h | 172 ++++++++++++++++------------------------- 1 file changed, 69 insertions(+), 103 deletions(-) diff --git a/include/trace/events/sunrpc.h b/include/trace/events/sunrpc.h index 28e3841..88bda93 100644 --- a/include/trace/events/sunrpc.h +++ b/include/trace/events/sunrpc.h @@ -16,40 +16,6 @@ DECLARE_EVENT_CLASS(rpc_task_status, - TP_PROTO(struct rpc_task *task), - - TP_ARGS(task), - - TP_STRUCT__entry( - __field(unsigned int, task_id) - __field(unsigned int, client_id) - __field(int, status) - ), - - TP_fast_assign( - __entry->task_id = task->tk_pid; - __entry->client_id = task->tk_client->cl_clid; - __entry->status = task->tk_status; - ), - - TP_printk("task:%u@%u status=%d", - __entry->task_id, __entry->client_id, - __entry->status) -); - -DEFINE_EVENT(rpc_task_status, rpc_call_status, - TP_PROTO(struct rpc_task *task), - - TP_ARGS(task) -); - -DEFINE_EVENT(rpc_task_status, rpc_bind_status, - TP_PROTO(struct rpc_task *task), - - TP_ARGS(task) -); - -TRACE_EVENT(rpc_connect_status, TP_PROTO(const struct rpc_task *task), TP_ARGS(task), @@ -70,6 +36,16 @@ __entry->task_id, __entry->client_id, __entry->status) ); +#define DEFINE_RPC_STATUS_EVENT(name) \ + DEFINE_EVENT(rpc_task_status, rpc_##name##_status, \ + TP_PROTO( \ + const struct rpc_task *task \ + ), \ + TP_ARGS(task)) + +DEFINE_RPC_STATUS_EVENT(call); +DEFINE_RPC_STATUS_EVENT(bind); +DEFINE_RPC_STATUS_EVENT(connect); TRACE_EVENT(rpc_request, TP_PROTO(const struct rpc_task *task), @@ -134,30 +110,17 @@ __entry->action ) ); +#define DEFINE_RPC_RUNNING_EVENT(name) \ + DEFINE_EVENT(rpc_task_running, rpc_task_##name, \ + TP_PROTO( \ + const struct rpc_task *task, \ + const void *action \ + ), \ + TP_ARGS(task, action)) -DEFINE_EVENT(rpc_task_running, rpc_task_begin, - - TP_PROTO(const struct rpc_task *task, const void *action), - - TP_ARGS(task, action) - -); - -DEFINE_EVENT(rpc_task_running, rpc_task_run_action, - - TP_PROTO(const struct rpc_task *task, const void *action), - - TP_ARGS(task, action) - -); - -DEFINE_EVENT(rpc_task_running, rpc_task_complete, - - TP_PROTO(const struct rpc_task *task, const void *action), - - TP_ARGS(task, action) - -); +DEFINE_RPC_RUNNING_EVENT(begin); +DEFINE_RPC_RUNNING_EVENT(run_action); +DEFINE_RPC_RUNNING_EVENT(complete); DECLARE_EVENT_CLASS(rpc_task_queued, @@ -195,22 +158,16 @@ __get_str(q_name) ) ); +#define DEFINE_RPC_QUEUED_EVENT(name) \ + DEFINE_EVENT(rpc_task_queued, rpc_task_##name, \ + TP_PROTO( \ + const struct rpc_task *task, \ + const struct rpc_wait_queue *q \ + ), \ + TP_ARGS(task, q)) -DEFINE_EVENT(rpc_task_queued, rpc_task_sleep, - - TP_PROTO(const struct rpc_task *task, const struct rpc_wait_queue *q), - - TP_ARGS(task, q) - -); - -DEFINE_EVENT(rpc_task_queued, rpc_task_wakeup, - - TP_PROTO(const struct rpc_task *task, const struct rpc_wait_queue *q), - - TP_ARGS(task, q) - -); +DEFINE_RPC_QUEUED_EVENT(sleep); +DEFINE_RPC_QUEUED_EVENT(wakeup); TRACE_EVENT(rpc_stats_latency, @@ -410,7 +367,11 @@ DEFINE_RPC_SOCKET_EVENT(rpc_socket_shutdown); DECLARE_EVENT_CLASS(rpc_xprt_event, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), + TP_PROTO( + const struct rpc_xprt *xprt, + __be32 xid, + int status + ), TP_ARGS(xprt, xid, status), @@ -432,22 +393,19 @@ __get_str(port), __entry->xid, __entry->status) ); +#define DEFINE_RPC_XPRT_EVENT(name) \ + DEFINE_EVENT(rpc_xprt_event, xprt_##name, \ + TP_PROTO( \ + const struct rpc_xprt *xprt, \ + __be32 xid, \ + int status \ + ), \ + TP_ARGS(xprt, xid, status)) -DEFINE_EVENT(rpc_xprt_event, xprt_timer, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); - -DEFINE_EVENT(rpc_xprt_event, xprt_lookup_rqst, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); - -DEFINE_EVENT(rpc_xprt_event, xprt_transmit, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); - -DEFINE_EVENT(rpc_xprt_event, xprt_complete_rqst, - TP_PROTO(struct rpc_xprt *xprt, __be32 xid, int status), - TP_ARGS(xprt, xid, status)); +DEFINE_RPC_XPRT_EVENT(timer); +DEFINE_RPC_XPRT_EVENT(lookup_rqst); +DEFINE_RPC_XPRT_EVENT(transmit); +DEFINE_RPC_XPRT_EVENT(complete_rqst); TRACE_EVENT(xprt_ping, TP_PROTO(const struct rpc_xprt *xprt, int status), @@ -587,7 +545,9 @@ DECLARE_EVENT_CLASS(svc_rqst_event, - TP_PROTO(struct svc_rqst *rqst), + TP_PROTO( + const struct svc_rqst *rqst + ), TP_ARGS(rqst), @@ -607,14 +567,15 @@ __get_str(addr), __entry->xid, show_rqstp_flags(__entry->flags)) ); +#define DEFINE_SVC_RQST_EVENT(name) \ + DEFINE_EVENT(svc_rqst_event, svc_##name, \ + TP_PROTO( \ + const struct svc_rqst *rqst \ + ), \ + TP_ARGS(rqst)) -DEFINE_EVENT(svc_rqst_event, svc_defer, - TP_PROTO(struct svc_rqst *rqst), - TP_ARGS(rqst)); - -DEFINE_EVENT(svc_rqst_event, svc_drop, - TP_PROTO(struct svc_rqst *rqst), - TP_ARGS(rqst)); +DEFINE_SVC_RQST_EVENT(defer); +DEFINE_SVC_RQST_EVENT(drop); DECLARE_EVENT_CLASS(svc_rqst_status, @@ -801,7 +762,9 @@ ); DECLARE_EVENT_CLASS(svc_deferred_event, - TP_PROTO(struct svc_deferred_req *dr), + TP_PROTO( + const struct svc_deferred_req *dr + ), TP_ARGS(dr), @@ -818,13 +781,16 @@ TP_printk("addr=%s xid=0x%08x", __get_str(addr), __entry->xid) ); +#define DEFINE_SVC_DEFERRED_EVENT(name) \ + DEFINE_EVENT(svc_deferred_event, svc_##name##_deferred, \ + TP_PROTO( \ + const struct svc_deferred_req *dr \ + ), \ + TP_ARGS(dr)) + +DEFINE_SVC_DEFERRED_EVENT(drop); +DEFINE_SVC_DEFERRED_EVENT(revisit); -DEFINE_EVENT(svc_deferred_event, svc_drop_deferred, - TP_PROTO(struct svc_deferred_req *dr), - TP_ARGS(dr)); -DEFINE_EVENT(svc_deferred_event, svc_revisit_deferred, - TP_PROTO(struct svc_deferred_req *dr), - TP_ARGS(dr)); #endif /* _TRACE_SUNRPC_H */ #include From patchwork Mon Dec 17 16:41:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733995 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A54513AD for ; Mon, 17 Dec 2018 16:41:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EFA962A215 for ; Mon, 17 Dec 2018 16:41:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E26C42A22D; Mon, 17 Dec 2018 16:41:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 735A52A215 for ; Mon, 17 Dec 2018 16:41:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388235AbeLQQlh (ORCPT ); Mon, 17 Dec 2018 11:41:37 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:50381 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlh (ORCPT ); Mon, 17 Dec 2018 11:41:37 -0500 Received: by mail-it1-f193.google.com with SMTP id z7so19484525iti.0; Mon, 17 Dec 2018 08:41:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=SJcMrwxu+G9rJtVuVfQqtypngoSzGPrG1TFAd0Mn5u8=; b=AC6GUX2+Z0KIUx2uQXi2iDj2gpSb4IRVyCWzX2KyLiQ323xYiMCibPklI9E+jzLCLr O3fSLgXlaa6526OIA1+yNv6+YGfLWZhwq3Fzp1IHmm6h7dkwAMxqa3L7sgRNTKtIquDz nd3yNnTfj80mIKxgwRPnoRJ0CbfnCdn0G5TR8lWaDr5+qVf7UbNmvpHIkeFjuED0KLpB lv0W4Bd7Nupr0vC5nwb72sckRe3b8362Rkv4XX0w/Xp4PzzP/OGFjbWyG1F+y10JBazL /yf6tEuTR+Ksmuo+oLfAJXIQBQubKzzhFd9ZNFzIGsQ6OBqgZmadmrbQRUxNRbyf4GMs cGkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=SJcMrwxu+G9rJtVuVfQqtypngoSzGPrG1TFAd0Mn5u8=; b=gQCwRha1w3X0d7F+bJfQHylJeHF8kURWsPty+RA8K67fl9AtTlqg/ea0kGljm8SkrZ WnxjDK6Cr46zMa/kHJxI4lRThBTTZH1VGXUP67oJ4Yr4yECNuzZ5yeOPstOqc64Cjq1F +TPMsgRphCJVRedauW47fs2/y+0mmSCrtufNbapebOTbulPfxz2Ng4jzgKTcx+xlkUXn RayV1fIvoLPdnrUXM3rywRSR9HrclKq2sBye2YjNmlQn3T5XMqPCqgGiHKsMaS5sJGR6 FxXrB1bNRkT0ohvKJD5gmmKy3uUpifFxVZkHhg6CX+BQB8CApoDgSSDlOI1F+NxdKkXm Pkbw== X-Gm-Message-State: AA+aEWZB5BHhaMhR5eTU5xXZKY1kiB94JWaksswUk4bFqusNbqKZqdoS dPhJZfOT0fbiCX5tQVVfiO21SwaI X-Google-Smtp-Source: AFSGD/XZPmjJaHmSNkQ/1BXQKpzjMC6YqYsviZAADuYh+SbITKGM8vIFpqSCnn9IH1XcHTCa89WoQw== X-Received: by 2002:a24:750f:: with SMTP id y15mr11932067itc.177.1545064896285; Mon, 17 Dec 2018 08:41:36 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id n124sm7686601itn.4.2018.12.17.08.41.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:35 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfZV6018654; Mon, 17 Dec 2018 16:41:35 GMT Subject: [PATCH v4 25/30] SUNRPC: Fix some kernel doc complaints From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:35 -0500 Message-ID: <20181217164134.24133.91936.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up some warnings observed when building with "make W=1". Signed-off-by: Chuck Lever --- net/sunrpc/auth_gss/gss_mech_switch.c | 2 +- net/sunrpc/backchannel_rqst.c | 2 +- net/sunrpc/xprtmultipath.c | 4 ++-- net/sunrpc/xprtsock.c | 2 ++ 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/auth_gss/gss_mech_switch.c b/net/sunrpc/auth_gss/gss_mech_switch.c index 16ac0f4..379318d 100644 --- a/net/sunrpc/auth_gss/gss_mech_switch.c +++ b/net/sunrpc/auth_gss/gss_mech_switch.c @@ -244,7 +244,7 @@ struct gss_api_mech * /** * gss_mech_list_pseudoflavors - Discover registered GSS pseudoflavors - * @array: array to fill in + * @array_ptr: array to fill in * @size: size of "array" * * Returns the number of array items filled in, or a negative errno. diff --git a/net/sunrpc/backchannel_rqst.c b/net/sunrpc/backchannel_rqst.c index fa5ba6e..ec451b8 100644 --- a/net/sunrpc/backchannel_rqst.c +++ b/net/sunrpc/backchannel_rqst.c @@ -197,7 +197,7 @@ int xprt_setup_bc(struct rpc_xprt *xprt, unsigned int min_reqs) /** * xprt_destroy_backchannel - Destroys the backchannel preallocated structures. * @xprt: the transport holding the preallocated strucures - * @max_reqs the maximum number of preallocated structures to destroy + * @max_reqs: the maximum number of preallocated structures to destroy * * Since these structures may have been allocated by multiple calls * to xprt_setup_backchannel, we only destroy up to the maximum number diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c index e2d64c7..8394124 100644 --- a/net/sunrpc/xprtmultipath.c +++ b/net/sunrpc/xprtmultipath.c @@ -383,7 +383,7 @@ void xprt_iter_init_listall(struct rpc_xprt_iter *xpi, /** * xprt_iter_xchg_switch - Atomically swap out the rpc_xprt_switch * @xpi: pointer to rpc_xprt_iter - * @xps: pointer to a new rpc_xprt_switch or NULL + * @newswitch: pointer to a new rpc_xprt_switch or NULL * * Swaps out the existing xpi->xpi_xpswitch with a new value. */ @@ -401,7 +401,7 @@ struct rpc_xprt_switch *xprt_iter_xchg_switch(struct rpc_xprt_iter *xpi, /** * xprt_iter_destroy - Destroys the xprt iterator - * @xpi pointer to rpc_xprt_iter + * @xpi: pointer to rpc_xprt_iter */ void xprt_iter_destroy(struct rpc_xprt_iter *xpi) { diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index 8a5e823..8ee9831 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1602,6 +1602,7 @@ static void xs_udp_set_buffer_size(struct rpc_xprt *xprt, size_t sndsize, size_t /** * xs_udp_timer - called when a retransmit timeout occurs on a UDP transport + * @xprt: controlling transport * @task: task that timed out * * Adjust the congestion window after a retransmit timeout has occurred. @@ -2259,6 +2260,7 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) /** * xs_tcp_setup_socket - create a TCP socket and connect to a remote endpoint + * @work: queued work item * * Invoked by a work queue tasklet. */ From patchwork Mon Dec 17 16:41:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10733999 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2869B13AD for ; Mon, 17 Dec 2018 16:41:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18A7A2A225 for ; Mon, 17 Dec 2018 16:41:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D2BC2A228; Mon, 17 Dec 2018 16:41:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B54012A215 for ; Mon, 17 Dec 2018 16:41:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388326AbeLQQlo (ORCPT ); Mon, 17 Dec 2018 11:41:44 -0500 Received: from mail-io1-f68.google.com ([209.85.166.68]:39253 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQln (ORCPT ); Mon, 17 Dec 2018 11:41:43 -0500 Received: by mail-io1-f68.google.com with SMTP id k7so10458700iob.6; Mon, 17 Dec 2018 08:41:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=P/nO2EcoT+jAHQTgWRUXGGXa4Ij30Kh35hLNAzyaj+Q=; b=ZNyjqZUxRUWM5ThDn2TrKT00tEE/WAS/pUtinsEKhy+nLzsJ/Ik/gxhLhIaRMJpbIF yQOhLO1rXqr9pbInKyXYDmcBk5x78JVpM/WEKhTnSgpq56Wf2OsnD/dq3RV4wyLKCWXu 63Ev7k3NSMmjlAvN0xmfpgiowxmgM8J71zE2esViDN/RCThsxG/ugrafjrLN2ylo5WRl ywZjHFx3ew+s09F97zxK11kDEBc9TOaAFNs//C0gy0mmjPAyT9SBaGuvxOZd3Kc9rmoZ 17WjhS0SsEifS/9dg6Uq8NwTlk9ddW0pfKLxsMmDuM4Qjs9zINMPt1IJRsMVVjWLRaJC dnpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=P/nO2EcoT+jAHQTgWRUXGGXa4Ij30Kh35hLNAzyaj+Q=; b=MiSgt4r1Vvokqc6+LDzSMzQkTZEqmuXgeNp6ha123bSSzUbtHTLf/L2gU20m3DVoUx IXAV3IW4zgHyQwnVC8O8784Md7TKNINZVDaSErwS0QiCinRhXLA4bixNif4dQTDIgqLp ++2m0ZprEO+rdlB+iPamW8FeRdbKfERPYkt7Ssn/jcLTKHKT/CyaFlNn4qdoujORyq/2 rD93Z2OAP1a9czmBX8jBW29FoeR6thCClhCWKIS+trtpNLY7tO9Qm/d7l8MQWLYBDLKb Vg9YcGMN37MmwImWrDqJpbaMz0B9HL/Xdq9xgDosRScfnLJPF4i4B/qjhzOqnGtA0VRH tCeg== X-Gm-Message-State: AA+aEWZ7Bijay/9Ag9FFHzN3xZqSgEQJPxkQhE6Nrb8GQlLcPuoeARK5 SHitbiJ8iWKu8Aw0RsMd9tWBYTN9 X-Google-Smtp-Source: AFSGD/Vvuuwl8DzYTya0o3VZEul2kR7+tejMeXo/2oN1Sx77X/bnySY+w6fbjxd5fyc5+0mVW/Q/lQ== X-Received: by 2002:a6b:d20f:: with SMTP id q15mr12354834iob.100.1545064901464; Mon, 17 Dec 2018 08:41:41 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id l11sm5828081iob.24.2018.12.17.08.41.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:40 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfers018657; Mon, 17 Dec 2018 16:41:40 GMT Subject: [PATCH v4 26/30] xprtrdma: Update comments in frwr_op_send From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:40 -0500 Message-ID: <20181217164140.24133.39887.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit f2877623082b ("xprtrdma: Chain Send to FastReg WRs") was written before commit ce5b37178283 ("xprtrdma: Replace all usage of "frmr" with "frwr""), but was merged afterwards. Thus it still refers to FRMR and MWs. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 8a0f1a6..35c8f62 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -479,7 +479,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, * @ia: interface adapter * @req: Prepared RPC Call * - * For FRMR, chain any FastReg WRs to the Send WR. Only a + * For FRWR, chain any FastReg WRs to the Send WR. Only a * single ib_post_send call is needed to register memory * and then post the Send WR. * @@ -507,7 +507,7 @@ int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) } /* If ib_post_send fails, the next ->send_request for - * @req will queue these MWs for recovery. + * @req will queue these MRs for recovery. */ return ib_post_send(ia->ri_id->qp, post_wr, NULL); } From patchwork Mon Dec 17 16:41:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10734003 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C7301575 for ; Mon, 17 Dec 2018 16:41:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D3F72A215 for ; Mon, 17 Dec 2018 16:41:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3201F2A22D; Mon, 17 Dec 2018 16:41:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DBC662A215 for ; Mon, 17 Dec 2018 16:41:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388330AbeLQQlt (ORCPT ); Mon, 17 Dec 2018 11:41:49 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:54190 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlt (ORCPT ); Mon, 17 Dec 2018 11:41:49 -0500 Received: by mail-it1-f196.google.com with SMTP id g85so19469926ita.3; Mon, 17 Dec 2018 08:41:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=7uArD0olifUE5YxboWcMnzA5vma1vSehByVkKR/ay80=; b=qM5uSpuPWVEHpxDCBchgTVNL2RQWcZ6+RQE8jTImmLpGgG7yfJjvZQQQtb9V44QEQf 6Wd4D6mJKb8hlwitJtY5HIRTttJ4ZgjSLKiNMjBcoCwrHtfqt+iAgwe7o2APpvuUQ9Lr SPlBjyM3v3xMyU0MJ4Ri5/v95fRQVCz+TTVBudXm7wlpJD1EANU4DCMdGEHL5bbpSIrP Wq6MeZ5GaHE3gSbmVtsQaFWnqSKae6vUOkl95h9GuzPfCu3glVerPGnexkqnFaG1Z5jb IdkQ7vpUbfdxClG/eBQuocOJoKxdIEUuoXedKlTDWrVAAwwavvjtHFzT66KMwiiefaEY H72A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=7uArD0olifUE5YxboWcMnzA5vma1vSehByVkKR/ay80=; b=cmyD/09Gw9NUnm6cHl9roICQKQ9sx+cecnKMYTu1sNJYf/aku+dcUgAQLHSrAFaHLh +BQWFcN2UmbzyEPREZQc+Fpg4xCCNuzze5fAOf7j4dc4ivyEdsGTRTaxXZdZN1aWKJPP g6Jg/u7tSuEpuHW7g3oIQF3poQE7/h8G5Q3mN3lXsi/DFB2uv1iU2lVuDRfutL7B6xGP E6cSUQnHB0xhGaIURvxBvcxbqPAJdGlPf4b93twyyxx8SVj2SE18MS2gVHxW6An9ZUF3 BWVjhIOJqviK2XerHQyYL2MmfOcbJmJz2qS4AQ0MwdKZ8ji8o3m8sAe0LF8X2vXtiLLg K+NA== X-Gm-Message-State: AA+aEWYo0u1K1tp92spny3zLdP1DUO06nJBsBuODL3PCbKbPx6klevAG i8kDi1dL8Zmygto1JUrbs8QLj57q X-Google-Smtp-Source: AFSGD/UiKAvoQS3+dDF3YWIbdFmC8jITPY5yhEYFBcyW0sTYi4VieDh0BSDnLAv5E6NxrMBVXJpCUA== X-Received: by 2002:a24:bd0e:: with SMTP id x14mr6047915ite.123.1545064906889; Mon, 17 Dec 2018 08:41:46 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id u3sm7141816ita.18.2018.12.17.08.41.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:46 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfjCf018660; Mon, 17 Dec 2018 16:41:45 GMT Subject: [PATCH v4 27/30] xprtrdma: Replace outdated comment for rpcrdma_ep_post From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:45 -0500 Message-ID: <20181217164145.24133.74915.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since commit 7c8d9e7c8863 ("xprtrdma: Move Receive posting to Receive handler"), rpcrdma_ep_post is no longer responsible for posting Receive buffers. Update the documenting comment to reflect this change. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index b46e2f9..c69b985 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1427,10 +1427,14 @@ struct rpcrdma_regbuf * kfree(rb); } -/* - * Prepost any receive buffer, then post send. +/** + * rpcrdma_ep_post - Post WRs to a transport's Send Queue + * @ia: transport's device information + * @ep: transport's RDMA endpoint information + * @req: rpcrdma_req containing the Send WR to post * - * Receive buffer is donated to hardware, reclaimed upon recv completion. + * Returns 0 if the post was successful, otherwise -ENOTCONN + * is returned. */ int rpcrdma_ep_post(struct rpcrdma_ia *ia, From patchwork Mon Dec 17 16:41:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10734007 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 23CCF13AD for ; Mon, 17 Dec 2018 16:41:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 138652A215 for ; Mon, 17 Dec 2018 16:41:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 068DA2A22D; Mon, 17 Dec 2018 16:41:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A61C02A215 for ; Mon, 17 Dec 2018 16:41:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388329AbeLQQlx (ORCPT ); Mon, 17 Dec 2018 11:41:53 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:35605 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQlx (ORCPT ); Mon, 17 Dec 2018 11:41:53 -0500 Received: by mail-it1-f196.google.com with SMTP id p197so20946607itp.0; Mon, 17 Dec 2018 08:41:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=zl8j8je6pfA0Wi9IzJb3FMhbqTE3c/KKDSDa4e1eAIg=; b=piEGMR/p/W7nsqva7Ldjl+XONIPE+oILpDt/jClVSFqfhZG7hN+ha7xdI/fCaLCW/X tWEGWqRJ5KpNxLlR0rhRPa9O+1WdrtBTBzBzT8dcI5Il3kzd0SgpRshZ08x66WB+KYuC TVpPqnmx1l3LIJSUwp3dMmiq3f81zQ5ORSSx5g2S2QuV5nBqI59OV5UaTajGwT3lOaLh Eyk99uaYpjVtsmCpTCu7hYmnzVjwRyIv4oN768k7ExQWTUsgM/XoBKqTr5oPl1DR9ssN PWXlGcEjjzBf4jbLTukTpASz7WnyPs3NuRSDf2N/ZB+YjLxSi6IAJL/qlqZ0UkuGvZl9 Luog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=zl8j8je6pfA0Wi9IzJb3FMhbqTE3c/KKDSDa4e1eAIg=; b=mCw1fOllESoOFg/PqmdD7FRW5tiJzeXF2/BG6IWxIp7jCCaELmABBcB4P94jsojrtm HgS6D/gIYqwyXa4WYkqlgQe2aOgx1D4KtE8SmRQx7oZdzImmS4SBP6bjfY2x4mn65Ft1 XH+7nXT3xdPju7o4e7/ymqCXN42DKIfqx3ysi6Uor7QjnZysTWMMUuctdN3/eKH9tfd1 NkrOPAdYh8hiBAjHIus8ghFp6wAl9HGutrSdXdXCirD+4bIxz1LeMc1ImLWKKipFFFV7 oI4zDxclEPgMGW0zDvoJVSc4Q5c0+aIaERzdZshL7Q+iKRUHvD5Hp8/y8f4k54NgCf/t YacA== X-Gm-Message-State: AA+aEWYtny5q4uuZuzzT6buKgTIkNRECel916bbpcEyI08rJVfMuUFm9 r5jN6GW6DuZMzPX9PtAl+ANhv6tb X-Google-Smtp-Source: AFSGD/XveRTjmxb+GTSw+E72w4inBbHu+DzErbMP1I9CASYGL/MJumwWJaZxxDgAYrnc+o4anfb+oA== X-Received: by 2002:a24:9fc1:: with SMTP id c184mr12386661ite.8.1545064912167; Mon, 17 Dec 2018 08:41:52 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id t6sm6397677ioc.87.2018.12.17.08.41.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:51 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfoS4018663; Mon, 17 Dec 2018 16:41:50 GMT Subject: [PATCH v4 28/30] xprtrdma: Add documenting comment for rpcrdma_buffer_destroy From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:50 -0500 Message-ID: <20181217164150.24133.67463.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make a note of the function's dependency on an earlier ib_drain_qp. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index c69b985..339c40e 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1177,6 +1177,14 @@ struct rpcrdma_req * dprintk("RPC: %s: released %u MRs\n", __func__, count); } +/** + * rpcrdma_buffer_destroy - Release all hw resources + * @buf: root control block for resources + * + * ORDERING: relies on a prior ib_drain_qp : + * - No more Send or Receive completions can occur + * - All MRs, reps, and reqs are returned to their free lists + */ void rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) { From patchwork Mon Dec 17 16:41:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10734011 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 46A821575 for ; Mon, 17 Dec 2018 16:42:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 333B92A215 for ; Mon, 17 Dec 2018 16:42:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 22A952A22D; Mon, 17 Dec 2018 16:42:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE7F92A215 for ; Mon, 17 Dec 2018 16:42:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388331AbeLQQmA (ORCPT ); Mon, 17 Dec 2018 11:42:00 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:35176 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQl7 (ORCPT ); Mon, 17 Dec 2018 11:41:59 -0500 Received: by mail-io1-f67.google.com with SMTP id o13so10494636ioh.2; Mon, 17 Dec 2018 08:41:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Hg+/62w4wVOPn0SmI/QIaJsKJ5mlxxB+x5Gu8gJqcDo=; b=a/Znnd8mn+JEb7fwLisp4e4zhqoizQuIpDCh1vInFW1C1nf7AFqtNV1n5UCyTsW9c7 bU9peQpQrQlKpOSWw5KRt6ifsMTuY7f8NzR/cKLqv/vd3sQBgP1GfF5TvFEzIvBFhw6X KuFLDZLSIKbs/cAnUxl/sTZ/TNRW5u0jZEp/0T518beDX4z40ZRtCoWd4EGG9cRIYsa/ Rph8+Lo2uDu6R7kMkoPgCCoSRWY13ivfNjP3JJz9qa/7ZUDVjnnyJpOGMeKstEa0MngW WOM4+zIZ/M5QpRv+I/x7VwjeE/XAPkO5NU4RF7USk063U8f35tgATCy1MT/z5VQCvzJZ 26eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Hg+/62w4wVOPn0SmI/QIaJsKJ5mlxxB+x5Gu8gJqcDo=; b=Emd02x2G9VzRGt6gg5ZLhfWTJtgZPs/NJG+CXcu7RuifNdTI3bO9Wg451zbwtPCh09 6UZ3TV9ZNekn/EdmcrRKPyYVX4UGGNKuILpPRe0dgR6chS8pkKQ5tuYR7l5r9bCjaaLa fSyay1WN7ZwLQ/Z5vzSJTcRXWCFW8k6LmkqcW9a/Ddat1QsoHslSz/es8sIaUAcaSkEr AfkaOQXjrReRrh+iP+GB1yMID4pQ8RcddfqXCV16vU4zICN55wkeOZK2ulheju1R7ynn qU/J8rCwAFWtSt+Rtru+XgJehgYwZMDSQfQlgjoczMGHWRx3ovW1FOCkt+X1SWJlRc6x EkPQ== X-Gm-Message-State: AA+aEWZP6yDQ5t3boUpXr96HMc6XloSfddsWoBsJRTdZ2tnXvN1LrrNW N8sn13KKXijyQwkBehhZ+eLFukoK X-Google-Smtp-Source: AFSGD/XtWLsXtqTcpRWWXiGcgM0qdeSWl4SSbOISjhsVjfWHrrb1zOyhnMK711oKkWD0XwAYtlCPpQ== X-Received: by 2002:a5e:9902:: with SMTP id t2mr11921118ioj.153.1545064917361; Mon, 17 Dec 2018 08:41:57 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id o128sm7702159itb.39.2018.12.17.08.41.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:41:56 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGfuaW018666; Mon, 17 Dec 2018 16:41:56 GMT Subject: [PATCH v4 29/30] xprtrdma: Clarify comments in rpcrdma_ia_remove From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:41:56 -0500 Message-ID: <20181217164156.24133.2578.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Comments are clarified to note how transport data structures are protected. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 339c40e..b700ade 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -402,8 +402,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) * rpcrdma_ia_remove - Handle device driver unload * @ia: interface adapter being removed * - * Divest transport H/W resources associated with this adapter, - * but allow it to be restored later. + * Callers must serialize calls to this function. */ void rpcrdma_ia_remove(struct rpcrdma_ia *ia) @@ -434,16 +433,23 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) ib_free_cq(ep->rep_attr.send_cq); ep->rep_attr.send_cq = NULL; - /* The ULP is responsible for ensuring all DMA - * mappings and MRs are gone. + /* The ib_drain_qp above guarantees that all posted + * Receives have flushed, which returns the transport's + * rpcrdma_reps to the rb_recv_bufs list. */ list_for_each_entry(rep, &buf->rb_recv_bufs, rr_list) rpcrdma_dma_unmap_regbuf(rep->rr_rdmabuf); + + /* DMA mapping happens in ->send_request with the + * transport send lock held. Our caller is holding + * the transport send lock. + */ list_for_each_entry(req, &buf->rb_allreqs, rl_all) { rpcrdma_dma_unmap_regbuf(req->rl_rdmabuf); rpcrdma_dma_unmap_regbuf(req->rl_sendbuf); rpcrdma_dma_unmap_regbuf(req->rl_recvbuf); } + rpcrdma_mrs_destroy(buf); ib_dealloc_pd(ia->ri_pd); ia->ri_pd = NULL; From patchwork Mon Dec 17 16:42:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 10734017 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 664F8161F for ; Mon, 17 Dec 2018 16:42:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58DDC2A215 for ; Mon, 17 Dec 2018 16:42:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4CF682A226; Mon, 17 Dec 2018 16:42:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E178D2A22D for ; Mon, 17 Dec 2018 16:42:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388334AbeLQQmE (ORCPT ); Mon, 17 Dec 2018 11:42:04 -0500 Received: from mail-io1-f65.google.com ([209.85.166.65]:38630 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388011AbeLQQmE (ORCPT ); Mon, 17 Dec 2018 11:42:04 -0500 Received: by mail-io1-f65.google.com with SMTP id l14so10471549ioj.5; Mon, 17 Dec 2018 08:42:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=p6lkr/16ohwqBFo2MRfmrj0EAPREXCOKqFIKpKn96C8=; b=sBCjh2KK6Ku9KrJWvfE6qDN9tz6XCq/Qn+JWd2LJ1LUFeWunlgBQ8spCTLyuwsBg5d 3RG0itlPbUAS/mRVV4XPnJOOX0KywOGJo4AG4WY/PCkQYoJAOeC12Ix5oDvIvjVRvmDK L8a3vCu8iaq4RDVoGqaVI4RsxH1NlA68d4tNa0NXm8Vx/b372jnKPLGb6ZLYVDCQFQvd C0zVFW5tPI7UArd3P8dBhJaOAVFV+d9DIYlBwApbD7E0alhfvyv1jdYytgY9PAe14jND TCqZU+tnuW85WFWQsGScP/8LcCATZF+dFLJrtrVBKAyEe6y2JCugEt7FoCe5C4cK1G2l R5XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=p6lkr/16ohwqBFo2MRfmrj0EAPREXCOKqFIKpKn96C8=; b=aCbTU3mRV00KFr3Ny3cko+fs6mqkZsK9M0XftRafsY8/qYdwx/NfktFbC/9F2pHbZy SeeVRuqVN+tjwWHSbeH8OPUMkx0Ec9+5bX3ZXWB43Dar8zHb0/DJUq5S6b9uNL5ym0uo Cft0wOKDIKhxALxMaK7j5NJh0yHnT4HGpofKWfu8L+TMO7aFH+egC5N7lFW9BBf4Nb/r O7k3ctR5yggOPQJU3Vg/I4ez6ZGpnF7XBrLOte2I14Lo3MruYTqgtPM0B1MMHIC+iBlK UkzX48VhrcVvVsTzCZ8vAKq3ApALLkH88JgMhwI6kAy5hLoCf6VerbPHiJKWNZempdmi TJuw== X-Gm-Message-State: AA+aEWZRXY95MjOtTnWpuJ98k32xQ/1v5RQtH8nPY0aa5/ZQ4XsmbWf1 tYArJxr0sSYLTHR683MOOBGC049W X-Google-Smtp-Source: AFSGD/WfmpaFlrZpKI2pgejl2Qnir7qjW5+CRlM3B1VrNchKI9hS4+hvhLugRtj1Bysfz1m+jY2q9Q== X-Received: by 2002:a6b:3f06:: with SMTP id m6mr11097430ioa.117.1545064922786; Mon, 17 Dec 2018 08:42:02 -0800 (PST) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id f10sm7233182itf.41.2018.12.17.08.42.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 08:42:02 -0800 (PST) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id wBHGg1JE018669; Mon, 17 Dec 2018 16:42:01 GMT Subject: [PATCH v4 30/30] xprtrdma: Don't leak freed MRs From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 17 Dec 2018 11:42:01 -0500 Message-ID: <20181217164201.24133.618.stgit@manet.1015granger.net> In-Reply-To: <20181217162406.24133.27356.stgit@manet.1015granger.net> References: <20181217162406.24133.27356.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Defensive clean up. Don't set frwr->fr_mr until we know that the scatterlist allocation has succeeded. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 35c8f62..6a56105 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -155,36 +155,39 @@ void frwr_release_mr(struct rpcrdma_mr *mr) int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) { unsigned int depth = ia->ri_max_frwr_depth; - struct rpcrdma_frwr *frwr = &mr->frwr; + struct scatterlist *sg; + struct ib_mr *frmr; int rc; - frwr->fr_mr = ib_alloc_mr(ia->ri_pd, ia->ri_mrtype, depth); - if (IS_ERR(frwr->fr_mr)) + frmr = ib_alloc_mr(ia->ri_pd, ia->ri_mrtype, depth); + if (IS_ERR(frmr)) goto out_mr_err; - mr->mr_sg = kcalloc(depth, sizeof(*mr->mr_sg), GFP_KERNEL); - if (!mr->mr_sg) + sg = kcalloc(depth, sizeof(*sg), GFP_KERNEL); + if (!sg) goto out_list_err; - frwr->fr_state = FRWR_IS_INVALID; + mr->frwr.fr_mr = frmr; + mr->frwr.fr_state = FRWR_IS_INVALID; mr->mr_dir = DMA_NONE; INIT_LIST_HEAD(&mr->mr_list); INIT_WORK(&mr->mr_recycle, frwr_mr_recycle_worker); - sg_init_table(mr->mr_sg, depth); - init_completion(&frwr->fr_linv_done); + init_completion(&mr->frwr.fr_linv_done); + + sg_init_table(sg, depth); + mr->mr_sg = sg; return 0; out_mr_err: - rc = PTR_ERR(frwr->fr_mr); + rc = PTR_ERR(frmr); trace_xprtrdma_frwr_alloc(mr, rc); return rc; out_list_err: - rc = -ENOMEM; dprintk("RPC: %s: sg allocation failure\n", __func__); - ib_dereg_mr(frwr->fr_mr); - return rc; + ib_dereg_mr(frmr); + return -ENOMEM; } /**