From patchwork Thu Oct 17 18:31:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11196817 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9158C13B1 for ; Thu, 17 Oct 2019 18:31:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 62E522089C for ; Thu, 17 Oct 2019 18:31:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dHZRb88N" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2404464AbfJQSbN (ORCPT ); Thu, 17 Oct 2019 14:31:13 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:45994 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731227AbfJQSbM (ORCPT ); Thu, 17 Oct 2019 14:31:12 -0400 Received: by mail-qk1-f196.google.com with SMTP id z67so2784068qkb.12; Thu, 17 Oct 2019 11:31:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Pz4rG8h6xekipFWcDb/qNVz1ZYjqPG2oa7RN9sxsHR4=; b=dHZRb88N/fvG+1roIUxqVOcpOWKb4YuDzo5ObbEjRxbRWUKHne2/TIJDQpNxYaLcJq G9BWgJF5wklcDe03MK/XJfJt7u0aBmyBg053vaKkxAVPf9QoAjTuEOjwCaW6gKvTohEP cjMovt4rdUKMJIBeIB29gc4CgacoTCw1J8uwEmB0XxOEp61QreoVV4TXx3yOX1jX9wcq vZs3cDFlrBkRPXRl3ZJg5EG7SLuYk76w64bxgquQ5DRyQmwcMfOPDcICYzI3yezqvukw QsaXmgzF8YK7/uHcj1GpATmldmk0j3ZAaOgixigzP3G5BEXuyxQT0v6Ura2JIdnlaiNi SMRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Pz4rG8h6xekipFWcDb/qNVz1ZYjqPG2oa7RN9sxsHR4=; b=QDWBhdheamc31EF6kmHieJ2VH3ljlpwb/uYgxKv/HFdtQ1CPI1ExbbD4vWqyjQQb2V 7GwwaJMyH1Cg/A28ebfHOeKi4+yrT4OUStXclR9RKWA7RnJkAmJsOtixKZRdcDKwJb8j Agb2nF/k/+m+QT1FpHSmfRRdbTEt30P+JMvQr7j2gPvjH4u6uJ16apJA+o+aSA+mw/Vy ZIVzKb+TPUaShM/YgnpxT4SwFnsSimDYjMVc6Xs+Ks9lK7L2jfSC6KNsg9RXW/MxnBDT a3UiHwQOlG0Brx221+lZE1wbwwKar7OnFkCjPYDsiZw9+rkqM178sD6eJTLbWMprzN4C 8hFA== X-Gm-Message-State: APjAAAVlVugSzWpMuJjsQepLfQNm+nNDtkFaTBQ+I3MlkMSW3EnEGBZU nfc7uKpvsVzV8KtbSWurTskaaKzZ X-Google-Smtp-Source: APXvYqyYu9M3IDKFFS9TdUH90kAGUotlYXp4FZfHlJ047PlU7GcNZKPBtYHr34k8znerISUnSRQlIw== X-Received: by 2002:a05:620a:20d5:: with SMTP id f21mr4642324qka.209.1571337070882; Thu, 17 Oct 2019 11:31:10 -0700 (PDT) Received: from oracle-102.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id t40sm1863363qta.36.2019.10.17.11.31.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 11:31:10 -0700 (PDT) Subject: [PATCH v1 1/6] xprtrdma: Ensure ri_id is stable during MR recycling From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Oct 2019 14:31:09 -0400 Message-ID: <20191017183109.2517.8846.stgit@oracle-102.nfsv4bat.org> In-Reply-To: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> References: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org ia->ri_id is replaced during a reconnect. The connect_worker runs with the transport send lock held to prevent ri_id from being dereferenced by the send_request path during this process. Currently, however, there is no guarantee that ia->ri_id is stable in the MR recycling worker, which operates in the background and is not serialized with the connect_worker in any way. But now that Local_Inv completions are being done in process context, we can handle the recycling operation there instead of deferring the recycling work to another process. Because the disconnect path drains all work before allowing tear down to proceed, it is guaranteed that Local Invalidations complete only while the ri_id pointer is stable. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 23 ++++++----------------- net/sunrpc/xprtrdma/xprt_rdma.h | 7 ------- 2 files changed, 6 insertions(+), 24 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 37ba82d..5cd8715 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -88,8 +88,10 @@ void frwr_release_mr(struct rpcrdma_mr *mr) kfree(mr); } -static void frwr_mr_recycle(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr) +static void frwr_mr_recycle(struct rpcrdma_mr *mr) { + struct rpcrdma_xprt *r_xprt = mr->mr_xprt; + trace_xprtrdma_mr_recycle(mr); if (mr->mr_dir != DMA_NONE) { @@ -107,18 +109,6 @@ static void frwr_mr_recycle(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr) frwr_release_mr(mr); } -/* MRs are dynamically allocated, so simply clean up and release the MR. - * A replacement MR will subsequently be allocated on demand. - */ -static void -frwr_mr_recycle_worker(struct work_struct *work) -{ - struct rpcrdma_mr *mr = container_of(work, struct rpcrdma_mr, - mr_recycle); - - frwr_mr_recycle(mr->mr_xprt, mr); -} - /* frwr_reset - Place MRs back on the free list * @req: request to reset * @@ -163,7 +153,6 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) mr->frwr.fr_mr = frmr; mr->mr_dir = DMA_NONE; INIT_LIST_HEAD(&mr->mr_list); - INIT_WORK(&mr->mr_recycle, frwr_mr_recycle_worker); init_completion(&mr->frwr.fr_linv_done); sg_init_table(sg, depth); @@ -448,7 +437,7 @@ void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) static void __frwr_release_mr(struct ib_wc *wc, struct rpcrdma_mr *mr) { if (wc->status != IB_WC_SUCCESS) - rpcrdma_mr_recycle(mr); + frwr_mr_recycle(mr); else rpcrdma_mr_put(mr); } @@ -570,7 +559,7 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) bad_wr = bad_wr->next; list_del_init(&mr->mr_list); - rpcrdma_mr_recycle(mr); + frwr_mr_recycle(mr); } } @@ -664,7 +653,7 @@ void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) mr = container_of(frwr, struct rpcrdma_mr, frwr); bad_wr = bad_wr->next; - rpcrdma_mr_recycle(mr); + frwr_mr_recycle(mr); } /* The final LOCAL_INV WR in the chain is supposed to diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index a7ef965..b8e768d 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -257,7 +257,6 @@ struct rpcrdma_mr { u32 mr_handle; u32 mr_length; u64 mr_offset; - struct work_struct mr_recycle; struct list_head mr_all; }; @@ -490,12 +489,6 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, void rpcrdma_mr_put(struct rpcrdma_mr *mr); void rpcrdma_mrs_refresh(struct rpcrdma_xprt *r_xprt); -static inline void -rpcrdma_mr_recycle(struct rpcrdma_mr *mr) -{ - schedule_work(&mr->mr_recycle); -} - struct rpcrdma_req *rpcrdma_buffer_get(struct rpcrdma_buffer *); void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req); From patchwork Thu Oct 17 18:31:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11196821 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF2CB1575 for ; Thu, 17 Oct 2019 18:31:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C712320854 for ; Thu, 17 Oct 2019 18:31:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uvDpYsZ7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2392111AbfJQSbW (ORCPT ); Thu, 17 Oct 2019 14:31:22 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:39489 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731227AbfJQSbW (ORCPT ); Thu, 17 Oct 2019 14:31:22 -0400 Received: by mail-qk1-f196.google.com with SMTP id 4so2815285qki.6; Thu, 17 Oct 2019 11:31:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=osoCTJXMINvNwg2jj7tYRSSlZ5yCfx+CAfcFhP304Yc=; b=uvDpYsZ7IGv6VQOxWVjQXKilSc2qV/EVTo0cvJvPKUvYMRGeOivFVasQe80HX0FNJ8 KAMfR3tsUJvk0o8ZpfaoCmXgd3WYPlTowoiHSwbnJaNlzYX9AFhnZHNtJeok43obrEwJ 3sD6Rg3a/E6pA7OnvbZ2esHZ10Ba2GVIiSwKNZhoBbzNjjndck6CVMCGarns+z/nPF7p F4TD7Vq9/jenpEkCmlKTjKTL0ow6q6Mo9Z74u6FwOcHkMyzIuDj0uobhWeYdAAJ5UAjq pkdqDlkrus0Zhyz5FWzT7KfzLedbJXldjo8qMaRaaa4mTuUefZGGmEWSzSXfhEF/ARog bzrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=osoCTJXMINvNwg2jj7tYRSSlZ5yCfx+CAfcFhP304Yc=; b=dTdnnVooz0bXJ4CSTBJlVT2HjevNeYOxnWUisRlfJn0h2rqbBgvqtLPVToC+vjyZP+ glBDrQ6HDM+GVH4+1qwSwBQQvuCnNC2vhdLlorqxt1TlQlaQnk8Q2DgnrLrZZ289LkJE PfpOXRWJj7D9+wFsmcnglu2ZIh6wwtXuljbcS3ScvSzo1s8/+x0tKjy3lFqCi8nmpNQb Ivr44GMtWPnkC4C+JA5SZDrTXU1pY85FauVuZdl7fFf6DOCTVq3V+V+vLeLTqKSZO1M5 LzyEdCqtrsiLnhLrq2zWirIKDRn/3auM46MbjvMR8jG/9WuMHNCLqj7/EYG6YRRy7f8N RY1w== X-Gm-Message-State: APjAAAXPl4SdBRgl/A36S252CMnUbBxP0XmcMVTrb9ZXX99M1Dcp8fmG qk80PQINmODJZdLtjr7JDlcB1Lhr X-Google-Smtp-Source: APXvYqyNQuVikiOkm3HCSYatck+pMKUuRzslKKnTyiAsn584+BeOoZLln0mEGW4BM2jGV1hfdp/F6w== X-Received: by 2002:a37:9642:: with SMTP id y63mr4465868qkd.387.1571337080629; Thu, 17 Oct 2019 11:31:20 -0700 (PDT) Received: from oracle-102.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id u17sm1689319qkj.71.2019.10.17.11.31.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 11:31:18 -0700 (PDT) Subject: [PATCH v1 2/6] xprtrdma: Remove rpcrdma_sendctx::sc_xprt From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Oct 2019 14:31:18 -0400 Message-ID: <20191017183118.2517.45660.stgit@oracle-102.nfsv4bat.org> In-Reply-To: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> References: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Micro-optimization: Save eight bytes in a frequently allocated structure. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 19 ++++++++++--------- net/sunrpc/xprtrdma/xprt_rdma.h | 2 -- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index c79d862..3ab086a 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -74,7 +74,8 @@ /* * internal functions */ -static void rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc); +static void rpcrdma_sendctx_put_locked(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_sendctx *sc); static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt); static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); @@ -124,7 +125,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) /** * rpcrdma_wc_send - Invoked by RDMA provider for each polled Send WC - * @cq: completion queue (ignored) + * @cq: completion queue * @wc: completed WR * */ @@ -137,7 +138,7 @@ static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt) /* WARNING: Only wr_cqe and status are reliable at this point */ trace_xprtrdma_wc_send(sc, wc); - rpcrdma_sendctx_put_locked(sc); + rpcrdma_sendctx_put_locked((struct rpcrdma_xprt *)cq->cq_context, sc); } /** @@ -518,7 +519,7 @@ int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt) init_waitqueue_head(&ep->rep_connect_wait); ep->rep_receive_count = 0; - sendcq = ib_alloc_cq_any(ia->ri_id->device, NULL, + sendcq = ib_alloc_cq_any(ia->ri_id->device, r_xprt, ep->rep_attr.cap.max_send_wr + 1, IB_POLL_WORKQUEUE); if (IS_ERR(sendcq)) { @@ -840,7 +841,6 @@ static int rpcrdma_sendctxs_create(struct rpcrdma_xprt *r_xprt) if (!sc) return -ENOMEM; - sc->sc_xprt = r_xprt; buf->rb_sc_ctxs[i] = sc; } @@ -903,6 +903,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) /** * rpcrdma_sendctx_put_locked - Release a send context + * @r_xprt: controlling transport instance * @sc: send context to release * * Usage: Called from Send completion to return a sendctxt @@ -910,10 +911,10 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) * * The caller serializes calls to this function (per transport). */ -static void -rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc) +static void rpcrdma_sendctx_put_locked(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_sendctx *sc) { - struct rpcrdma_buffer *buf = &sc->sc_xprt->rx_buf; + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; unsigned long next_tail; /* Unmap SGEs of previously completed but unsignaled @@ -931,7 +932,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt) /* Paired with READ_ONCE */ smp_store_release(&buf->rb_sc_tail, next_tail); - xprt_write_space(&sc->sc_xprt->rx_xprt); + xprt_write_space(&r_xprt->rx_xprt); } static void diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index b8e768d..4897c09 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -218,12 +218,10 @@ enum { /* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes */ struct rpcrdma_req; -struct rpcrdma_xprt; struct rpcrdma_sendctx { struct ib_send_wr sc_wr; struct ib_cqe sc_cqe; struct ib_device *sc_device; - struct rpcrdma_xprt *sc_xprt; struct rpcrdma_req *sc_req; unsigned int sc_unmap_count; struct ib_sge sc_sges[]; From patchwork Thu Oct 17 18:31:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11196825 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B88941575 for ; Thu, 17 Oct 2019 18:31:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 974C020854 for ; Thu, 17 Oct 2019 18:31:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Mw7uz7kJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2404552AbfJQSba (ORCPT ); Thu, 17 Oct 2019 14:31:30 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:45084 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731227AbfJQSb3 (ORCPT ); Thu, 17 Oct 2019 14:31:29 -0400 Received: by mail-qt1-f194.google.com with SMTP id c21so4984450qtj.12; Thu, 17 Oct 2019 11:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=tg+NkdbkgiRH9rURMmOHWE9wg09lS+D0EM34LImt5Zc=; b=Mw7uz7kJ/EPMNk2obrHeVOc8jYLtLXsG94FYBO3fyurEbCni6othi6/R8DsVvzyU2X 2yXokeIlKrF7bl6nISGkuL79VDhnehhA87xvfWr6fKYo6siss7TwlrhkvaEIgP50Qjg1 l8vpHWUj/SIihzc+cQfl8vkcan6aIKZDRtOx9w65pQFxyl/yYA/nudHpYnd9M7aUKCNj jVClJvVu9cW3RD3moNiw3i1KK0ikFp6In02MIAh9QZVyJIvrC3FM4PCkcA8yJKLOnlV2 EyrZaC5n/evNbbVcPIA3NOHDuVi7VN3g//qeNbBsKBznm51ILbzEP8FLlV4BVk9rXLpm uL2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=tg+NkdbkgiRH9rURMmOHWE9wg09lS+D0EM34LImt5Zc=; b=uXQQuTM30AffUEAKDHCddomrU2WbJ4RNwKefIRWP4epcTcvpJZEo87hYS4xVKj6m6d UWoHG/wEDrhFfgpWAOYIlE+gtN6m4Sz0GB/+9jk60dcT1h6/YqkIHJQ6ZSv014BCpRXJ QRzRfxqeEfRuoh0op6IRD2qXpb95J0fGrQgEC1/xJ0ytwfzMZGvUWjR1GYnMDKrlZGCB lIglkfBJm3IPjBN7Zfemc5OrTDLFjApZLLNWkkD91K3TNXlXqrvOvvO09vXW4AI/ahvd J9yAcxkBbvHX9KsXkIi3aJWSmsXI+L91aBy2TxGhmNmO+u7ttV4L9mNtNjoAME/3gzxO zg0Q== X-Gm-Message-State: APjAAAX5Pr8kQLcu6cVN9gTBBUyzHpDMrl9FliJyj6eMVS2DjA1SoUOV 9PIaRN8QTIA/C+6yaRioSllp+mqt X-Google-Smtp-Source: APXvYqxRE5YuEBKE6QW1nt6M0YM2uFFxG84sBysU2ExniLqXweB5x0Id447urK4eZcTGAa73gEWaFw== X-Received: by 2002:ac8:6c4:: with SMTP id j4mr5349748qth.235.1571337088825; Thu, 17 Oct 2019 11:31:28 -0700 (PDT) Received: from oracle-102.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id p36sm2492111qtc.0.2019.10.17.11.31.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 11:31:28 -0700 (PDT) Subject: [PATCH v1 3/6] xprtrdma: Remove rpcrdma_sendctx::sc_device From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Oct 2019 14:31:27 -0400 Message-ID: <20191017183127.2517.53756.stgit@oracle-102.nfsv4bat.org> In-Reply-To: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> References: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Micro-optimization: Save eight bytes in a frequently allocated structure. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 4 ++-- net/sunrpc/xprtrdma/xprt_rdma.h | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 7b13582..1941b22 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -564,6 +564,7 @@ static void rpcrdma_sendctx_done(struct kref *kref) */ void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc) { + struct rpcrdma_regbuf *rb = sc->sc_req->rl_sendbuf; struct ib_sge *sge; if (!sc->sc_unmap_count) @@ -575,7 +576,7 @@ void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc) */ for (sge = &sc->sc_sges[2]; sc->sc_unmap_count; ++sge, --sc->sc_unmap_count) - ib_dma_unmap_page(sc->sc_device, sge->addr, sge->length, + ib_dma_unmap_page(rdmab_device(rb), sge->addr, sge->length, DMA_TO_DEVICE); kref_put(&sc->sc_req->rl_kref, rpcrdma_sendctx_done); @@ -625,7 +626,6 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, */ if (!rpcrdma_regbuf_dma_map(r_xprt, rb)) goto out_regbuf; - sc->sc_device = rdmab_device(rb); sge_no = 1; sge[sge_no].addr = rdmab_addr(rb); sge[sge_no].length = xdr->head[0].iov_len; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 4897c09..0e5b7f3 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -221,7 +221,6 @@ enum { struct rpcrdma_sendctx { struct ib_send_wr sc_wr; struct ib_cqe sc_cqe; - struct ib_device *sc_device; struct rpcrdma_req *sc_req; unsigned int sc_unmap_count; struct ib_sge sc_sges[]; From patchwork Thu Oct 17 18:31:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11196831 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C76BF1668 for ; Thu, 17 Oct 2019 18:31:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A467220854 for ; Thu, 17 Oct 2019 18:31:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jzVg+dr4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2406687AbfJQSbk (ORCPT ); Thu, 17 Oct 2019 14:31:40 -0400 Received: from mail-qk1-f194.google.com ([209.85.222.194]:37914 "EHLO mail-qk1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731227AbfJQSbk (ORCPT ); Thu, 17 Oct 2019 14:31:40 -0400 Received: by mail-qk1-f194.google.com with SMTP id p4so2829115qkf.5; Thu, 17 Oct 2019 11:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Pn5eLKTK1EeeO2p0zq+GeIFZ+Xs+ZFVG/1mx3hQz3mk=; b=jzVg+dr4BPcss4fG5wo5uRG+yIiblDINfqMGFxY07ukJAtNYS2qgwkjt8brcUxUQwE z5rmr8I7pLXaWGdIGE1GPr4ppi64GOfv60miHenSuUVJT9oPFKoncomGPkWxSdiuiC/5 j5nUSNMyGf+3guF9umrSSV+vrY92rFNFyWPecPYpydrDFxjxp+B3m6xYKTvzHx044nyN /i6xeDrlbffguLXb5RCNFMbi593ElVcze0RQruTIKWJqX6VVlXOSrH/ssJVmUJ22wo/X wD74txzAbngsLJmLj89QU+lIPft4f3/ttAmiHwqN0vadtmAY6UobrjRlFF6e7P/FQ9Ov NoNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Pn5eLKTK1EeeO2p0zq+GeIFZ+Xs+ZFVG/1mx3hQz3mk=; b=HZp0z43NPt+PeQdW/QwERU8qR3pVORr74pPi6cNaNyui5Zxq7AtSvL9eFZIajA5p+n 1kRxLwpAwcOhhgJ5TUx8pp+76+V/jqRkNcoWaFUx792geChI1kDKdONcuMNcki6W01MN JMqDzq47FU5YbG/W+ZAhL8H9CTCXV7Qz92KD+i9RcRfj5cLzv8G8pN5oVV69J5p6zw9L J5CXaQmeIDKfuWynq5p+OkgvEdgXNCbD3Jw98X1wBrOukAcX+BGw7d6LD1RAMGZEZjSI vKpdNfxK8OnC4SVFMUaBd9ZfX/lE6/8++AsSacGFt8j6Js6XTyUfdAFdHXwtqLHjbBU6 dNKw== X-Gm-Message-State: APjAAAWVApmvdztyGBgkiPsLmWpvME9AGOYDt/IpJ/y5VpDRj4ybaemz c0XOz5x6ZRAQvt+oiRF0lq1RyS8F X-Google-Smtp-Source: APXvYqys78o/pzhzntw3fSbGMPFXDYzHnYmGC9KcoLuLw8nsNnQzYH2SBXn22PnR7Pa4ZZRQhkaykA== X-Received: by 2002:a37:8785:: with SMTP id j127mr4631696qkd.126.1571337097389; Thu, 17 Oct 2019 11:31:37 -0700 (PDT) Received: from oracle-102.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id x8sm1356871qkx.77.2019.10.17.11.31.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 11:31:36 -0700 (PDT) Subject: [PATCH v1 4/6] xprtrdma: Move the rpcrdma_sendctx::sc_wr field From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Oct 2019 14:31:35 -0400 Message-ID: <20191017183135.2517.60416.stgit@oracle-102.nfsv4bat.org> In-Reply-To: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> References: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up: This field is not needed in the Send completion handler, so it can be moved to struct rpcrdma_req to reduce the size of struct rpcrdma_sendctx, and to reduce the amount of memory that is sloshed between the sending process and the Send completion process. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- net/sunrpc/xprtrdma/rpc_rdma.c | 9 ++++++--- net/sunrpc/xprtrdma/verbs.c | 5 +---- net/sunrpc/xprtrdma/xprt_rdma.h | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 31681cb..16572ae7 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -667,9 +667,8 @@ __entry->client_id = rqst->rq_task->tk_client ? rqst->rq_task->tk_client->cl_clid : -1; __entry->req = req; - __entry->num_sge = req->rl_sendctx->sc_wr.num_sge; - __entry->signaled = req->rl_sendctx->sc_wr.send_flags & - IB_SEND_SIGNALED; + __entry->num_sge = req->rl_wr.num_sge; + __entry->signaled = req->rl_wr.send_flags & IB_SEND_SIGNALED; __entry->status = status; ), diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 5cd8715..523722b 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -396,7 +396,7 @@ int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req) struct ib_send_wr *post_wr; struct rpcrdma_mr *mr; - post_wr = &req->rl_sendctx->sc_wr; + post_wr = &req->rl_wr; list_for_each_entry(mr, &req->rl_registered, mr_list) { struct rpcrdma_frwr *frwr; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 1941b22..53cd2e3 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -599,7 +599,7 @@ static bool rpcrdma_prepare_hdr_sge(struct rpcrdma_xprt *r_xprt, ib_dma_sync_single_for_device(rdmab_device(rb), sge->addr, sge->length, DMA_TO_DEVICE); - sc->sc_wr.num_sge++; + req->rl_wr.num_sge++; return true; out_regbuf: @@ -711,7 +711,7 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, } out: - sc->sc_wr.num_sge += sge_no; + req->rl_wr.num_sge += sge_no; if (sc->sc_unmap_count) kref_get(&req->rl_kref); return true; @@ -752,10 +752,13 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, req->rl_sendctx = rpcrdma_sendctx_get_locked(r_xprt); if (!req->rl_sendctx) goto err; - req->rl_sendctx->sc_wr.num_sge = 0; req->rl_sendctx->sc_unmap_count = 0; req->rl_sendctx->sc_req = req; kref_init(&req->rl_kref); + req->rl_wr.wr_cqe = &req->rl_sendctx->sc_cqe; + req->rl_wr.sg_list = req->rl_sendctx->sc_sges; + req->rl_wr.num_sge = 0; + req->rl_wr.opcode = IB_WR_SEND; ret = -EIO; if (!rpcrdma_prepare_hdr_sge(r_xprt, req, hdrlen)) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3ab086a..2f46582 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -811,9 +811,6 @@ static struct rpcrdma_sendctx *rpcrdma_sendctx_create(struct rpcrdma_ia *ia) if (!sc) return NULL; - sc->sc_wr.wr_cqe = &sc->sc_cqe; - sc->sc_wr.sg_list = sc->sc_sges; - sc->sc_wr.opcode = IB_WR_SEND; sc->sc_cqe.done = rpcrdma_wc_send; return sc; } @@ -1469,7 +1466,7 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) struct rpcrdma_ep *ep, struct rpcrdma_req *req) { - struct ib_send_wr *send_wr = &req->rl_sendctx->sc_wr; + struct ib_send_wr *send_wr = &req->rl_wr; int rc; if (!ep->rep_send_count || kref_read(&req->rl_kref) > 1) { diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 0e5b7f3..cdd6a3d 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -219,7 +219,6 @@ enum { */ struct rpcrdma_req; struct rpcrdma_sendctx { - struct ib_send_wr sc_wr; struct ib_cqe sc_cqe; struct rpcrdma_req *sc_req; unsigned int sc_unmap_count; @@ -314,6 +313,7 @@ struct rpcrdma_req { struct rpcrdma_rep *rl_reply; struct xdr_stream rl_stream; struct xdr_buf rl_hdrbuf; + struct ib_send_wr rl_wr; struct rpcrdma_sendctx *rl_sendctx; struct rpcrdma_regbuf *rl_rdmabuf; /* xprt header */ struct rpcrdma_regbuf *rl_sendbuf; /* rq_snd_buf */ From patchwork Thu Oct 17 18:31:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11196833 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F0381575 for ; Thu, 17 Oct 2019 18:31:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 40F1021925 for ; Thu, 17 Oct 2019 18:31:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="teyzA88N" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389368AbfJQSbr (ORCPT ); Thu, 17 Oct 2019 14:31:47 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:34056 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731227AbfJQSbr (ORCPT ); Thu, 17 Oct 2019 14:31:47 -0400 Received: by mail-qt1-f194.google.com with SMTP id 3so5090900qta.1; Thu, 17 Oct 2019 11:31:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=otlt8/jDB1f+82HmzxFeICdchXOW0YwjEpb2xQOlzog=; b=teyzA88Nmc/qPzW8QcRgUp++LmNMi1XLekpPbNmkKyVH6GCS3e0WRDi7umshXC2MoP g5vupwI2pO7izVYhMkIAq0HX1VtWLFVQy2SnHuZBV+0XL4C0mqnk6hcWT8XeeLhCoCjB tO7ZciQyR0WuHsYUymWO3lDoD5NoR1ug62qdMJOGsYAU3jLrc721ryHGI57AQPSGUTDA PuQ/RMD/gURGCV+NtlzLqXtZ2bZnqU6nSkpmlTaP9SkxHJIzPjBfv290KNQJv8uLfXfE qviWsy5Mh/GgQpgmURT7UsqyXPE7TruyLUmlk2wZmDoj2UwQIodbMmYBIINGZ4ffRsg2 jwFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=otlt8/jDB1f+82HmzxFeICdchXOW0YwjEpb2xQOlzog=; b=muPBPK/cox38W4mslDVQELw7aBUywYf6pgaqDdKssJTDe/hL7l5UMbH5aWHX1+8Xec JAIJJLNSMeuztnbQcbyzQEZbFFlBr7PCtH8qZQGbQv8vuZKdrtWKcPB7S2lzCa1cQ5QN exRk+aMBUCU1/JBhBMNXgYNn68JtoGWFXw0RxfoCm1hOvYw38YPEyR0dyCgAMuhvEBlL lFk95UnRUWj62h1V1FzuNsgHfmWCedJBg4xW1NsupOB1e68MAMJH4WNbG+AicY1kwIFR kdkSxEQhV22KVyvhh3ppJHNcQ7UY3RuwTDAshygZntlIoKKQFvIqKoP4LfwXDmYC2NFD 0Idg== X-Gm-Message-State: APjAAAU2BT1s7HLmo4HMvunUrAJMgrVH+tOSVpRME6PItc//4oo8buZO Eots/cjbl+t3Ncq/oc9fCb7iu2an X-Google-Smtp-Source: APXvYqzKET+Od9esIlgh2zQ0DH/bucDE6Flue44Yr8cVwUgjGgmn/sONcvLwairkHher84nFZBsYPg== X-Received: by 2002:ac8:237b:: with SMTP id b56mr5625583qtb.264.1571337105840; Thu, 17 Oct 2019 11:31:45 -0700 (PDT) Received: from oracle-102.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id d10sm1403350qko.29.2019.10.17.11.31.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 11:31:45 -0700 (PDT) Subject: [PATCH v1 5/6] xprtrdma: Refactor rpcrdma_prepare_msg_sges() From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Oct 2019 14:31:44 -0400 Message-ID: <20191017183144.2517.16352.stgit@oracle-102.nfsv4bat.org> In-Reply-To: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> References: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Refactor: Replace spaghetti with code that makes it plain what needs to be done for each rtype. This makes it easier to add features and optimizations. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 263 ++++++++++++++++++++++------------------ 1 file changed, 146 insertions(+), 117 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 53cd2e3..a441dbf 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -589,148 +589,162 @@ static bool rpcrdma_prepare_hdr_sge(struct rpcrdma_xprt *r_xprt, { struct rpcrdma_sendctx *sc = req->rl_sendctx; struct rpcrdma_regbuf *rb = req->rl_rdmabuf; - struct ib_sge *sge = sc->sc_sges; + struct ib_sge *sge = &sc->sc_sges[req->rl_wr.num_sge++]; if (!rpcrdma_regbuf_dma_map(r_xprt, rb)) - goto out_regbuf; + return false; sge->addr = rdmab_addr(rb); sge->length = len; sge->lkey = rdmab_lkey(rb); ib_dma_sync_single_for_device(rdmab_device(rb), sge->addr, sge->length, DMA_TO_DEVICE); - req->rl_wr.num_sge++; return true; - -out_regbuf: - pr_err("rpcrdma: failed to DMA map a Send buffer\n"); - return false; } -/* Prepare the Send SGEs. The head and tail iovec, and each entry - * in the page list, gets its own SGE. +/* The head iovec is straightforward, as it is usually already + * DMA-mapped. Sync the content that has changed. */ -static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, - struct rpcrdma_req *req, - struct xdr_buf *xdr, - enum rpcrdma_chunktype rtype) +static bool rpcrdma_prepare_head_iov(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, unsigned int len) { struct rpcrdma_sendctx *sc = req->rl_sendctx; - unsigned int sge_no, page_base, len, remaining; + struct ib_sge *sge = &sc->sc_sges[req->rl_wr.num_sge++]; struct rpcrdma_regbuf *rb = req->rl_sendbuf; - struct ib_sge *sge = sc->sc_sges; - struct page *page, **ppages; - /* The head iovec is straightforward, as it is already - * DMA-mapped. Sync the content that has changed. - */ if (!rpcrdma_regbuf_dma_map(r_xprt, rb)) - goto out_regbuf; - sge_no = 1; - sge[sge_no].addr = rdmab_addr(rb); - sge[sge_no].length = xdr->head[0].iov_len; - sge[sge_no].lkey = rdmab_lkey(rb); - ib_dma_sync_single_for_device(rdmab_device(rb), sge[sge_no].addr, - sge[sge_no].length, DMA_TO_DEVICE); - - /* If there is a Read chunk, the page list is being handled - * via explicit RDMA, and thus is skipped here. However, the - * tail iovec may include an XDR pad for the page list, as - * well as additional content, and may not reside in the - * same page as the head iovec. - */ - if (rtype == rpcrdma_readch) { - len = xdr->tail[0].iov_len; + return false; - /* Do not include the tail if it is only an XDR pad */ - if (len < 4) - goto out; + sge->addr = rdmab_addr(rb); + sge->length = len; + sge->lkey = rdmab_lkey(rb); - page = virt_to_page(xdr->tail[0].iov_base); - page_base = offset_in_page(xdr->tail[0].iov_base); + ib_dma_sync_single_for_device(rdmab_device(rb), sge->addr, sge->length, + DMA_TO_DEVICE); + return true; +} - /* If the content in the page list is an odd length, - * xdr_write_pages() has added a pad at the beginning - * of the tail iovec. Force the tail's non-pad content - * to land at the next XDR position in the Send message. - */ - page_base += len & 3; - len -= len & 3; - goto map_tail; - } +/* If there is a page list present, DMA map and prepare an + * SGE for each page to be sent. + */ +static bool rpcrdma_prepare_pagelist(struct rpcrdma_req *req, + struct xdr_buf *xdr) +{ + struct rpcrdma_sendctx *sc = req->rl_sendctx; + struct rpcrdma_regbuf *rb = req->rl_sendbuf; + unsigned int page_base, len, remaining; + struct page **ppages; + struct ib_sge *sge; - /* If there is a page list present, temporarily DMA map - * and prepare an SGE for each page to be sent. - */ - if (xdr->page_len) { - ppages = xdr->pages + (xdr->page_base >> PAGE_SHIFT); - page_base = offset_in_page(xdr->page_base); - remaining = xdr->page_len; - while (remaining) { - sge_no++; - if (sge_no > RPCRDMA_MAX_SEND_SGES - 2) - goto out_mapping_overflow; - - len = min_t(u32, PAGE_SIZE - page_base, remaining); - sge[sge_no].addr = - ib_dma_map_page(rdmab_device(rb), *ppages, - page_base, len, DMA_TO_DEVICE); - if (ib_dma_mapping_error(rdmab_device(rb), - sge[sge_no].addr)) - goto out_mapping_err; - sge[sge_no].length = len; - sge[sge_no].lkey = rdmab_lkey(rb); - - sc->sc_unmap_count++; - ppages++; - remaining -= len; - page_base = 0; - } - } + ppages = xdr->pages + (xdr->page_base >> PAGE_SHIFT); + page_base = offset_in_page(xdr->page_base); + remaining = xdr->page_len; + while (remaining) { + sge = &sc->sc_sges[req->rl_wr.num_sge++]; + len = min_t(unsigned int, PAGE_SIZE - page_base, remaining); + sge->addr = ib_dma_map_page(rdmab_device(rb), *ppages, + page_base, len, DMA_TO_DEVICE); + if (ib_dma_mapping_error(rdmab_device(rb), sge->addr)) + goto out_mapping_err; - /* The tail iovec is not always constructed in the same - * page where the head iovec resides (see, for example, - * gss_wrap_req_priv). To neatly accommodate that case, - * DMA map it separately. - */ - if (xdr->tail[0].iov_len) { - page = virt_to_page(xdr->tail[0].iov_base); - page_base = offset_in_page(xdr->tail[0].iov_base); - len = xdr->tail[0].iov_len; + sge->length = len; + sge->lkey = rdmab_lkey(rb); -map_tail: - sge_no++; - sge[sge_no].addr = - ib_dma_map_page(rdmab_device(rb), page, page_base, len, - DMA_TO_DEVICE); - if (ib_dma_mapping_error(rdmab_device(rb), sge[sge_no].addr)) - goto out_mapping_err; - sge[sge_no].length = len; - sge[sge_no].lkey = rdmab_lkey(rb); sc->sc_unmap_count++; + ppages++; + remaining -= len; + page_base = 0; } -out: - req->rl_wr.num_sge += sge_no; - if (sc->sc_unmap_count) - kref_get(&req->rl_kref); return true; -out_regbuf: - pr_err("rpcrdma: failed to DMA map a Send buffer\n"); +out_mapping_err: + trace_xprtrdma_dma_maperr(sge->addr); return false; +} -out_mapping_overflow: - rpcrdma_sendctx_unmap(sc); - pr_err("rpcrdma: too many Send SGEs (%u)\n", sge_no); - return false; +/* The tail iovec may include an XDR pad for the page list, + * as well as additional content, and may not reside in the + * same page as the head iovec. + */ +static bool rpcrdma_prepare_tail_iov(struct rpcrdma_req *req, + struct xdr_buf *xdr, + unsigned int page_base, unsigned int len) +{ + struct rpcrdma_sendctx *sc = req->rl_sendctx; + struct ib_sge *sge = &sc->sc_sges[req->rl_wr.num_sge++]; + struct rpcrdma_regbuf *rb = req->rl_sendbuf; + struct page *page = virt_to_page(xdr->tail[0].iov_base); + + sge->addr = ib_dma_map_page(rdmab_device(rb), page, page_base, len, + DMA_TO_DEVICE); + if (ib_dma_mapping_error(rdmab_device(rb), sge->addr)) + goto out_mapping_err; + + sge->length = len; + sge->lkey = rdmab_lkey(rb); + ++sc->sc_unmap_count; + return true; out_mapping_err: - rpcrdma_sendctx_unmap(sc); - trace_xprtrdma_dma_maperr(sge[sge_no].addr); + trace_xprtrdma_dma_maperr(sge->addr); return false; } +static bool rpcrdma_prepare_noch_mapped(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct xdr_buf *xdr) +{ + struct kvec *tail = &xdr->tail[0]; + + if (!rpcrdma_prepare_head_iov(r_xprt, req, xdr->head[0].iov_len)) + return false; + if (xdr->page_len) + if (!rpcrdma_prepare_pagelist(req, xdr)) + return false; + if (tail->iov_len) + if (!rpcrdma_prepare_tail_iov(req, xdr, + offset_in_page(tail->iov_base), + tail->iov_len)) + return false; + + if (req->rl_sendctx->sc_unmap_count) + kref_get(&req->rl_kref); + return true; +} + +static bool rpcrdma_prepare_readch(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct xdr_buf *xdr) +{ + if (!rpcrdma_prepare_head_iov(r_xprt, req, xdr->head[0].iov_len)) + return false; + + /* If there is a Read chunk, the page list is being handled + * via explicit RDMA, and thus is skipped here. + */ + + /* Do not include the tail if it is only an XDR pad */ + if (xdr->tail[0].iov_len > 3) { + unsigned int page_base, len; + + /* If the content in the page list is an odd length, + * xdr_write_pages() adds a pad at the beginning of + * the tail iovec. Force the tail's non-pad content to + * land at the next XDR position in the Send message. + */ + page_base = offset_in_page(xdr->tail[0].iov_base); + len = xdr->tail[0].iov_len; + page_base += len & 3; + len -= len & 3; + if (!rpcrdma_prepare_tail_iov(req, xdr, page_base, len)) + return false; + kref_get(&req->rl_kref); + } + + return true; +} + /** * rpcrdma_prepare_send_sges - Construct SGEs for a Send WR * @r_xprt: controlling transport @@ -741,17 +755,17 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, * * Returns 0 on success; otherwise a negative errno is returned. */ -int -rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, - struct rpcrdma_req *req, u32 hdrlen, - struct xdr_buf *xdr, enum rpcrdma_chunktype rtype) +inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, u32 hdrlen, + struct xdr_buf *xdr, + enum rpcrdma_chunktype rtype) { int ret; ret = -EAGAIN; req->rl_sendctx = rpcrdma_sendctx_get_locked(r_xprt); if (!req->rl_sendctx) - goto err; + goto out_nosc; req->rl_sendctx->sc_unmap_count = 0; req->rl_sendctx->sc_req = req; kref_init(&req->rl_kref); @@ -762,13 +776,28 @@ static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt, ret = -EIO; if (!rpcrdma_prepare_hdr_sge(r_xprt, req, hdrlen)) - goto err; - if (rtype != rpcrdma_areadch) - if (!rpcrdma_prepare_msg_sges(r_xprt, req, xdr, rtype)) - goto err; + goto out_unmap; + + switch (rtype) { + case rpcrdma_noch: + if (!rpcrdma_prepare_noch_mapped(r_xprt, req, xdr)) + goto out_unmap; + break; + case rpcrdma_readch: + if (!rpcrdma_prepare_readch(r_xprt, req, xdr)) + goto out_unmap; + break; + case rpcrdma_areadch: + break; + default: + goto out_unmap; + } + return 0; -err: +out_unmap: + rpcrdma_sendctx_unmap(req->rl_sendctx); +out_nosc: trace_xprtrdma_prepsend_failed(&req->rl_slot, ret); return ret; } From patchwork Thu Oct 17 18:31:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11196837 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EAF1813B1 for ; Thu, 17 Oct 2019 18:31:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BCC8321925 for ; Thu, 17 Oct 2019 18:31:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="vgaQlvxj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390554AbfJQSb4 (ORCPT ); Thu, 17 Oct 2019 14:31:56 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:42715 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731227AbfJQSb4 (ORCPT ); Thu, 17 Oct 2019 14:31:56 -0400 Received: by mail-qk1-f196.google.com with SMTP id f16so2803330qkl.9; Thu, 17 Oct 2019 11:31:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=lLcZy06AHylCycj31ZmQsaJzF4hrjnC7RjraMBkZdVU=; b=vgaQlvxjPFDDhf/Dl9y+h6VlYhig7lWlJ5aStPlJQDWdLuYcBORSNlKfwi1z8N4k5C NM3qyTNAYpaFHnhc+7jTcmHIIDADD2u7YBdygIQit+ytMSHNxFkfwiunrwlFTlENKnqE 3NCtgneRFd6Cj0VdKIO+yszhm7pScEjuNAF2KGE/w6/5WdzPzi0h9bJvaUuPfcA7ayj4 9kiWTbrt5NAwfW2+1VBOi5SwUAuQW914tG+SL5gSvsYnT5/TXvBBQKx9Ag0whXw8knMk tyYsEM1ADehp0BJuJbZC9KklYTqrLTVAnRz1aCxMdIrbA1vFuixp/1MvDx6LG7kG56JV b1ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=lLcZy06AHylCycj31ZmQsaJzF4hrjnC7RjraMBkZdVU=; b=di9TeM7ExXDlFu7q5ZFf3/1xSRuBEjdal/SF8FdxI3WTPNG9VMhrmTq8tzgfL2Fc9I 4tdaYChDqKr+99xm0fJjQe0xjIqL4JrljwX1kV8GsUMxohuyHilXKtq7JofHTthJyf8m fzVoBflOz8ZTbazra0NYxfA7mSp+vN+SBEsaMCSgMUL5UyujLUKxWoiLE5PX3cSKHcB2 +6ak3AwX+ixrK3+ZJMbge2tNuZrJESu7vMar0Ki2P3x/9L6AuPxhA41IMDoscsaUiCl9 Sk8wlsvcN32qnMEYtu52V4QblcdwJjAFIUEIWboeY4DF3PG6gNaq2Ze/W4G1zp2Zdhe6 1V9w== X-Gm-Message-State: APjAAAVWZpQJW28Mk6xixkONmay1dS85U92EZQdNBQ8pYx8bB5JpkCpi ZAMGUS/wXq/MdhNh+UaEOsZ674mD X-Google-Smtp-Source: APXvYqwLuwj/Bv07kEsshQ59Y9AjKhlx+Xb87ZqnooIo1P5XB/B1HN7rUkHo/TELMuvbZ0JbpQkiCw== X-Received: by 2002:a05:620a:743:: with SMTP id i3mr4786286qki.268.1571337114203; Thu, 17 Oct 2019 11:31:54 -0700 (PDT) Received: from oracle-102.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id x19sm1397853qkf.26.2019.10.17.11.31.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Oct 2019 11:31:53 -0700 (PDT) Subject: [PATCH v1 6/6] xprtrdma: Pull up sometimes From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Oct 2019 14:31:53 -0400 Message-ID: <20191017183152.2517.67599.stgit@oracle-102.nfsv4bat.org> In-Reply-To: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> References: <20191017182811.2517.25676.stgit@oracle-102.nfsv4bat.org> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org On some platforms, DMA mapping part of a page is more costly than copying bytes. Restore the pull-up code and use that when we think it's going to be faster. The heuristic for now is to pull-up when the size of the RPC message body fits in the buffer underlying the head iovec. Indeed, not involving the I/O MMU can help the RPC/RDMA transport scale better for tiny I/Os across more RDMA devices. This is because interaction with the I/O MMU is eliminated, as is handling a Send completion, for each of these small I/Os. Without the explicit unmapping, the NIC no longer needs to do a costly internal TLB shoot down for buffers that are just a handful of bytes. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 2 - net/sunrpc/xprtrdma/rpc_rdma.c | 82 +++++++++++++++++++++++++++++++++++-- net/sunrpc/xprtrdma/verbs.c | 2 - net/sunrpc/xprtrdma/xprt_rdma.h | 2 + 4 files changed, 81 insertions(+), 7 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 16572ae7..336a65d 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -532,6 +532,8 @@ DEFINE_WRCH_EVENT(reply); TRACE_DEFINE_ENUM(rpcrdma_noch); +TRACE_DEFINE_ENUM(rpcrdma_noch_pullup); +TRACE_DEFINE_ENUM(rpcrdma_noch_mapped); TRACE_DEFINE_ENUM(rpcrdma_readch); TRACE_DEFINE_ENUM(rpcrdma_areadch); TRACE_DEFINE_ENUM(rpcrdma_writech); @@ -540,6 +542,8 @@ #define xprtrdma_show_chunktype(x) \ __print_symbolic(x, \ { rpcrdma_noch, "inline" }, \ + { rpcrdma_noch_pullup, "pullup" }, \ + { rpcrdma_noch_mapped, "mapped" }, \ { rpcrdma_readch, "read list" }, \ { rpcrdma_areadch, "*read list" }, \ { rpcrdma_writech, "write list" }, \ diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 50e075f..1168524 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -79,7 +79,7 @@ static int rpcrdma_bc_marshal_reply(struct rpc_rqst *rqst) *p = xdr_zero; if (rpcrdma_prepare_send_sges(r_xprt, req, RPCRDMA_HDRLEN_MIN, - &rqst->rq_snd_buf, rpcrdma_noch)) + &rqst->rq_snd_buf, rpcrdma_noch_pullup)) return -EIO; trace_xprtrdma_cb_reply(rqst); diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index a441dbf..4ad8889 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -392,7 +392,7 @@ static int rpcrdma_encode_read_list(struct rpcrdma_xprt *r_xprt, unsigned int pos; int nsegs; - if (rtype == rpcrdma_noch) + if (rtype == rpcrdma_noch_pullup || rtype == rpcrdma_noch_mapped) goto done; pos = rqst->rq_snd_buf.head[0].iov_len; @@ -691,6 +691,72 @@ static bool rpcrdma_prepare_tail_iov(struct rpcrdma_req *req, return false; } +/* Copy the tail to the end of the head buffer. + */ +static void rpcrdma_pullup_tail_iov(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct xdr_buf *xdr) +{ + unsigned char *dst; + + dst = (unsigned char *)xdr->head[0].iov_base; + dst += xdr->head[0].iov_len + xdr->page_len; + memmove(dst, xdr->tail[0].iov_base, xdr->tail[0].iov_len); + r_xprt->rx_stats.pullup_copy_count += xdr->tail[0].iov_len; +} + +/* Copy pagelist content into the head buffer. + */ +static void rpcrdma_pullup_pagelist(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct xdr_buf *xdr) +{ + unsigned int len, page_base, remaining; + struct page **ppages; + unsigned char *src, *dst; + + dst = (unsigned char *)xdr->head[0].iov_base; + dst += xdr->head[0].iov_len; + ppages = xdr->pages + (xdr->page_base >> PAGE_SHIFT); + page_base = offset_in_page(xdr->page_base); + remaining = xdr->page_len; + while (remaining) { + src = page_address(*ppages); + src += page_base; + len = min_t(unsigned int, PAGE_SIZE - page_base, remaining); + memcpy(dst, src, len); + r_xprt->rx_stats.pullup_copy_count += len; + + ppages++; + dst += len; + remaining -= len; + page_base = 0; + } +} + +/* Copy the contents of @xdr into @rl_sendbuf and DMA sync it. + * When the head, pagelist, and tail are small, a pull-up copy + * is considerably less costly than DMA mapping the components + * of @xdr. + * + * Assumptions: + * - the caller has already verified that the total length + * of the RPC Call body will fit into @rl_sendbuf. + */ +static bool rpcrdma_prepare_noch_pullup(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct xdr_buf *xdr) +{ + if (unlikely(xdr->tail[0].iov_len)) + rpcrdma_pullup_tail_iov(r_xprt, req, xdr); + + if (unlikely(xdr->page_len)) + rpcrdma_pullup_pagelist(r_xprt, req, xdr); + + /* The whole RPC message resides in the head iovec now */ + return rpcrdma_prepare_head_iov(r_xprt, req, xdr->len); +} + static bool rpcrdma_prepare_noch_mapped(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, struct xdr_buf *xdr) @@ -779,7 +845,11 @@ inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, goto out_unmap; switch (rtype) { - case rpcrdma_noch: + case rpcrdma_noch_pullup: + if (!rpcrdma_prepare_noch_pullup(r_xprt, req, xdr)) + goto out_unmap; + break; + case rpcrdma_noch_mapped: if (!rpcrdma_prepare_noch_mapped(r_xprt, req, xdr)) goto out_unmap; break; @@ -827,6 +897,7 @@ inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req = rpcr_to_rdmar(rqst); struct xdr_stream *xdr = &req->rl_stream; enum rpcrdma_chunktype rtype, wtype; + struct xdr_buf *buf = &rqst->rq_snd_buf; bool ddp_allowed; __be32 *p; int ret; @@ -884,8 +955,9 @@ inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, */ if (rpcrdma_args_inline(r_xprt, rqst)) { *p++ = rdma_msg; - rtype = rpcrdma_noch; - } else if (ddp_allowed && rqst->rq_snd_buf.flags & XDRBUF_WRITE) { + rtype = buf->len < rdmab_length(req->rl_sendbuf) ? + rpcrdma_noch_pullup : rpcrdma_noch_mapped; + } else if (ddp_allowed && buf->flags & XDRBUF_WRITE) { *p++ = rdma_msg; rtype = rpcrdma_readch; } else { @@ -927,7 +999,7 @@ inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt, goto out_err; ret = rpcrdma_prepare_send_sges(r_xprt, req, req->rl_hdrbuf.len, - &rqst->rq_snd_buf, rtype); + buf, rtype); if (ret) goto out_err; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 2f46582..a514e2c 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1165,7 +1165,7 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) for (i = 0; i < buf->rb_max_requests; i++) { struct rpcrdma_req *req; - req = rpcrdma_req_create(r_xprt, RPCRDMA_V1_DEF_INLINE_SIZE, + req = rpcrdma_req_create(r_xprt, RPCRDMA_V1_DEF_INLINE_SIZE * 2, GFP_KERNEL); if (!req) goto out; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index cdd6a3d..5d15140 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -554,6 +554,8 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, enum rpcrdma_chunktype { rpcrdma_noch = 0, + rpcrdma_noch_pullup, + rpcrdma_noch_mapped, rpcrdma_readch, rpcrdma_areadch, rpcrdma_writech,