From patchwork Wed Nov 9 17:34:10 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 9419909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 59F9F60512 for ; Wed, 9 Nov 2016 17:52:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4C07F28973 for ; Wed, 9 Nov 2016 17:52:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E44B29361; Wed, 9 Nov 2016 17:52:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B4F7628973 for ; Wed, 9 Nov 2016 17:52:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752963AbcKIRwN (ORCPT ); Wed, 9 Nov 2016 12:52:13 -0500 Received: from mail-it0-f66.google.com ([209.85.214.66]:33329 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752634AbcKIRwM (ORCPT ); Wed, 9 Nov 2016 12:52:12 -0500 Received: by mail-it0-f66.google.com with SMTP id e187so23255686itc.0; Wed, 09 Nov 2016 09:52:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=a53HBhNfOn/Vlx79Jp/nALRJSFfxzu+9h156a4W16Hg=; b=clolrC+vIcSkMaRxTtY1JetpY2M+J838IY/LOvI5e0boHkmttzZE5bHDaCJptf2ZgK D3PPyKxLCOtBXRV64ZO1Qe4t+ouJmrNgoY2RP+0DpRcn3q1UQ/pT0/CmCLHkLzIrpDOw PcQp+T6Q6oDhfyBq0AEwXP+9SMpPJkZNU0nNPcRjPqjT2b55P4vahtyeBvdAD5qk+XPg bkdm6gLTfv9Re0oDiBH9MmrHXWmuTpvfINFePPXTyk2aM2i+pBmOL3dJXzLQFhiBTick SW3lvxDQiJOJ4UX56dvegVTYlC1GUlV9cJTvoH9Js2cf1rBAAMqk9+/IbpfUv7dTwZ4T j5fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=a53HBhNfOn/Vlx79Jp/nALRJSFfxzu+9h156a4W16Hg=; b=j+1zKLrhfnR/xPVbe0AisCPvNpwsy1fIVaRCFhCDvGbwb2b41xo2kgY2V3Nmv3PhpJ hRJ2OBemzZxhfSpXonMX7AYPsh/6diPtZZbYryVIZFqIwmJdkmwZFIJQeK3qgICLSxIS yXw/1PSqJ9KuRMViw1FDyL/9hGL1VmgOfDXn/iYncWjPEaQnhzQ8D5l0Qcm7OKg9G4gY Hk1mwuQRdj0DdKmDWRQ5J5+ZBBrktgXlORRdNyuP6b525I52D7S666Nl6Ta0erBtsh6W gOKZ9XitDKC9rp0OGf0E2clU055Y+m9PxHp4AAUDq3aH2oEvmwHeLZJYnpHO/OIvR+X6 m5AA== X-Gm-Message-State: ABUngvcT3WfoXKCai3TljNpVaMUWfP8hAYC8oCuGSe5jWewSataVobF8bhAB+17wBXShdg== X-Received: by 10.36.70.18 with SMTP id j18mr15743043itb.97.1478712851468; Wed, 09 Nov 2016 09:34:11 -0800 (PST) Received: from klimt.1015granger.net ([2604:8800:100:81fc:ec4:7aff:fe6c:6aa0]) by smtp.gmail.com with ESMTPSA id r3sm426601iod.24.2016.11.09.09.34.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Nov 2016 09:34:11 -0800 (PST) Subject: [PATCH v1 4/7] svcrdma: Remove BH-disabled spin locking in svc_rdma_send() From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Wed, 09 Nov 2016 12:34:10 -0500 Message-ID: <20161109173410.15735.12286.stgit@klimt.1015granger.net> In-Reply-To: <20161109165814.15735.47815.stgit@klimt.1015granger.net> References: <20161109165814.15735.47815.stgit@klimt.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP svcrdma's current SQ accounting algorithm takes sc_lock and disables bottom-halves while posting all RDMA Read, Write, and Send WRs. This is relatively heavyweight serialization. And note that Write and Send are already fully serialized by the xpt_mutex. Using a single atomic_t should be all that is necessary to guarantee that ib_post_send() is called only when there is enough space on the send queue. This is what the other RDMA-enabled storage targets do. Signed-off-by: Chuck Lever --- include/linux/sunrpc/svc_rdma.h | 2 +- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 7 ++++++- net/sunrpc/xprtrdma/svc_rdma_transport.c | 18 +++++++----------- 3 files changed, 14 insertions(+), 13 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index 6aef63b..601cb07 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -139,7 +139,7 @@ struct svcxprt_rdma { int sc_max_sge_rd; /* max sge for read target */ bool sc_snd_w_inv; /* OK to use Send With Invalidate */ - atomic_t sc_sq_count; /* Number of SQ WR on queue */ + atomic_t sc_sq_avail; /* SQEs ready to be consumed */ unsigned int sc_sq_depth; /* Depth of SQ */ unsigned int sc_rq_depth; /* Depth of RQ */ u32 sc_max_requests; /* Forward credits */ diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index 0a58d40..30eeab5 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -594,7 +594,12 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) goto err0; inline_bytes = rqstp->rq_res.len; - /* Create the RDMA response header */ + /* Create the RDMA response header. xprt->xpt_mutex, + * acquired in svc_send(), serializes RPC replies. The + * code path below that inserts the credit grant value + * into each transport header runs only inside this + * critical section. + */ ret = -ENOMEM; res_page = alloc_page(GFP_KERNEL); if (!res_page) diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index 6864fb9..f7dd6494 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -434,7 +434,7 @@ static void svc_rdma_send_wc_common(struct svcxprt_rdma *xprt, goto err; out: - atomic_dec(&xprt->sc_sq_count); + atomic_inc(&xprt->sc_sq_avail); wake_up(&xprt->sc_send_wait); return; @@ -1008,6 +1008,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_rq_depth = newxprt->sc_max_requests + newxprt->sc_max_bc_requests; newxprt->sc_sq_depth = RPCRDMA_SQ_DEPTH_MULT * newxprt->sc_rq_depth; + atomic_set(&newxprt->sc_sq_avail, newxprt->sc_sq_depth); if (!svc_rdma_prealloc_ctxts(newxprt)) goto errout; @@ -1333,15 +1334,13 @@ int svc_rdma_send(struct svcxprt_rdma *xprt, struct ib_send_wr *wr) /* If the SQ is full, wait until an SQ entry is available */ while (1) { - spin_lock_bh(&xprt->sc_lock); - if (xprt->sc_sq_depth < atomic_read(&xprt->sc_sq_count) + wr_count) { - spin_unlock_bh(&xprt->sc_lock); + if ((atomic_sub_return(wr_count, &xprt->sc_sq_avail) < 0)) { atomic_inc(&rdma_stat_sq_starve); /* Wait until SQ WR available if SQ still full */ + atomic_add(wr_count, &xprt->sc_sq_avail); wait_event(xprt->sc_send_wait, - atomic_read(&xprt->sc_sq_count) < - xprt->sc_sq_depth); + atomic_read(&xprt->sc_sq_avail) > wr_count); if (test_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags)) return -ENOTCONN; continue; @@ -1351,19 +1350,16 @@ int svc_rdma_send(struct svcxprt_rdma *xprt, struct ib_send_wr *wr) svc_xprt_get(&xprt->sc_xprt); /* Bump used SQ WR count and post */ - atomic_add(wr_count, &xprt->sc_sq_count); ret = ib_post_send(xprt->sc_qp, wr, &bad_wr); if (ret) { set_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags); - atomic_sub(wr_count, &xprt->sc_sq_count); for (i = 0; i < wr_count; i ++) svc_xprt_put(&xprt->sc_xprt); dprintk("svcrdma: failed to post SQ WR rc=%d, " - "sc_sq_count=%d, sc_sq_depth=%d\n", - ret, atomic_read(&xprt->sc_sq_count), + "sc_sq_avail=%d, sc_sq_depth=%d\n", + ret, atomic_read(&xprt->sc_sq_avail), xprt->sc_sq_depth); } - spin_unlock_bh(&xprt->sc_lock); if (ret) wake_up(&xprt->sc_send_wait); break;