From patchwork Mon Aug 19 22:36:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102195 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E5EF13B1 for ; Mon, 19 Aug 2019 22:36:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5D29B22CEC for ; Mon, 19 Aug 2019 22:36:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EoDZczF5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728656AbfHSWgl (ORCPT ); Mon, 19 Aug 2019 18:36:41 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:44864 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728435AbfHSWgl (ORCPT ); Mon, 19 Aug 2019 18:36:41 -0400 Received: by mail-ot1-f65.google.com with SMTP id w4so3209037ote.11; Mon, 19 Aug 2019 15:36:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=v/LwbD9loDDrhZ9x90qmsZdLimN0wXa2LzcUKgQfxQc=; b=EoDZczF5XYhNzKSOJKgZp4lVYMH6JteEpKYL23AgOlFpEt2C2Qg8DWoHCfMrhCKtgF m5EkayDhhIj81vaOqlT7gdwRsflgWI8deRMbWmGQAM8709SpBM6VzjZ4lfg9HIv2pGgZ pt655Gyb5tglMhXdqMxy0/COunKOq3Mk3AkV1Xcjl6Oh8GjhMaNZ/NX5DRqi15gd6t4q DT3/bR0rnvd76ZzmyoVa5z+6RL8gwqs0HD1uHzdQHfeZ9WGK4uTB5xyl6xxBziv81oJV owuL9bTO1SgBZnsLt+tEdpQukQt+ccv7zJU0NmR+FuuNjid3LRU6YvWgWj/1E0sDCZfE XdoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=v/LwbD9loDDrhZ9x90qmsZdLimN0wXa2LzcUKgQfxQc=; b=g6TAqYdJCohfUK1mLuprB5lno/9lFUXktGbsama3oiqIXpHmbQCs+cEAAKLDMy/YcM xdcG42wkTGzrgBT9LPhBVnl3l1SKH1KgfZ458iBT/AE7esrcvDBw0VtZm1teLueJ++Yd bq1dEsaBRU/eov/4wyQgLQuO4xKhjC1f+CtzCBhvvV6MIuB85Fn3Ce5+ub4g8zUR5MYL VCcM1n3mPauCHqhB58cUqSCFg9ja84lgsrCFOqQ+m5nL4SlilRi24HugCOUKBKiExJFE 0BsZmtwuBD9Rh7VQSdhaYQP3QE0yr8gpJzAzw/vwHzpS4vGDmPwZJsbEWz05xY8i88N0 ftRQ== X-Gm-Message-State: APjAAAWTL7drtdfi/VxZp5ZDRlfDLkV+6PxP3EAnDuqPiwpY1MeEyVmu GWqdRB0E0OWGjwk9Cheircd5fZao X-Google-Smtp-Source: APXvYqzZTwKdznpHNOaf3S1X3ioDVgrnRrA/tU3eV2d/AsA5PY5eDzAEeQOHHScdn1UnDhbZPkZ3+g== X-Received: by 2002:a9d:591a:: with SMTP id t26mr17062173oth.170.1566254200348; Mon, 19 Aug 2019 15:36:40 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id v17sm4713375oif.1.2019.08.19.15.36.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:36:40 -0700 (PDT) Subject: [PATCH v2 01/21] SUNRPC: Remove rpc_wake_up_queued_task_on_wq() From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:36:19 -0400 Message-ID: <156625415896.8161.6772555466035123844.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up: commit c544577daddb ("SUNRPC: Clean up transport write space handling") appears to have removed the last caller of rpc_wake_up_queued_task_on_wq(). Signed-off-by: Chuck Lever --- include/linux/sunrpc/sched.h | 3 --- net/sunrpc/sched.c | 27 ++++----------------------- 2 files changed, 4 insertions(+), 26 deletions(-) diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h index baa3ecdb882f..d1283bddd218 100644 --- a/include/linux/sunrpc/sched.h +++ b/include/linux/sunrpc/sched.h @@ -243,9 +243,6 @@ void rpc_sleep_on_priority_timeout(struct rpc_wait_queue *queue, void rpc_sleep_on_priority(struct rpc_wait_queue *, struct rpc_task *, int priority); -void rpc_wake_up_queued_task_on_wq(struct workqueue_struct *wq, - struct rpc_wait_queue *queue, - struct rpc_task *task); void rpc_wake_up_queued_task(struct rpc_wait_queue *, struct rpc_task *); void rpc_wake_up_queued_task_set_status(struct rpc_wait_queue *, diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c index 1f275aba786f..f25c4b9ba185 100644 --- a/net/sunrpc/sched.c +++ b/net/sunrpc/sched.c @@ -541,33 +541,14 @@ rpc_wake_up_task_on_wq_queue_action_locked(struct workqueue_struct *wq, return NULL; } -static void -rpc_wake_up_task_on_wq_queue_locked(struct workqueue_struct *wq, - struct rpc_wait_queue *queue, struct rpc_task *task) -{ - rpc_wake_up_task_on_wq_queue_action_locked(wq, queue, task, NULL, NULL); -} - /* * Wake up a queued task while the queue lock is being held */ -static void rpc_wake_up_task_queue_locked(struct rpc_wait_queue *queue, struct rpc_task *task) +static void rpc_wake_up_task_queue_locked(struct rpc_wait_queue *queue, + struct rpc_task *task) { - rpc_wake_up_task_on_wq_queue_locked(rpciod_workqueue, queue, task); -} - -/* - * Wake up a task on a specific queue - */ -void rpc_wake_up_queued_task_on_wq(struct workqueue_struct *wq, - struct rpc_wait_queue *queue, - struct rpc_task *task) -{ - if (!RPC_IS_QUEUED(task)) - return; - spin_lock(&queue->lock); - rpc_wake_up_task_on_wq_queue_locked(wq, queue, task); - spin_unlock(&queue->lock); + rpc_wake_up_task_on_wq_queue_action_locked(rpciod_workqueue, queue, + task, NULL, NULL); } /* From patchwork Mon Aug 19 22:37:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102199 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F33DA1399 for ; Mon, 19 Aug 2019 22:37:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D2AF72070B for ; Mon, 19 Aug 2019 22:37:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Sn4TwetY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728508AbfHSWh2 (ORCPT ); Mon, 19 Aug 2019 18:37:28 -0400 Received: from mail-oi1-f196.google.com ([209.85.167.196]:45136 "EHLO mail-oi1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728494AbfHSWh2 (ORCPT ); Mon, 19 Aug 2019 18:37:28 -0400 Received: by mail-oi1-f196.google.com with SMTP id v12so2587191oic.12; Mon, 19 Aug 2019 15:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=KGw+O8zhRdR6qOzf1+hFXDgB8d55DXqyorgudiaLglA=; b=Sn4TwetYV1xBZAep9BXXAFmwF7kGpp8l16LWM65wutSt1AceQdBSJmwzayzUwQ6vay Qvji3v0RFTt9PHNniGiETy7AsVGMHXa3mqnoWFsPGiqcL1aORVPMrGHo1Bd+j7Eb87aR 7LMfud0rJ66jp1xdKoj5rvQZUmqzqJcxz09csJO6vTT8pB069BjgwfAXMb5MChRg+5u/ rbkYatrun4C+8kKRlNE5zkDldDiDWUw/Ncte8jz5eRDhI3XuWn/hdIIoXnG7qF9KBFoE h4zbYVQcupqLiHkN97vWg/QUga63q9a4/i5Z43OZih1iQz9WZ2PVnAh3PK+p7MjQKAG2 TVtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=KGw+O8zhRdR6qOzf1+hFXDgB8d55DXqyorgudiaLglA=; b=JxzcF5h50vWurRtJY/coxqqZOuYkWl4Nt4HSzR5XGiHHDWAVI0yredWHwyHrGeYOsL mo8Q1qS6Dhwsp9u17TgldBz4aT4yAAsn10QnmMuk0FPdLo8CiCe7xH++L9ijvGAB0/nu AUOSrac1v6FMnM9LUYYcnOR4dDWOZuz0e5mzoOAx8TDcDtyE7uns+50mT3BC1TGiCV6p q5QAWWX4bX3QUrUGVlAUPtvUI2AkM3SWlTAWOI0yP2NNQnhVZLRZ63j91QZsbAbpUft6 YF5dgmlA99Yrc1oEYBwirw1gftLWEVW36vjuU12vkM0vS0PIfgmTjUR+VPm6l7P3HcIQ O3Mg== X-Gm-Message-State: APjAAAWoXZAomIl9aYpT0Oue8KU20coS+V7nGQoVm9oNA/MUr/Ceg+8t 2dXaGFzMgYszz0JQe5IMcQFfDX3G X-Google-Smtp-Source: APXvYqx8rizhqOOm6eIIZGHJ1F5x0Tdwk1fM5A/EW+r4/16jx3mdyJtZ9QKxCeMMw1+gIK+F775nxw== X-Received: by 2002:aca:3804:: with SMTP id f4mr14365915oia.144.1566254246924; Mon, 19 Aug 2019 15:37:26 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id 23sm1541199oiz.8.2019.08.19.15.37.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:37:26 -0700 (PDT) Subject: [PATCH v2 02/21] SUNRPC: Inline xdr_commit_encode From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:37:05 -0400 Message-ID: <156625420544.8161.7549333388864452214.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Micro-optimization: For xdr_commit_encode call sites in net/sunrpc/xdr.c, eliminate the extra calling sequence. On my client, this change saves about a microsecond for every 30 calls to xdr_reserve_space(). Signed-off-by: Chuck Lever --- net/sunrpc/xdr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 48c93b9e525e..7ba0ede6b417 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -560,7 +560,7 @@ EXPORT_SYMBOL_GPL(xdr_init_encode); * required at the end of encoding, or any other time when the xdr_buf * data might be read. */ -void xdr_commit_encode(struct xdr_stream *xdr) +inline void xdr_commit_encode(struct xdr_stream *xdr) { int shift = xdr->scratch.iov_len; void *page; From patchwork Mon Aug 19 22:37:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102203 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1694C13B1 for ; Mon, 19 Aug 2019 22:38:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DEC3122CEC for ; Mon, 19 Aug 2019 22:38:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BXXTLHch" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728494AbfHSWiP (ORCPT ); Mon, 19 Aug 2019 18:38:15 -0400 Received: from mail-oi1-f195.google.com ([209.85.167.195]:44482 "EHLO mail-oi1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728136AbfHSWiP (ORCPT ); Mon, 19 Aug 2019 18:38:15 -0400 Received: by mail-oi1-f195.google.com with SMTP id k22so2597539oiw.11; Mon, 19 Aug 2019 15:38:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=hgHtWU4sAHy10ecYDRHFxG9KkrfPY8CSPtGGStpLVYw=; b=BXXTLHchIy/c4RbPtg8BMoLS14JswsGA/ilPtziWZpFe3JSLy8ZoB6bWb3y2bOODIK yWaPLCc3d5bVJzIpyHeHtGX1Hc2w0RCIRAjno1VZE3neAGz52+SV1xDgKIbfHqdk8IIE ve2WWQ2LIkNa9ORlL31wGKa8uWHkqxcyeNzOVICi0we4FXFalsrNadwBcZV1udKpxizs wWSskVNZDg0FYYTNHP7Y0Adx6H9NS03nUw6aIL5nK8B0fVA5rjC75h2dTKxpxoMbhEBt CUomNyFvRID992pNwastj0p1N+lNv5emCmKMMzZ18KsNxDfWPfoop146aADmBjYTvjG3 IoCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=hgHtWU4sAHy10ecYDRHFxG9KkrfPY8CSPtGGStpLVYw=; b=M2mufifC29qRG7vpktP6Ulw+BrJ+cJ7AgiVvpcWX+51X8FmATdS7L7EHsTLQqtCyx/ MSP8VXUX3AqxoSe1u1rwaI5+pa6I9U2NbRFtB8PKdFC/eD7OIlBGmomtIX8JmjC5NnKI r6B6IhSURtSnJNSd+5b3N1iJpLsiBa2/noI59vYBx1UOxjFZoUuXZS/V2R0YHRS66gXR ktoR0WTa5+BtdWpao86t4o/iAeVOA4xsL03Xj82luQUTcA+n2pOE+pTGdujDroqy6+1E 6Or6zn3ctG9+2VNFukn6asW62Yhy/TlIDtMWkyBD9B3onkeV0+uK3yMqHmlX5yDJxTn+ tEAw== X-Gm-Message-State: APjAAAVg8LvNrur6vrzdyNYv4vB3PHxapTqgO2zp/eZk7ZhJQ8HMGCAI SlC0vUv5JSo+zC8Eg0NaoPjqh4c1 X-Google-Smtp-Source: APXvYqw9lGMuBEzap6KCcuRzBXEXOccZ/VkzBnE91dG029oqm89Wy9HthEEYQd1Za/WCKO0FR8KvxQ== X-Received: by 2002:aca:2104:: with SMTP id 4mr15217357oiz.12.1566254293472; Mon, 19 Aug 2019 15:38:13 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id b10sm5920111oti.61.2019.08.19.15.38.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:38:13 -0700 (PDT) Subject: [PATCH v2 03/21] xprtrdma: Refresh the documenting comment in frwr_ops.c From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:37:52 -0400 Message-ID: <156625425207.8161.7734951575643318770.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Things have changed since this comment was written. In particular, the reworking of connection closing, on-demand creation of MRs, and the removal of fr_state all mean that deferring MR recovery to frwr_map is no longer needed. The description is obsolete. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 66 +++++++++++----------------------------- 1 file changed, 18 insertions(+), 48 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 0b6dad7580a1..a30f2ae49578 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -7,67 +7,37 @@ /* Lightweight memory registration using Fast Registration Work * Requests (FRWR). * - * FRWR features ordered asynchronous registration and deregistration - * of arbitrarily sized memory regions. This is the fastest and safest + * FRWR features ordered asynchronous registration and invalidation + * of arbitrarily-sized memory regions. This is the fastest and safest * but most complex memory registration mode. */ /* Normal operation * - * A Memory Region is prepared for RDMA READ or WRITE using a FAST_REG + * A Memory Region is prepared for RDMA Read or Write using a FAST_REG * Work Request (frwr_map). When the RDMA operation is finished, this * Memory Region is invalidated using a LOCAL_INV Work Request - * (frwr_unmap_sync). + * (frwr_unmap_async and frwr_unmap_sync). * - * Typically these Work Requests are not signaled, and neither are RDMA - * SEND Work Requests (with the exception of signaling occasionally to - * prevent provider work queue overflows). This greatly reduces HCA + * Typically FAST_REG Work Requests are not signaled, and neither are + * RDMA Send Work Requests (with the exception of signaling occasionally + * to prevent provider work queue overflows). This greatly reduces HCA * interrupt workload. - * - * As an optimization, frwr_unmap marks MRs INVALID before the - * LOCAL_INV WR is posted. If posting succeeds, the MR is placed on - * rb_mrs immediately so that no work (like managing a linked list - * under a spinlock) is needed in the completion upcall. - * - * But this means that frwr_map() can occasionally encounter an MR - * that is INVALID but the LOCAL_INV WR has not completed. Work Queue - * ordering prevents a subsequent FAST_REG WR from executing against - * that MR while it is still being invalidated. */ /* Transport recovery * - * ->op_map and the transport connect worker cannot run at the same - * time, but ->op_unmap can fire while the transport connect worker - * is running. Thus MR recovery is handled in ->op_map, to guarantee - * that recovered MRs are owned by a sending RPC, and not one where - * ->op_unmap could fire at the same time transport reconnect is - * being done. - * - * When the underlying transport disconnects, MRs are left in one of - * four states: - * - * INVALID: The MR was not in use before the QP entered ERROR state. - * - * VALID: The MR was registered before the QP entered ERROR state. - * - * FLUSHED_FR: The MR was being registered when the QP entered ERROR - * state, and the pending WR was flushed. - * - * FLUSHED_LI: The MR was being invalidated when the QP entered ERROR - * state, and the pending WR was flushed. - * - * When frwr_map encounters FLUSHED and VALID MRs, they are recovered - * with ib_dereg_mr and then are re-initialized. Because MR recovery - * allocates fresh resources, it is deferred to a workqueue, and the - * recovered MRs are placed back on the rb_mrs list when recovery is - * complete. frwr_map allocates another MR for the current RPC while - * the broken MR is reset. - * - * To ensure that frwr_map doesn't encounter an MR that is marked - * INVALID but that is about to be flushed due to a previous transport - * disconnect, the transport connect worker attempts to drain all - * pending send queue WRs before the transport is reconnected. + * frwr_map and frwr_unmap_* cannot run at the same time the transport + * connect worker is running. The connect worker holds the transport + * send lock, just as ->send_request does. This prevents frwr_map and + * the connect worker from running concurrently. When a connection is + * closed, the Receive completion queue is drained before the allowing + * the connect worker to get control. This prevents frwr_unmap and the + * connect worker from running concurrently. + * + * When the underlying transport disconnects, MRs that are in flight + * are flushed and are likely unusable. Thus all flushed MRs are + * destroyed. New MRs are created on demand. */ #include From patchwork Mon Aug 19 22:38:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102207 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B69191399 for ; Mon, 19 Aug 2019 22:39:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 943F222CEC for ; Mon, 19 Aug 2019 22:39:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="t2oTA9jY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728508AbfHSWjB (ORCPT ); Mon, 19 Aug 2019 18:39:01 -0400 Received: from mail-oi1-f196.google.com ([209.85.167.196]:35689 "EHLO mail-oi1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728136AbfHSWjA (ORCPT ); Mon, 19 Aug 2019 18:39:00 -0400 Received: by mail-oi1-f196.google.com with SMTP id a127so2602519oii.2; Mon, 19 Aug 2019 15:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=9ArGAmPXjVE1LatQBgsFDGnSvkfYJFMMdFUO/FkUeIo=; b=t2oTA9jYCBDGjR3YUZqtFbmXXj9r0UGs4aEn6GneWcJRlDBwm0luLwnFgF/BRe/rFp AVUpsDMQ8rS20r1ShuHHBgmuCUf9+d9QtGqB3HSq3UZCzdX8y+fLRRsuiUcoRgDqnBmd PU9Rp2GacIquWCqZA+cgtdUAMDb9omucAxyp412mcpO5lk2AL7PE7vtG8zqOSkbnhY16 fs7J63V2elaEL2RF1xxVk1BTXgPrFy+5TsrCcZ3d3kAGhRTrkCjjEkgnCs3/NsOdVGBp nj9RCfn4CAHlsq0rBBNXS/05sImjKrslsGRhyWY/TIc1mNt0zigRCSc1VTWALNou5YC9 vrLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=9ArGAmPXjVE1LatQBgsFDGnSvkfYJFMMdFUO/FkUeIo=; b=IFnQprt1QJMa3NmnNIyTDBfBxx/yf11tKNa/OvRDRY5NZuNFJfIPniNBBiYJ4oUvoG Ut85B+ZWs3j9nb3XJxC+w7pX1t5R7D6dz4QGyyDYvygdcc8uT1gOJCpbVTa81ZVM6hAE 4+ZlwyWecMmZEAeCqs+ibxknjxtAsrOfcu2c+SfcSZi2hh4r7OWZTVpwc8yra61Yn8za ya4xeXJ+Yavc9Bmy/Nk9FQchtafFaorIR/4uiuKmOvf54dijovs2C3IWcoEXrToXU9gg b2+bX6tink5oGadxmzdcilikIyj7xzh4yMhWV5YmO3Sos0GjIINjwPfDXOUDMGXNjMOs Ffeg== X-Gm-Message-State: APjAAAWH0YScYizPqGAx+LZIJDdxh9G9zeYCIgVzDNINw0V18xH7iFVP A5NixjpTtldjzI8N0S6rUZwkdrrS X-Google-Smtp-Source: APXvYqxvlXrxOglFfGM47hXsqhu1EIqUHQ+aOqjgLWlyvGAGupKCjboRCJU9JZxwbt7h78q8aftd/w== X-Received: by 2002:aca:50cb:: with SMTP id e194mr7646726oib.48.1566254339926; Mon, 19 Aug 2019 15:38:59 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id s22sm3344653oij.37.2019.08.19.15.38.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:38:59 -0700 (PDT) Subject: [PATCH v2 04/21] xprtrdma: Update obsolete comment From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:38:38 -0400 Message-ID: <156625429858.8161.12698423687311798381.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Comment was made obsolete by commit 8cec3dba76a4 ("xprtrdma: rpcrdma_regbuf alignment"). Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/xprt_rdma.h | 3 --- 1 file changed, 3 deletions(-) diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 92ce09fcea74..3b2f2041e889 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -117,9 +117,6 @@ struct rpcrdma_ep { #endif /* Registered buffer -- registered kmalloc'd memory for RDMA SEND/RECV - * - * The below structure appears at the front of a large region of kmalloc'd - * memory, which always starts on a good alignment boundary. */ struct rpcrdma_regbuf { From patchwork Mon Aug 19 22:39:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102211 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6441913B1 for ; Mon, 19 Aug 2019 22:39:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 41F5222CEC for ; Mon, 19 Aug 2019 22:39:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XI455noK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728469AbfHSWjr (ORCPT ); Mon, 19 Aug 2019 18:39:47 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:34968 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728136AbfHSWjr (ORCPT ); Mon, 19 Aug 2019 18:39:47 -0400 Received: by mail-ot1-f67.google.com with SMTP id g17so3236793otl.2; Mon, 19 Aug 2019 15:39:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=bTFse4JKhV2BT/xERQHbbHpDOri/QkVxMmcYVlM2Efk=; b=XI455noKSKIgemwRKoRycfSkEoM4J7nMXYJ9p5+5yJpAwcX3SKbJ24ExvCwhqVUfAO Jy9cYq8XiRoo8p18MgtoB4zbupl9ZYoUVn+EHLLLOO6fsB/p47ZlYdN2HL8YrKzPNWR0 VzNJIdcC2i+ZhRMmrINjLOFPFaa/pdoBAv9PqRTVWENQUFWXEmslO5JACuo7laJMAmcm NqqKRc2wTDlAqg8YqqBJlo3BAnDDBU4vJxJ14UFTNDKewVrZijFfPB3gsKtR8kxwa0Bn qknly/ku7n3EkbTF6uaYbNqgF/hKA5XK8C5gT4R98goYQTNH5MkHobJQHHlxoshp5Bi/ vUYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=bTFse4JKhV2BT/xERQHbbHpDOri/QkVxMmcYVlM2Efk=; b=FTHV659YciLfftPY7Ms7y1FsgkiJtykiyPT1gPwCVlVPkQzZSmziZOETwAI+7gm7s7 5b12h6JQGTjb9allof67/UJNmawJ1mE9DrxH3c+eH7LOk/V2JZJs6krBNTddkUJOr7UL O9evrdOW+iMdG6BO6IN3oY9NDXXQ9A74nFhkct3mrZuaQfKN8XCXz/4W44bsOBs+HLjM rmjGkt612ekmXTUUrNMl58KsRqfF0/FVlS0sy8DIrwEyGz4hdE+wGk6a7dRSHqhovxrA LNmZG3QpybcAXj5frg6mowLMQCkqhN1tMqTvV7aD5idViJbDkY2+elM9/wdI4kAwtTD6 6eFg== X-Gm-Message-State: APjAAAWIPwibiim89BfbVrPfh+hdPK4K+/1M6H2uIUy7NeelHSlZMQ5l hiOTpSP3iT9Eah6OyMYY9p+GqWON X-Google-Smtp-Source: APXvYqzQ/FQcsGkbOOOKmkRElcP6WzfbaTVWChLnmzP16JdCSJ9A15K4e4SXLAP7UuVk+RuQdCPiUA== X-Received: by 2002:a9d:6854:: with SMTP id c20mr14348094oto.120.1566254386384; Mon, 19 Aug 2019 15:39:46 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id w139sm5094055oiw.0.2019.08.19.15.39.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:39:46 -0700 (PDT) Subject: [PATCH v2 05/21] xprtrdma: Fix calculation of ri_max_segs again From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:39:25 -0400 Message-ID: <156625434505.8161.16198361086528272640.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Commit 302d3deb206 ("xprtrdma: Prevent inline overflow") added this calculation back in 2016, but got it wrong. I tested only the lower bound, which is why there is a max_t there. The upper bound should be rounded up too. Now, when using DIV_ROUND_UP, that takes care of the lower bound as well. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index a30f2ae49578..3a10bfff2125 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -260,8 +260,8 @@ int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep) ep->rep_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS; ep->rep_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */ - ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / - ia->ri_max_frwr_depth); + ia->ri_max_segs = + DIV_ROUND_UP(RPCRDMA_MAX_DATA_SEGS, ia->ri_max_frwr_depth); /* Reply chunks require segments for head and tail buffers */ ia->ri_max_segs += 2; if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS) From patchwork Mon Aug 19 22:40:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102215 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 905521399 for ; Mon, 19 Aug 2019 22:40:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6FAF22070B for ; Mon, 19 Aug 2019 22:40:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FtJTBZZm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728526AbfHSWkf (ORCPT ); Mon, 19 Aug 2019 18:40:35 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:35842 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728494AbfHSWke (ORCPT ); Mon, 19 Aug 2019 18:40:34 -0400 Received: by mail-ot1-f67.google.com with SMTP id k18so3234770otr.3; Mon, 19 Aug 2019 15:40:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=9dF8UwmpXvMDAG/BLxG85Y8rvbTT5sYtQ6wFZaELboE=; b=FtJTBZZmI8Jq+d+M/q60vhpV+UBN0W3eoyqsx6k1KIKJp2pTst+TNNdvZaqAWfYvkU TyEooNK+A0+I0tI3UmqdQJaecISs7aFscZR3OmzTYwIeCgzTA8PzZ67aSjNndGJRdnZk rd4cp6ZX9yCJPl/rqElQEnpJO8pgpWlj6yAkFbRx/KsSolGLAuY+OgeaoPTDQ37b0E5J CPRaUAIxoJuL+o1rdaPxXPAaLDRpJnu8OIepgS58pmpTEj3VOPSc5f9V54xVO1uKwaJD jFRTFcnVdlbaBjyJ/Ixn0AynVIcJ4AJMFY6xnL8esZI4z3qJSAAGMT5/52TpnYGQ6yZB rvVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=9dF8UwmpXvMDAG/BLxG85Y8rvbTT5sYtQ6wFZaELboE=; b=JatNmefujEwtR1IIWwL2iZ77TrVln0nNqANXuEBsd54H+HwE222FHnwy7WlPPY3NUq 1wNERpLc+EB9T2IjYgGys+UxE4bEh9wCnAo1G1CuToOkp4NaYLmCSF7B92nzZ0B+6mMV GltP0sQu/YlQ0pXXhZ1jCHNAl1uvCmD6vcRWN9BIxvMT/5ddLswQcv8G3kqx9xoSoYUW KM6sLEQuQMPDrYGtxHUSWM9sHRCFtnYlxeZ1225i3cR+Xdn6Gz26SuF60X1X7IQEox4Z 08VNL7sNC1hHQIjqmufVmqfztKs3uUl9Sklrg3geyePGD/pOjwkYC5+cPz8u7eV8Auiz YevA== X-Gm-Message-State: APjAAAUOhCSJYTx8xzrJixe9LtcuOCwmOaNbf5uvIItxrPevulrvswVO wD1qcXA4lcj459ukIfmDAZf024iu X-Google-Smtp-Source: APXvYqxJJNNIWge7HF8KHU92k4cinZrR/JqC5xDP8eyZs+Njo1QCxj69YMLRO9wnas2rlgOMsDBgdQ== X-Received: by 2002:a9d:518e:: with SMTP id y14mr13182679otg.259.1566254433050; Mon, 19 Aug 2019 15:40:33 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id t18sm5839587otk.73.2019.08.19.15.40.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:40:32 -0700 (PDT) Subject: [PATCH v2 06/21] xprtrdma: Boost maximum transport header size From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:40:11 -0400 Message-ID: <156625439150.8161.9923129489297297655.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Although I haven't seen any performance results that justify it, I've received several complaints that NFS/RDMA no longer supports a maximum rsize and wsize of 1MB. These days it is somewhat smaller. To simplify the logic that determines whether a chunk list is necessary, the implementation uses a fixed maximum size of the transport header. Currently that maximum size is 256 bytes, one quarter of the default inline threshold size for RPC/RDMA v1. Since commit a78868497c2e ("xprtrdma: Reduce max_frwr_depth"), the size of chunks is also smaller to take advantage of inline page lists in device internal MR data structures. The combination of these two design choices has reduced the maximum NFS rsize and wsize that can be used for most RNIC/HCAs. Increasing the maximum transport header size and the maximum number of RDMA segments it can contain increases the negotiated maximum rsize/wsize on common RNIC/HCAs. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 9 ++++++++- net/sunrpc/xprtrdma/xprt_rdma.h | 23 ++++++++++------------- 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 805b1f35e1ca..e639ea0faf19 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include @@ -1000,12 +1001,18 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, struct rpcrdma_buffer *buffer = &r_xprt->rx_buf; struct rpcrdma_regbuf *rb; struct rpcrdma_req *req; + size_t maxhdrsize; req = kzalloc(sizeof(*req), flags); if (req == NULL) goto out1; - rb = rpcrdma_regbuf_alloc(RPCRDMA_HDRBUF_SIZE, DMA_TO_DEVICE, flags); + /* Compute maximum header buffer size in bytes */ + maxhdrsize = rpcrdma_fixed_maxsz + 3 + + r_xprt->rx_ia.ri_max_segs * rpcrdma_readchunk_maxsz; + maxhdrsize *= sizeof(__be32); + rb = rpcrdma_regbuf_alloc(__roundup_pow_of_two(maxhdrsize), + DMA_TO_DEVICE, flags); if (!rb) goto out2; req->rl_rdmabuf = rb; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 3b2f2041e889..eaf6b907a76e 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -155,25 +155,22 @@ static inline void *rdmab_data(const struct rpcrdma_regbuf *rb) /* To ensure a transport can always make forward progress, * the number of RDMA segments allowed in header chunk lists - * is capped at 8. This prevents less-capable devices and - * memory registrations from overrunning the Send buffer - * while building chunk lists. + * is capped at 16. This prevents less-capable devices from + * overrunning the Send buffer while building chunk lists. * * Elements of the Read list take up more room than the - * Write list or Reply chunk. 8 read segments means the Read - * list (or Write list or Reply chunk) cannot consume more - * than + * Write list or Reply chunk. 16 read segments means the + * chunk lists cannot consume more than * - * ((8 + 2) * read segment size) + 1 XDR words, or 244 bytes. + * ((16 + 2) * read segment size) + 1 XDR words, * - * And the fixed part of the header is another 24 bytes. - * - * The smallest inline threshold is 1024 bytes, ensuring that - * at least 750 bytes are available for RPC messages. + * or about 400 bytes. The fixed part of the header is + * another 24 bytes. Thus when the inline threshold is + * 1024 bytes, at least 600 bytes are available for RPC + * message bodies. */ enum { - RPCRDMA_MAX_HDR_SEGS = 8, - RPCRDMA_HDRBUF_SIZE = 256, + RPCRDMA_MAX_HDR_SEGS = 16, }; /* From patchwork Mon Aug 19 22:40:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102219 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3ACAF1399 for ; Mon, 19 Aug 2019 22:41:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1A8CF22CEC for ; Mon, 19 Aug 2019 22:41:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VuIGbQ0h" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728538AbfHSWlU (ORCPT ); Mon, 19 Aug 2019 18:41:20 -0400 Received: from mail-oi1-f195.google.com ([209.85.167.195]:44668 "EHLO mail-oi1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728402AbfHSWlU (ORCPT ); Mon, 19 Aug 2019 18:41:20 -0400 Received: by mail-oi1-f195.google.com with SMTP id k22so2602463oiw.11; Mon, 19 Aug 2019 15:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=XwonKW5E3QU6ws/Gywd0FV6ntfQVppu+1SLDZCu7wNk=; b=VuIGbQ0hRsOODEPtA3aFhe+GcTKcRyJsXycC9v3WDyf9JKrqYeneZ04DPqthoNjOnn r/xA1XbPLidUfTg2TM9Iw6HkEcLsKCRjd2f+lv2iyJ29ewMHLGJ1Aash58fKM3eY3ScZ r2LLxLMk2twedq5ctJsRnkDNsr1TmWhvrUZosecvmguKNZb9ObHArE/cQAycmfVvj1dp uf6oThfM0CxuQEFNjSeUZ1H5B/nfiX+XvoMnIIOnID8Y43zEk3V6BlmHD4xUz+PlpXV+ tY+trSZCYD10YupqRhyxx+tjkO5HfIATNW8jpC2ZUTCXoBxqo2Hkh8Otv/WiWwEhFXI3 EY/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=XwonKW5E3QU6ws/Gywd0FV6ntfQVppu+1SLDZCu7wNk=; b=ZSZQ4eNZCRJDbY+OLKSV7pV+ci5OR11xZi0jR1UwzKpomt5KhxHLLnpZ4P5bVH/FGK TKFSaRbCiqDhvjdlwccMkJHSzaSNJydMXeeJo+7+G5UBYn1TZiUXCoAeBry9gF3I1TFF FCdm5eksWrZhoSlw12GTNZPDE1KbDRt04Udnv6dGmDoGbbVv2ul4B8qblroaL3TfTekD pdzHS7JAohZgdLTBWoipsKYW0iS2WfBBoB4cO6XYDdtyFpuIhqltMPk+QShdkX+mpwsQ s6utPMbkx57krgzl127BREZqhnMSwch9bT7bu3EG1reLRXERzFAxl14OLq37qeeEdTVj fqyw== X-Gm-Message-State: APjAAAUh6qeI8SXqdFzPqp52TBunDmQyMori0LTqHg76MqGbmR4DNzzx YDxH41LGwZG25AvfB+2PZxcrl4TR X-Google-Smtp-Source: APXvYqzCFUSW6iqGsIYX+XwSmETDf9wMbH1eND1jHODm5YypBG49i13QLCbEdYIiwFeSLDawwQd5/w== X-Received: by 2002:a54:4f09:: with SMTP id e9mr15430015oiy.89.1566254479449; Mon, 19 Aug 2019 15:41:19 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id q85sm4924518oic.52.2019.08.19.15.41.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:41:19 -0700 (PDT) Subject: [PATCH v2 07/21] xprtrdma: Boost client's max slot table size to match Linux server From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:40:58 -0400 Message-ID: <156625443812.8161.1304836419453818534.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org I've heard rumors of an NFS/RDMA server implementation that has a default credit limit of 1024. The client's default setting remains at 128. Signed-off-by: Chuck Lever --- include/linux/sunrpc/xprtrdma.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/sunrpc/xprtrdma.h b/include/linux/sunrpc/xprtrdma.h index 86fc38ff0355..16c239e0d6dd 100644 --- a/include/linux/sunrpc/xprtrdma.h +++ b/include/linux/sunrpc/xprtrdma.h @@ -49,9 +49,9 @@ * fully-chunked NFS message (read chunks are the largest). Note only * a single chunk type per message is supported currently. */ -#define RPCRDMA_MIN_SLOT_TABLE (2U) +#define RPCRDMA_MIN_SLOT_TABLE (4U) #define RPCRDMA_DEF_SLOT_TABLE (128U) -#define RPCRDMA_MAX_SLOT_TABLE (256U) +#define RPCRDMA_MAX_SLOT_TABLE (16384U) #define RPCRDMA_MIN_INLINE (1024) /* min inline thresh */ #define RPCRDMA_DEF_INLINE (4096) /* default inline thresh */ From patchwork Mon Aug 19 22:41:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102229 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED40F13B1 for ; Mon, 19 Aug 2019 22:42:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CBF252070B for ; Mon, 19 Aug 2019 22:42:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="F9MR4XMj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728710AbfHSWmH (ORCPT ); Mon, 19 Aug 2019 18:42:07 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:39654 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728700AbfHSWmH (ORCPT ); Mon, 19 Aug 2019 18:42:07 -0400 Received: by mail-ot1-f67.google.com with SMTP id b1so3232875otp.6; Mon, 19 Aug 2019 15:42:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=/m0hsTGit+/kYdO/kQGAejffnizn6DrsEepbXnB8zzE=; b=F9MR4XMj87DV3uiuvwdY9mykbjbw9jpvwGwNBvsmcqisp+c9V6JpNqwEPv1JOvgQ24 RSia5daJQl7Ch+vKH27g/NowKxwaPixP+k3290igh4QAm2SQRDHQTgvPEEJ97/7GpPiT +f/0WtPX+X7mb2idUxP+HDetZflU8iitWDhCrZYNzjHOyHipj+OtmEqYIxzTwNqih5Lw DgSU+VP5bI8KKoklgdyTZLCNqKn7XI+CFSJHxJt/0+eLupNHqUwOq+VlNKVkjgEyyZO7 K7L58E7gkTt4Cdt+UEg9xzM9Z4j2BMOmZ1eKIAh/3vBSeO8yhF5vXrSLMKWRiaAA9s7A 1e/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=/m0hsTGit+/kYdO/kQGAejffnizn6DrsEepbXnB8zzE=; b=g/zVMUU2rWYpUjRfufV0Cb5hIgH3p+cuv3yoylYVSqry9ryjKblhFyY1ufJzaDkfKs KgDDZNNzeujvhCTV3lOfOhHzXrNmGb59DwLwrmUue5qlmi7btzQ4rSsbjqwH9Bp3nqPH XABL6uFEGnJW3IKQYmwAJjpd8wkDKZyXCvCmGoPYt6u3b9SQpg9U7k9D/eYp0bAs+Ds5 Cusgl8UVEuzCcO/dEFasFN5weNWdzWfnPOr7vIwkInlPLFQyW/3Ju84sev9We+U80fWA cPTxr62ZiMIO85dCACzcrGrQXjNNv+JHo4dPUj713kpM33sye+h2tnS9ELIfWwyACy2u Hb+w== X-Gm-Message-State: APjAAAVH/wrC3WFvnqCXy3KSFbxF77Wr/CuHfx5OCO29MCMXN7wPg1BR sm2ZIFBlrXFqnZYkaxqVzVYrWaOu X-Google-Smtp-Source: APXvYqzd7sS9BJpdXNmrPV0z5cdCxCXX1U01H4Z39wgjNaI45FJu35oy67+UKjRM3f2gSCH5f8Cvjg== X-Received: by 2002:a05:6830:2055:: with SMTP id f21mr19256117otp.53.1566254526015; Mon, 19 Aug 2019 15:42:06 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id p2sm5617047otl.59.2019.08.19.15.42.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:42:05 -0700 (PDT) Subject: [PATCH v2 08/21] xprtrdma: Rename CQE field in Receive trace points From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:41:44 -0400 Message-ID: <156625448455.8161.8551241541431387600.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Make the field name the same for all trace points that handle pointers to struct rpcrdma_rep. That makes it easy to grep for matching rep points in trace output. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 21 +++++++++++---------- net/sunrpc/xprtrdma/verbs.c | 2 +- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index f6a4eaa85a3e..6e6055eb67e7 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -623,21 +623,21 @@ TRACE_EVENT(xprtrdma_post_send, TRACE_EVENT(xprtrdma_post_recv, TP_PROTO( - const struct ib_cqe *cqe + const struct rpcrdma_rep *rep ), - TP_ARGS(cqe), + TP_ARGS(rep), TP_STRUCT__entry( - __field(const void *, cqe) + __field(const void *, rep) ), TP_fast_assign( - __entry->cqe = cqe; + __entry->rep = rep; ), - TP_printk("cqe=%p", - __entry->cqe + TP_printk("rep=%p", + __entry->rep ) ); @@ -715,14 +715,15 @@ TRACE_EVENT(xprtrdma_wc_receive, TP_ARGS(wc), TP_STRUCT__entry( - __field(const void *, cqe) + __field(const void *, rep) __field(u32, byte_len) __field(unsigned int, status) __field(u32, vendor_err) ), TP_fast_assign( - __entry->cqe = wc->wr_cqe; + __entry->rep = container_of(wc->wr_cqe, struct rpcrdma_rep, + rr_cqe); __entry->status = wc->status; if (wc->status) { __entry->byte_len = 0; @@ -733,8 +734,8 @@ TRACE_EVENT(xprtrdma_wc_receive, } ), - TP_printk("cqe=%p %u bytes: %s (%u/0x%x)", - __entry->cqe, __entry->byte_len, + TP_printk("rep=%p %u bytes: %s (%u/0x%x)", + __entry->rep, __entry->byte_len, rdma_show_wc_status(__entry->status), __entry->status, __entry->vendor_err ) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index e639ea0faf19..3c275a7a4e4c 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1531,7 +1531,7 @@ rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) if (!rpcrdma_regbuf_dma_map(r_xprt, rep->rr_rdmabuf)) goto release_wrs; - trace_xprtrdma_post_recv(rep->rr_recv_wr.wr_cqe); + trace_xprtrdma_post_recv(rep); ++count; } From patchwork Mon Aug 19 22:42:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102233 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74AE713B1 for ; Mon, 19 Aug 2019 22:42:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 53A4A2070B for ; Mon, 19 Aug 2019 22:42:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="T7SF6NRw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728438AbfHSWmx (ORCPT ); Mon, 19 Aug 2019 18:42:53 -0400 Received: from mail-oi1-f194.google.com ([209.85.167.194]:41900 "EHLO mail-oi1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWmx (ORCPT ); Mon, 19 Aug 2019 18:42:53 -0400 Received: by mail-oi1-f194.google.com with SMTP id g7so2617046oia.8; Mon, 19 Aug 2019 15:42:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=HcRL7mEklA22xsf0ewjatYoR4UwmJXPgoFIVsj/Roas=; b=T7SF6NRw+HCY2YXOMlfH9CNtDwywRKUlYjoddyzJLrd5vGbPJxGchgFML1tccXtiAl ZyjpkFQLWlQx26d8wc/SXfrISRoiCQCPsyMpjXkdsbsFwfvhtrrBRxNTrrUb87J9RRh8 /lV5XJJPygcQl591CvDCx4jWQ35yRYoA5oNM5qOw174EiJNon8JPi0X0wWpuO7qizIky qRhJZdPqPDlsNjTQYvM7Ov2PnHWRRU2HHSpMyFKWCOizjzI9KSBDrZA7uXpzI/7L9LtP Scq4NEoKsh+HCyP7crJdE4Up8h98M7b6xj39JCSzVi2VrdK72DXiuXviRFvVIbbc0oLl FHNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=HcRL7mEklA22xsf0ewjatYoR4UwmJXPgoFIVsj/Roas=; b=X+RPh52jNtqqDKUwBDIYvNQHU7munKvASnBovTCkVYx+a3Ge2EQR/GdAID2UefDCss 6wdVN69Pa/6IiFMgPpWkZ1Jq/HIEKPFBHdURHD/j2mXuHnxEa3PcllUrM27xZM4O4uJn 39Dpv9E9V3JEFgAKTrZHFa/xd7mcJPVyrXSAVmRwxrQvX89RpRr5/NWOqI9ouvCzKfE2 Et+wjvz6gUgU8PlftrxdH0Ps2yuQcX7WJL0c9//j6p18VP/7ANYKvG1JYiEWS0df1aZz pJlN/stMsvBOtGpw8jBDFM6YZCV6zqceCH3mXwiu/nEDtQVlBQyYhRp/KWMWr01720QY 1YBg== X-Gm-Message-State: APjAAAV3cj6/KUax8hc9krCJ4zKapx4FYc3K3okkekslpbTloESoZl0A UFmfqLOBi6Hn0NsjXQ5afO7Jt4U4 X-Google-Smtp-Source: APXvYqzHg+PjTHw9Hkwnki40OCPdCv0h7uimWK/ZbaUgJ0xaHM+kGi9SmW/izVce51ikTZtJFZDHwg== X-Received: by 2002:aca:3c1:: with SMTP id 184mr15305455oid.113.1566254572472; Mon, 19 Aug 2019 15:42:52 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id w5sm4442399oic.36.2019.08.19.15.42.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:42:52 -0700 (PDT) Subject: [PATCH v2 09/21] xprtrdma: Rename rpcrdma_buffer::rb_all From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:42:31 -0400 Message-ID: <156625453108.8161.5618620590594132378.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up: There are other "all" list heads. For code clarity distinguish this one as for use only for MRs by renaming it. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 26 ++++++++------------------ net/sunrpc/xprtrdma/xprt_rdma.h | 2 +- 2 files changed, 9 insertions(+), 19 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3c275a7a4e4c..e004873cc4f0 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -944,8 +944,6 @@ rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ia *ia = &r_xprt->rx_ia; unsigned int count; - LIST_HEAD(free); - LIST_HEAD(all); for (count = 0; count < ia->ri_max_segs; count++) { struct rpcrdma_mr *mr; @@ -963,15 +961,13 @@ rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) mr->mr_xprt = r_xprt; - list_add(&mr->mr_list, &free); - list_add(&mr->mr_all, &all); + spin_lock(&buf->rb_mrlock); + list_add(&mr->mr_list, &buf->rb_mrs); + list_add(&mr->mr_all, &buf->rb_all_mrs); + spin_unlock(&buf->rb_mrlock); } - spin_lock(&buf->rb_mrlock); - list_splice(&free, &buf->rb_mrs); - list_splice(&all, &buf->rb_all); r_xprt->rx_stats.mrs_allocated += count; - spin_unlock(&buf->rb_mrlock); trace_xprtrdma_createmrs(r_xprt, count); } @@ -1089,7 +1085,7 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) spin_lock_init(&buf->rb_mrlock); spin_lock_init(&buf->rb_lock); INIT_LIST_HEAD(&buf->rb_mrs); - INIT_LIST_HEAD(&buf->rb_all); + INIT_LIST_HEAD(&buf->rb_all_mrs); INIT_DELAYED_WORK(&buf->rb_refresh_worker, rpcrdma_mr_refresh_worker); @@ -1156,24 +1152,18 @@ rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) count = 0; spin_lock(&buf->rb_mrlock); - while (!list_empty(&buf->rb_all)) { - mr = list_entry(buf->rb_all.next, struct rpcrdma_mr, mr_all); + while ((mr = list_first_entry_or_null(&buf->rb_all_mrs, + struct rpcrdma_mr, + mr_all)) != NULL) { list_del(&mr->mr_all); - spin_unlock(&buf->rb_mrlock); - /* Ensure MW is not on any rl_registered list */ - if (!list_empty(&mr->mr_list)) - list_del(&mr->mr_list); - frwr_release_mr(mr); count++; spin_lock(&buf->rb_mrlock); } spin_unlock(&buf->rb_mrlock); r_xprt->rx_stats.mrs_allocated = 0; - - dprintk("RPC: %s: released %u MRs\n", __func__, count); } /** diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index eaf6b907a76e..5aaa53b8ae12 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -360,7 +360,7 @@ rpcrdma_mr_pop(struct list_head *list) struct rpcrdma_buffer { spinlock_t rb_mrlock; /* protect rb_mrs list */ struct list_head rb_mrs; - struct list_head rb_all; + struct list_head rb_all_mrs; unsigned long rb_sc_head; unsigned long rb_sc_tail; From patchwork Mon Aug 19 22:43:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102237 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B0E41399 for ; Mon, 19 Aug 2019 22:43:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DA5DF2070B for ; Mon, 19 Aug 2019 22:43:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QSsPnzfj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728402AbfHSWnk (ORCPT ); Mon, 19 Aug 2019 18:43:40 -0400 Received: from mail-ot1-f68.google.com ([209.85.210.68]:39776 "EHLO mail-ot1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWnk (ORCPT ); Mon, 19 Aug 2019 18:43:40 -0400 Received: by mail-ot1-f68.google.com with SMTP id b1so3235894otp.6; Mon, 19 Aug 2019 15:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=s9bIFjIETVLbW91cl2wEIB0Q9eUpvnPCnK8hNB6ULRg=; b=QSsPnzfjHMjaka59dEObAA8y70A1WlqtroaOEVml3m47y8HQzivyqIDmW7DpxkFtR2 KNmLW+xjiXLxz3TKEIOXrxD+pPDVMMhfC3pOhLkU3+X2Ts6dmxAIziiWVEBobvSS/HiP lvqnXl2VFi0moj9ijGft57j9q1MbcJSjMUcGkhmQVnRkxfEC8HvM9CDfvDw5FqW5JqTc ygnlijfa9We2z28K20tL63jXawLrvMH951T26wOn+TiDgAM9FkE7SEo3GqLp/IipgBgP PXpNqy3INFvWvDn73lY9C7dn4PDDeyL8dhKI21fvlA22uAogA5h+C7u1QgHfUWxozaSu eFlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=s9bIFjIETVLbW91cl2wEIB0Q9eUpvnPCnK8hNB6ULRg=; b=fQnPgHYMFSo1ppF6xG4QOQ7lcJxpDsaFHvmWYjjGFgmNbI7yYrlJ24ZVsfhnn/QOAa Qps/E45DnQNRS3Iwl6WN/eBTOiPGdiRDmMhscdrEdpJcONAEDnIe75428OZxOgDxIoXb uQUQ2Ehaayvloe7S3EWChQy4QRHkDMYUqoskZDZs95guR2N37/Vg+zyaPC1sJ2gQwsBg FX1sp83V31KIe+ZE7k5tEmGbO0adHkE107zq53Z2u3oE1Xjr+uFDy+VwQdEpef3hQU4U YptnAsA39/o4hf/gri2Xw9S/cZVfX031jg93GkfrMEBO7Tp4Yq1MABkXasLxIMPvKTtv DriQ== X-Gm-Message-State: APjAAAUA1LjU/XRRbyOHqsu4LS6ujd9BSnjmvZ07wjqwLhf9L3A6A0hn E61BPCwbOK3l5aiWzWC/omownZV0 X-Google-Smtp-Source: APXvYqxvKvTjJI57D4KMVzEgC6wGevPMdgTgFF67L0OG/s1Rw0CXuqR/SkzfhCk5iQKpKmqXabjKDQ== X-Received: by 2002:a05:6830:22e3:: with SMTP id t3mr2134022otc.347.1566254619040; Mon, 19 Aug 2019 15:43:39 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id 11sm6097897otc.45.2019.08.19.15.43.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:43:38 -0700 (PDT) Subject: [PATCH v2 10/21] xprtrdma: Toggle XPRT_CONGESTED in xprtrdma's slot methods From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:43:17 -0400 Message-ID: <156625457758.8161.9892376897799599275.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Commit 48be539dd44a ("xprtrdma: Introduce ->alloc_slot call-out for xprtrdma") added a separate alloc_slot and free_slot to the RPC/RDMA transport. Later, commit 75891f502f5f ("SUNRPC: Support for congestion control when queuing is enabled") modified the generic alloc/free_slot methods, but neglected the methods in xprtrdma. Found via code review. Fixes: 75891f502f5f ("SUNRPC: Support for congestion control ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/transport.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 2ec349ed4770..f4763e8a6761 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -571,6 +571,7 @@ xprt_rdma_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task) return; out_sleep: + set_bit(XPRT_CONGESTED, &xprt->state); rpc_sleep_on(&xprt->backlog, task, NULL); task->tk_status = -EAGAIN; } @@ -589,7 +590,8 @@ xprt_rdma_free_slot(struct rpc_xprt *xprt, struct rpc_rqst *rqst) memset(rqst, 0, sizeof(*rqst)); rpcrdma_buffer_put(&r_xprt->rx_buf, rpcr_to_rdmar(rqst)); - rpc_wake_up_next(&xprt->backlog); + if (unlikely(!rpc_wake_up_next(&xprt->backlog))) + clear_bit(XPRT_CONGESTED, &xprt->state); } static bool rpcrdma_check_regbuf(struct rpcrdma_xprt *r_xprt, From patchwork Mon Aug 19 22:44:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102241 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74E741399 for ; Mon, 19 Aug 2019 22:44:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 521EC2070B for ; Mon, 19 Aug 2019 22:44:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tTrQ8Bbl" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728539AbfHSWo0 (ORCPT ); Mon, 19 Aug 2019 18:44:26 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:35325 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWo0 (ORCPT ); Mon, 19 Aug 2019 18:44:26 -0400 Received: by mail-ot1-f65.google.com with SMTP id g17so3245992otl.2; Mon, 19 Aug 2019 15:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=BEP0xwlQ4b6InfkIJQy/Guf2YYQMYLXVg18uYu4Cs1c=; b=tTrQ8Bble+y/ksy75w2DzKGbEMGSRZGoRxTBqPWYQ6BPqUt5VWOCPxLCZjMu1LylPX FEI3P/oiBgfohSjGb4e/+uFEgJB5iK5ujAB4lDXk2IOvPafcGDfoYRbLQ9+Py4pFBZMa 8pP+MoBEaFi0eG1wTkST6AyRE0dGFbY1wqMasrazn9Z6DPEQpJ8xQwchoww5VBsG8tKu 5JsoLEkzOSRnLm7Sfx6el0xp13KoR9zqmhzBuAO/BSGbbJ8ihRrkUNDdxwSYBgx9ojpC mo+pNOwQoNXdqkPML6SgU9RMZRkBlolTKMylGY365K2IO3ebbhH+RI+pbW6iE4SFZRYJ ITMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=BEP0xwlQ4b6InfkIJQy/Guf2YYQMYLXVg18uYu4Cs1c=; b=ef1dtvy7ueu3brFUe9DPbTNSZFPjoyQIwp9QFJ5uAAFHXS6ppkFuRSIIIn6n24sjPV nLrk3XQbPHRtqozlCpgky9O4NdzgBfR6FdlFpM9NiooubgoOQ4fGHmr5zHNFFomT5+SY xmf9yBcHYIuOH2ekYnKOZAnVx4TVYK07nu4EoTclkiGiGCZUdACZAvmOt8pSufT+e2su N6DDILUQghobwa7PAIkDHEQO4h/4elbwNOEVsC7KZNgdCqgl5vZnYwb9ZR0zpsxmT+P8 xx2NOtdsjedc/su/c/SPBCAC+Kbfck+AjSoacNcm/TxqV+0z1tMWnVaODKDcdnqpanFz cMug== X-Gm-Message-State: APjAAAWVo2jK3s8v32T+6/oHZZ6IQZwxzvSbZnul5BoAtSv6qNht7G38 fnDr2DLSN4J9IkpfpaGaK+Js5sGY X-Google-Smtp-Source: APXvYqz4hw3N1Q65A1cSKiWSnEV/Eoab5IoGTcbTn+GVltUsKvcx6cf7fcsdIkFPyCCNySsIcr9drQ== X-Received: by 2002:a9d:4913:: with SMTP id e19mr13194541otf.1.1566254665577; Mon, 19 Aug 2019 15:44:25 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id g3sm6437033oti.41.2019.08.19.15.44.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:44:25 -0700 (PDT) Subject: [PATCH v2 11/21] xprtrdma: Simplify rpcrdma_mr_pop From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:44:04 -0400 Message-ID: <156625462416.8161.10214523289584503091.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up: rpcrdma_mr_pop call sites check if the list is empty first. Let's replace the list_empty with less costly logic. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 12 ++++-------- net/sunrpc/xprtrdma/rpc_rdma.c | 7 +------ net/sunrpc/xprtrdma/verbs.c | 6 ++---- net/sunrpc/xprtrdma/xprt_rdma.h | 7 ++++--- 4 files changed, 11 insertions(+), 21 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 3a10bfff2125..d7e763fafa04 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -126,12 +126,10 @@ frwr_mr_recycle_worker(struct work_struct *work) */ void frwr_reset(struct rpcrdma_req *req) { - while (!list_empty(&req->rl_registered)) { - struct rpcrdma_mr *mr; + struct rpcrdma_mr *mr; - mr = rpcrdma_mr_pop(&req->rl_registered); + while ((mr = rpcrdma_mr_pop(&req->rl_registered))) rpcrdma_mr_unmap_and_put(mr); - } } /** @@ -532,8 +530,7 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) */ frwr = NULL; prev = &first; - while (!list_empty(&req->rl_registered)) { - mr = rpcrdma_mr_pop(&req->rl_registered); + while ((mr = rpcrdma_mr_pop(&req->rl_registered))) { trace_xprtrdma_mr_localinv(mr); r_xprt->rx_stats.local_inv_needed++; @@ -632,8 +629,7 @@ void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) */ frwr = NULL; prev = &first; - while (!list_empty(&req->rl_registered)) { - mr = rpcrdma_mr_pop(&req->rl_registered); + while ((mr = rpcrdma_mr_pop(&req->rl_registered))) { trace_xprtrdma_mr_localinv(mr); r_xprt->rx_stats.local_inv_needed++; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 4345e6912392..0ac096a6348a 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -841,12 +841,7 @@ rpcrdma_marshal_req(struct rpcrdma_xprt *r_xprt, struct rpc_rqst *rqst) * chunks. Very likely the connection has been replaced, * so these registrations are invalid and unusable. */ - while (unlikely(!list_empty(&req->rl_registered))) { - struct rpcrdma_mr *mr; - - mr = rpcrdma_mr_pop(&req->rl_registered); - rpcrdma_mr_recycle(mr); - } + frwr_reset(req); /* This implementation supports the following combinations * of chunk lists in one RPC-over-RDMA Call message: diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index e004873cc4f0..ee6fcf10425b 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1213,13 +1213,11 @@ struct rpcrdma_mr * rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; - struct rpcrdma_mr *mr = NULL; + struct rpcrdma_mr *mr; spin_lock(&buf->rb_mrlock); - if (!list_empty(&buf->rb_mrs)) - mr = rpcrdma_mr_pop(&buf->rb_mrs); + mr = rpcrdma_mr_pop(&buf->rb_mrs); spin_unlock(&buf->rb_mrlock); - if (!mr) goto out_nomrs; return mr; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 5aaa53b8ae12..9663b8ddd733 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -338,7 +338,7 @@ rpcr_to_rdmar(const struct rpc_rqst *rqst) static inline void rpcrdma_mr_push(struct rpcrdma_mr *mr, struct list_head *list) { - list_add_tail(&mr->mr_list, list); + list_add(&mr->mr_list, list); } static inline struct rpcrdma_mr * @@ -346,8 +346,9 @@ rpcrdma_mr_pop(struct list_head *list) { struct rpcrdma_mr *mr; - mr = list_first_entry(list, struct rpcrdma_mr, mr_list); - list_del_init(&mr->mr_list); + mr = list_first_entry_or_null(list, struct rpcrdma_mr, mr_list); + if (mr) + list_del_init(&mr->mr_list); return mr; } From patchwork Mon Aug 19 22:44:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102245 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42D7D13A0 for ; Mon, 19 Aug 2019 22:45:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 20EBB2070B for ; Mon, 19 Aug 2019 22:45:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AlUal8MH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728438AbfHSWpN (ORCPT ); Mon, 19 Aug 2019 18:45:13 -0400 Received: from mail-oi1-f193.google.com ([209.85.167.193]:40147 "EHLO mail-oi1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728484AbfHSWpN (ORCPT ); Mon, 19 Aug 2019 18:45:13 -0400 Received: by mail-oi1-f193.google.com with SMTP id h21so2610123oie.7; Mon, 19 Aug 2019 15:45:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=zmFja180y9h2DsLU8o+XPO/YzzGcxYC2qni3J+OcF8M=; b=AlUal8MHSRx1lnRXcOv8L7WlGoqDJLCO8biAWvoPZr/AcouZgxyCO8OChgEuyo1KhY 2OID5aZhs96By9b1UOY2xWIpAT9M3QL4jcb6ByY5wZp1QP/KQdxuxggQ0n14AChwAnS8 EsdOSIRvwrgeSdw3PBSGrsOWvj4MRqkknq2YZpbjiQqkqepVPmF85Sbo5yAHcfRyBQli 2WsTm6zTu9VmX39EnUffErOTDiWegbM9YdiJicVq7ctqu1MAvNrIAnzACgNhuzc1YDxN hluNHz1bF/YaHNUpKq9pfifj7G87vJlSNaaoScjPtnW1nYRLpRirQV3BfW/q9GisjX28 68Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=zmFja180y9h2DsLU8o+XPO/YzzGcxYC2qni3J+OcF8M=; b=foX3dynGgcudlLBFHhqpOFmziPa8SSgIZ55p5siUOCaGJHuV/NOkMbNzQq2NZTjqvc gWucZC4TxoqpjQx65K/iYRldEZpruIxzZM3fEmI7I4yppc85IIZ5UDUaWRj9ebdzm85G fRwty77RZp4N7VnixxlBLu8InUwNG4aQM+4Jlm8bAB0RIhR0RpKRZzMduXVsN0a3QwIk d2nYkcFDwphfYH8ScAw14Kipod0WTBcHS/bdD6cfcuZz6JoGD1lWMSH3QytVqXF8FLrh 3bJV1kv6JJCYKlYMlSTLsVBiJCYbcV9cpwDvA68cYMWl1JcDX/uueblxvmdoqQg71zNe 2jPA== X-Gm-Message-State: APjAAAVggEyv2b1hHfv1LF6tZ1hT24CjbtF7MdGd/KY6ESdrnbI46mht anisRuuPhbALpToC8HHl36UlLlYs X-Google-Smtp-Source: APXvYqxYNRkUoIuUtFcE/yGr+MzSUepFqpOfxF/KuZbVriIC2M85dOgm3/6nVJRmQrOW7DR/cBLMcQ== X-Received: by 2002:aca:5744:: with SMTP id l65mr15446127oib.159.1566254712147; Mon, 19 Aug 2019 15:45:12 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id a22sm5867379otr.3.2019.08.19.15.45.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:45:11 -0700 (PDT) Subject: [PATCH v2 12/21] xprtrdma: Combine rpcrdma_mr_put and rpcrdma_mr_unmap_and_put From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:44:50 -0400 Message-ID: <156625467065.8161.9671734536400171055.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up. There is only one remaining rpcrdma_mr_put call site, and it can be directly replaced with unmap_and_put because mr->mr_dir is set to DMA_NONE just before the call. Now all the call sites do a DMA unmap, and we can just rename mr_unmap_and_put to mr_put, which nicely matches mr_get. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 6 +++--- net/sunrpc/xprtrdma/verbs.c | 32 ++++++++------------------------ net/sunrpc/xprtrdma/xprt_rdma.h | 1 - 3 files changed, 11 insertions(+), 28 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index d7e763fafa04..97e1804139b8 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -129,7 +129,7 @@ void frwr_reset(struct rpcrdma_req *req) struct rpcrdma_mr *mr; while ((mr = rpcrdma_mr_pop(&req->rl_registered))) - rpcrdma_mr_unmap_and_put(mr); + rpcrdma_mr_put(mr); } /** @@ -453,7 +453,7 @@ void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs) if (mr->mr_handle == rep->rr_inv_rkey) { list_del_init(&mr->mr_list); trace_xprtrdma_mr_remoteinv(mr); - rpcrdma_mr_unmap_and_put(mr); + rpcrdma_mr_put(mr); break; /* only one invalidated MR per RPC */ } } @@ -463,7 +463,7 @@ static void __frwr_release_mr(struct ib_wc *wc, struct rpcrdma_mr *mr) if (wc->status != IB_WC_SUCCESS) rpcrdma_mr_recycle(mr); else - rpcrdma_mr_unmap_and_put(mr); + rpcrdma_mr_put(mr); } /** diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index ee6fcf10425b..5e0b774ed522 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1233,34 +1233,15 @@ rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt) return NULL; } -static void -__rpcrdma_mr_put(struct rpcrdma_buffer *buf, struct rpcrdma_mr *mr) -{ - spin_lock(&buf->rb_mrlock); - rpcrdma_mr_push(mr, &buf->rb_mrs); - spin_unlock(&buf->rb_mrlock); -} - -/** - * rpcrdma_mr_put - Release an rpcrdma_mr object - * @mr: object to release - * - */ -void -rpcrdma_mr_put(struct rpcrdma_mr *mr) -{ - __rpcrdma_mr_put(&mr->mr_xprt->rx_buf, mr); -} - /** - * rpcrdma_mr_unmap_and_put - DMA unmap an MR and release it - * @mr: object to release + * rpcrdma_mr_put - DMA unmap an MR and release it + * @mr: MR to release * */ -void -rpcrdma_mr_unmap_and_put(struct rpcrdma_mr *mr) +void rpcrdma_mr_put(struct rpcrdma_mr *mr) { struct rpcrdma_xprt *r_xprt = mr->mr_xprt; + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; if (mr->mr_dir != DMA_NONE) { trace_xprtrdma_mr_unmap(mr); @@ -1268,7 +1249,10 @@ rpcrdma_mr_unmap_and_put(struct rpcrdma_mr *mr) mr->mr_sg, mr->mr_nents, mr->mr_dir); mr->mr_dir = DMA_NONE; } - __rpcrdma_mr_put(&r_xprt->rx_buf, mr); + + spin_lock(&buf->rb_mrlock); + rpcrdma_mr_push(mr, &buf->rb_mrs); + spin_unlock(&buf->rb_mrlock); } /** diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 9663b8ddd733..3e0839c2cda2 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -485,7 +485,6 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt); struct rpcrdma_mr *rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt); void rpcrdma_mr_put(struct rpcrdma_mr *mr); -void rpcrdma_mr_unmap_and_put(struct rpcrdma_mr *mr); static inline void rpcrdma_mr_recycle(struct rpcrdma_mr *mr) From patchwork Mon Aug 19 22:45:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102251 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86D1814DB for ; Mon, 19 Aug 2019 22:46:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6432322CF4 for ; Mon, 19 Aug 2019 22:46:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kO1RYwjk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728469AbfHSWqA (ORCPT ); Mon, 19 Aug 2019 18:46:00 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:33915 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728402AbfHSWqA (ORCPT ); Mon, 19 Aug 2019 18:46:00 -0400 Received: by mail-ot1-f67.google.com with SMTP id c7so3248744otp.1; Mon, 19 Aug 2019 15:45:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=K76ockKaA/dKCETBDEOaS16txechlXh3Dbav1CV2dDY=; b=kO1RYwjkQhokuEPBEw3TrOzPrpB8qX6r9gvwRBMwLqo1hOolFiBqtsuhZ8APSD5Iln kq/BzCzXi/2nOaOpAGqBLVhUkcwLdRegdaZ9JgS93FvFP6+Rhme1OsWpZH7XMHmrEtlR RjOf7E5DyndzjGwQCBoJ9/s7O9oWGbe7HZEClpyt1qI9b6c1iSes2xV7QLg9gSsHSE/u Tr0TvX1f2GlZGK5W9XV3C+XWLyvYhUYKiLhPE6xhnpWz1LBbcZ5zbKPEetUDvT9lGzLs VOwaTF+9RO53l8OwTdLpwf9Igmyp8HPXJb0o+i/llskygk5x+YP4OuRiqbgTjD0gc6gM aTWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=K76ockKaA/dKCETBDEOaS16txechlXh3Dbav1CV2dDY=; b=q3YTkLt71uCtY+/Wr2Wvo8C4cfT8LQOP+4sqgvMDaW5pEojwRKUIGhjQHJQN0Izrlu qLqy/ATWu/h9qfrTTXf+3VGmqFlW+DwFm8VrRW7fazen9kfgqVOxkUeJdkfwgPgV3i4C fjYlSKkbmu/eRpunsXbwHPMI9X++x7eu8jAv7Z8aOjyFrQh3HNrdXN3EhbV9zDvftwTn HPAM1Wh2q80R4fgmjBjIvOTQcC/0lNfAblrBD6AnXGPafNrnWVYiPA6GVOZFc5jX0opr L/uvo35eGoCHC+Mg0mcSO6b0BBJ9EjAnVOKc0Mmu6IOjvigQxBCah6RfuPDUSrre8uz7 2Muw== X-Gm-Message-State: APjAAAWcHXd1ay2HNO7INuHKWUQILH41KFH3Six2DvGUrF797i1zqP2a uDaHHIkBSj/opBnq0rlqmnaGC/rb X-Google-Smtp-Source: APXvYqxLcJqeCCm+4/iGTm1ZuW5zWD0Qtzf2FH6FQkS12RagFdWAguHW7A6jrargiYmb0+Eh+2QjMQ== X-Received: by 2002:a9d:65ca:: with SMTP id z10mr19043049oth.167.1566254758861; Mon, 19 Aug 2019 15:45:58 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id g7sm5922208otp.20.2019.08.19.15.45.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:45:58 -0700 (PDT) Subject: [PATCH v2 13/21] xprtrdma: Move rpcrdma_mr_get out of frwr_map From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:45:37 -0400 Message-ID: <156625471731.8161.15138263031991137209.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Refactor: Retrieve an MR and handle error recovery entirely in rpc_rdma.c, as this is not a device-specific function. Note that since commit 89f90fe1ad8b ("SUNRPC: Allow calls to xprt_transmit() to drain the entire transmit queue"), the xprt_transmit function handles the cond_resched. The transport no longer has to do this itself. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 29 ++++++++++++++++++++++++++++- net/sunrpc/xprtrdma/frwr_ops.c | 23 +++++------------------ net/sunrpc/xprtrdma/rpc_rdma.c | 30 ++++++++++++++++++++++++------ net/sunrpc/xprtrdma/verbs.c | 21 ++++----------------- net/sunrpc/xprtrdma/xprt_rdma.h | 4 ++-- 5 files changed, 63 insertions(+), 44 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 6e6055eb67e7..83c4dfd7feea 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -464,7 +464,34 @@ TRACE_EVENT(xprtrdma_createmrs, ) ); -DEFINE_RXPRT_EVENT(xprtrdma_nomrs); +TRACE_EVENT(xprtrdma_nomrs, + TP_PROTO( + const struct rpcrdma_req *req + ), + + TP_ARGS(req), + + TP_STRUCT__entry( + __field(const void *, req) + __field(unsigned int, task_id) + __field(unsigned int, client_id) + __field(u32, xid) + ), + + TP_fast_assign( + const struct rpc_rqst *rqst = &req->rl_slot; + + __entry->req = req; + __entry->task_id = rqst->rq_task->tk_pid; + __entry->client_id = rqst->rq_task->tk_client->cl_clid; + __entry->xid = be32_to_cpu(rqst->rq_xid); + ), + + TP_printk("task:%u@%u xid=0x%08x req=%p", + __entry->task_id, __entry->client_id, __entry->xid, + __entry->req + ) +); DEFINE_RDCH_EVENT(read); DEFINE_WRCH_EVENT(write); diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 97e1804139b8..362056f4f48d 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -291,31 +291,25 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt) * @nsegs: number of segments remaining * @writing: true when RDMA Write will be used * @xid: XID of RPC using the registered memory - * @out: initialized MR + * @mr: MR to fill in * * Prepare a REG_MR Work Request to register a memory region * for remote access via RDMA READ or RDMA WRITE. * * Returns the next segment or a negative errno pointer. - * On success, the prepared MR is planted in @out. + * On success, @mr is filled in. */ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, int nsegs, bool writing, __be32 xid, - struct rpcrdma_mr **out) + struct rpcrdma_mr *mr) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; - bool holes_ok = ia->ri_mrtype == IB_MR_TYPE_SG_GAPS; - struct rpcrdma_mr *mr; - struct ib_mr *ibmr; struct ib_reg_wr *reg_wr; + struct ib_mr *ibmr; int i, n; u8 key; - mr = rpcrdma_mr_get(r_xprt); - if (!mr) - goto out_getmr_err; - if (nsegs > ia->ri_max_frwr_depth) nsegs = ia->ri_max_frwr_depth; for (i = 0; i < nsegs;) { @@ -330,7 +324,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, ++seg; ++i; - if (holes_ok) + if (ia->ri_mrtype == IB_MR_TYPE_SG_GAPS) continue; if ((i < nsegs && offset_in_page(seg->mr_offset)) || offset_in_page((seg-1)->mr_offset + (seg-1)->mr_len)) @@ -365,22 +359,15 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, mr->mr_offset = ibmr->iova; trace_xprtrdma_mr_map(mr); - *out = mr; return seg; -out_getmr_err: - xprt_wait_for_buffer_space(&r_xprt->rx_xprt); - return ERR_PTR(-EAGAIN); - out_dmamap_err: mr->mr_dir = DMA_NONE; trace_xprtrdma_frwr_sgerr(mr, i); - rpcrdma_mr_put(mr); return ERR_PTR(-EIO); out_mapmr_err: trace_xprtrdma_frwr_maperr(mr, n); - rpcrdma_mr_recycle(mr); return ERR_PTR(-EIO); } diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 0ac096a6348a..34772cb19286 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -342,6 +342,27 @@ encode_read_segment(struct xdr_stream *xdr, struct rpcrdma_mr *mr, return 0; } +static struct rpcrdma_mr_seg *rpcrdma_mr_prepare(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct rpcrdma_mr_seg *seg, + int nsegs, bool writing, + struct rpcrdma_mr **mr) +{ + *mr = rpcrdma_mr_get(r_xprt); + if (!*mr) + goto out_getmr_err; + + rpcrdma_mr_push(*mr, &req->rl_registered); + return frwr_map(r_xprt, seg, nsegs, writing, req->rl_slot.rq_xid, *mr); + +out_getmr_err: + trace_xprtrdma_nomrs(req); + xprt_wait_for_buffer_space(&r_xprt->rx_xprt); + if (r_xprt->rx_ep.rep_connected != -ENODEV) + schedule_work(&r_xprt->rx_buf.rb_refresh_worker); + return ERR_PTR(-EAGAIN); +} + /* Register and XDR encode the Read list. Supports encoding a list of read * segments that belong to a single read chunk. * @@ -379,10 +400,9 @@ rpcrdma_encode_read_list(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, return nsegs; do { - seg = frwr_map(r_xprt, seg, nsegs, false, rqst->rq_xid, &mr); + seg = rpcrdma_mr_prepare(r_xprt, req, seg, nsegs, false, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); - rpcrdma_mr_push(mr, &req->rl_registered); if (encode_read_segment(xdr, mr, pos) < 0) return -EMSGSIZE; @@ -440,10 +460,9 @@ rpcrdma_encode_write_list(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, nchunks = 0; do { - seg = frwr_map(r_xprt, seg, nsegs, true, rqst->rq_xid, &mr); + seg = rpcrdma_mr_prepare(r_xprt, req, seg, nsegs, true, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); - rpcrdma_mr_push(mr, &req->rl_registered); if (encode_rdma_segment(xdr, mr) < 0) return -EMSGSIZE; @@ -501,10 +520,9 @@ rpcrdma_encode_reply_chunk(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, nchunks = 0; do { - seg = frwr_map(r_xprt, seg, nsegs, true, rqst->rq_xid, &mr); + seg = rpcrdma_mr_prepare(r_xprt, req, seg, nsegs, true, &mr); if (IS_ERR(seg)) return PTR_ERR(seg); - rpcrdma_mr_push(mr, &req->rl_registered); if (encode_rdma_segment(xdr, mr) < 0) return -EMSGSIZE; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 5e0b774ed522..c9fa0f27b10a 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -408,7 +408,7 @@ rpcrdma_ia_remove(struct rpcrdma_ia *ia) struct rpcrdma_req *req; struct rpcrdma_rep *rep; - cancel_delayed_work_sync(&buf->rb_refresh_worker); + cancel_work_sync(&buf->rb_refresh_worker); /* This is similar to rpcrdma_ep_destroy, but: * - Don't cancel the connect worker. @@ -975,7 +975,7 @@ static void rpcrdma_mr_refresh_worker(struct work_struct *work) { struct rpcrdma_buffer *buf = container_of(work, struct rpcrdma_buffer, - rb_refresh_worker.work); + rb_refresh_worker); struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt, rx_buf); @@ -1086,8 +1086,7 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) spin_lock_init(&buf->rb_lock); INIT_LIST_HEAD(&buf->rb_mrs); INIT_LIST_HEAD(&buf->rb_all_mrs); - INIT_DELAYED_WORK(&buf->rb_refresh_worker, - rpcrdma_mr_refresh_worker); + INIT_WORK(&buf->rb_refresh_worker, rpcrdma_mr_refresh_worker); rpcrdma_mrs_create(r_xprt); @@ -1177,7 +1176,7 @@ rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) void rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) { - cancel_delayed_work_sync(&buf->rb_refresh_worker); + cancel_work_sync(&buf->rb_refresh_worker); rpcrdma_sendctxs_destroy(buf); @@ -1218,19 +1217,7 @@ rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt) spin_lock(&buf->rb_mrlock); mr = rpcrdma_mr_pop(&buf->rb_mrs); spin_unlock(&buf->rb_mrlock); - if (!mr) - goto out_nomrs; return mr; - -out_nomrs: - trace_xprtrdma_nomrs(r_xprt); - if (r_xprt->rx_ep.rep_connected != -ENODEV) - schedule_delayed_work(&buf->rb_refresh_worker, 0); - - /* Allow the reply handler and refresh worker to run */ - cond_resched(); - - return NULL; } /** diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 3e0839c2cda2..9573587ca602 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -379,7 +379,7 @@ struct rpcrdma_buffer { u32 rb_bc_srv_max_requests; u32 rb_bc_max_requests; - struct delayed_work rb_refresh_worker; + struct work_struct rb_refresh_worker; }; /* @@ -548,7 +548,7 @@ size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt); struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg, int nsegs, bool writing, __be32 xid, - struct rpcrdma_mr **mr); + struct rpcrdma_mr *mr); int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req); void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs); void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req); From patchwork Mon Aug 19 22:46:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102253 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4470813A0 for ; Mon, 19 Aug 2019 22:46:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 22BC722CEC for ; Mon, 19 Aug 2019 22:46:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GYayVWd5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728363AbfHSWqq (ORCPT ); Mon, 19 Aug 2019 18:46:46 -0400 Received: from mail-ot1-f66.google.com ([209.85.210.66]:44619 "EHLO mail-ot1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWqq (ORCPT ); Mon, 19 Aug 2019 18:46:46 -0400 Received: by mail-ot1-f66.google.com with SMTP id w4so3228847ote.11; Mon, 19 Aug 2019 15:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=zxkSkyCfPKPerjvC8Yo1fxnGEm55eqARL6sFVpESOm0=; b=GYayVWd5fme4TfxOLU2G4rxKnQkHPAxdPsUemQfVkAca62EXfQqeN5wZUaWrL5BySY xsUYN3BRK7RxH3lEin8DU3VhDJ0p2VDELez/XQ6s8MJH8FVHZNuVVQB/E3i2D5vCZ+40 Muue4m/hR1EvX2z0A/p3T9Xjim9GPSyJSFmlQ6q83O7kSFkdeOjB4opUDdq8n8tD9Tom VWms4jpW1DLxdsmuWUXtautlgaO7Ah2+dY03ClNQO8ENXpecZepaTXb4tXdTV6i/tiDG Jq5n/dGcmZ8geaPWQkvbjwFFCwQDkzDYMKxGiRVNZa3EK2ehVARYnly55ZK4fiX/A4su sTEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=zxkSkyCfPKPerjvC8Yo1fxnGEm55eqARL6sFVpESOm0=; b=R1/HZALtUTrBWp53aZag9iv4XqPI4P1bXqYlEz3HjxtSkHHOpUiPs9CoWdFFUz2R7C cdT0kwHqlQMgMtScvFt83C6lECJ6tZL/+rPlVRQcV02scBEnvlPznU/aklQvpZFB16wj NSX435kGcVIRuT3vT2Q4dnlugC1ZYBQk0GQ2JlsxUQlrgb4CAWUYrn/SRoe1V4QFboip m4iCQddXZIf0bsKFHMfJA0FWWKKM4RBh8ex7cU9tS+Xji2+yqMUuyGaHZ+IAgCVYpftJ 1nQj54dUH9qLdBI5qsf5sk7D7CWBUin5zaSWkBQzc4zzLcP0jgqHreeKHUpMki9QcJPO X+zQ== X-Gm-Message-State: APjAAAVR4z+6Y7Wo7GBcC3iCns2sj/aQNsSHJnMB4wM2fd2fXoV3ct7n AyLEEXpU2+oTPiXYXYvKeD6wAAGm X-Google-Smtp-Source: APXvYqxyX6nHvwLzrUHw/fGny6/uJjEjBphGXTRI/KDlYoIr5ohX5fBWJJoyCyiJiSD/0FR62u+2Ng== X-Received: by 2002:a9d:d51:: with SMTP id 75mr20439529oti.46.1566254805327; Mon, 19 Aug 2019 15:46:45 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id f84sm4732118oig.43.2019.08.19.15.46.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:46:44 -0700 (PDT) Subject: [PATCH v2 14/21] xprtrdma: Ensure creating an MR does not trigger FS writeback From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:46:24 -0400 Message-ID: <156625476397.8161.12423510142910031118.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Probably would be good to also pass GFP flags to ib_alloc_mr. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 7 ++++--- net/sunrpc/xprtrdma/verbs.c | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 362056f4f48d..1f2e3dda7401 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -147,11 +147,14 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) struct ib_mr *frmr; int rc; + /* NB: ib_alloc_mr and device drivers typically allocate + * memory with GFP_KERNEL. + */ frmr = ib_alloc_mr(ia->ri_pd, ia->ri_mrtype, depth); if (IS_ERR(frmr)) goto out_mr_err; - sg = kcalloc(depth, sizeof(*sg), GFP_KERNEL); + sg = kcalloc(depth, sizeof(*sg), GFP_NOFS); if (!sg) goto out_list_err; @@ -171,8 +174,6 @@ int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr) return rc; out_list_err: - dprintk("RPC: %s: sg allocation failure\n", - __func__); ib_dereg_mr(frmr); return -ENOMEM; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index c9fa0f27b10a..cb6df58488bb 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -949,7 +949,7 @@ rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) struct rpcrdma_mr *mr; int rc; - mr = kzalloc(sizeof(*mr), GFP_KERNEL); + mr = kzalloc(sizeof(*mr), GFP_NOFS); if (!mr) break; From patchwork Mon Aug 19 22:47:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102259 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C5E4814F7 for ; Mon, 19 Aug 2019 22:47:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A41432070B for ; Mon, 19 Aug 2019 22:47:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DeSVbJGW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728363AbfHSWrd (ORCPT ); Mon, 19 Aug 2019 18:47:33 -0400 Received: from mail-ot1-f66.google.com ([209.85.210.66]:32823 "EHLO mail-ot1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWrd (ORCPT ); Mon, 19 Aug 2019 18:47:33 -0400 Received: by mail-ot1-f66.google.com with SMTP id q20so3261320otl.0; Mon, 19 Aug 2019 15:47:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=OHp3dI4bNSLqfi7jMEsyKOlcekkjcDgj+SJCxIQrhWU=; b=DeSVbJGWXYaPORN2UCo0rskzIABFZEOIJBnqt5wNp/xSDq1fXC49T7IR0ZWctcP4O9 qC9kzaYdr4KALHsujN/hYthPxCRd2ceI35ML2ZQPWWpkgZ2Kg+M7v+gkVRrimXOCqxde HFf5SO6vIR1qS3GyepkSZ9DVu4WxQnmFnGY7xD2mYZn3YKy5gKonYLt6Eikne32XUYaR zfMF6ov/pUCvvi9dDcP0Ydc4qvR1++eYIM4O7he6Zh//SSLr8Jq8EokrbuUHEgGbosZU S/OjLFqvEKWpKOyt4TLDdfyjBTKrxC/W0AuufAHJU+ATneShB8IqUdBRKpL/sVeoV1nz Tl7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=OHp3dI4bNSLqfi7jMEsyKOlcekkjcDgj+SJCxIQrhWU=; b=L7lkcH2/xia9/Bw2volKBFqhywzAZzbWvFoG8Yp2FmQoEOo0k7Bvp4Y4+Kz6cOdHS0 SEW623AVVcGG1Kt4kKYU0qYJnrsbuZFzjiECts47F4g4yH+yVjcAHjY3GeQ3E4490W0e tRsz5MvtTQKL7gb10Njl/z0B30hqU0TQr6/5Y4qkVpkUTTOWUi8+Px5YbI4Ni6lD5akO GwJ2jdwxcsHy/kgjdaEpIZIqWpc30lSfJlN2Kfx6BnkjIL+JBMbeTMzn3Sgx43/VNgmH yQVnPbzKj056MFk6UACDilDHmc3GN/XfuhCZWHjou+9pWi1vbcglI/iUWa6gjC5PnHin eqrA== X-Gm-Message-State: APjAAAW+ATIoiYiaSV2t8gRlXa2YDaB6XpEOFdW7mLamHoZzCvRLymZB QNEFKW5tsEF4/zAS5Bg7JAp78awQ X-Google-Smtp-Source: APXvYqy3lrPcPZT3WdyqFbEvfIq2rshVtyMV/lHMeepQXxz7DIqMK8o7GwmWr1X2HFtDqP+2a5FT6w== X-Received: by 2002:a9d:5c11:: with SMTP id o17mr3758247otk.107.1566254851884; Mon, 19 Aug 2019 15:47:31 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id 20sm5997463oth.43.2019.08.19.15.47.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:47:31 -0700 (PDT) Subject: [PATCH v2 15/21] xprtrdma: Cache free MRs in each rpcrdma_req From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:47:10 -0400 Message-ID: <156625481040.8161.13621119408884656142.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Instead of a globally-contended MR free list, cache MRs in each rpcrdma_req as they are released. This means acquiring and releasing an MR will be lock-free in the common case, even outside the transport send lock. The original idea of per-rpcrdma_req MR free lists was suggested by Shirley Ma several years ago. I just now figured out how to make that idea work with on-demand MR allocation. Signed-off-by: Chuck Lever --- include/trace/events/rpcrdma.h | 38 ++++++++++++++++++++++++++++++++++++-- net/sunrpc/xprtrdma/frwr_ops.c | 9 ++++++--- net/sunrpc/xprtrdma/rpc_rdma.c | 11 ++++++++--- net/sunrpc/xprtrdma/verbs.c | 18 +++++++++++++++--- net/sunrpc/xprtrdma/xprt_rdma.h | 7 ++++--- 5 files changed, 69 insertions(+), 14 deletions(-) diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 83c4dfd7feea..a13830616107 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -451,16 +451,50 @@ TRACE_EVENT(xprtrdma_createmrs, TP_STRUCT__entry( __field(const void *, r_xprt) + __string(addr, rpcrdma_addrstr(r_xprt)) + __string(port, rpcrdma_portstr(r_xprt)) __field(unsigned int, count) ), TP_fast_assign( __entry->r_xprt = r_xprt; __entry->count = count; + __assign_str(addr, rpcrdma_addrstr(r_xprt)); + __assign_str(port, rpcrdma_portstr(r_xprt)); ), - TP_printk("r_xprt=%p: created %u MRs", - __entry->r_xprt, __entry->count + TP_printk("peer=[%s]:%s r_xprt=%p: created %u MRs", + __get_str(addr), __get_str(port), __entry->r_xprt, + __entry->count + ) +); + +TRACE_EVENT(xprtrdma_mr_get, + TP_PROTO( + const struct rpcrdma_req *req + ), + + TP_ARGS(req), + + TP_STRUCT__entry( + __field(const void *, req) + __field(unsigned int, task_id) + __field(unsigned int, client_id) + __field(u32, xid) + ), + + TP_fast_assign( + const struct rpc_rqst *rqst = &req->rl_slot; + + __entry->req = req; + __entry->task_id = rqst->rq_task->tk_pid; + __entry->client_id = rqst->rq_task->tk_client->cl_clid; + __entry->xid = be32_to_cpu(rqst->rq_xid); + ), + + TP_printk("task:%u@%u xid=0x%08x req=%p", + __entry->task_id, __entry->client_id, __entry->xid, + __entry->req ) ); diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 1f2e3dda7401..0e740bae2d80 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -488,8 +488,8 @@ static void frwr_wc_localinv_wake(struct ib_cq *cq, struct ib_wc *wc) /* WARNING: Only wr_cqe and status are reliable at this point */ trace_xprtrdma_wc_li_wake(wc, frwr); - complete(&frwr->fr_linv_done); __frwr_release_mr(wc, mr); + complete(&frwr->fr_linv_done); } /** @@ -587,11 +587,15 @@ static void frwr_wc_localinv_done(struct ib_cq *cq, struct ib_wc *wc) struct rpcrdma_frwr *frwr = container_of(cqe, struct rpcrdma_frwr, fr_cqe); struct rpcrdma_mr *mr = container_of(frwr, struct rpcrdma_mr, frwr); + struct rpcrdma_rep *rep = mr->mr_req->rl_reply; /* WARNING: Only wr_cqe and status are reliable at this point */ trace_xprtrdma_wc_li_done(wc, frwr); - rpcrdma_complete_rqst(frwr->fr_req->rl_reply); __frwr_release_mr(wc, mr); + + /* Ensure @rep is generated before __frwr_release_mr */ + smp_rmb(); + rpcrdma_complete_rqst(rep); } /** @@ -624,7 +628,6 @@ void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) frwr = &mr->frwr; frwr->fr_cqe.done = frwr_wc_localinv; - frwr->fr_req = req; last = &frwr->fr_invwr; last->next = NULL; last->wr_cqe = &frwr->fr_cqe; diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 34772cb19286..ffeb4dfebd46 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -348,9 +348,14 @@ static struct rpcrdma_mr_seg *rpcrdma_mr_prepare(struct rpcrdma_xprt *r_xprt, int nsegs, bool writing, struct rpcrdma_mr **mr) { - *mr = rpcrdma_mr_get(r_xprt); - if (!*mr) - goto out_getmr_err; + *mr = rpcrdma_mr_pop(&req->rl_free_mrs); + if (!*mr) { + *mr = rpcrdma_mr_get(r_xprt); + if (!*mr) + goto out_getmr_err; + trace_xprtrdma_mr_get(req); + (*mr)->mr_req = req; + } rpcrdma_mr_push(*mr, &req->rl_registered); return frwr_map(r_xprt, seg, nsegs, writing, req->rl_slot.rq_xid, *mr); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index cb6df58488bb..69753ec73c36 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -77,6 +77,7 @@ static void rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); +static void rpcrdma_mr_free(struct rpcrdma_mr *mr); static struct rpcrdma_regbuf * rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction, gfp_t flags); @@ -1022,6 +1023,7 @@ struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size, if (!req->rl_recvbuf) goto out4; + INIT_LIST_HEAD(&req->rl_free_mrs); INIT_LIST_HEAD(&req->rl_registered); spin_lock(&buffer->rb_lock); list_add(&req->rl_all, &buffer->rb_allreqs); @@ -1130,11 +1132,13 @@ static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep) * This function assumes that the caller prevents concurrent device * unload and transport tear-down. */ -void -rpcrdma_req_destroy(struct rpcrdma_req *req) +void rpcrdma_req_destroy(struct rpcrdma_req *req) { list_del(&req->rl_all); + while (!list_empty(&req->rl_free_mrs)) + rpcrdma_mr_free(rpcrdma_mr_pop(&req->rl_free_mrs)); + rpcrdma_regbuf_free(req->rl_recvbuf); rpcrdma_regbuf_free(req->rl_sendbuf); rpcrdma_regbuf_free(req->rl_rdmabuf); @@ -1228,7 +1232,6 @@ rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt) void rpcrdma_mr_put(struct rpcrdma_mr *mr) { struct rpcrdma_xprt *r_xprt = mr->mr_xprt; - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; if (mr->mr_dir != DMA_NONE) { trace_xprtrdma_mr_unmap(mr); @@ -1237,6 +1240,15 @@ void rpcrdma_mr_put(struct rpcrdma_mr *mr) mr->mr_dir = DMA_NONE; } + rpcrdma_mr_push(mr, &mr->mr_req->rl_free_mrs); +} + +static void rpcrdma_mr_free(struct rpcrdma_mr *mr) +{ + struct rpcrdma_xprt *r_xprt = mr->mr_xprt; + struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + + mr->mr_req = NULL; spin_lock(&buf->rb_mrlock); rpcrdma_mr_push(mr, &buf->rb_mrs); spin_unlock(&buf->rb_mrlock); diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 9573587ca602..c375b0e434ac 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -234,20 +234,20 @@ struct rpcrdma_sendctx { * An external memory region is any buffer or page that is registered * on the fly (ie, not pre-registered). */ -struct rpcrdma_req; struct rpcrdma_frwr { struct ib_mr *fr_mr; struct ib_cqe fr_cqe; struct completion fr_linv_done; - struct rpcrdma_req *fr_req; union { struct ib_reg_wr fr_regwr; struct ib_send_wr fr_invwr; }; }; +struct rpcrdma_req; struct rpcrdma_mr { struct list_head mr_list; + struct rpcrdma_req *mr_req; struct scatterlist *mr_sg; int mr_nents; enum dma_data_direction mr_dir; @@ -325,7 +325,8 @@ struct rpcrdma_req { struct list_head rl_all; struct kref rl_kref; - struct list_head rl_registered; /* registered segments */ + struct list_head rl_free_mrs; + struct list_head rl_registered; struct rpcrdma_mr_seg rl_segments[RPCRDMA_MAX_SEGS]; }; From patchwork Mon Aug 19 22:47:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102261 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43108912 for ; Mon, 19 Aug 2019 22:48:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2311A22CEC for ; Mon, 19 Aug 2019 22:48:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OVaey6Pw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728603AbfHSWsT (ORCPT ); Mon, 19 Aug 2019 18:48:19 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:42870 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWsT (ORCPT ); Mon, 19 Aug 2019 18:48:19 -0400 Received: by mail-ot1-f67.google.com with SMTP id j7so3239235ota.9; Mon, 19 Aug 2019 15:48:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=tlYQvSFDVmirL961EYb6E6NXJ8a88YWm44bFByD4wzY=; b=OVaey6PwRYSt5idQU7SDEHeLR09BNeMdvcOkX5t+bkdSMiSd3E2jwvb2qS3luzUE9b NEbdS267G1z0lX6JryY9wpkypHSqjgL3dbkrC15ZK9cE6zQCoDV1ROPIPQwBa0eobNz+ 33d4gieTDGxFa61wK/rWcMVRY5mWCiCeZrbWbUc0Z4tnCOWdp4rtZfwm1eZu1XV+B0pI TZxshs4YHX3VLJXdz/PMdDPWGEyR+V5oPcA9i4BKFn60rXTVmNRndSilVAsedxWwD72m OeOAfpDOuCDnl4L1MNqwyoPHD9GEkWAHF37j+pY8lqpo9ROZWiJld+JrW4TbMa4tfFaH UOSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=tlYQvSFDVmirL961EYb6E6NXJ8a88YWm44bFByD4wzY=; b=H9vIaKhLIpIyB1uAy0QIpWCL6oT9SBSqUNlwMwaIqR3YUBs3MBRY9CKNSu/0+sqBLw fAcCGrNHe1bNNUZboax35R9eRBWCYJ+BwRvZfpcDF6siP3YX/GuUoXwHwVEa7UlRx2aj MELGZRMkOvN/4TQtkFXsyrLjTUBqVM0Za8xKs1fn6m8jY35WMw4FHT+ERfZC/RLCHaU6 beKencZhQ2hDgF6nNqIExPEsh8CboRbwBbKT01K0ZOEjmDe4iY20oIrj0P1VqVM3rFTw W5JgPteY9EEsJ0fDtR1PNc/5bejoZBLYAy5iQyJzsbZCHN8HxmXbbYsGrntZgdhpGJZJ nm6w== X-Gm-Message-State: APjAAAWzSIovEEI0NuVPJQ5W8CtGoa7CeCtd6FhM+TfYsWsFjhajrm/B yUtx63LHoMSudOhnheuXqZeinuVk X-Google-Smtp-Source: APXvYqwcdYo9Ve/O4x3S0RZ/BQMLCOAeq5rf7msEsBp7tXlA16st8zO8dhKDdFPa+G2RZLx3l4A2jQ== X-Received: by 2002:a05:6830:1d6e:: with SMTP id l14mr1775810oti.77.1566254898320; Mon, 19 Aug 2019 15:48:18 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id e27sm3502316oig.53.2019.08.19.15.48.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:48:17 -0700 (PDT) Subject: [PATCH v2 16/21] xprtrdma: Remove rpcrdma_buffer::rb_mrlock From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:47:57 -0400 Message-ID: <156625485698.8161.16540137464222574550.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up: Now that the free list is used sparingly, get rid of the separate spin lock protecting it. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- net/sunrpc/xprtrdma/verbs.c | 21 ++++++++++----------- net/sunrpc/xprtrdma/xprt_rdma.h | 9 ++++----- 3 files changed, 16 insertions(+), 18 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 0e740bae2d80..368cdf3edfc9 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -106,10 +106,10 @@ frwr_mr_recycle_worker(struct work_struct *work) mr->mr_dir = DMA_NONE; } - spin_lock(&r_xprt->rx_buf.rb_mrlock); + spin_lock(&r_xprt->rx_buf.rb_lock); list_del(&mr->mr_all); r_xprt->rx_stats.mrs_recycled++; - spin_unlock(&r_xprt->rx_buf.rb_mrlock); + spin_unlock(&r_xprt->rx_buf.rb_lock); frwr_release_mr(mr); } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 69753ec73c36..52444c4d1be2 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -962,10 +962,10 @@ rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) mr->mr_xprt = r_xprt; - spin_lock(&buf->rb_mrlock); + spin_lock(&buf->rb_lock); list_add(&mr->mr_list, &buf->rb_mrs); list_add(&mr->mr_all, &buf->rb_all_mrs); - spin_unlock(&buf->rb_mrlock); + spin_unlock(&buf->rb_lock); } r_xprt->rx_stats.mrs_allocated += count; @@ -1084,7 +1084,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) buf->rb_max_requests = r_xprt->rx_ep.rep_max_requests; buf->rb_bc_srv_max_requests = 0; - spin_lock_init(&buf->rb_mrlock); spin_lock_init(&buf->rb_lock); INIT_LIST_HEAD(&buf->rb_mrs); INIT_LIST_HEAD(&buf->rb_all_mrs); @@ -1154,18 +1153,18 @@ rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf) unsigned int count; count = 0; - spin_lock(&buf->rb_mrlock); + spin_lock(&buf->rb_lock); while ((mr = list_first_entry_or_null(&buf->rb_all_mrs, struct rpcrdma_mr, mr_all)) != NULL) { list_del(&mr->mr_all); - spin_unlock(&buf->rb_mrlock); + spin_unlock(&buf->rb_lock); frwr_release_mr(mr); count++; - spin_lock(&buf->rb_mrlock); + spin_lock(&buf->rb_lock); } - spin_unlock(&buf->rb_mrlock); + spin_unlock(&buf->rb_lock); r_xprt->rx_stats.mrs_allocated = 0; } @@ -1218,9 +1217,9 @@ rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt) struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_mr *mr; - spin_lock(&buf->rb_mrlock); + spin_lock(&buf->rb_lock); mr = rpcrdma_mr_pop(&buf->rb_mrs); - spin_unlock(&buf->rb_mrlock); + spin_unlock(&buf->rb_lock); return mr; } @@ -1249,9 +1248,9 @@ static void rpcrdma_mr_free(struct rpcrdma_mr *mr) struct rpcrdma_buffer *buf = &r_xprt->rx_buf; mr->mr_req = NULL; - spin_lock(&buf->rb_mrlock); + spin_lock(&buf->rb_lock); rpcrdma_mr_push(mr, &buf->rb_mrs); - spin_unlock(&buf->rb_mrlock); + spin_unlock(&buf->rb_lock); } /** diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index c375b0e434ac..200d075bbe31 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -360,19 +360,18 @@ rpcrdma_mr_pop(struct list_head *list) * One of these is associated with a transport instance */ struct rpcrdma_buffer { - spinlock_t rb_mrlock; /* protect rb_mrs list */ + spinlock_t rb_lock; + struct list_head rb_send_bufs; + struct list_head rb_recv_bufs; struct list_head rb_mrs; - struct list_head rb_all_mrs; unsigned long rb_sc_head; unsigned long rb_sc_tail; unsigned long rb_sc_last; struct rpcrdma_sendctx **rb_sc_ctxs; - spinlock_t rb_lock; /* protect buf lists */ - struct list_head rb_send_bufs; - struct list_head rb_recv_bufs; struct list_head rb_allreqs; + struct list_head rb_all_mrs; u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ From patchwork Mon Aug 19 22:48:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102265 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A8D0912 for ; Mon, 19 Aug 2019 22:49:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0A02322CEC for ; Mon, 19 Aug 2019 22:49:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MHeBxx8V" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728484AbfHSWtG (ORCPT ); Mon, 19 Aug 2019 18:49:06 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:40375 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWtG (ORCPT ); Mon, 19 Aug 2019 18:49:06 -0400 Received: by mail-ot1-f65.google.com with SMTP id c34so3241378otb.7; Mon, 19 Aug 2019 15:49:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=KtCwL8c69F3COLA77qkJ81XZUTUH1fP6yQHmJ3Q9TRA=; b=MHeBxx8V+WLyrABwc4enReMYqD25fNsuHxc0dm7Y9gQJmDMBC71Al3D/tlStuGtmw9 //noduB2DZg2WQpBltOmImdTJNyfxnnY/RIvKC8enlfVGFRTGFLm787it0psfGeiyXV5 RWtYCpTDuDoji/lj3f3IqR2mihcQrUr2AjzeBI/+1KEm32JeetDU7FKHrsnqahouf4Kw DE3/b0WjTGzzVVRupSFPs9aIEJ+f7TlBbsrlY184GNyrKdfcT3hYYElyhvwsysGA4FV/ 5Yr0ptZGGHfzQOyotWoG7h+lGHgZg565bpCt5cm/LeJ+nxntN1xPt59xwQGg14NqMrA+ POrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=KtCwL8c69F3COLA77qkJ81XZUTUH1fP6yQHmJ3Q9TRA=; b=MnEPUktIRM5EifB2rmQiHCFF1Wh/1xdysCR3wwjTk5McULgqiV3Xnr7W5WqmcsN4jr OwmQjDKbXaMdhous3r92eaGuZskBqD0mMl/J4TRuSQcHzuc9FP28N31i4jQ6b/JlukHM NoT/vxrMR6WzpoUBuKc1WgRom1YXS5gOGwK+FaDu0yvDet426t+PkMzk1PR1vC0ETteM 5GUJNWVJAXEMqGDyjX1oCEIkS5UzOwZyn3NvtP0hR6op9uNpVtxhhApzSrAPOSLl1gm5 RK/Y0+4luatCD0r+aH/ZsHCXltwuaDqq68Rqz4/fqblN+DmCnx4Y4jIEWFm3NOghrICb 54tw== X-Gm-Message-State: APjAAAXRiVAze2WwrrNNnVKtUVSwVJeeI1NZv50YqZ1d3l5wOxzFfP5P PMzUrXg1IuquLpD2W5ZWQyAm5ItA X-Google-Smtp-Source: APXvYqwZ3UMVJCPpn0rzGbYSsj7Kb6GIoetc9CZm3vLEQGJZjPL3GJW8as6jvK6tMXL/GZ1jd/vKow== X-Received: by 2002:a9d:4718:: with SMTP id a24mr1362102otf.12.1566254945124; Mon, 19 Aug 2019 15:49:05 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id g93sm6023429otb.39.2019.08.19.15.49.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:49:04 -0700 (PDT) Subject: [PATCH v2 17/21] xprtrdma: Use an llist to manage free rpcrdma_reps From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:48:43 -0400 Message-ID: <156625490339.8161.16802609239404263834.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org rpcrdma_rep objects are removed from their free list by only a single thread: the Receive completion handler. Thus that free list can be converted to an llist, where a single-threaded consumer and a multi-threaded producer (rpcrdma_buffer_put) can both access the llist without the need for any serialization. This eliminates spin lock contention between the Receive completion handler and rpcrdma_buffer_get, and makes the rep consumer wait- free. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 106 ++++++++++++++++++--------------------- net/sunrpc/xprtrdma/xprt_rdma.h | 6 +- 2 files changed, 53 insertions(+), 59 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 52444c4d1be2..db90083ed35b 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -75,6 +75,7 @@ * internal functions */ static void rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc); +static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf); static void rpcrdma_mr_free(struct rpcrdma_mr *mr); @@ -407,7 +408,6 @@ rpcrdma_ia_remove(struct rpcrdma_ia *ia) struct rpcrdma_ep *ep = &r_xprt->rx_ep; struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_req *req; - struct rpcrdma_rep *rep; cancel_work_sync(&buf->rb_refresh_worker); @@ -431,8 +431,7 @@ rpcrdma_ia_remove(struct rpcrdma_ia *ia) /* The ULP is responsible for ensuring all DMA * mappings and MRs are gone. */ - list_for_each_entry(rep, &buf->rb_recv_bufs, rr_list) - rpcrdma_regbuf_dma_unmap(rep->rr_rdmabuf); + rpcrdma_reps_destroy(buf); list_for_each_entry(req, &buf->rb_allreqs, rl_all) { rpcrdma_regbuf_dma_unmap(req->rl_rdmabuf); rpcrdma_regbuf_dma_unmap(req->rl_sendbuf); @@ -1071,6 +1070,40 @@ static struct rpcrdma_rep *rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, return NULL; } +static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep) +{ + rpcrdma_regbuf_free(rep->rr_rdmabuf); + kfree(rep); +} + +static struct rpcrdma_rep *rpcrdma_rep_get_locked(struct rpcrdma_buffer *buf) +{ + struct llist_node *node; + + /* Calls to llist_del_first are required to be serialized */ + node = llist_del_first(&buf->rb_free_reps); + if (!node) + return NULL; + return llist_entry(node, struct rpcrdma_rep, rr_node); +} + +static void rpcrdma_rep_put(struct rpcrdma_buffer *buf, + struct rpcrdma_rep *rep) +{ + if (!rep->rr_temp) + llist_add(&rep->rr_node, &buf->rb_free_reps); + else + rpcrdma_rep_destroy(rep); +} + +static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf) +{ + struct rpcrdma_rep *rep; + + while ((rep = rpcrdma_rep_get_locked(buf)) != NULL) + rpcrdma_rep_destroy(rep); +} + /** * rpcrdma_buffer_create - Create initial set of req/rep objects * @r_xprt: transport instance to (re)initialize @@ -1106,7 +1139,7 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) } buf->rb_credits = 1; - INIT_LIST_HEAD(&buf->rb_recv_bufs); + init_llist_head(&buf->rb_free_reps); rc = rpcrdma_sendctxs_create(r_xprt); if (rc) @@ -1118,12 +1151,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) return rc; } -static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep) -{ - rpcrdma_regbuf_free(rep->rr_rdmabuf); - kfree(rep); -} - /** * rpcrdma_req_destroy - Destroy an rpcrdma_req object * @req: unused object to be destroyed @@ -1182,15 +1209,7 @@ rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) cancel_work_sync(&buf->rb_refresh_worker); rpcrdma_sendctxs_destroy(buf); - - while (!list_empty(&buf->rb_recv_bufs)) { - struct rpcrdma_rep *rep; - - rep = list_first_entry(&buf->rb_recv_bufs, - struct rpcrdma_rep, rr_list); - list_del(&rep->rr_list); - rpcrdma_rep_destroy(rep); - } + rpcrdma_reps_destroy(buf); while (!list_empty(&buf->rb_send_bufs)) { struct rpcrdma_req *req; @@ -1281,39 +1300,24 @@ rpcrdma_buffer_get(struct rpcrdma_buffer *buffers) */ void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req) { - struct rpcrdma_rep *rep = req->rl_reply; - + if (req->rl_reply) + rpcrdma_rep_put(buffers, req->rl_reply); req->rl_reply = NULL; spin_lock(&buffers->rb_lock); list_add(&req->rl_list, &buffers->rb_send_bufs); - if (rep) { - if (!rep->rr_temp) { - list_add(&rep->rr_list, &buffers->rb_recv_bufs); - rep = NULL; - } - } spin_unlock(&buffers->rb_lock); - if (rep) - rpcrdma_rep_destroy(rep); } -/* - * Put reply buffers back into pool when not attached to - * request. This happens in error conditions. +/** + * rpcrdma_recv_buffer_put - Release rpcrdma_rep back to free list + * @rep: rep to release + * + * Used after error conditions. */ -void -rpcrdma_recv_buffer_put(struct rpcrdma_rep *rep) +void rpcrdma_recv_buffer_put(struct rpcrdma_rep *rep) { - struct rpcrdma_buffer *buffers = &rep->rr_rxprt->rx_buf; - - if (!rep->rr_temp) { - spin_lock(&buffers->rb_lock); - list_add(&rep->rr_list, &buffers->rb_recv_bufs); - spin_unlock(&buffers->rb_lock); - } else { - rpcrdma_rep_destroy(rep); - } + rpcrdma_rep_put(&rep->rr_rxprt->rx_buf, rep); } /* Returns a pointer to a rpcrdma_regbuf object, or NULL. @@ -1469,22 +1473,10 @@ rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) /* fast path: all needed reps can be found on the free list */ wr = NULL; - spin_lock(&buf->rb_lock); while (needed) { - rep = list_first_entry_or_null(&buf->rb_recv_bufs, - struct rpcrdma_rep, rr_list); + rep = rpcrdma_rep_get_locked(buf); if (!rep) - break; - - list_del(&rep->rr_list); - rep->rr_recv_wr.next = wr; - wr = &rep->rr_recv_wr; - --needed; - } - spin_unlock(&buf->rb_lock); - - while (needed) { - rep = rpcrdma_rep_create(r_xprt, temp); + rep = rpcrdma_rep_create(r_xprt, temp); if (!rep) break; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 200d075bbe31..bd1befa83d24 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -47,6 +47,7 @@ #include /* atomic_t, etc */ #include /* struct kref */ #include /* struct work_struct */ +#include #include /* RDMA connection api */ #include /* RDMA verbs api */ @@ -200,7 +201,7 @@ struct rpcrdma_rep { struct rpc_rqst *rr_rqst; struct xdr_buf rr_hdrbuf; struct xdr_stream rr_stream; - struct list_head rr_list; + struct llist_node rr_node; struct ib_recv_wr rr_recv_wr; }; @@ -362,7 +363,6 @@ rpcrdma_mr_pop(struct list_head *list) struct rpcrdma_buffer { spinlock_t rb_lock; struct list_head rb_send_bufs; - struct list_head rb_recv_bufs; struct list_head rb_mrs; unsigned long rb_sc_head; @@ -373,6 +373,8 @@ struct rpcrdma_buffer { struct list_head rb_allreqs; struct list_head rb_all_mrs; + struct llist_head rb_free_reps; + u32 rb_max_requests; u32 rb_credits; /* most recent credit grant */ From patchwork Mon Aug 19 22:49:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102269 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D17514DB for ; Mon, 19 Aug 2019 22:49:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4AAA722CEC for ; Mon, 19 Aug 2019 22:49:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CdgpGeLS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728402AbfHSWtw (ORCPT ); Mon, 19 Aug 2019 18:49:52 -0400 Received: from mail-oi1-f196.google.com ([209.85.167.196]:40453 "EHLO mail-oi1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWtw (ORCPT ); Mon, 19 Aug 2019 18:49:52 -0400 Received: by mail-oi1-f196.google.com with SMTP id h21so2617442oie.7; Mon, 19 Aug 2019 15:49:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=sa44db4rf5pgBQCqMZ77sm8NvSn2T4kdYjbQifvbiGM=; b=CdgpGeLSgTnaWtglBd8rWGa1Nj6zfw6ULmn7Fc4I/TiF81OZHMDRZXrQjJ4/cySGSp 50gF8p645rmHxVp5YtHSM4YLVfBK9sEtG6AU8oRGooBl6HazhsWPbQUzd2pMFydguVux yNTWBuNeXXcUCTvjoh5Q8OPdYTBFcsyZo/C+VFJQhKch09NJwxvxc+y6LcEUtrNx0FKH 0f+h82gARixFh9NrKBd1UJjARMkFLO8h1WrJGYjEVfdo+gcLWLXTAKKPGL45yDp6pjds 6VXF7mvNawJRmOkEwQI8vGuIj4PPTf2PYj9DbBqX9TrkidZKmODD4LwkSWoWgk0JG91s fffg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=sa44db4rf5pgBQCqMZ77sm8NvSn2T4kdYjbQifvbiGM=; b=TjuxBs0IsrlAM5tDrvqVtbVqJQONk+gRZfhKimIfQWgq80AAlKH4RseJ+XdWpEWaAr 2xt6TDN/xyW1n+luNwr2hJgYV07fLzhLAugg8Mp5xTLFy6zdYUd53dWepZyHCijfuK6C YnYwa5sqVJYzJdV3SnKtDtp5w5ZTz5dOW5y2UpkKeIEGz1kGHqHdt8AYxTXaHx1pkx/M MY1BdRlZjX65hr9aLlgs4+NGcf0Qs5OQSD2lf/FHHtNLnqU6DTWz/yEMbhM/+sJ2/MpJ U9g8qTrRbfpE/DOuzJM+ETcgACnyN1nJB2hTMNEDYedbDdZMeerG5WicH60wAmjRSvVW m+Mw== X-Gm-Message-State: APjAAAWZALt6LttZjXYkTSK7+j67UIVKFT4Ol3vthr0XhReLUWv5WYJq W/ju+oNCe8RgWIIq2rlgn/AS0W/7 X-Google-Smtp-Source: APXvYqw6qZFPcALrR8AyC9WLJ0Lk3fSBFmBg1MajT4YBT2/MsKEaHThovtg81OXkFvBPwrskPGQjqQ== X-Received: by 2002:aca:37c5:: with SMTP id e188mr14292607oia.66.1566254991522; Mon, 19 Aug 2019 15:49:51 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id t18sm5848187otk.73.2019.08.19.15.49.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:49:51 -0700 (PDT) Subject: [PATCH v2 18/21] xprtrdma: Clean up xprt_rdma_set_connect_timeout() From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:49:30 -0400 Message-ID: <156625495023.8161.16052575877778151008.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Clean up: The function name should match the documenting comment. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/transport.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index f4763e8a6761..993b96ff6760 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -494,9 +494,9 @@ xprt_rdma_timer(struct rpc_xprt *xprt, struct rpc_task *task) * @reconnect_timeout: reconnect timeout after server disconnects * */ -static void xprt_rdma_tcp_set_connect_timeout(struct rpc_xprt *xprt, - unsigned long connect_timeout, - unsigned long reconnect_timeout) +static void xprt_rdma_set_connect_timeout(struct rpc_xprt *xprt, + unsigned long connect_timeout, + unsigned long reconnect_timeout) { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); @@ -805,7 +805,7 @@ static const struct rpc_xprt_ops xprt_rdma_procs = { .send_request = xprt_rdma_send_request, .close = xprt_rdma_close, .destroy = xprt_rdma_destroy, - .set_connect_timeout = xprt_rdma_tcp_set_connect_timeout, + .set_connect_timeout = xprt_rdma_set_connect_timeout, .print_stats = xprt_rdma_print_stats, .enable_swap = xprt_rdma_enable_swap, .disable_swap = xprt_rdma_disable_swap, From patchwork Mon Aug 19 22:50:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102273 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2130D14DB for ; Mon, 19 Aug 2019 22:50:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EFF282070B for ; Mon, 19 Aug 2019 22:50:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oIw1/8UE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728402AbfHSWuj (ORCPT ); Mon, 19 Aug 2019 18:50:39 -0400 Received: from mail-ot1-f68.google.com ([209.85.210.68]:40496 "EHLO mail-ot1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWuj (ORCPT ); Mon, 19 Aug 2019 18:50:39 -0400 Received: by mail-ot1-f68.google.com with SMTP id c34so3244198otb.7; Mon, 19 Aug 2019 15:50:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=QtZX51bpSlxp8tlege0urIMMQj34aAPxmb0sf5uCYXM=; b=oIw1/8UEGIxajiHg1UCPkV4cItEpw6oYHjHFsGLPyZ/b+w4n6AGOJN66MwdA2wfNlX 026K/4knO4J0Hx1tX0oomDjmomje2hhN6sIuoI3R4MO2nNKlPFDqLMwhY9ZEq7GlB1CF VWIcnCL+FMhXO7ObFk28L1naMyZqWemcZaWujndftCLFxRH9oyCaJg6hPdcaYj08OBL2 GV2rf4WQZ2mbTdlvHiByvFGopjQiFkIORzm6uPpRUUMp9xmIDXaAfvMjC1dwgdg5Y++a 3ob1wmlP87bezeNgiuXykYrFsqI2jfAJpo8hcTtp60QCONU0UH/2mQVLYJoXoIA3588X NGYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=QtZX51bpSlxp8tlege0urIMMQj34aAPxmb0sf5uCYXM=; b=IKeV/S3ddJwGDv0DzvXc8m4kbCQkqM9jhWN3zWInUw424to/7QMwxxQEELK6BywvRC RQsT+gseIHuLACqifg/WJfdKtWJZj/1Gq2N2cP3VPVx5o0HCEhKWcL1eqXH+M0EJJzyA UX/F02MdY85F7AfDMCYQN4I6OHBptri5j6OJS6ezf9laCQBW4xenzRmRoovkWaBA1Wkj HIV+1lZdWzr8/5Q5nfbIs122jtfUfDDz95EGEKBxujS8xRqFqP7pvPFt6EHtiLGsXEfk /U/BzhBisR2ylPdt1us3ZxdeixCI05puBvfrgpUqeaxoObl0ESpGK7UcH+D6pfyo5Le6 YEtw== X-Gm-Message-State: APjAAAVt8nKHf1qLVY5xLNptiT8WnHTjtjN2ZkI+UVy3w7UD0/tdeoPk M6B68wTzbTNtQnu1BRkNCf/qFu3m X-Google-Smtp-Source: APXvYqxfLChYNY3Dhx0o7TaSJ8oIEBkd0TN61/5xsGIMHvaGnYT9YgrGmVDbwHs4ZdRngej30RLs/w== X-Received: by 2002:a05:6830:1085:: with SMTP id y5mr19068922oto.214.1566255038223; Mon, 19 Aug 2019 15:50:38 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id w18sm5977745otk.22.2019.08.19.15.50.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:50:37 -0700 (PDT) Subject: [PATCH v2 19/21] xprtrdma: Fix bc_max_slots return value From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:50:16 -0400 Message-ID: <156625499660.8161.16372104759177233644.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org For the moment the returned value just happens to be correct because the current backchannel server implementation does not vary the number of credits it offers. The spec does permit this value to change during the lifetime of a connection, however. The actual maximum is fixed for all RPC/RDMA transports, because each transport instance has to pre-allocate the resources for processing BC requests. That's the value that should be returned. Fixes: 7402a4fedc2b ("SUNRPC: Fix up backchannel slot table ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 59e624b1d7a0..50e075fcdd8f 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -54,9 +54,7 @@ size_t xprt_rdma_bc_maxpayload(struct rpc_xprt *xprt) unsigned int xprt_rdma_bc_max_slots(struct rpc_xprt *xprt) { - struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - - return r_xprt->rx_buf.rb_bc_srv_max_requests; + return RPCRDMA_BACKWARD_WRS >> 1; } static int rpcrdma_bc_marshal_reply(struct rpc_rqst *rqst) From patchwork Mon Aug 19 22:51:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102277 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6ECCD912 for ; Mon, 19 Aug 2019 22:51:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4C7112070B for ; Mon, 19 Aug 2019 22:51:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nAmIvl5q" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728351AbfHSWvZ (ORCPT ); Mon, 19 Aug 2019 18:51:25 -0400 Received: from mail-ot1-f66.google.com ([209.85.210.66]:41424 "EHLO mail-ot1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWvZ (ORCPT ); Mon, 19 Aug 2019 18:51:25 -0400 Received: by mail-ot1-f66.google.com with SMTP id o101so3247357ota.8; Mon, 19 Aug 2019 15:51:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=TcWcDgT7uZqe+BWiSZ2azCfOJ01Li7gzQpK20RYlSx0=; b=nAmIvl5qLMEXHFzq2uP7oJfc5F2YZ0cbB95CmlQGTM6KRYIcoSC/W4UCWePPwZp4aC sWNJaEa2QhbbUo41uuX79q2MPwraDRvVF0AlLIGMa00yV+E99Tf9ozrN6v7fHKUc/Y81 7bTT3vkKBI3UKvC1PkI3SPhDgrJKLkBIBfvbzIK1HV1ezB7YidMd1ftf4Vz5IlGjOHYp 4SppFQQMez18ZLclzGmHS7o3XZI7NOX69xSc8l9PLE5VIG8f2s1PNf7Jr5EpJAVEKFvX zm8wNzNRqOT5LvzXnbEv4M569BtWeybN+MHYSmcIq2ya5fde+l+strdi+preZn5Vno0e bfjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=TcWcDgT7uZqe+BWiSZ2azCfOJ01Li7gzQpK20RYlSx0=; b=NA+g1nMBMv/FfGwBJCqaOq+oMCrVvocThPc8OFysXq8PNDiOJ8xGc1JGlw9zVUgAkG L/FPUxIK0R65WQbxRIZ8RG8BBWSnXOI7lOYuzEHnVySblSYgipR4CiaBVP7Pi5r4u+ER 4RA1p4ZbjjHuOKZTPRLCrexccvJ7d7OnUL0EOCxlx2Yosjqjystksi96/M9b5XPx65EG Used0C0B/kLP68Rq7BHoEhnirIDthjMM2yM9wmjgwDxRke4U42ToYSfu9nFUQYzyd5RZ 3ScZf4b8HQWGQzs5gTe58slC5UU+zDgu+c4TA6LXP7Q12XnAz5TuC+Y7D1Una4Sjlli6 AWvA== X-Gm-Message-State: APjAAAVB9kueE9pmu6ov5sH3bJCZcYeNnyPNHU32toDLPr0SlLDZ4w/q rIV5HqbrpW6zEWCGMeI/LGVutssr X-Google-Smtp-Source: APXvYqzC+p51PdJUZBXU6NStGNEpikVwuLpk21NTHt6doi40No7awd5HwPDUwFetKCPZSFPHoF1Lnw== X-Received: by 2002:a9d:4c0f:: with SMTP id l15mr2314616otf.138.1566255084702; Mon, 19 Aug 2019 15:51:24 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id d22sm4763839oig.38.2019.08.19.15.51.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:51:24 -0700 (PDT) Subject: [PATCH v2 20/21] xprtrdma: Inline XDR chunk encoder functions From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:51:03 -0400 Message-ID: <156625504326.8161.3307243638540897095.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Micro-optimization: Save the cost of three function calls during transport header encoding. These were "noinline" before to generate more meaningful call stacks during debugging, but this code is now pretty stable. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index ffeb4dfebd46..67e1684aee6d 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -382,9 +382,10 @@ static struct rpcrdma_mr_seg *rpcrdma_mr_prepare(struct rpcrdma_xprt *r_xprt, * * Only a single @pos value is currently supported. */ -static noinline int -rpcrdma_encode_read_list(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, - struct rpc_rqst *rqst, enum rpcrdma_chunktype rtype) +static int rpcrdma_encode_read_list(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct rpc_rqst *rqst, + enum rpcrdma_chunktype rtype) { struct xdr_stream *xdr = &req->rl_stream; struct rpcrdma_mr_seg *seg; @@ -436,9 +437,10 @@ rpcrdma_encode_read_list(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, * * Only a single Write chunk is currently supported. */ -static noinline int -rpcrdma_encode_write_list(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, - struct rpc_rqst *rqst, enum rpcrdma_chunktype wtype) +static int rpcrdma_encode_write_list(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct rpc_rqst *rqst, + enum rpcrdma_chunktype wtype) { struct xdr_stream *xdr = &req->rl_stream; struct rpcrdma_mr_seg *seg; @@ -498,9 +500,10 @@ rpcrdma_encode_write_list(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, * Returns zero on success, or a negative errno if a failure occurred. * @xdr is advanced to the next position in the stream. */ -static noinline int -rpcrdma_encode_reply_chunk(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, - struct rpc_rqst *rqst, enum rpcrdma_chunktype wtype) +static int rpcrdma_encode_reply_chunk(struct rpcrdma_xprt *r_xprt, + struct rpcrdma_req *req, + struct rpc_rqst *rqst, + enum rpcrdma_chunktype wtype) { struct xdr_stream *xdr = &req->rl_stream; struct rpcrdma_mr_seg *seg; From patchwork Mon Aug 19 22:51:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 11102281 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 424B214DB for ; Mon, 19 Aug 2019 22:52:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 21D5422CEC for ; Mon, 19 Aug 2019 22:52:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WxqFndBn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728484AbfHSWwM (ORCPT ); Mon, 19 Aug 2019 18:52:12 -0400 Received: from mail-oi1-f196.google.com ([209.85.167.196]:37091 "EHLO mail-oi1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728014AbfHSWwM (ORCPT ); Mon, 19 Aug 2019 18:52:12 -0400 Received: by mail-oi1-f196.google.com with SMTP id b25so2632858oib.4; Mon, 19 Aug 2019 15:52:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=XrEdW8AMlQpqsqIQf8IMdqgYvv8EcJ1wwexhKa1n9xE=; b=WxqFndBn57xHAtgEYnjGe/vywndmOIag0QWMQ99o6nShwbt0Hi+ybraZqHw2mPrccI 3VkOxDNVS+zmt1awXsMWM7cMO0vyZik1DTN7QB3vRCUaq1/7TSomVRstZrAk+Kn9aM2C 7z2u4rjhSEUdM6XpFQPEYvFGx1v8Sd+bMTHmTIjXzMpBte8PPKfw9q7l9qJk+3Ilh1jX 0qeULHBzWJJfLwPMxauazwxaai8NFeiGu6sbrp7aDb9pA9YZ6grZpJVtFMs3RJEhhfn8 fC0pCO5a0po36q3BO5WcXXvlvaPdHPzwxxBCt2tM/gFciKS+EtYtMZE4EW0aEt4SDULg Hp/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=XrEdW8AMlQpqsqIQf8IMdqgYvv8EcJ1wwexhKa1n9xE=; b=d3lV/Csb7dwTKCLH7kagDVZtk8C2eNhuu8rVFjpfglf35HZVTrRHeAmba4UOHXBhiT jAPIzs/5wYWaoQagKzq9KkKOsWjSvZNU4ylfzVbooMSfeLxpvHy8NaGgGK6H+Y+8+AfK J3KeDjjbHuo4K83VlfY64Upj3eOO2+KXKjLGQodL7OqG+svMhSf5TtXn/q4ZEyXVQPIO VXmFf8I2j8snj03fTPXam3NH5MU6689psg0H9flLaYmKKd2QLm8hh8BHQFp7l9jM9cKJ CbKKf5Y+xcUFJTMw8BCdcVUmWPD5TCNLbc+y4RyeThrbwme7/EtdNczklgBAvMz1gRH3 8YuA== X-Gm-Message-State: APjAAAVw8gyv/BEgMdEZ+Htcwpg5/RrMHhLtP2020S+coWx7C3ACY53s bwxnWlhRzeMruakZ49cFixCqZTn8 X-Google-Smtp-Source: APXvYqw5PFOKtgIUITMaymkDfEgHAbJJqWo2WcrMvJDhUipwniayzGr11OVe/BHS6gXPgGoi6FFrOQ== X-Received: by 2002:aca:ea0b:: with SMTP id i11mr14175212oih.102.1566255131200; Mon, 19 Aug 2019 15:52:11 -0700 (PDT) Received: from seurat29.1015granger.net ([12.235.16.3]) by smtp.gmail.com with ESMTPSA id c15sm5737736otf.35.2019.08.19.15.52.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 15:52:10 -0700 (PDT) Subject: [PATCH v2 21/21] xprtrdma: Optimize rpcrdma_post_recvs() From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Date: Mon, 19 Aug 2019 18:51:49 -0400 Message-ID: <156625508978.8161.1876379934869374429.stgit@seurat29.1015granger.net> In-Reply-To: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> References: <156625401091.8161.14744201497689200191.stgit@seurat29.1015granger.net> User-Agent: StGit/unknown-version MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Micro-optimization: In rpcrdma_post_recvs, since commit e340c2d6ef2a ("xprtrdma: Reduce the doorbell rate (Receive)"), the common case is to return without doing anything. Found with perf. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index db90083ed35b..ac2abf4578b9 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1465,7 +1465,7 @@ rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp) count = 0; needed = buf->rb_credits + (buf->rb_bc_srv_max_requests << 1); - if (ep->rep_receive_count > needed) + if (likely(ep->rep_receive_count > needed)) goto out; needed -= ep->rep_receive_count; if (!temp)