From patchwork Tue Aug 17 18:24:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chuck Lever III X-Patchwork-Id: 12441903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48F20C4320A for ; Tue, 17 Aug 2021 18:24:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2785261051 for ; Tue, 17 Aug 2021 18:24:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233075AbhHQSZ2 (ORCPT ); Tue, 17 Aug 2021 14:25:28 -0400 Received: from mail.kernel.org ([198.145.29.99]:46580 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232301AbhHQSZ0 (ORCPT ); Tue, 17 Aug 2021 14:25:26 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id D8AA4601FE; Tue, 17 Aug 2021 18:24:52 +0000 (UTC) Subject: [PATCH RFC] xprtrdma: Move initial Receive posting From: Chuck Lever To: haakon.bugge@oracle.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 17 Aug 2021 14:24:51 -0400 Message-ID: <162922469165.515267.14848799693507242987.stgit@manet.1015granger.net> User-Agent: StGit/1.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Håkon Bugge points out that rdma_create_qp() is not supposed to return a QP that is ready for Receives to be posted. It so happens that ours does that, but the IBTA spec (12.9.7.1) states that a transition to INIT happens only after REQ has been sent. In future kernels, QPs returned from rdma_create_qp() might not be in a state where posting Receives will succeed. This patch is a pre-requisite to changing the legacy behavior of rdma_create_qp(). The first available moment after cm_send_req where xprtrdma can post Receives is when the RDMA core reports the QP connection has been established. Note that xprtrdma has posted Receives just after rdma_create_qp() since 8d4fb8ff427a ("xprtrdma: Fix disconnect regression"). To avoid regressing 8d4fb8ff427a, xprtrdma needs to ensure that initial Receive WRs are posted before pending RPCs are awoken. It appears that the current logic does provide that guarantee. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index aaec3c9be8db..87ae62cdea18 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -520,12 +520,6 @@ int rpcrdma_xprt_connect(struct rpcrdma_xprt *r_xprt) xprt_clear_connected(xprt); rpcrdma_reset_cwnd(r_xprt); - /* Bump the ep's reference count while there are - * outstanding Receives. - */ - rpcrdma_ep_get(ep); - rpcrdma_post_recvs(r_xprt, 1, true); - rc = rdma_connect(ep->re_id, &ep->re_remote_cma); if (rc) goto out; @@ -539,6 +533,12 @@ int rpcrdma_xprt_connect(struct rpcrdma_xprt *r_xprt) goto out; } + /* Bump the ep's reference count while there are + * outstanding Receives. + */ + rpcrdma_ep_get(ep); + rpcrdma_post_recvs(r_xprt, 1, true); + rc = rpcrdma_sendctxs_create(r_xprt); if (rc) { rc = -ENOTCONN;