From patchwork Sat Jul 28 14:46:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10547987 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E17B514BC for ; Sat, 28 Jul 2018 14:46:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0E652B4F4 for ; Sat, 28 Jul 2018 14:46:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C4FE12B504; Sat, 28 Jul 2018 14:46:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48A492B4C1 for ; Sat, 28 Jul 2018 14:46:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729001AbeG1QNg (ORCPT ); Sat, 28 Jul 2018 12:13:36 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:53317 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728900AbeG1QNg (ORCPT ); Sat, 28 Jul 2018 12:13:36 -0400 Received: by mail-it0-f66.google.com with SMTP id 72-v6so11610829itw.3; Sat, 28 Jul 2018 07:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=ZPkKQP+tIH8Pd6LtbOQsNOdOhjQKFdHS97dbP3UaEAo=; b=ea2HY7SXzdJkUhkksdQVGFsBvi6Al+nPsPf7keTgFab/MDL+9E8yUtO+iCPJot2gJP HPOwTVKRsa8+o+m2qcb5HpW1ZO4AcLAzuEjIsymNuOCbfp3u0DecpRht+JdFI7J8aXxU 7MFVkUi9q4XXI5zA+LMfHj4W3yCqRSrLcbw566mKw8ReaChIDLnds5VlUZr6weM2EdHc ocuNVTrklaZ/k3qFBvoQFGNPpaan6gi+7jqHccW9OJSeKhrCKElxUl3hpYi0ij2dAI0g J9MNAbKg7SuTvcUQsYne6FCd7l5du5AEm3AVKMkzam8L7PnRo2q/VyApDQoPw6T9GbF3 WwHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :user-agent:mime-version:content-transfer-encoding; bh=ZPkKQP+tIH8Pd6LtbOQsNOdOhjQKFdHS97dbP3UaEAo=; b=qayvngpeJ3L1f9A9zhZ8cF5dqKkTCnoZCDqxKoj5zjFS1f3W8m2BQt0w9NOjAbCyA8 Tk+PG6dgEW7D8fe+MYDMeitHd62/rrxmXJ8uTXfcCz+5FoQIAKavz0j2ydb5BY9D9Y6K BVOE5aheOfQQH2sTRQMBIROdScVjUsGPa3CiCEZnoYiuax1HNPC1neOrXO1BjrwdA6cC T+PfMIjt6D6fSBB416x85j5dzThVaxW5R9gwD123MqhJYYisD4reb1cpJkEnDBQMDze9 4Esbk9acpbfTiHhNQbN6HXC+IctfZHFffgBp8x+knlrGg4ZeWnqfgkKj8JnIEe0AV/SY pQRw== X-Gm-Message-State: AOUpUlGhQP6dm3Fhg6waPY7VG//STzbUynroqhxAQKL24q9uRjUrzOc+ bIUv46vuaphaVblWvt0jzpsrMna+ X-Google-Smtp-Source: AAOMgpctFNOca8BZVfX2dUxZkm29AfkJlpwDo4ll3z4bBLmc4u//LHvvUYFB7DS1QCy/eikDx8FkWA== X-Received: by 2002:a24:ce41:: with SMTP id v62-v6mr8569540itg.57.1532789210373; Sat, 28 Jul 2018 07:46:50 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id a7-v6sm2102995ioc.25.2018.07.28.07.46.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Jul 2018 07:46:49 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id w6SEkl1Q001926; Sat, 28 Jul 2018 14:46:48 GMT Subject: [PATCH] xprtrdma: Fix disconnect regression From: Chuck Lever To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Sat, 28 Jul 2018 10:46:47 -0400 Message-ID: <20180728144051.3612.82362.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP I found that injecting disconnects with v4.18-rc resulted in random failures of the multi-threaded git regression test. The root cause appears to be that, after a reconnect, the RPC/RDMA transport is waking pending RPCs before the transport has posted enough Receive buffers to receive the Replies. If a Reply arrives before enough Receive buffers are posted, the connection is dropped. A few connection drops happen in quick succession as the client and server struggle to regain credit synchronization. This regression was introduced with commit 7c8d9e7c8863 ("xprtrdma: Move Receive posting to Receive handler"). The client is supposed to post a single Receive when a connection is established because it's not supposed to send more than one RPC Call before it gets a fresh credit grant in the first RPC Reply [RFC 8166, Section 3.3.3]. Unfortunately there appears to be a longstanding bug in the Linux client's credit accounting mechanism. On connect, it simply dumps all pending RPC Calls onto the new connection. It's possible it has done this ever since the RPC/RDMA transport was added to the kernel ten years ago. Servers have so far been tolerant of this bad behavior. Currently no server implementation ever changes its credit grant over reconnects, and servers always repost enough Receives before connections are fully established. The Linux client implementation used to post a Receive before each of these Calls. This has covered up the flooding send behavior. I could try to correct this old bug so that the client sends exactly one RPC Call and waits for a Reply. Since we are so close to the next merge window, I'm going to instead provide a simple patch to post enough Receives before a reconnect completes (based on the number of credits granted to the previous connection). The spurious disconnects will be gone, but the client will still send multiple RPC Calls immediately after a reconnect. Addressing the latter problem will wait for a merge window because a) I expect it to be a large change requiring lots of testing, and b) obviously the Linux client has interoperated successfully since day zero while still being broken. Fixes: 7c8d9e7c8863 ("xprtrdma: Move Receive posting to ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) Hi Anna- Would it be possible to get this into linux-next and then v4.18-rc ? I know this is very late, but I found this issue only recently and it took a few days to nail down a simple fix. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 16161a3..e8d1024 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -280,7 +280,6 @@ ++xprt->rx_xprt.connect_cookie; connstate = -ECONNABORTED; connected: - xprt->rx_buf.rb_credits = 1; ep->rep_connected = connstate; rpcrdma_conn_func(ep); wake_up_all(&ep->rep_connect_wait); @@ -755,6 +754,7 @@ } ep->rep_connected = 0; + rpcrdma_post_recvs(r_xprt, true); rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma); if (rc) { @@ -773,8 +773,6 @@ dprintk("RPC: %s: connected\n", __func__); - rpcrdma_post_recvs(r_xprt, true); - out: if (rc) ep->rep_connected = rc; @@ -1171,6 +1169,7 @@ struct rpcrdma_req * list_add(&req->rl_list, &buf->rb_send_bufs); } + buf->rb_credits = 1; buf->rb_posted_receives = 0; INIT_LIST_HEAD(&buf->rb_recv_bufs);