From patchwork Mon Aug 27 23:29:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10577655 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 693C913B8 for ; Mon, 27 Aug 2018 23:29:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BED429C23 for ; Mon, 27 Aug 2018 23:29:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5FCAD29C54; Mon, 27 Aug 2018 23:29:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F32C829C26 for ; Mon, 27 Aug 2018 23:29:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727317AbeH1DSU (ORCPT ); Mon, 27 Aug 2018 23:18:20 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:52901 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727305AbeH1DST (ORCPT ); Mon, 27 Aug 2018 23:18:19 -0400 Received: by mail-it0-f65.google.com with SMTP id h3-v6so61311ita.2; Mon, 27 Aug 2018 16:29:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=8xx20eRdGA6Tcm9wuLWo8j5vysn0IE/m/rt2lpFWZnw=; b=E4GtDjVOauSAPFQSIWwKvVGMUgH8cwdZ/ujV0+xP/u1HT46+6jhgN7gqwmmpwDixsO +5bdSoAvapH4KNwtv/bSsnOSMVizj/cgqrMflGDZ40nUwgLtPnxARkM7rdLPArAf9DGn OfWABLI5uVR+cx1cHL4RmC+JUVpiIMv/w0aHHViSsD4fbKguPRlRREokAfEbrVCFXW1u 2fVibw1V2G2qZZgN5PezNUtEdS0QPMYwEQ7GwCbLXXfKV8ax/JovQnSVydtJgquFROpt QrElwGN6DBu0nDpUvpe+fRzX3j+3VALnXxtst1M98TrVenyEmwQBD4AseDn0ANZR5mmT Gfzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :user-agent:mime-version:content-transfer-encoding; bh=8xx20eRdGA6Tcm9wuLWo8j5vysn0IE/m/rt2lpFWZnw=; b=P6DYAaW4ruFfDWdPS7ChOYUidTebXTjyX5/oG27zGT9IvpaWN09fTti1QIImcWLMTL TOCSCnnaM0ukz+B2YHbOOLlrRMQ0zA991hOsecqpRy8pAZAZNqhqyBcush2Zeknr4HWS JlZFpDbc8G/CNQnq52v7pggbOVZOFL8O+NvyEhI06H3vNhjh8M7xgRR/Y9Y+uIA/ZB/7 Dxu4uuWWp8WyriqDVGfILidjBDitymX8n8lflqJJ1uflAVFc6dY7N9N3X4YNlUlyulhn GoYoKG0NrYg1fKWIdqo8RsPuHaibX1Fge1jdsDf4xV7BtkDy4lnDF4wd8Lvsih71wE3B i9VA== X-Gm-Message-State: APzg51AIKqr9muBJm9ru7c2KdIDH6V/vDFXdxtKi0uH2GQFZ7XsL1GGy HaC4GmTmmCGCUmNPQFBJyYjEghO5 X-Google-Smtp-Source: ANB0VdZzV6FLL5i+iQl4p3wQX/JL6OfJqnmspK86v1zhdxlwjczYk/JvlXnBP+n6EIpcROJ6XDYvIA== X-Received: by 2002:a02:4009:: with SMTP id n9-v6mr12732142jaa.19.1535412570058; Mon, 27 Aug 2018 16:29:30 -0700 (PDT) Received: from gateway.1015granger.net (c-68-61-232-219.hsd1.mi.comcast.net. [68.61.232.219]) by smtp.gmail.com with ESMTPSA id c3-v6sm239805itd.8.2018.08.27.16.29.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Aug 2018 16:29:29 -0700 (PDT) Received: from manet.1015granger.net (manet.1015granger.net [192.168.1.51]) by gateway.1015granger.net (8.14.7/8.14.7) with ESMTP id w7RNTRTe007918; Mon, 27 Aug 2018 23:29:28 GMT Subject: [PATCH] xprtrdma: Fix disconnect regression From: Chuck Lever To: stable@vger.kernel.org Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 27 Aug 2018 19:29:27 -0400 Message-ID: <20180827232321.12635.40263.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP I found that injecting disconnects with v4.18-rc resulted in random failures of the multi-threaded git regression test. The root cause appears to be that, after a reconnect, the RPC/RDMA transport is waking pending RPCs before the transport has posted enough Receive buffers to receive the Replies. If a Reply arrives before enough Receive buffers are posted, the connection is dropped. A few connection drops happen in quick succession as the client and server struggle to regain credit synchronization. This regression was introduced with commit 7c8d9e7c8863 ("xprtrdma: Move Receive posting to Receive handler"). The client is supposed to post a single Receive when a connection is established because it's not supposed to send more than one RPC Call before it gets a fresh credit grant in the first RPC Reply [RFC 8166, Section 3.3.3]. Unfortunately there appears to be a longstanding bug in the Linux client's credit accounting mechanism. On connect, it simply dumps all pending RPC Calls onto the new connection. It's possible it has done this ever since the RPC/RDMA transport was added to the kernel ten years ago. Servers have so far been tolerant of this bad behavior. Currently no server implementation ever changes its credit grant over reconnects, and servers always repost enough Receives before connections are fully established. The Linux client implementation used to post a Receive before each of these Calls. This has covered up the flooding send behavior. I could try to correct this old bug so that the client sends exactly one RPC Call and waits for a Reply. Since we are so close to the next merge window, I'm going to instead provide a simple patch to post enough Receives before a reconnect completes (based on the number of credits granted to the previous connection). The spurious disconnects will be gone, but the client will still send multiple RPC Calls immediately after a reconnect. Addressing the latter problem will wait for a merge window because a) I expect it to be a large change requiring lots of testing, and b) obviously the Linux client has interoperated successfully since day zero while still being broken. Fixes: 7c8d9e7c8863 ("xprtrdma: Move Receive posting to ... ") Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/verbs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) Hi stable@ - This fix has been merged into v4.19 as upstream commit 8d4fb8ff427a ("xprtrdma: Fix disconnect regression"). It addresses a regression in v4.18. I expected it to go into late v4.18-rc, which is why there is no "cc: stable" on the original submission. Could you please apply it to 4.18.y ? Thank you! diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 042bb24..1386c6e 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -279,7 +279,6 @@ ++xprt->rx_xprt.connect_cookie; connstate = -ECONNABORTED; connected: - xprt->rx_buf.rb_credits = 1; ep->rep_connected = connstate; rpcrdma_conn_func(ep); wake_up_all(&ep->rep_connect_wait); @@ -754,6 +753,7 @@ } ep->rep_connected = 0; + rpcrdma_post_recvs(r_xprt, true); rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma); if (rc) { @@ -772,8 +772,6 @@ dprintk("RPC: %s: connected\n", __func__); - rpcrdma_post_recvs(r_xprt, true); - out: if (rc) ep->rep_connected = rc; @@ -1170,6 +1168,7 @@ struct rpcrdma_req * list_add(&req->rl_list, &buf->rb_send_bufs); } + buf->rb_credits = 1; buf->rb_posted_receives = 0; INIT_LIST_HEAD(&buf->rb_recv_bufs);