From patchwork Wed Sep 12 20:54:14 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trond Myklebust X-Patchwork-Id: 1446611 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 06D853FCFC for ; Wed, 12 Sep 2012 20:54:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753704Ab2ILUyQ (ORCPT ); Wed, 12 Sep 2012 16:54:16 -0400 Received: from mx2.netapp.com ([216.240.18.37]:13579 "EHLO mx2.netapp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753330Ab2ILUyP (ORCPT ); Wed, 12 Sep 2012 16:54:15 -0400 X-IronPort-AV: E=Sophos;i="4.80,413,1344236400"; d="scan'208";a="688905244" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 12 Sep 2012 13:54:15 -0700 Received: from vmwexceht02-prd.hq.netapp.com (vmwexceht02-prd.hq.netapp.com [10.106.76.240]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8CKsEdN007291; Wed, 12 Sep 2012 13:54:15 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.158]) by vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) with mapi id 14.02.0309.002; Wed, 12 Sep 2012 13:54:14 -0700 From: "Myklebust, Trond" To: Simon Kirby CC: Yan-Pai Chen , "linux-nfs@vger.kernel.org" Subject: Re: [3.2.5] NFSv3 CLOSE_WAIT hang Thread-Topic: [3.2.5] NFSv3 CLOSE_WAIT hang Thread-Index: AQHNizvPivRwaVteykiGe6jCimUevZd8T8YAgAKbN9eAAAPSIIAABmFGgACJk4CAAVydkoACdKLQgAK5kYCAACu+AIABexgA Date: Wed, 12 Sep 2012 20:54:14 +0000 Message-ID: <4FA345DA4F4AE44899BD2B03EEEC2FA908F9F734@SACEXCMBX04-PRD.hq.netapp.com> References: <4FA345DA4F4AE44899BD2B03EEEC2FA908F86381@SACEXCMBX04-PRD.hq.netapp.com> <6cb9.5049fd40.b47c1@altium.nl> <6cb9.5049fd40.b47c1@altium.nl> <4FA345DA4F4AE44899BD2B03EEEC2FA908F8E302@SACEXCMBX04-PRD.hq.netapp.com> <447c.504a05c9.dd0a9@altium.nl> <447c.504a05c9.dd0a9@altium.nl> <4FA345DA4F4AE44899BD2B03EEEC2FA908F8E833@SACEXCMBX04-PRD.hq.netapp.com> <74c7.504b9d45.a5956@altium.nl> <20120911194051.GB11160@hostway.ca> <1347401844.15208.17.camel@lade.trondhjem.org> In-Reply-To: <1347401844.15208.17.camel@lade.trondhjem.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.104.60.116] Content-ID: <5120E7C3BAC53747A125D5AA54DE7A3D@tahoe.netapp.com> MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org On Tue, 2012-09-11 at 18:17 -0400, Trond Myklebust wrote: > On Tue, 2012-09-11 at 12:40 -0700, Simon Kirby wrote: > > Hello! > > > > This problem still bites us rarely, and we've been using TCP NFS for some > > time. However, our case seems to be narrowed it down to a very long > > storage hang on the knfsd side. If storage never has any problems, we > > don't see the NFS client hang. I was going to try to make a test-case by > > forcing the server to hang, but I never got around to this. Meanwhile, > > I've been running the clients with the debugging patches I posted > > earlier, and it always prints the 'xprt_force_disconnect(): setting > > XPRT_CLOSE_WAIT" warning before hanging. If Apache is in sendfile() at > > the time, it seems to get stuck forever; otherwise, it might recover. > > Does the "if (test_and_set_bit(XPRT_LOCK) == 0)" condition immediately > following that succeed so that queue_work() is called? > > > http://www.spinics.net/lists/linux-nfs/msg29495.html > > http://0x.ca/sim/ref/3.2.10/dmesg > > > > I suppose we could try 3.5 at this point. > > If you've been keeping up with the 3.2 stable releases, then I wouldn't > expect any major differences to the sunrpc code, but it might be worth a > try in case the networking layer has changed. Hi Simon, Can you try the following patch, and see if it addresses the TCP "server hangs" case? Cheers Trond 8<---------------------------------------------------------------------- From 99330d09cc1074fbdc64089fa0a3f8dbdc74daaf Mon Sep 17 00:00:00 2001 From: Trond Myklebust Date: Wed, 12 Sep 2012 16:49:15 -0400 Subject: [PATCH] SUNRPC: Ensure that the TCP socket is closed when in CLOSE_WAIT Instead of doing a shutdown() call, we need to do an actual close(). Ditto if/when the server is sending us junk RPC headers. Signed-off-by: Trond Myklebust --- net/sunrpc/xprtsock.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) -- 1.7.11.4 -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index a35b8e5..d1988cf 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1025,6 +1025,16 @@ static void xs_udp_data_ready(struct sock *sk, int len) read_unlock_bh(&sk->sk_callback_lock); } +/* + * Helper function to force a TCP close if the server is sending + * junk and/or it has put us in CLOSE_WAIT + */ +static void xs_tcp_force_close(struct rpc_xprt *xprt) +{ + set_bit(XPRT_CONNECTION_CLOSE, &xprt->state); + xprt_force_disconnect(xprt); +} + static inline void xs_tcp_read_fraghdr(struct rpc_xprt *xprt, struct xdr_skb_reader *desc) { struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); @@ -1051,7 +1061,7 @@ static inline void xs_tcp_read_fraghdr(struct rpc_xprt *xprt, struct xdr_skb_rea /* Sanity check of the record length */ if (unlikely(transport->tcp_reclen < 8)) { dprintk("RPC: invalid TCP record fragment length\n"); - xprt_force_disconnect(xprt); + xs_tcp_force_close(xprt); return; } dprintk("RPC: reading TCP record fragment of length %d\n", @@ -1132,7 +1142,7 @@ static inline void xs_tcp_read_calldir(struct sock_xprt *transport, break; default: dprintk("RPC: invalid request message type\n"); - xprt_force_disconnect(&transport->xprt); + xs_tcp_force_close(&transport->xprt); } xs_tcp_check_fraghdr(transport); } @@ -1455,6 +1465,8 @@ static void xs_tcp_cancel_linger_timeout(struct rpc_xprt *xprt) static void xs_sock_mark_closed(struct rpc_xprt *xprt) { smp_mb__before_clear_bit(); + clear_bit(XPRT_CONNECTION_ABORT, &xprt->state); + clear_bit(XPRT_CONNECTION_CLOSE, &xprt->state); clear_bit(XPRT_CLOSE_WAIT, &xprt->state); clear_bit(XPRT_CLOSING, &xprt->state); smp_mb__after_clear_bit(); @@ -1512,8 +1524,8 @@ static void xs_tcp_state_change(struct sock *sk) break; case TCP_CLOSE_WAIT: /* The server initiated a shutdown of the socket */ - xprt_force_disconnect(xprt); xprt->connect_cookie++; + xs_tcp_force_close(xprt); case TCP_CLOSING: /* * If the server closed down the connection, make sure that @@ -2199,8 +2211,7 @@ static void xs_tcp_setup_socket(struct work_struct *work) /* We're probably in TIME_WAIT. Get rid of existing socket, * and retry */ - set_bit(XPRT_CONNECTION_CLOSE, &xprt->state); - xprt_force_disconnect(xprt); + xs_tcp_force_close(xprt); break; case -ECONNREFUSED: case -ECONNRESET: