From patchwork Fri Sep 19 01:54:56 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Baron X-Patchwork-Id: 4935171 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id D599FBEEA5 for ; Fri, 19 Sep 2014 01:55:01 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id F326720166 for ; Fri, 19 Sep 2014 01:55:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CDB2D20122 for ; Fri, 19 Sep 2014 01:54:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750895AbaISBy6 (ORCPT ); Thu, 18 Sep 2014 21:54:58 -0400 Received: from prod-mail-xrelay02.akamai.com ([72.246.2.14]:52878 "EHLO prod-mail-xrelay02.akamai.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750732AbaISBy6 (ORCPT ); Thu, 18 Sep 2014 21:54:58 -0400 Received: from prod-mail-xrelay02.akamai.com (localhost [127.0.0.1]) by postfix.imss70 (Postfix) with ESMTP id 1C9DC28520; Fri, 19 Sep 2014 01:54:57 +0000 (GMT) Received: from prod-mail-relay06.akamai.com (prod-mail-relay06.akamai.com [172.17.120.126]) by prod-mail-xrelay02.akamai.com (Postfix) with ESMTP id F38C42851E; Fri, 19 Sep 2014 01:54:56 +0000 (GMT) Received: from [0.0.0.0] (callahan.kendall.corp.akamai.com [172.17.12.11]) by prod-mail-relay06.akamai.com (Postfix) with ESMTP id BF2092027; Fri, 19 Sep 2014 01:54:56 +0000 (GMT) Message-ID: <541B8CF0.3090502@akamai.com> Date: Thu, 18 Sep 2014 21:54:56 -0400 From: Jason Baron User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Trond Myklebust CC: Bruce Fields , Linux NFS Mailing List Subject: Re: [PATCH 2/2] rpc: Add -EPERM processing for xs_udp_send_request() References: <541B484E.90202@akamai.com> In-Reply-To: Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 09/18/2014 05:20 PM, Trond Myklebust wrote: > On Thu, Sep 18, 2014 at 5:02 PM, Jason Baron wrote: >> On 09/18/2014 04:51 PM, Trond Myklebust wrote: >>> On Thu, Sep 18, 2014 at 3:51 PM, Jason Baron wrote: >>>> If an iptables drop rule is added for an nfs server, the client can end up in >>>> a softlockup. Because of the way that xs_sendpages() is structured, the -EPERM >>>> is ignored since the prior bits of the packet may have been successfully queued >>>> and thus xs_sendpages() returns a non-zero value. Then, xs_udp_send_request() >>>> thinks that because some bits were queued it should return -EAGAIN. We then try >>>> the request and again and a softlockup occurs. The test sequence is simply: >>>> >>>> 1) open a file on the nfs server '/nfs/foo' (mounted using udp) >>>> 2) iptables -A OUTPUT -d -j DROP >>>> 3) write to /nfs/foo >>>> 4) close /nfs/foo >>>> 5) iptables -D OUTPUT -d -j DROP >>>> >>>> The softlockup occurs in step 4 above. >>> For UDP, the expected and documented behaviour in the case above is as follows: >>> - if the mount is soft, then return EIO on the first major timeout. >> yeah - so this case is a softlockup in my testing :( >> >>> - if the mount is hard, then retry indefinitely on timeout. >>> >>> Won't these 2 patches end up propagating an EPERM to the application? >>> That would be a definite violation of both hard and soft semantics. >> ok, yeah it does propogate the -EPERM up - I wasn't aware of the correct >> semantics - thanks. >> >> I can rework the patches such that they return -EIO instead for a soft mount, >> and verify that we keep retrying for a hard one. >> > Doesn't the soft timeout currently trigger after the major timeout? If > not, do we understand why it isn't doing so? No, the soft timeout does not currently trigger after the major timeout. Instead, the kernel spins indefinitely, and triggers a softlockup. The reason is that xs_sendpages() returns a positive value in this case and xs_udp_send_request() turns it in an -EAGAIN for the write operation. Subsequently, we call call_transmit_status() and then call_status() which sees the EAGAIN, which just starts all over again with a 'call_transmit()'. So we are stuck spinning indefinitely in kernel space. Simply moving the -EPERM up in this patch, results in the behavior you described above - EIO after a major timeout on a soft mount, and indefinte retries on a hard mount - but without the cpu consumption. IE applying this on top of this patch: We could also 'translate' the -EPERM into an '-ENETUNREACH' or such, in the return from xs_udp_send_request(), if you think that would make more sense? Hopefully, I've explained things better. Thanks, -Jason --- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -2019,6 +2019,7 @@ call_status(struct rpc_task *task) case -EHOSTDOWN: case -EHOSTUNREACH: case -ENETUNREACH: + case -EPERM: if (RPC_IS_SOFTCONN(task)) { rpc_exit(task, status); break; @@ -2048,7 +2049,6 @@ call_status(struct rpc_task *task) case -EAGAIN: task->tk_action = call_transmit; break; - case -EPERM: case -EIO: /* shutdown or soft timeout */ rpc_exit(task, status);