From patchwork Fri Dec 15 19:15:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dai Ngo X-Patchwork-Id: 13494803 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 935EE3FE2F for ; Fri, 15 Dec 2023 19:17:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="RDcCwtW7" Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFI48Pq004535; Fri, 15 Dec 2023 19:15:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=FMO8+QZuPvsTjEiPng0xTvGLmJ61p5PwJT51KIIg2nI=; b=RDcCwtW78tGkYYfBXKGReHSB3YsncvYmnbuJj4wbErM3GotpthwOE5rvraeWoEiIaynd WTEA6qsiqHnK1yx6c8adsHlpNEomDpG/opMM1V1wED2PGNZvUcNJgrfmf75nnaDy/wBG oqxpoK7bu5g3TgvCWcwqqp10RuoEEETbvQTOhDW4Pbk+ncRIcxJkhHZMNsO3YIP+3iUW 9Mc5owPHDY9OWQozIxl/v/fdFsYsMopCkTwgwDTZyAWjfNeZiu/Ku3KxbHcPF0rBm+JT R9fdQy609iFYRK2Vz9L3cGaHZrDXsrtS0/Fc8KZma6YThhIht9yPRiSILStSMitkC3xl VA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3uvfuue3ss-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 19:15:25 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFIA2aF016620; Fri, 15 Dec 2023 19:15:24 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3uvepcfqnc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 19:15:24 +0000 Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3BFJEAbS013855; Fri, 15 Dec 2023 19:15:23 GMT Received: from ca-common-hq.us.oracle.com (ca-common-hq.us.oracle.com [10.211.9.209]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3uvepcfqmh-2; Fri, 15 Dec 2023 19:15:23 +0000 From: Dai Ngo To: chuck.lever@oracle.com, jlayton@kernel.org Cc: linux-nfs@vger.kernel.org, linux-nfs@stwm.de Subject: [PATCH 1/3] SUNRPC: remove printk when back channel request not found Date: Fri, 15 Dec 2023 11:15:01 -0800 Message-Id: <1702667703-17978-2-git-send-email-dai.ngo@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com> References: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-15_10,2023-12-14_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312150135 X-Proofpoint-GUID: _GNp4umQKwFrhmFQoT6xL_px2UcfehMF X-Proofpoint-ORIG-GUID: _GNp4umQKwFrhmFQoT6xL_px2UcfehMF Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: If the client interface is down, or there is a network partition between the client and server, that prevents the callback request to reach the client TCP on the server will keep re-transmitting the callback for about ~9 minutes before giving up and closes the connection. If the connection between the client and the server is re-established before the connection is closed and after the callback timed out (9 secs) then the re-transmitted callback request will arrive at the client. When the server receives the reply of the callback, receive_cb_reply prints the "Got unrecognized reply..." message in the system log since the callback request was already removed from the server xprt's recv_queue. Even though this scenario has no effect on the server operation, a malicious client can take advantage of this behavior and send thousand of callback replies with random XIDs to fill up the server's system log. Signed-off-by: Dai Ngo Reviewed-by: Jeff Layton --- net/sunrpc/svcsock.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 998687421fa6..3e89dc0afbef 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -1060,7 +1060,7 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp) spin_lock(&bc_xprt->queue_lock); req = xprt_lookup_rqst(bc_xprt, xid); if (!req) - goto unlock_notfound; + goto unlock_eagain; memcpy(&req->rq_private_buf, &req->rq_rcv_buf, sizeof(struct xdr_buf)); /* @@ -1077,12 +1077,6 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp) rqstp->rq_arg.len = 0; spin_unlock(&bc_xprt->queue_lock); return 0; -unlock_notfound: - printk(KERN_NOTICE - "%s: Got unrecognized reply: " - "calldir 0x%x xpt_bc_xprt %p xid %08x\n", - __func__, ntohl(calldir), - bc_xprt, ntohl(xid)); unlock_eagain: spin_unlock(&bc_xprt->queue_lock); return -EAGAIN; From patchwork Fri Dec 15 19:15:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dai Ngo X-Patchwork-Id: 13494802 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 933A43FE2C for ; Fri, 15 Dec 2023 19:17:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="mZJ+Vzbp" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFI49iE022609; Fri, 15 Dec 2023 19:15:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=n4ujrx9voqvIVS/sjBPNAir6ur/tgh0jeTvzuCmTNo4=; b=mZJ+VzbpfbgiWt8dZD/Ky2rB+LYPxB7nJSsSprzfJjhxx4yQHiRMz5Fpzn8cLnd6C313 dcc6Io9TSJJBdu8chhe6y7VgDEQ87lCCup3riS7AhCV/2LUF68UDXdLjDl0b57bdijxL D0CSETfsAm4NAkrJnXPjKmOTTfZSzPeVwUumfMwURQnXTpKuPqp295EZAunlYyTZOrO2 dfawFfOt4dmEgMLQJXVS4T63tFLfzuncKA6E72Z5Rb1JPhTyWfVSaJt7aj8bbh7+tuz0 jPaRaNjktOEYo1GOfc1woGhDifshjkaGIHrx/vHZ6eVdNBQ5fcPIfF1Lw6zd2WfrmNkm kw== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3uvg9de7xr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 19:15:25 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFIJI7O016695; Fri, 15 Dec 2023 19:15:24 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3uvepcfqnu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 19:15:24 +0000 Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3BFJEAbU013855; Fri, 15 Dec 2023 19:15:24 GMT Received: from ca-common-hq.us.oracle.com (ca-common-hq.us.oracle.com [10.211.9.209]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3uvepcfqmh-3; Fri, 15 Dec 2023 19:15:24 +0000 From: Dai Ngo To: chuck.lever@oracle.com, jlayton@kernel.org Cc: linux-nfs@vger.kernel.org, linux-nfs@stwm.de Subject: [PATCH 2/3] NFSD: restore delegation's sc_count if nfsd4_run_cb fails Date: Fri, 15 Dec 2023 11:15:02 -0800 Message-Id: <1702667703-17978-3-git-send-email-dai.ngo@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com> References: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-15_10,2023-12-14_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312150135 X-Proofpoint-GUID: 5RL8H_Rni1u-eSEmEczIdcd_isvvHk3Y X-Proofpoint-ORIG-GUID: 5RL8H_Rni1u-eSEmEczIdcd_isvvHk3Y Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Under some load conditions the callback work request can not be queued and nfsd4_run_cb returns 0 to caller. When this happens, the sc_count of the delegation state was left with an extra reference count preventing the state to be freed later. Signed-off-by: Dai Ngo --- fs/nfsd/nfs4state.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 40415929e2ae..175f3e9f5822 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -2947,8 +2947,14 @@ void nfs4_cb_getattr(struct nfs4_cb_fattr *ncf) if (test_and_set_bit(CB_GETATTR_BUSY, &ncf->ncf_cb_flags)) return; + refcount_inc(&dp->dl_stid.sc_count); - nfsd4_run_cb(&ncf->ncf_getattr); + if (!nfsd4_run_cb(&ncf->ncf_getattr)) { + refcount_dec(&dp->dl_stid.sc_count); + clear_bit(CB_GETATTR_BUSY, &ncf->ncf_cb_flags); + wake_up_bit(&ncf->ncf_cb_flags, CB_GETATTR_BUSY); + WARN_ON_ONCE(1); + } } static struct nfs4_client *create_client(struct xdr_netobj name, @@ -4967,7 +4973,10 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp) * we know it's safe to take a reference. */ refcount_inc(&dp->dl_stid.sc_count); - WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall)); + if (!nfsd4_run_cb(&dp->dl_recall)) { + refcount_dec(&dp->dl_stid.sc_count); + WARN_ON_ONCE(1); + } } /* Called from break_lease() with flc_lock held. */ @@ -8543,12 +8552,12 @@ nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct inode *inode, return 0; } break_lease: - spin_unlock(&ctx->flc_lock); nfsd_stats_wdeleg_getattr_inc(); - dp = fl->fl_owner; ncf = &dp->dl_cb_fattr; nfs4_cb_getattr(&dp->dl_cb_fattr); + spin_unlock(&ctx->flc_lock); + wait_on_bit(&ncf->ncf_cb_flags, CB_GETATTR_BUSY, TASK_INTERRUPTIBLE); if (ncf->ncf_cb_status) { status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ)); From patchwork Fri Dec 15 19:15:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dai Ngo X-Patchwork-Id: 13494804 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23ED23FB34 for ; Fri, 15 Dec 2023 19:17:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="HbEcpcOv" Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFI4IUl024130; Fri, 15 Dec 2023 19:15:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=b/H5JPUH+b32MlkB1jZGr68PwhnJY7JoOoPE47bH6io=; b=HbEcpcOv+dPOElfdH5YIAAePqptUtsALRV75eADZP1cJ0oZEBZMeLqX+lvJ8Gq17m58H QR7vNQ34oTpibfmjCRx32uGlHYMvpncy+Mngb1MNEIc9ns1FW38uKYQgaIoruVhfaBvU I4BzHSKDx6AIDGz1W0lex8u4XYtPHwXpooZfDazXen6q5J95eXMNH0CM1GFIIQCzHzWd ezjXkkVZZ34K55TCp91CR7/abmzPaTjpv6fT0/XgKo5h7jdVxZJcYMZm7IkgWz4bUlYh jNWwQpXiYKFIytf5QWNO1lmZCLEod751Qrib5IHaaPd62Tg9VcWDLjFEvUeZEiYLzRNf nw== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3uvf5ce5sq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 19:15:25 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFIHBI5017003; Fri, 15 Dec 2023 19:15:25 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3uvepcfqpr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 19:15:25 +0000 Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3BFJEAbW013855; Fri, 15 Dec 2023 19:15:24 GMT Received: from ca-common-hq.us.oracle.com (ca-common-hq.us.oracle.com [10.211.9.209]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3uvepcfqmh-4; Fri, 15 Dec 2023 19:15:24 +0000 From: Dai Ngo To: chuck.lever@oracle.com, jlayton@kernel.org Cc: linux-nfs@vger.kernel.org, linux-nfs@stwm.de Subject: [PATCH 3/3] NFSD: Fix server reboot hang problem when callback workqueue is stuck Date: Fri, 15 Dec 2023 11:15:03 -0800 Message-Id: <1702667703-17978-4-git-send-email-dai.ngo@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com> References: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-15_10,2023-12-14_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312150135 X-Proofpoint-ORIG-GUID: YaRd3K1SZcdbetrtd8a31Zb_Hve2nSqT X-Proofpoint-GUID: YaRd3K1SZcdbetrtd8a31Zb_Hve2nSqT Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: If the callback workqueue is stuck, nfsd4_deleg_getattr_conflict will also stuck waiting for the callback request to be executed. This causes the client to hang waiting for the reply of the GETATTR and also causes the reboot of the NFS server to hang due to the pending NFS request. Fix by replacing wait_on_bit with wait_on_bit_timeout with 20 seconds time out. Reported-by: Wolfgang Walter Fixes: 6c41d9a9bd02 ("NFSD: handle GETATTR conflict with write delegation") Signed-off-by: Dai Ngo Reviewed-by: Jeff Layton --- fs/nfsd/nfs4state.c | 6 +++++- fs/nfsd/state.h | 2 ++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 175f3e9f5822..0cc7d4953807 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -2948,6 +2948,9 @@ void nfs4_cb_getattr(struct nfs4_cb_fattr *ncf) if (test_and_set_bit(CB_GETATTR_BUSY, &ncf->ncf_cb_flags)) return; + /* set to proper status when nfsd4_cb_getattr_done runs */ + ncf->ncf_cb_status = NFS4ERR_IO; + refcount_inc(&dp->dl_stid.sc_count); if (!nfsd4_run_cb(&ncf->ncf_getattr)) { refcount_dec(&dp->dl_stid.sc_count); @@ -8558,7 +8561,8 @@ nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct inode *inode, nfs4_cb_getattr(&dp->dl_cb_fattr); spin_unlock(&ctx->flc_lock); - wait_on_bit(&ncf->ncf_cb_flags, CB_GETATTR_BUSY, TASK_INTERRUPTIBLE); + wait_on_bit_timeout(&ncf->ncf_cb_flags, CB_GETATTR_BUSY, + TASK_INTERRUPTIBLE, NFSD_CB_GETATTR_TIMEOUT); if (ncf->ncf_cb_status) { status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ)); if (status != nfserr_jukebox || diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index f96eaa8e9413..94563a6813a6 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -135,6 +135,8 @@ struct nfs4_cb_fattr { /* bits for ncf_cb_flags */ #define CB_GETATTR_BUSY 0 +#define NFSD_CB_GETATTR_TIMEOUT msecs_to_jiffies(20000) /* 20 secs */ + /* * Represents a delegation stateid. The nfs4_client holds references to these * and they are put when it is being destroyed or when the delegation is