From patchwork Thu Jul 14 16:17:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dai Ngo X-Patchwork-Id: 12918183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 133A2CCA482 for ; Thu, 14 Jul 2022 16:17:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238387AbiGNQRu (ORCPT ); Thu, 14 Jul 2022 12:17:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229493AbiGNQRt (ORCPT ); Thu, 14 Jul 2022 12:17:49 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40E3D61D70 for ; Thu, 14 Jul 2022 09:17:48 -0700 (PDT) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 26EFtIlv008504 for ; Thu, 14 Jul 2022 16:17:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2021-07-09; bh=wTnO+vGs+evI9bG/16zycQE4sc6jO6zqko/lPb6PI88=; b=kKQ0jUaGT7dTuzoIO3ZoLG/kf8vtM5n2qw6+Fb9ZuN3PsZABM73SvjQKsMjvDVRUdfRt KRMOcXCqgP77NyynJiAMyFoh/vJ2ci8tr7mOk/1WgExju3bPpbpAGwTgM+QZR4HdOdsz LF/pIeWLlUNiBwG+A3Iljz3jp5j8ZtnwK2l8IWpuDEWVEMC8AdU+z8I7R12C5/J1LW4S wFxy5d3SOSPB/GdZVbPw6qEnszE/86iYpe/x76p14VBgW3LFoLnmkE+vO8sVByM2ij9d 8bszQZpTg2SyTC4qm7tZFUCSE1go3mKhqnyTxcqnoLgBxiukVlcSqLhafQQQ9u291Rh+ xw== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3h71scdsf8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 14 Jul 2022 16:17:47 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 26EGFf0T019014 for ; Thu, 14 Jul 2022 16:17:46 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3h70467f6g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 14 Jul 2022 16:17:46 +0000 Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 26EGHjDq024660 for ; Thu, 14 Jul 2022 16:17:46 GMT Received: from ca-common-hq.us.oracle.com (ca-common-hq.us.oracle.com [10.211.9.209]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3h70467f5u-2; Thu, 14 Jul 2022 16:17:46 +0000 From: Dai Ngo To: chuck.lever@oracle.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH v2 1/2] NFSD: keep track of the number of v4 clients in the system Date: Thu, 14 Jul 2022 09:17:41 -0700 Message-Id: <1657815462-14069-2-git-send-email-dai.ngo@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657815462-14069-1-git-send-email-dai.ngo@oracle.com> References: <1657815462-14069-1-git-send-email-dai.ngo@oracle.com> X-Proofpoint-GUID: nuv_4nvXamyr6h-tgeMNIrobJdoA222A X-Proofpoint-ORIG-GUID: nuv_4nvXamyr6h-tgeMNIrobJdoA222A Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Add counter nfs4_client_count to keep track of the total number of v4 clients, including courtesy clients, in the system. Signed-off-by: Dai Ngo --- fs/nfsd/netns.h | 2 ++ fs/nfsd/nfs4state.c | 8 ++++++-- fs/nfsd/nfsctl.c | 2 ++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 1b1a962a1804..ce864f001a3e 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -189,6 +189,8 @@ struct nfsd_net { struct nfsd_fcache_disposal *fcache_disposal; siphash_key_t siphash_key; + + atomic_t nfs4_client_count; }; /* Simple check to find out if a given net was properly initialized */ diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 9409a0dc1b76..30e16d9e8657 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -2053,7 +2053,8 @@ STALE_CLIENTID(clientid_t *clid, struct nfsd_net *nn) * This type of memory management is somewhat inefficient, but we use it * anyway since SETCLIENTID is not a common operation. */ -static struct nfs4_client *alloc_client(struct xdr_netobj name) +static struct nfs4_client *alloc_client(struct xdr_netobj name, + struct nfsd_net *nn) { struct nfs4_client *clp; int i; @@ -2076,6 +2077,7 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name) atomic_set(&clp->cl_rpc_users, 0); clp->cl_cb_state = NFSD4_CB_UNKNOWN; clp->cl_state = NFSD4_ACTIVE; + atomic_inc(&nn->nfs4_client_count); atomic_set(&clp->cl_delegs_in_recall, 0); INIT_LIST_HEAD(&clp->cl_idhash); INIT_LIST_HEAD(&clp->cl_openowners); @@ -2187,6 +2189,7 @@ __destroy_client(struct nfs4_client *clp) struct nfs4_openowner *oo; struct nfs4_delegation *dp; struct list_head reaplist; + struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id); INIT_LIST_HEAD(&reaplist); spin_lock(&state_lock); @@ -2226,6 +2229,7 @@ __destroy_client(struct nfs4_client *clp) nfsd4_shutdown_callback(clp); if (clp->cl_cb_conn.cb_xprt) svc_xprt_put(clp->cl_cb_conn.cb_xprt); + atomic_add_unless(&nn->nfs4_client_count, -1, 0); free_client(clp); wake_up_all(&expiry_wq); } @@ -2848,7 +2852,7 @@ static struct nfs4_client *create_client(struct xdr_netobj name, struct nfsd_net *nn = net_generic(net, nfsd_net_id); struct dentry *dentries[ARRAY_SIZE(client_files)]; - clp = alloc_client(name); + clp = alloc_client(name, nn); if (clp == NULL) return NULL; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 0621c2faf242..547f4c4b9668 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1487,6 +1487,8 @@ static __net_init int nfsd_init_net(struct net *net) get_random_bytes(&nn->siphash_key, sizeof(nn->siphash_key)); seqlock_init(&nn->writeverf_lock); + atomic_set(&nn->nfs4_client_count, 0); + return 0; out_drc_error: From patchwork Thu Jul 14 16:17:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dai Ngo X-Patchwork-Id: 12918184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90B2CCCA47B for ; Thu, 14 Jul 2022 16:17:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235114AbiGNQRu (ORCPT ); Thu, 14 Jul 2022 12:17:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238921AbiGNQRt (ORCPT ); Thu, 14 Jul 2022 12:17:49 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A916461D74 for ; Thu, 14 Jul 2022 09:17:48 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 26EFs7Ng000894 for ; Thu, 14 Jul 2022 16:17:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2021-07-09; bh=09zGNvtNF1jafOS6jzYUooKqtTgc2Z8D2gVr7pmQ1gQ=; b=fb1ZsjwXuqU7sbrTErNImVNQmwfT3dkxcY5B+deVEUAN5mC/Xg/G1EF4XJym5umG/4i7 V+yM9BcZ4bEfg2ympR7pDtrbuRowKaLiz6ll6L+Q0EKWDPoxMiVOuc9mi4qC5slx9VFL xesPJyoLgAolI1kuJOl3CHULmRyyBFKBJ3q1FjL+z6V2Jfh2ZfHoOY6b9qNny9TuLZF2 nf2BU0qFN+BF9GWQ6efGTp4B2He+pjAzabACtSS7WvJBf/aQriYK+BVtEJF7aUn947e4 47kDrlw90iPHVuRZQHChJvD8UDA/nzcX0BUtKSen85pUlhicjtqcZwtdT/L1r7ntjlBp /A== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3h71xrnnjf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 14 Jul 2022 16:17:47 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 26EGFfne018976 for ; Thu, 14 Jul 2022 16:17:46 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3h70467f6p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 14 Jul 2022 16:17:46 +0000 Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 26EGHjDs024660 for ; Thu, 14 Jul 2022 16:17:46 GMT Received: from ca-common-hq.us.oracle.com (ca-common-hq.us.oracle.com [10.211.9.209]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3h70467f5u-3; Thu, 14 Jul 2022 16:17:46 +0000 From: Dai Ngo To: chuck.lever@oracle.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH v2 2/2] NFSD: limit the number of v4 clients to 1024 per 1GB of system memory Date: Thu, 14 Jul 2022 09:17:42 -0700 Message-Id: <1657815462-14069-3-git-send-email-dai.ngo@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1657815462-14069-1-git-send-email-dai.ngo@oracle.com> References: <1657815462-14069-1-git-send-email-dai.ngo@oracle.com> X-Proofpoint-GUID: sJm4fvku7PE2tgT58cEpynbw_hLxTRNM X-Proofpoint-ORIG-GUID: sJm4fvku7PE2tgT58cEpynbw_hLxTRNM Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Currently there is no limit on how many v4 clients are supported by the system. This can be a problem in systems with small memory configuration to function properly when a very large number of clients exist that creates memory shortage conditions. This patch enforces a limit of 1024 NFSv4 clients, including courtesy clients, per 1GB of system memory. When the number of the clients reaches the limit, requests that create new clients are returned with NFS4ERR_DELAY and the laundromat is kicked start to trim old clients. Due to the overhead of the upcall to remove the client record, the maximun number of clients the laundromat removes on each run is limited to 128. This is done to ensure the laundromat can still process the other tasks in a timely manner. Since there is now a limit of the number of clients, the 24-hr idle time limit of courtesy client is no longer needed and was removed. Signed-off-by: Dai Ngo --- fs/nfsd/netns.h | 1 + fs/nfsd/nfs4state.c | 20 ++++++++++++++------ fs/nfsd/nfsctl.c | 6 ++++++ fs/nfsd/nfsd.h | 2 ++ 4 files changed, 23 insertions(+), 6 deletions(-) diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index ce864f001a3e..ffe17743cc74 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -191,6 +191,7 @@ struct nfsd_net { siphash_key_t siphash_key; atomic_t nfs4_client_count; + int nfs4_max_clients; }; /* Simple check to find out if a given net was properly initialized */ diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 30e16d9e8657..19807f7f618d 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -2059,6 +2059,10 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name, struct nfs4_client *clp; int i; + if (atomic_read(&nn->nfs4_client_count) >= nn->nfs4_max_clients) { + mod_delayed_work(laundry_wq, &nn->laundromat_work, 0); + return NULL; + } clp = kmem_cache_zalloc(client_slab, GFP_KERNEL); if (clp == NULL) return NULL; @@ -5796,9 +5800,12 @@ static void nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist, struct laundry_time *lt) { + unsigned int maxreap, reapcnt = 0; struct list_head *pos, *next; struct nfs4_client *clp; + maxreap = (atomic_read(&nn->nfs4_client_count) >= nn->nfs4_max_clients) ? + NFSD_CLIENT_MAX_TRIM_PER_RUN : 0; INIT_LIST_HEAD(reaplist); spin_lock(&nn->client_lock); list_for_each_safe(pos, next, &nn->client_lru) { @@ -5809,14 +5816,15 @@ nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist, break; if (!atomic_read(&clp->cl_rpc_users)) clp->cl_state = NFSD4_COURTESY; - if (!client_has_state(clp) || - ktime_get_boottime_seconds() >= - (clp->cl_time + NFSD_COURTESY_CLIENT_TIMEOUT)) + if (!client_has_state(clp)) goto exp_client; - if (nfs4_anylock_blockers(clp)) { + if (!nfs4_anylock_blockers(clp)) + if (reapcnt >= maxreap) + continue; exp_client: - if (!mark_client_expired_locked(clp)) - list_add(&clp->cl_lru, reaplist); + if (!mark_client_expired_locked(clp)) { + list_add(&clp->cl_lru, reaplist); + reapcnt++; } } spin_unlock(&nn->client_lock); diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 547f4c4b9668..bbd251da86e4 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1463,6 +1463,8 @@ static __net_init int nfsd_init_net(struct net *net) { int retval; struct nfsd_net *nn = net_generic(net, nfsd_net_id); + u64 max_clients; + struct sysinfo si; retval = nfsd_export_init(net); if (retval) @@ -1488,6 +1490,10 @@ static __net_init int nfsd_init_net(struct net *net) seqlock_init(&nn->writeverf_lock); atomic_set(&nn->nfs4_client_count, 0); + si_meminfo(&si); + max_clients = (u64)si.totalram * si.mem_unit / (1024 * 1024 * 1024); + max_clients *= NFS4_CLIENTS_PER_GB; + nn->nfs4_max_clients = max_t(int, max_clients, NFS4_CLIENTS_PER_GB); return 0; diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index 847b482155ae..bbada18225b1 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -341,6 +341,8 @@ void nfsd_lockd_shutdown(void); #define NFSD_LAUNDROMAT_MINTIMEOUT 1 /* seconds */ #define NFSD_COURTESY_CLIENT_TIMEOUT (24 * 60 * 60) /* seconds */ +#define NFSD_CLIENT_MAX_TRIM_PER_RUN 128 +#define NFS4_CLIENTS_PER_GB 1024 /* * The following attributes are currently not supported by the NFSv4 server: