From patchwork Wed Oct 23 02:37:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13846340 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9F12132103 for ; Wed, 23 Oct 2024 02:42:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651372; cv=none; b=EisdzC5mBmc80YKOJ7nQWqPETSlLSCGp6PbnCFN0vS0q/IBHobydKlt4ePPOXMwCEBVRx9NfhhwX94Qf7c4KTMNHmrdCZATWJW8ZX/rJbQAicYLgHJtD+LWYzgOnjtj9R2zcHYW1Urdgk/KHy9lSkMthRymN/4olFZ3BpzEGAn0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651372; c=relaxed/simple; bh=R0QWJCHGnLosegOS6+CDcchbtgV6xLPMRHXQpughKFA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e0SnJ9vHMN9zouYHQ2dftHhyb6Vdjx25hUoHAn9VP0fV99PZQtLrBTYKsS35P668dmX8gxqTxsvlH8hqNc9y47kUAZ9oMe3Z05vsA8N3pS7RrkMpSJoRCggaWoRrR+ekWsddDpxxz8l9NXxq61Xex5hABEX2n3wmLSbFvcQ0jw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=rJviWy2t; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=lzXNj9xm; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=rJviWy2t; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=lzXNj9xm; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="rJviWy2t"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="lzXNj9xm"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="rJviWy2t"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="lzXNj9xm" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 288F41F83F; Wed, 23 Oct 2024 02:42:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651369; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jh5mvXC8c4OCgkYvqWMXEw+6Hi8aQ6F5ZL1ewgBnx1Q=; b=rJviWy2t6xA6uFfIu6J6t6D4LbJjRCxHoJgtpLW6xE9DHiueuOWVOS9nMpJgHcNWlbAOha yX2w5n6ndllp5toqbKHZkEOmMEe347b95y9v++ioxJ6NTIFpFJLLAYQ81UHc3DpjEvwATV 5LHVojLYcWW/51EXb+17a9uVsRLzLSk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651369; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jh5mvXC8c4OCgkYvqWMXEw+6Hi8aQ6F5ZL1ewgBnx1Q=; b=lzXNj9xmYMV9cWgltw1Q51OKqOwcQ8oeCrdlkosxXHI6hKaSmkkpEbTt0SXcGxk8sD4NDp qnOSTRauXnBgTiBA== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651369; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jh5mvXC8c4OCgkYvqWMXEw+6Hi8aQ6F5ZL1ewgBnx1Q=; b=rJviWy2t6xA6uFfIu6J6t6D4LbJjRCxHoJgtpLW6xE9DHiueuOWVOS9nMpJgHcNWlbAOha yX2w5n6ndllp5toqbKHZkEOmMEe347b95y9v++ioxJ6NTIFpFJLLAYQ81UHc3DpjEvwATV 5LHVojLYcWW/51EXb+17a9uVsRLzLSk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651369; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jh5mvXC8c4OCgkYvqWMXEw+6Hi8aQ6F5ZL1ewgBnx1Q=; b=lzXNj9xmYMV9cWgltw1Q51OKqOwcQ8oeCrdlkosxXHI6hKaSmkkpEbTt0SXcGxk8sD4NDp qnOSTRauXnBgTiBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1270513A63; Wed, 23 Oct 2024 02:42:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id fY+NLqZiGGcjOgAAD6G6ig (envelope-from ); Wed, 23 Oct 2024 02:42:46 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey Subject: [PATCH 1/6] SUNRPC: move nrthreads counting to start/stop threads. Date: Wed, 23 Oct 2024 13:37:01 +1100 Message-ID: <20241023024222.691745-2-neilb@suse.de> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241023024222.691745-1-neilb@suse.de> References: <20241023024222.691745-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,suse.de:mid]; RCVD_TLS_ALL(0.00)[] X-Spam-Score: -2.80 X-Spam-Flag: NO sp_nrthreads and sv_nrthreads are the number of threads that have been explicitly requested. Future patches will allow extra threads to be created as needed. So move the updating of these fields to code which is for updating configuration rather that code that is for starting/stopping threads. Reviewed-by: Jeff Layton Signed-off-by: NeilBrown --- net/sunrpc/svc.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 79879b7d39cb..bd4f02b34f44 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -713,9 +713,6 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node) rqstp->rq_err = -EAGAIN; /* No error yet */ - serv->sv_nrthreads += 1; - pool->sp_nrthreads += 1; - /* Protected by whatever lock the service uses when calling * svc_set_num_threads() */ @@ -815,6 +812,8 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) svc_exit_thread(rqstp); return PTR_ERR(task); } + serv->sv_nrthreads += 1; + chosen_pool->sp_nrthreads += 1; rqstp->rq_task = task; if (serv->sv_nrpools > 1) @@ -844,6 +843,8 @@ svc_stop_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) victim = svc_pool_victim(serv, pool, &state); if (!victim) break; + victim->sp_nrthreads -= 1; + serv->sv_nrthreads -= 1; svc_pool_wake_idle_thread(victim); wait_on_bit(&victim->sp_flags, SP_VICTIM_REMAINS, TASK_IDLE); @@ -959,8 +960,6 @@ svc_exit_thread(struct svc_rqst *rqstp) list_del_rcu(&rqstp->rq_all); - pool->sp_nrthreads -= 1; - serv->sv_nrthreads -= 1; svc_sock_update_bufs(serv); svc_rqst_free(rqstp); From patchwork Wed Oct 23 02:37:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13846341 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AA3F847B for ; Wed, 23 Oct 2024 02:43:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651382; cv=none; b=Wp5LYVd+Mdg68ZAnuck+EX/rdt4VbHbVO0Jv1cYJ78GUL3UZ6Ml9SP3FUf4lB88tkffdNYaiihhLZPlyiySCbjaU1af7QoPh3WLvLd+B+3bjDVq6WM+vP2Pd8H+JZZ1THItY71bZuaOcnK6eMS1/d74JB9wnUKKlr6l9DP9nD3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651382; c=relaxed/simple; bh=Rn84/pLRmFhXmIUP6uRi1iaA4rK7hQehJ9yaBq/VHCo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=myQMBueFcF03YjWEUVEnDqMopt0a1Al6HEfIaFdUyE++V+rXKNnPGfrmvHAwDIuCaiFd5n+v5IOKUkfD84meG2KpGZ00CfjH6gR9j7eK0pY8HAbJvBoxfI6qJKibt0Cbg1R0XG7LzmZaVkpleQl0L2DmQS8v06ZP8WJdmXdg7KE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=eeKndd2k; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=nUwA2jaV; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=eeKndd2k; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=nUwA2jaV; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="eeKndd2k"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="nUwA2jaV"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="eeKndd2k"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="nUwA2jaV" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A2BE81F83F; Wed, 23 Oct 2024 02:42:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651378; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ctdBhoJoqhe2atqg/Y55XyUbRucmK/lbMKezvDNLgOc=; b=eeKndd2k1+ZGs+JLXMFUO1Jto8wRSfQ6vju4mZ02+tGXqp5p5WVWxmCIoDZeudKBt+ARel OH81zRMkW2RPuEXdc1ipcXeiM09ZLh76bzRPoCMs6yOw3/snBWPlN2zpf1rsTplmfJB2+V 0Us9N8WPjdN4xHfF/3MH37V4iWCFUhU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651378; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ctdBhoJoqhe2atqg/Y55XyUbRucmK/lbMKezvDNLgOc=; b=nUwA2jaVc83wW0tINF3Xw82v5R96XQrTrQcSKanfBGsYZwTCKGzhiXjwThKpSMhcVczMT+ NMHIYPa7U6l6SPBQ== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=eeKndd2k; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=nUwA2jaV DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651378; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ctdBhoJoqhe2atqg/Y55XyUbRucmK/lbMKezvDNLgOc=; b=eeKndd2k1+ZGs+JLXMFUO1Jto8wRSfQ6vju4mZ02+tGXqp5p5WVWxmCIoDZeudKBt+ARel OH81zRMkW2RPuEXdc1ipcXeiM09ZLh76bzRPoCMs6yOw3/snBWPlN2zpf1rsTplmfJB2+V 0Us9N8WPjdN4xHfF/3MH37V4iWCFUhU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651378; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ctdBhoJoqhe2atqg/Y55XyUbRucmK/lbMKezvDNLgOc=; b=nUwA2jaVc83wW0tINF3Xw82v5R96XQrTrQcSKanfBGsYZwTCKGzhiXjwThKpSMhcVczMT+ NMHIYPa7U6l6SPBQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8CCC113A63; Wed, 23 Oct 2024 02:42:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2xLDELBiGGcvOgAAD6G6ig (envelope-from ); Wed, 23 Oct 2024 02:42:56 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey Subject: [PATCH 2/6] nfsd: return hard failure for OP_SETCLIENTID when there are too many clients. Date: Wed, 23 Oct 2024 13:37:02 +1100 Message-ID: <20241023024222.691745-3-neilb@suse.de> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241023024222.691745-1-neilb@suse.de> References: <20241023024222.691745-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: A2BE81F83F X-Spam-Score: -3.01 X-Rspamd-Action: no action X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:dkim,suse.de:mid,suse.de:email]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; DKIM_TRACE(0.00)[suse.de:+] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spam-Flag: NO X-Spam-Level: If there are more non-courteous clients than the calculated limit, we should fail the request rather than report a soft failure and encouraging the client to retry indefinitely. The only hard failure allowed for EXCHANGE_ID that doesn't clearly have some other meaning is NFS4ERR_SERVERFAULT. So use that, but explain why in a comment at each place that it is returned. If there are courteous clients which push us over the limit, then expedite their removal. This is not known to have caused a problem is production use, but testing of lots of clients reports repeated NFS4ERR_DELAY responses which doesn't seem helpful. Also remove an outdated comment - we do use a slab cache. Reviewed-by: Jeff Layton Signed-off-by: NeilBrown --- fs/nfsd/nfs4state.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 56b261608af4..ca6b5b52f77d 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -2212,21 +2212,20 @@ STALE_CLIENTID(clientid_t *clid, struct nfsd_net *nn) return 1; } -/* - * XXX Should we use a slab cache ? - * This type of memory management is somewhat inefficient, but we use it - * anyway since SETCLIENTID is not a common operation. - */ static struct nfs4_client *alloc_client(struct xdr_netobj name, struct nfsd_net *nn) { struct nfs4_client *clp; int i; - if (atomic_read(&nn->nfs4_client_count) >= nn->nfs4_max_clients) { + if (atomic_read(&nn->nfs4_client_count) - + atomic_read(&nn->nfsd_courtesy_clients) >= nn->nfs4_max_clients) + return ERR_PTR(-EUSERS); + + if (atomic_read(&nn->nfs4_client_count) >= nn->nfs4_max_clients && + atomic_read(&nn->nfsd_courtesy_clients) > 0) mod_delayed_work(laundry_wq, &nn->laundromat_work, 0); - return NULL; - } + clp = kmem_cache_zalloc(client_slab, GFP_KERNEL); if (clp == NULL) return NULL; @@ -3121,8 +3120,8 @@ static struct nfs4_client *create_client(struct xdr_netobj name, struct dentry *dentries[ARRAY_SIZE(client_files)]; clp = alloc_client(name, nn); - if (clp == NULL) - return NULL; + if (IS_ERR_OR_NULL(clp)) + return clp; ret = copy_cred(&clp->cl_cred, &rqstp->rq_cred); if (ret) { @@ -3504,6 +3503,11 @@ nfsd4_exchange_id(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new = create_client(exid->clname, rqstp, &verf); if (new == NULL) return nfserr_jukebox; + if (IS_ERR(new)) + /* Protocol has no specific error for "client limit reached". + * NFS4ERR_RESOURCE is not permitted for EXCHANGE_ID + */ + return nfserr_serverfault; status = copy_impl_id(new, exid); if (status) goto out_nolock; @@ -4422,6 +4426,12 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new = create_client(clname, rqstp, &clverifier); if (new == NULL) return nfserr_jukebox; + if (IS_ERR(new)) + /* Protocol has no specific error for "client limit reached". + * NFS4ERR_RESOURCE, while allowed for SETCLIENTID, implies + * that a smaller COMPOUND might be successful. + */ + return nfserr_serverfault; spin_lock(&nn->client_lock); conf = find_confirmed_client_by_name(&clname, nn); if (conf && client_has_state(conf)) { From patchwork Wed Oct 23 02:37:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13846342 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE22B847B for ; Wed, 23 Oct 2024 02:43:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651387; cv=none; b=YsaY6uxucmLr/3Tu/8N/xvAPdkgsqNh8Dc4kYhFDmTkRpvxyk2Z2OVQc8GQnJQGlNILaT6dtmSSvf1aKhaa0+yW7veXQEbK1qbGnqhAep8BtTCnP8M59eHQI8oHQ4JOLkhA/rEU/LDTw+7q6UoBe2iGwsB1frhJrdY2oBVPIzkg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651387; c=relaxed/simple; bh=mBcDz/zY761U9LuZ38gL5yJcQRFp/LsDvrN9xgEXYjw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kLk0w5ByZHBRzL57SNB6Ju3e/vGhmGwJX2zkzZeQb0m1S4wRELYau7BaM9ETY0ZaSGdZ5BJSs0ayYMLDNNzwTWoNCaDUyIdpO3ZsGLn6glnWel4ZaNUnvDNPv1yclJM0N03Ce3FEnI301fugfdFjQ73O802FuhwR5Dxvavzw5eY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Xlxra0Oc; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=5gBGc6ZS; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Xlxra0Oc; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=5gBGc6ZS; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Xlxra0Oc"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="5gBGc6ZS"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Xlxra0Oc"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="5gBGc6ZS" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 3B99021DB6; Wed, 23 Oct 2024 02:43:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651384; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+yza3dYffB0Jg/yZQs/BbmPTa484s0Y8qNnmEwQJGZM=; b=Xlxra0Ocr0PRSjfNUFrd8rJldXdh7xvjvEShIFxsP++XMCdWVW9OAx4mrLu446cUbe9me3 O+wNFnpW7XNrCnpHDWckXLd7O9rCZBjUtuEUPlFAkf9iGtzlnZulAoeSQHmYfs4zrvkmQk PbL4czGLt1ivD7eOR3/h4qirvwD1DNE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651384; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+yza3dYffB0Jg/yZQs/BbmPTa484s0Y8qNnmEwQJGZM=; b=5gBGc6ZSW4ivIEAFnI4YknyLJ3FqQEmnbBL8GkhNLUpQlvw38opBUPRP8xCj5CxHqUqbWw 6RqoygVy44q20nDw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651384; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+yza3dYffB0Jg/yZQs/BbmPTa484s0Y8qNnmEwQJGZM=; b=Xlxra0Ocr0PRSjfNUFrd8rJldXdh7xvjvEShIFxsP++XMCdWVW9OAx4mrLu446cUbe9me3 O+wNFnpW7XNrCnpHDWckXLd7O9rCZBjUtuEUPlFAkf9iGtzlnZulAoeSQHmYfs4zrvkmQk PbL4czGLt1ivD7eOR3/h4qirvwD1DNE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651384; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+yza3dYffB0Jg/yZQs/BbmPTa484s0Y8qNnmEwQJGZM=; b=5gBGc6ZSW4ivIEAFnI4YknyLJ3FqQEmnbBL8GkhNLUpQlvw38opBUPRP8xCj5CxHqUqbWw 6RqoygVy44q20nDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 249DA13A63; Wed, 23 Oct 2024 02:43:01 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id Hx/zMrViGGc1OgAAD6G6ig (envelope-from ); Wed, 23 Oct 2024 02:43:01 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey Subject: [PATCH 3/6] nfs: dynamically adjust per-client DRC slot limits. Date: Wed, 23 Oct 2024 13:37:03 +1100 Message-ID: <20241023024222.691745-4-neilb@suse.de> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241023024222.691745-1-neilb@suse.de> References: <20241023024222.691745-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:email]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Level: Currently per-client DRC slot limits (for v4.1+) are calculated when the client connects, and they are left unchanged. So earlier clients can get a larger share when memory is tight. The heuristic for choosing a number includes the number of configured server threads. This is problematic for 2 reasons. 1/ sv_nrthreads is different in different net namespaces, but the memory allocation is global across all namespaces. So different namespaces get treated differently without good reason. 2/ a future patch will auto-configure number of threads based on load so that there will be no need to preconfigure a number. This will make the current heuristic even more arbitrary. NFSv4.1 allows the number of slots to be varied dynamically - in the reply to each SEQUENCE op. With this patch we provide a provisional upper limit in the EXCHANGE_ID reply which might end up being too big, and adjust it with each SEQUENCE reply. The goal for when memory is tight is to allow each client to have a similar number of slots. So clients that ask for larger slots get more memory. This may not be ideal. It could be changed later. So we track the sum of the slot sizes of all active clients, and share memory out based on the ratio of the slot size for a given client with the sum of slot sizes. We never allow more in a SEQUENCE reply than we did in the EXCHANGE_ID reply. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/nfs4state.c | 81 ++++++++++++++++++++++++--------------------- fs/nfsd/nfs4xdr.c | 2 +- fs/nfsd/nfsd.h | 6 +++- fs/nfsd/nfssvc.c | 7 ++-- fs/nfsd/state.h | 2 +- fs/nfsd/xdr4.h | 2 -- 6 files changed, 56 insertions(+), 44 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index ca6b5b52f77d..834e9aa12b82 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -1909,44 +1909,26 @@ static inline u32 slot_bytes(struct nfsd4_channel_attrs *ca) } /* - * XXX: If we run out of reserved DRC memory we could (up to a point) - * re-negotiate active sessions and reduce their slot usage to make - * room for new connections. For now we just fail the create session. + * When a client connects it gets a max_requests number that would allow + * it to use 1/8 of the memory we think can reasonably be used for the DRC. + * Later in response to SEQUENCE operations we further limit that when there + * are more than 8 concurrent clients. */ -static u32 nfsd4_get_drc_mem(struct nfsd4_channel_attrs *ca, struct nfsd_net *nn) +static u32 nfsd4_get_drc_mem(struct nfsd4_channel_attrs *ca) { u32 slotsize = slot_bytes(ca); u32 num = ca->maxreqs; - unsigned long avail, total_avail; - unsigned int scale_factor; + unsigned long avail; spin_lock(&nfsd_drc_lock); - if (nfsd_drc_max_mem > nfsd_drc_mem_used) - total_avail = nfsd_drc_max_mem - nfsd_drc_mem_used; - else - /* We have handed out more space than we chose in - * set_max_drc() to allow. That isn't really a - * problem as long as that doesn't make us think we - * have lots more due to integer overflow. - */ - total_avail = 0; - avail = min((unsigned long)NFSD_MAX_MEM_PER_SESSION, total_avail); - /* - * Never use more than a fraction of the remaining memory, - * unless it's the only way to give this client a slot. - * The chosen fraction is either 1/8 or 1/number of threads, - * whichever is smaller. This ensures there are adequate - * slots to support multiple clients per thread. - * Give the client one slot even if that would require - * over-allocation--it is better than failure. - */ - scale_factor = max_t(unsigned int, 8, nn->nfsd_serv->sv_nrthreads); - avail = clamp_t(unsigned long, avail, slotsize, - total_avail/scale_factor); - num = min_t(int, num, avail / slotsize); - num = max_t(int, num, 1); - nfsd_drc_mem_used += num * slotsize; + avail = min(NFSD_MAX_MEM_PER_SESSION, + nfsd_drc_max_mem / 8); + + num = clamp_t(int, num, 1, avail / slotsize); + + nfsd_drc_slotsize_sum += slotsize; + spin_unlock(&nfsd_drc_lock); return num; @@ -1957,10 +1939,33 @@ static void nfsd4_put_drc_mem(struct nfsd4_channel_attrs *ca) int slotsize = slot_bytes(ca); spin_lock(&nfsd_drc_lock); - nfsd_drc_mem_used -= slotsize * ca->maxreqs; + nfsd_drc_slotsize_sum -= slotsize; spin_unlock(&nfsd_drc_lock); } +/* + * Report the number of slots that we would like the client to limit + * itself to. When the number of clients is large, we start sharing + * memory so all clients get the same number of slots. + */ +static unsigned int nfsd4_get_drc_slots(struct nfsd4_session *session) +{ + u32 slotsize = slot_bytes(&session->se_fchannel); + unsigned long avail; + unsigned long slotsize_sum = READ_ONCE(nfsd_drc_slotsize_sum); + + if (slotsize_sum < slotsize) + slotsize_sum = slotsize; + + /* Find our share of avail mem across all active clients, + * then limit to 1/8 of total, and configured max + */ + avail = min3(nfsd_drc_max_mem * slotsize / slotsize_sum, + nfsd_drc_max_mem / 8, + NFSD_MAX_MEM_PER_SESSION); + return max3(1UL, avail / slotsize, session->se_fchannel.maxreqs); +} + static struct nfsd4_session *alloc_session(struct nfsd4_channel_attrs *fattrs, struct nfsd4_channel_attrs *battrs) { @@ -3735,7 +3740,7 @@ static __be32 check_forechannel_attrs(struct nfsd4_channel_attrs *ca, struct nfs * Note that we always allow at least one slot, because our * accounting is soft and provides no guarantees either way. */ - ca->maxreqs = nfsd4_get_drc_mem(ca, nn); + ca->maxreqs = nfsd4_get_drc_mem(ca); return nfs_ok; } @@ -4229,10 +4234,12 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, slot = session->se_slots[seq->slotid]; dprintk("%s: slotid %d\n", __func__, seq->slotid); - /* We do not negotiate the number of slots yet, so set the - * maxslots to the session maxreqs which is used to encode - * sr_highest_slotid and the sr_target_slot id to maxslots */ - seq->maxslots = session->se_fchannel.maxreqs; + /* Negotiate number of slots: set the target, and use the + * same for max as long as it doesn't decrease the max + * (that isn't allowed). + */ + seq->target_maxslots = nfsd4_get_drc_slots(session); + seq->maxslots = max(seq->maxslots, seq->target_maxslots); trace_nfsd_slot_seqid_sequence(clp, seq, slot); status = check_slot_seqid(seq->seqid, slot->sl_seqid, diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c index f118921250c3..e4e706872e54 100644 --- a/fs/nfsd/nfs4xdr.c +++ b/fs/nfsd/nfs4xdr.c @@ -4955,7 +4955,7 @@ nfsd4_encode_sequence(struct nfsd4_compoundres *resp, __be32 nfserr, if (nfserr != nfs_ok) return nfserr; /* sr_target_highest_slotid */ - nfserr = nfsd4_encode_slotid4(xdr, seq->maxslots - 1); + nfserr = nfsd4_encode_slotid4(xdr, seq->target_maxslots - 1); if (nfserr != nfs_ok) return nfserr; /* sr_status_flags */ diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index 4b56ba1e8e48..33c9db3ee8b6 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -90,7 +90,11 @@ extern const struct svc_version nfsd_version2, nfsd_version3, nfsd_version4; extern struct mutex nfsd_mutex; extern spinlock_t nfsd_drc_lock; extern unsigned long nfsd_drc_max_mem; -extern unsigned long nfsd_drc_mem_used; +/* Storing the sum of the slot sizes for all active clients (across + * all net-namespaces) allows a share of total available memory to + * be allocaed to each client based on its slot size. + */ +extern unsigned long nfsd_drc_slotsize_sum; extern atomic_t nfsd_th_cnt; /* number of available threads */ extern const struct seq_operations nfs_exports_op; diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 49e2f32102ab..e596eb5d10db 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -78,7 +78,7 @@ DEFINE_MUTEX(nfsd_mutex); */ DEFINE_SPINLOCK(nfsd_drc_lock); unsigned long nfsd_drc_max_mem; -unsigned long nfsd_drc_mem_used; +unsigned long nfsd_drc_slotsize_sum; #if IS_ENABLED(CONFIG_NFS_LOCALIO) static const struct svc_version *localio_versions[] = { @@ -589,10 +589,13 @@ void nfsd_reset_versions(struct nfsd_net *nn) */ static void set_max_drc(void) { + if (nfsd_drc_max_mem) + return; + #define NFSD_DRC_SIZE_SHIFT 7 nfsd_drc_max_mem = (nr_free_buffer_pages() >> NFSD_DRC_SIZE_SHIFT) * PAGE_SIZE; - nfsd_drc_mem_used = 0; + nfsd_drc_slotsize_sum = 0; dprintk("%s nfsd_drc_max_mem %lu \n", __func__, nfsd_drc_max_mem); } diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index 79c743c01a47..fe71ae3c577b 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -214,7 +214,7 @@ static inline struct nfs4_delegation *delegstateid(struct nfs4_stid *s) /* Maximum number of slots per session. 160 is useful for long haul TCP */ #define NFSD_MAX_SLOTS_PER_SESSION 160 /* Maximum session per slot cache size */ -#define NFSD_SLOT_CACHE_SIZE 2048 +#define NFSD_SLOT_CACHE_SIZE 2048UL /* Maximum number of NFSD_SLOT_CACHE_SIZE slots per session */ #define NFSD_CACHE_SIZE_SLOTS_PER_SESSION 32 #define NFSD_MAX_MEM_PER_SESSION \ diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h index 2a21a7662e03..71b87190a4a6 100644 --- a/fs/nfsd/xdr4.h +++ b/fs/nfsd/xdr4.h @@ -575,9 +575,7 @@ struct nfsd4_sequence { u32 slotid; /* request/response */ u32 maxslots; /* request/response */ u32 cachethis; /* request */ -#if 0 u32 target_maxslots; /* response */ -#endif /* not yet */ u32 status_flags; /* response */ }; From patchwork Wed Oct 23 02:37:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13846343 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A0A4847B for ; Wed, 23 Oct 2024 02:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651393; cv=none; b=uYuPU4WrhmH+LQ/CJefo7uzVPZ1WYLLTNgahsXjFMpM0FXnrQ8QPtKWthWqzeqRmAqbLsaa9SEyhzkT8v0bEpMeQvzb3tfqhpnwhyJVi/BXafNLyEg8XzoRZSClcVoxtmA2fkM5dzNxJHchMn8wzKjcBiHFdJZ6BtRu16HYJFsw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651393; c=relaxed/simple; bh=Hf2lh8gPchUA5ZuvI1/v/TQLy/uYvyGzWfS9HUwsxyA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NRxmXRQkrGejxwMkpXru1C3pd/W1UVVC6u22Y8IUgV04Haw3JgT8K4kdxEUtR2DCLnMT+ECtjeVXac7r+AXy2H3TYuwMWdDWgmD0/25JMUFzO7R9oHOtVvRvEVHzw+QlHauj9qohZ55PdfwzwB516YAe7N/FE55fTf6QCTglkgE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=iJ7+22FU; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=FbIEeER9; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=iJ7+22FU; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=FbIEeER9; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="iJ7+22FU"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="FbIEeER9"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="iJ7+22FU"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="FbIEeER9" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id CBB101F83F; Wed, 23 Oct 2024 02:43:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651389; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z7q8pw9d8BhjkTU/EPZvKyv6t8TQLrHQQsLfVtRA+Gg=; b=iJ7+22FURCfaW0F3KbtmWOsrV/8QIvNc46VHTwuAwf9Ia1pdt0N+XlIzp1pOhWXuYl2RAT fKveVhzDqqQF3iVPOBcP8yJA+2zLzyKl3EnXMmZZioayHblQ+0OWJDRpoREgc3kPlvTg0V Qj2vNoBvXzX9sPQjk4i8aJIuN2V+2tM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651389; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z7q8pw9d8BhjkTU/EPZvKyv6t8TQLrHQQsLfVtRA+Gg=; b=FbIEeER96lXfj/ZWUq70W9QMhZeLDjuoR88EvlPNcGLGqi68pmgXPybgqUTHEwUUmxM4vX /oqWbc06+MsotjAw== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651389; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z7q8pw9d8BhjkTU/EPZvKyv6t8TQLrHQQsLfVtRA+Gg=; b=iJ7+22FURCfaW0F3KbtmWOsrV/8QIvNc46VHTwuAwf9Ia1pdt0N+XlIzp1pOhWXuYl2RAT fKveVhzDqqQF3iVPOBcP8yJA+2zLzyKl3EnXMmZZioayHblQ+0OWJDRpoREgc3kPlvTg0V Qj2vNoBvXzX9sPQjk4i8aJIuN2V+2tM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651389; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z7q8pw9d8BhjkTU/EPZvKyv6t8TQLrHQQsLfVtRA+Gg=; b=FbIEeER96lXfj/ZWUq70W9QMhZeLDjuoR88EvlPNcGLGqi68pmgXPybgqUTHEwUUmxM4vX /oqWbc06+MsotjAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B47CF13A63; Wed, 23 Oct 2024 02:43:07 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id POd1GrtiGGc+OgAAD6G6ig (envelope-from ); Wed, 23 Oct 2024 02:43:07 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey Subject: [PATCH 4/6] nfsd: don't use sv_nrthreads in connection limiting calculations. Date: Wed, 23 Oct 2024 13:37:04 +1100 Message-ID: <20241023024222.691745-5-neilb@suse.de> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241023024222.691745-1-neilb@suse.de> References: <20241023024222.691745-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:email]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Level: The heuristic for limiting the number of incoming connections to nfsd currently uses sv_nrthreads - allowing more connections if more threads were configured. A future patch will allow number of threads to grow dynamically so that there will be no need to configure sv_nrthreads. So we need a different solution for limiting connections. It isn't clear what problem is solved by limiting connections (as mentioned in a code comment) but the most likely problem is a connection storm - many connections that are not doing productive work. These will be closed after about 6 minutes already but it might help to slow down a storm. This patch adds a per-connection flag XPT_PEER_VALID which indicates that the peer has presented a filehandle for which it has some sort of access. i.e the peer is known to be trusted in some way. We now only count connections which have NOT been determined to be valid. There should be relative few of these at any given time. If the number of non-validated peer exceed a limit - currently 64 - we close the oldest non-validated peer to avoid having too many of these useless connections. Note that this patch significantly changes the meaning of the various configuration parameters for "max connections". The next patch will remove all of these. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfs/callback.c | 4 ---- fs/nfs/callback_xdr.c | 1 + fs/nfsd/netns.h | 4 ++-- fs/nfsd/nfsfh.c | 2 ++ include/linux/sunrpc/svc.h | 2 +- include/linux/sunrpc/svc_xprt.h | 15 +++++++++++++++ net/sunrpc/svc_xprt.c | 33 +++++++++++++++++---------------- 7 files changed, 38 insertions(+), 23 deletions(-) diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c index 6cf92498a5ac..86bdc7d23fb9 100644 --- a/fs/nfs/callback.c +++ b/fs/nfs/callback.c @@ -211,10 +211,6 @@ static struct svc_serv *nfs_callback_create_svc(int minorversion) return ERR_PTR(-ENOMEM); } cb_info->serv = serv; - /* As there is only one thread we need to over-ride the - * default maximum of 80 connections - */ - serv->sv_maxconn = 1024; dprintk("nfs_callback_create_svc: service created\n"); return serv; } diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c index fdeb0b34a3d3..4254ba3ee7c5 100644 --- a/fs/nfs/callback_xdr.c +++ b/fs/nfs/callback_xdr.c @@ -984,6 +984,7 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp) nfs_put_client(cps.clp); goto out_invalidcred; } + svc_xprt_set_valid(rqstp->rq_xprt); } cps.minorversion = hdr_arg.minorversion; diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 26f7b34d1a03..a05a45bb1978 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -129,8 +129,8 @@ struct nfsd_net { unsigned char writeverf[8]; /* - * Max number of connections this nfsd container will allow. Defaults - * to '0' which is means that it bases this on the number of threads. + * Max number of non-validated connections this nfsd container + * will allow. Defaults to '0' gets mapped to 64. */ unsigned int max_connections; diff --git a/fs/nfsd/nfsfh.c b/fs/nfsd/nfsfh.c index 40ad58a6a036..2f44de99f709 100644 --- a/fs/nfsd/nfsfh.c +++ b/fs/nfsd/nfsfh.c @@ -383,6 +383,8 @@ __fh_verify(struct svc_rqst *rqstp, goto out; skip_pseudoflavor_check: + svc_xprt_set_valid(rqstp->rq_xprt); + /* Finally, check access permissions. */ error = nfsd_permission(cred, exp, dentry, access); out: diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index e68fecf6eab5..617ebfff2f30 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -81,7 +81,7 @@ struct svc_serv { unsigned int sv_xdrsize; /* XDR buffer size */ struct list_head sv_permsocks; /* all permanent sockets */ struct list_head sv_tempsocks; /* all temporary sockets */ - int sv_tmpcnt; /* count of temporary sockets */ + int sv_tmpcnt; /* count of temporary "valid" sockets */ struct timer_list sv_temptimer; /* timer for aging temporary sockets */ char * sv_name; /* service name */ diff --git a/include/linux/sunrpc/svc_xprt.h b/include/linux/sunrpc/svc_xprt.h index 0981e35a9fed..35929a7727c7 100644 --- a/include/linux/sunrpc/svc_xprt.h +++ b/include/linux/sunrpc/svc_xprt.h @@ -99,8 +99,23 @@ enum { XPT_HANDSHAKE, /* xprt requests a handshake */ XPT_TLS_SESSION, /* transport-layer security established */ XPT_PEER_AUTH, /* peer has been authenticated */ + XPT_PEER_VALID, /* peer has presented a filehandle that + * it has access to. It is NOT counted + * in ->sv_tmpcnt. + */ }; +static inline void svc_xprt_set_valid(struct svc_xprt *xpt) +{ + if (test_bit(XPT_TEMP, &xpt->xpt_flags) && + !test_and_set_bit(XPT_PEER_VALID, &xpt->xpt_flags)) { + struct svc_serv *serv = xpt->xpt_server; + spin_lock(&serv->sv_lock); + serv->sv_tmpcnt -= 1; + spin_unlock(&serv->sv_lock); + } +} + static inline void unregister_xpt_user(struct svc_xprt *xpt, struct svc_xpt_user *u) { spin_lock(&xpt->xpt_lock); diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 43c57124de52..ff5b8bb8a88f 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -606,7 +606,8 @@ int svc_port_is_privileged(struct sockaddr *sin) } /* - * Make sure that we don't have too many active connections. If we have, + * Make sure that we don't have too many connections that have not yet + * demonstrated that they have access the the NFS server. If we have, * something must be dropped. It's not clear what will happen if we allow * "too many" connections, but when dealing with network-facing software, * we have to code defensively. Here we do that by imposing hard limits. @@ -625,27 +626,26 @@ int svc_port_is_privileged(struct sockaddr *sin) */ static void svc_check_conn_limits(struct svc_serv *serv) { - unsigned int limit = serv->sv_maxconn ? serv->sv_maxconn : - (serv->sv_nrthreads+3) * 20; + unsigned int limit = serv->sv_maxconn ? serv->sv_maxconn : 64; if (serv->sv_tmpcnt > limit) { - struct svc_xprt *xprt = NULL; + struct svc_xprt *xprt = NULL, *xprti; spin_lock_bh(&serv->sv_lock); if (!list_empty(&serv->sv_tempsocks)) { - /* Try to help the admin */ - net_notice_ratelimited("%s: too many open connections, consider increasing the %s\n", - serv->sv_name, serv->sv_maxconn ? - "max number of connections" : - "number of threads"); /* * Always select the oldest connection. It's not fair, - * but so is life + * but nor is life. */ - xprt = list_entry(serv->sv_tempsocks.prev, - struct svc_xprt, - xpt_list); - set_bit(XPT_CLOSE, &xprt->xpt_flags); - svc_xprt_get(xprt); + list_for_each_entry_reverse(xprti, &serv->sv_tempsocks, + xpt_list) + { + if (!test_bit(XPT_PEER_VALID, &xprti->xpt_flags)) { + xprt = xprti; + set_bit(XPT_CLOSE, &xprt->xpt_flags); + svc_xprt_get(xprt); + break; + } + } } spin_unlock_bh(&serv->sv_lock); @@ -1039,7 +1039,8 @@ static void svc_delete_xprt(struct svc_xprt *xprt) spin_lock_bh(&serv->sv_lock); list_del_init(&xprt->xpt_list); - if (test_bit(XPT_TEMP, &xprt->xpt_flags)) + if (test_bit(XPT_TEMP, &xprt->xpt_flags) && + !test_bit(XPT_PEER_VALID, &xprt->xpt_flags)) serv->sv_tmpcnt--; spin_unlock_bh(&serv->sv_lock); From patchwork Wed Oct 23 02:37:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13846344 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F304D43AB9 for ; Wed, 23 Oct 2024 02:43:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651402; cv=none; b=X3Ns6nTGe4iPKNmrIm97GL5FmfW978+xXqVbrBonepKBE6HuweRckmSj144P6RBGLO1HVAultgKwzVU3fFmUtlrgT6fx5QsY+R2BGh43cXIrzrE8YEHKryrhXlvjPSRtx40UiiGRPoYSKSPmhj02vduUTCFBUiU8FZ4jIKGCJVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651402; c=relaxed/simple; bh=jCIP0S5ZwQ5QeW2JF3dsFOuZArRhikA/XVXWBQWyl14=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bvETwiCkPDHEWyqkTh0/j/Oo+YOAX7K/5HM00gHpUIypphG7/ZZ6jLqa3JIoHBPni1fQfRc6ExWfTgUSV6J/fGGRAtU6sXTVxIn9lak82p4xxPlUsABlhvIbT4YS/B6NxDqGiirgOosqf287DKE1tRM3+8MJQxoRj/TfHB80r8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=rHi/EEkL; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=JWVy89m5; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=rHi/EEkL; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=JWVy89m5; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="rHi/EEkL"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="JWVy89m5"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="rHi/EEkL"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="JWVy89m5" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 571FF1FD6C; Wed, 23 Oct 2024 02:43:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651399; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JAhqp9WSKcMvVHtAf8e5T8HyDVqOToWy0690/XemAtA=; b=rHi/EEkLjAk3oyfoaFYFu5860JSWmHFhJdbxlbWxOhp8TRXU9/nBHlqSzDRwmUKOb+Slxe GaC41IU1BxE8olKlDfp8Ta5vjaSfy3jpKExpUTBgNu4nN7tttCVo7I8HCqXHFTejb5hHve IcHFfQGi2D5Nn/Ete3K1YvNPRq1m818= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651399; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JAhqp9WSKcMvVHtAf8e5T8HyDVqOToWy0690/XemAtA=; b=JWVy89m5jaNlngPjf09CKLHYlcL4QaSens5tkz02gLR9pAi8sZ+4i/LIf28IbBzMwX5Dl3 ht/dfnj/6B8asXBw== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651399; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JAhqp9WSKcMvVHtAf8e5T8HyDVqOToWy0690/XemAtA=; b=rHi/EEkLjAk3oyfoaFYFu5860JSWmHFhJdbxlbWxOhp8TRXU9/nBHlqSzDRwmUKOb+Slxe GaC41IU1BxE8olKlDfp8Ta5vjaSfy3jpKExpUTBgNu4nN7tttCVo7I8HCqXHFTejb5hHve IcHFfQGi2D5Nn/Ete3K1YvNPRq1m818= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651399; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JAhqp9WSKcMvVHtAf8e5T8HyDVqOToWy0690/XemAtA=; b=JWVy89m5jaNlngPjf09CKLHYlcL4QaSens5tkz02gLR9pAi8sZ+4i/LIf28IbBzMwX5Dl3 ht/dfnj/6B8asXBw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 41BFA13A63; Wed, 23 Oct 2024 02:43:16 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id hD4ROsRiGGdIOgAAD6G6ig (envelope-from ); Wed, 23 Oct 2024 02:43:16 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey Subject: [PATCH 5/6] sunrpc: remove all connection limit configuration Date: Wed, 23 Oct 2024 13:37:05 +1100 Message-ID: <20241023024222.691745-6-neilb@suse.de> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241023024222.691745-1-neilb@suse.de> References: <20241023024222.691745-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:email] X-Spam-Flag: NO X-Spam-Level: Now that the connection limit only apply to unconfirmed connections, there is no need to configure it. So remove all the configuration and fix the number of unconfirmed connections as always 64 - which is now given a name: XPT_MAX_TMP_CONN Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/lockd/svc.c | 8 ------- fs/nfsd/netns.h | 6 ----- fs/nfsd/nfsctl.c | 42 --------------------------------- fs/nfsd/nfssvc.c | 5 ---- include/linux/sunrpc/svc.h | 4 ---- include/linux/sunrpc/svc_xprt.h | 6 +++++ net/sunrpc/svc_xprt.c | 8 +------ 7 files changed, 7 insertions(+), 72 deletions(-) diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c index 4ec22c2f2ea3..7ded57ec3a60 100644 --- a/fs/lockd/svc.c +++ b/fs/lockd/svc.c @@ -70,9 +70,6 @@ static unsigned long nlm_grace_period; unsigned long nlm_timeout = LOCKD_DFLT_TIMEO; static int nlm_udpport, nlm_tcpport; -/* RLIM_NOFILE defaults to 1024. That seems like a reasonable default here. */ -static unsigned int nlm_max_connections = 1024; - /* * Constants needed for the sysctl interface. */ @@ -136,9 +133,6 @@ lockd(void *vrqstp) * NFS mount or NFS daemon has gone away. */ while (!svc_thread_should_stop(rqstp)) { - /* update sv_maxconn if it has changed */ - rqstp->rq_server->sv_maxconn = nlm_max_connections; - nlmsvc_retry_blocked(rqstp); svc_recv(rqstp); } @@ -340,7 +334,6 @@ static int lockd_get(void) return -ENOMEM; } - serv->sv_maxconn = nlm_max_connections; error = svc_set_num_threads(serv, NULL, 1); if (error < 0) { svc_destroy(&serv); @@ -542,7 +535,6 @@ module_param_call(nlm_udpport, param_set_port, param_get_int, module_param_call(nlm_tcpport, param_set_port, param_get_int, &nlm_tcpport, 0644); module_param(nsm_use_hostnames, bool, 0644); -module_param(nlm_max_connections, uint, 0644); static int lockd_init_net(struct net *net) { diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index a05a45bb1978..4a07b8d0837b 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -128,12 +128,6 @@ struct nfsd_net { seqlock_t writeverf_lock; unsigned char writeverf[8]; - /* - * Max number of non-validated connections this nfsd container - * will allow. Defaults to '0' gets mapped to 64. - */ - unsigned int max_connections; - u32 clientid_base; u32 clientid_counter; u32 clverifier_counter; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 3adbc05ebaac..95ea4393305b 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -48,7 +48,6 @@ enum { NFSD_Versions, NFSD_Ports, NFSD_MaxBlkSize, - NFSD_MaxConnections, NFSD_Filecache, NFSD_Leasetime, NFSD_Gracetime, @@ -68,7 +67,6 @@ static ssize_t write_pool_threads(struct file *file, char *buf, size_t size); static ssize_t write_versions(struct file *file, char *buf, size_t size); static ssize_t write_ports(struct file *file, char *buf, size_t size); static ssize_t write_maxblksize(struct file *file, char *buf, size_t size); -static ssize_t write_maxconn(struct file *file, char *buf, size_t size); #ifdef CONFIG_NFSD_V4 static ssize_t write_leasetime(struct file *file, char *buf, size_t size); static ssize_t write_gracetime(struct file *file, char *buf, size_t size); @@ -87,7 +85,6 @@ static ssize_t (*const write_op[])(struct file *, char *, size_t) = { [NFSD_Versions] = write_versions, [NFSD_Ports] = write_ports, [NFSD_MaxBlkSize] = write_maxblksize, - [NFSD_MaxConnections] = write_maxconn, #ifdef CONFIG_NFSD_V4 [NFSD_Leasetime] = write_leasetime, [NFSD_Gracetime] = write_gracetime, @@ -902,44 +899,6 @@ static ssize_t write_maxblksize(struct file *file, char *buf, size_t size) nfsd_max_blksize); } -/* - * write_maxconn - Set or report the current max number of connections - * - * Input: - * buf: ignored - * size: zero - * OR - * - * Input: - * buf: C string containing an unsigned - * integer value representing the new - * number of max connections - * size: non-zero length of C string in @buf - * Output: - * On success: passed-in buffer filled with '\n'-terminated C string - * containing numeric value of max_connections setting - * for this net namespace; - * return code is the size in bytes of the string - * On error: return code is zero or a negative errno value - */ -static ssize_t write_maxconn(struct file *file, char *buf, size_t size) -{ - char *mesg = buf; - struct nfsd_net *nn = net_generic(netns(file), nfsd_net_id); - unsigned int maxconn = nn->max_connections; - - if (size > 0) { - int rv = get_uint(&mesg, &maxconn); - - if (rv) - return rv; - trace_nfsd_ctl_maxconn(netns(file), maxconn); - nn->max_connections = maxconn; - } - - return scnprintf(buf, SIMPLE_TRANSACTION_LIMIT, "%u\n", maxconn); -} - #ifdef CONFIG_NFSD_V4 static ssize_t __nfsd4_write_time(struct file *file, char *buf, size_t size, time64_t *time, struct nfsd_net *nn) @@ -1372,7 +1331,6 @@ static int nfsd_fill_super(struct super_block *sb, struct fs_context *fc) [NFSD_Versions] = {"versions", &transaction_ops, S_IWUSR|S_IRUSR}, [NFSD_Ports] = {"portlist", &transaction_ops, S_IWUSR|S_IRUGO}, [NFSD_MaxBlkSize] = {"max_block_size", &transaction_ops, S_IWUSR|S_IRUGO}, - [NFSD_MaxConnections] = {"max_connections", &transaction_ops, S_IWUSR|S_IRUGO}, [NFSD_Filecache] = {"filecache", &nfsd_file_cache_stats_fops, S_IRUGO}, #ifdef CONFIG_NFSD_V4 [NFSD_Leasetime] = {"nfsv4leasetime", &transaction_ops, S_IWUSR|S_IRUSR}, diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index e596eb5d10db..1a172a7e9e0c 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -671,7 +671,6 @@ int nfsd_create_serv(struct net *net) if (serv == NULL) return -ENOMEM; - serv->sv_maxconn = nn->max_connections; error = svc_bind(serv, net); if (error < 0) { svc_destroy(&serv); @@ -957,11 +956,7 @@ nfsd(void *vrqstp) * The main request loop */ while (!svc_thread_should_stop(rqstp)) { - /* Update sv_maxconn if it has changed */ - rqstp->rq_server->sv_maxconn = nn->max_connections; - svc_recv(rqstp); - nfsd_file_net_dispose(nn); } diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 617ebfff2f30..9d288a673705 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -72,10 +72,6 @@ struct svc_serv { spinlock_t sv_lock; unsigned int sv_nprogs; /* Number of sv_programs */ unsigned int sv_nrthreads; /* # of server threads */ - unsigned int sv_maxconn; /* max connections allowed or - * '0' causing max to be based - * on number of threads. */ - unsigned int sv_max_payload; /* datagram payload size */ unsigned int sv_max_mesg; /* max_payload + 1 page for overheads */ unsigned int sv_xdrsize; /* XDR buffer size */ diff --git a/include/linux/sunrpc/svc_xprt.h b/include/linux/sunrpc/svc_xprt.h index 35929a7727c7..114051ad985a 100644 --- a/include/linux/sunrpc/svc_xprt.h +++ b/include/linux/sunrpc/svc_xprt.h @@ -105,6 +105,12 @@ enum { */ }; +/* + * Maximum number of "tmp" connections - those without XPT_PEER_VALID - + * permitted on any service. + */ +#define XPT_MAX_TMP_CONN 64 + static inline void svc_xprt_set_valid(struct svc_xprt *xpt) { if (test_bit(XPT_TEMP, &xpt->xpt_flags) && diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index ff5b8bb8a88f..070bdeb50496 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -619,16 +619,10 @@ int svc_port_is_privileged(struct sockaddr *sin) * The only somewhat efficient mechanism would be if drop old * connections from the same IP first. But right now we don't even * record the client IP in svc_sock. - * - * single-threaded services that expect a lot of clients will probably - * need to set sv_maxconn to override the default value which is based - * on the number of threads */ static void svc_check_conn_limits(struct svc_serv *serv) { - unsigned int limit = serv->sv_maxconn ? serv->sv_maxconn : 64; - - if (serv->sv_tmpcnt > limit) { + if (serv->sv_tmpcnt > XPT_MAX_TMP_CONN) { struct svc_xprt *xprt = NULL, *xprti; spin_lock_bh(&serv->sv_lock); if (!list_empty(&serv->sv_tempsocks)) { From patchwork Wed Oct 23 02:37:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13846345 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 884B543AB9 for ; Wed, 23 Oct 2024 02:43:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651408; cv=none; b=BeOmqj9EeSwtUr3TRIKVXa46jCfunCGhtB2yc+f7WJw0zATrVh8DXoD90ICKIygzdHqrCQMBEjFxlXUwM+SSQuaBfIp9QzxgdYsgZH/MDB2l2fvakjlLAtIgaKXv1kvnLX9C0SOrGCY6UGk9yrVVnl0Okfb1jgDnWDzxwMmxA4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651408; c=relaxed/simple; bh=RLkKmu7UbbwjjBlkUh0fXZrdQdGhUfIsOQSIMhyehCs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qtxgugrJbUre9jNOPc10gXOhSRV0kAFSAleyhMygODaWscIWowIt/+3yH+rMCUwBz9WaiZqJyis1lbOISK6bPHDQV24KchLcmSJn7QOl8nZqqs2yYDFRAsuHzVy8tOgjjTkFTc7KK1lDuBWwkb2AmDNBqYSxsdXmv6yqJSJsQPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Reb+8l4h; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=g+7vtrqy; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Reb+8l4h; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=g+7vtrqy; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Reb+8l4h"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="g+7vtrqy"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Reb+8l4h"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="g+7vtrqy" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id CBBE41FD6C; Wed, 23 Oct 2024 02:43:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651404; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8MrjqwMdeSgI+QnmJd87IgiWz2gslumvoKMQSlvZtT4=; b=Reb+8l4hd+q5dAVR6oZLhUx/IvxdxawHsILXpSg7uzoxej4SEQPTDZ6NznpFTz4IIIFyuH WeUxLKYbk4uLIjcxY+R/zH6U80/it5F9p6nNtZVAfvE/a6jJf4T/FhBdB/XIpGEk9aWvG/ bGZqbsRCpZEUx/49kLO9wuGR0ZRQm9A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651404; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8MrjqwMdeSgI+QnmJd87IgiWz2gslumvoKMQSlvZtT4=; b=g+7vtrqy0MIa9v9vuLhWEgthk8/nLCTyj4+gSKRHy7aPgNL3B5kfYKdK7yieO7M9ejSi55 h8H5Cb7S9/P0HyDw== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1729651404; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8MrjqwMdeSgI+QnmJd87IgiWz2gslumvoKMQSlvZtT4=; b=Reb+8l4hd+q5dAVR6oZLhUx/IvxdxawHsILXpSg7uzoxej4SEQPTDZ6NznpFTz4IIIFyuH WeUxLKYbk4uLIjcxY+R/zH6U80/it5F9p6nNtZVAfvE/a6jJf4T/FhBdB/XIpGEk9aWvG/ bGZqbsRCpZEUx/49kLO9wuGR0ZRQm9A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1729651404; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8MrjqwMdeSgI+QnmJd87IgiWz2gslumvoKMQSlvZtT4=; b=g+7vtrqy0MIa9v9vuLhWEgthk8/nLCTyj4+gSKRHy7aPgNL3B5kfYKdK7yieO7M9ejSi55 h8H5Cb7S9/P0HyDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B5CF413A63; Wed, 23 Oct 2024 02:43:22 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id YzvUGspiGGdNOgAAD6G6ig (envelope-from ); Wed, 23 Oct 2024 02:43:22 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey Subject: [PATCH 6/6] sunrpc: introduce possibility that requested number of threads is different from actual Date: Wed, 23 Oct 2024 13:37:06 +1100 Message-ID: <20241023024222.691745-7-neilb@suse.de> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241023024222.691745-1-neilb@suse.de> References: <20241023024222.691745-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,suse.de:mid]; RCVD_TLS_ALL(0.00)[] X-Spam-Score: -2.80 X-Spam-Flag: NO New fields sp_nractual and sv_nractual track how many actual threads are running. sp_nrhtreads and sv_nrthreads will be the number that were explicitly request. Currently nractually == nrthreads. sv_nractual is used for sizing UDP incoming socket space - in the rare case that UDP is used. This is because each thread might need to keep a request in the skbs. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- include/linux/sunrpc/svc.h | 6 ++++-- net/sunrpc/svc.c | 22 +++++++++++++++------- net/sunrpc/svcsock.c | 2 +- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 9d288a673705..3f2c90061b4a 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -36,7 +36,8 @@ struct svc_pool { unsigned int sp_id; /* pool id; also node id on NUMA */ struct lwq sp_xprts; /* pending transports */ - unsigned int sp_nrthreads; /* # of threads in pool */ + unsigned int sp_nrthreads; /* # of threads requested for pool */ + unsigned int sp_nractual; /* # of threads running */ struct list_head sp_all_threads; /* all server threads */ struct llist_head sp_idle_threads; /* idle server threads */ @@ -71,7 +72,8 @@ struct svc_serv { struct svc_stat * sv_stats; /* RPC statistics */ spinlock_t sv_lock; unsigned int sv_nprogs; /* Number of sv_programs */ - unsigned int sv_nrthreads; /* # of server threads */ + unsigned int sv_nrthreads; /* # of server threads requested*/ + unsigned int sv_nractual; /* # of running threads */ unsigned int sv_max_payload; /* datagram payload size */ unsigned int sv_max_mesg; /* max_payload + 1 page for overheads */ unsigned int sv_xdrsize; /* XDR buffer size */ diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index bd4f02b34f44..d332f9d3d875 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -781,8 +781,12 @@ svc_pool_victim(struct svc_serv *serv, struct svc_pool *target_pool, } if (pool && pool->sp_nrthreads) { - set_bit(SP_VICTIM_REMAINS, &pool->sp_flags); - set_bit(SP_NEED_VICTIM, &pool->sp_flags); + if (pool->sp_nrthreads <= pool->sp_nractual) { + set_bit(SP_VICTIM_REMAINS, &pool->sp_flags); + set_bit(SP_NEED_VICTIM, &pool->sp_flags); + pool->sp_nractual -= 1; + serv->sv_nractual -= 1; + } return pool; } return NULL; @@ -803,6 +807,12 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) chosen_pool = svc_pool_next(serv, pool, &state); node = svc_pool_map_get_node(chosen_pool->sp_id); + serv->sv_nrthreads += 1; + chosen_pool->sp_nrthreads += 1; + + if (chosen_pool->sp_nrthreads <= chosen_pool->sp_nractual) + continue; + rqstp = svc_prepare_thread(serv, chosen_pool, node); if (!rqstp) return -ENOMEM; @@ -812,8 +822,8 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) svc_exit_thread(rqstp); return PTR_ERR(task); } - serv->sv_nrthreads += 1; - chosen_pool->sp_nrthreads += 1; + serv->sv_nractual += 1; + chosen_pool->sp_nractual += 1; rqstp->rq_task = task; if (serv->sv_nrpools > 1) @@ -850,6 +860,7 @@ svc_stop_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) TASK_IDLE); nrservs++; } while (nrservs < 0); + svc_sock_update_bufs(serv); return 0; } @@ -955,13 +966,10 @@ void svc_rqst_release_pages(struct svc_rqst *rqstp) void svc_exit_thread(struct svc_rqst *rqstp) { - struct svc_serv *serv = rqstp->rq_server; struct svc_pool *pool = rqstp->rq_pool; list_del_rcu(&rqstp->rq_all); - svc_sock_update_bufs(serv); - svc_rqst_free(rqstp); clear_and_wake_up_bit(SP_VICTIM_REMAINS, &pool->sp_flags); diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 825ec5357691..191dbc648bd0 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -588,7 +588,7 @@ static int svc_udp_recvfrom(struct svc_rqst *rqstp) * provides an upper bound on the number of threads * which will access the socket. */ - svc_sock_setbufsize(svsk, serv->sv_nrthreads + 3); + svc_sock_setbufsize(svsk, serv->sv_nractual + 3); clear_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags); err = kernel_recvmsg(svsk->sk_sock, &msg, NULL,