From patchwork Mon Jul 15 07:14:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733032 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13CB01836E6 for ; Mon, 15 Jul 2024 07:47:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029660; cv=none; b=ETOHGFNVD3Tox/ucks80qwPZIzqRsNnvlIR1L2VbdJsOWLFppKuFpdNWiQhhKeEmBRyNN/BZMFPaGhxrb3Hijf58g46M/sjkrZaI6Um27dCdPsFsHFJyg+SWcxe5LgFpgTN6/Qdn6ZTHBtbqf7G2sL4afeWMBL8kwPNDFxnWsgw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029660; c=relaxed/simple; bh=zb2KPj+r/OjAmUGPqWtkD2QKRgTxnfpygHbihaCSNVA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h6zgeeiTox+2XojHXbnnJKe+GifPtPX38FOoImKGN4KNn2RIhbiqO7Y+k9s7/JWAAg4HW6gwxqJWj0owNTrc7Iqr4HMDYfbTRwcH/sUT3eeiBAml5gnXi6Fu5FdGdS0Tt0ah5HAJtSl357VrvvCueO8N4tiEiPFtLAIE1ao9/sE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=xtmcQQbi; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=kQdnXnS8; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=xtmcQQbi; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=kQdnXnS8; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="xtmcQQbi"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="kQdnXnS8"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="xtmcQQbi"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="kQdnXnS8" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 24F621F80B; Mon, 15 Jul 2024 07:47:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029657; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+v14Rt4o3037+hhkNelW3hokj1y8/ArfmyTgs9MokJQ=; b=xtmcQQbifjxY6bO8JPrWN3OmDaKa1fSIZXO0KXf+owUsBJXHN7NtjL36Ct97kyyyCfBfwX Tc3Hz2nHWy35IZPVOcQnA29s3GlnLrvpnjse06KjMvtu8wY0wv/HMQ3Cc2rdUTc0MhTH/H XvzxGc3rXD9PRxKUJnGoqYfwmET1By4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029657; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+v14Rt4o3037+hhkNelW3hokj1y8/ArfmyTgs9MokJQ=; b=kQdnXnS842/0hquhPvhNfkZwGJ7/H0OXbgRmsfUd8OQa0aWsnmHHfGoFNZLTyuQWXxjTDM naGWGD7wemfTqnAg== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029657; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+v14Rt4o3037+hhkNelW3hokj1y8/ArfmyTgs9MokJQ=; b=xtmcQQbifjxY6bO8JPrWN3OmDaKa1fSIZXO0KXf+owUsBJXHN7NtjL36Ct97kyyyCfBfwX Tc3Hz2nHWy35IZPVOcQnA29s3GlnLrvpnjse06KjMvtu8wY0wv/HMQ3Cc2rdUTc0MhTH/H XvzxGc3rXD9PRxKUJnGoqYfwmET1By4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029657; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+v14Rt4o3037+hhkNelW3hokj1y8/ArfmyTgs9MokJQ=; b=kQdnXnS842/0hquhPvhNfkZwGJ7/H0OXbgRmsfUd8OQa0aWsnmHHfGoFNZLTyuQWXxjTDM naGWGD7wemfTqnAg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B1F99137EB; Mon, 15 Jul 2024 07:47:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 6f7fGRbUlGa2bQAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:47:34 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 01/14] lockd: discard nlmsvc_timeout Date: Mon, 15 Jul 2024 17:14:14 +1000 Message-ID: <20240715074657.18174-2-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:email]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Score: -2.80 X-Spam-Level: nlmsvc_timeout always has the same value as (nlm_timeout * HZ), so use that in the one place that nlmsvc_timeout is used. In truth it *might* not always be the same as nlmsvc_timeout is only set when lockd is started while nlm_timeout can be set at anytime via sysctl. I think this difference it not helpful so removing it is good. Also remove the test for nlm_timout being 0. This is not possible - unless a module parameter is used to set the minimum timeout to 0, and if that happens then it probably should be honoured. Signed-off-by: NeilBrown --- fs/lockd/host.c | 2 +- fs/lockd/svc.c | 7 +------ include/linux/lockd/lockd.h | 2 +- 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/fs/lockd/host.c b/fs/lockd/host.c index c11516801784..5e6877c37f73 100644 --- a/fs/lockd/host.c +++ b/fs/lockd/host.c @@ -440,7 +440,7 @@ nlm_bind_host(struct nlm_host *host) if ((clnt = host->h_rpcclnt) != NULL) { nlm_rebind_host(host); } else { - unsigned long increment = nlmsvc_timeout; + unsigned long increment = nlm_timeout * HZ; struct rpc_timeout timeparms = { .to_initval = increment, .to_increment = increment, diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c index ab8042a5b895..71713309967d 100644 --- a/fs/lockd/svc.c +++ b/fs/lockd/svc.c @@ -53,7 +53,6 @@ EXPORT_SYMBOL_GPL(nlmsvc_ops); static DEFINE_MUTEX(nlmsvc_mutex); static unsigned int nlmsvc_users; static struct svc_serv *nlmsvc_serv; -unsigned long nlmsvc_timeout; static void nlmsvc_request_retry(struct timer_list *tl) { @@ -68,7 +67,7 @@ unsigned int lockd_net_id; * and also changed through the sysctl interface. -- Jamie Lokier, Aug 2003 */ static unsigned long nlm_grace_period; -static unsigned long nlm_timeout = LOCKD_DFLT_TIMEO; +unsigned long nlm_timeout = LOCKD_DFLT_TIMEO; static int nlm_udpport, nlm_tcpport; /* RLIM_NOFILE defaults to 1024. That seems like a reasonable default here. */ @@ -333,10 +332,6 @@ static int lockd_get(void) printk(KERN_WARNING "lockd_up: no pid, %d users??\n", nlmsvc_users); - if (!nlm_timeout) - nlm_timeout = LOCKD_DFLT_TIMEO; - nlmsvc_timeout = nlm_timeout * HZ; - serv = svc_create(&nlmsvc_program, LOCKD_BUFSIZE, lockd); if (!serv) { printk(KERN_WARNING "lockd_up: create service failed\n"); diff --git a/include/linux/lockd/lockd.h b/include/linux/lockd/lockd.h index 1b95fe31051f..61c4b9c41904 100644 --- a/include/linux/lockd/lockd.h +++ b/include/linux/lockd/lockd.h @@ -200,7 +200,7 @@ extern const struct svc_procedure nlmsvc_procedures[24]; extern const struct svc_procedure nlmsvc_procedures4[24]; #endif extern int nlmsvc_grace_period; -extern unsigned long nlmsvc_timeout; +extern unsigned long nlm_timeout; extern bool nsm_use_hostnames; extern u32 nsm_local_state; From patchwork Mon Jul 15 07:14:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733033 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCDA3175A6 for ; Mon, 15 Jul 2024 07:47:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029670; cv=none; b=Kfol9R9fQZ62Wq3lTCXt1Ghk3jE06yohLqFJxseeRxPTcS5cK4A27PsjtdUYRjRoIPykCwXsmvBk87SRkpNKV1avpiCvMzvakTRJ5qLYcYJ9yWme9BKZuoMIpzDDZXX5wo7j7G8HE1O4tqiNWPo/uGevrjR4lhCa1H7TJ6oRAPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029670; c=relaxed/simple; bh=zGv4lgTOk5B4Vez9M9+1NBEq6MzGbBBrGiUT2VF1sAg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hVSnToag1/NEzMnA/krcWA3kGBZvZd4A5MLybWwuhLtcmbTPGyeFzQpkzjv6KRgik+HaevgJjwdOXk175Aws5l4ax2WWd1rizzhOJRaMN/PWVPjPzIx8UxwG053U52d/SN6tmtjCIDNqRWfQQfNBAwbCEF7aVdABLXmA2vMOOXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=VZftjmVW; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=X62gdTbO; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=VZftjmVW; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=X62gdTbO; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="VZftjmVW"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="X62gdTbO"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="VZftjmVW"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="X62gdTbO" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E59AC21B95; Mon, 15 Jul 2024 07:47:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029666; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X9uwQZxc45oL3RzIBCFXeXBOQabgo6olEaP7HqJa8nE=; b=VZftjmVWzogaGbOsR4HZNfeBu1PJwu0+IuqYArVvFvr9tnabJeRrMaQg7vO8W3+SV574xa NETVn5Xijro9JueKHu2uTsOpBfk1hUtSZo8CXSAMbM99927joHF3t+Qa+eQOmz5i+dknvm UWE3jb3cgYXU3Hlbgrz9rjZvwE1zP7U= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029666; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X9uwQZxc45oL3RzIBCFXeXBOQabgo6olEaP7HqJa8nE=; b=X62gdTbOFGKa2m9nA37Q3vP7QlpMQohsv5xTvus289W6m0Gwi60n6rUknq3N5RfzLUZi4y 4ahEZG47uW3vNkBg== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029666; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X9uwQZxc45oL3RzIBCFXeXBOQabgo6olEaP7HqJa8nE=; b=VZftjmVWzogaGbOsR4HZNfeBu1PJwu0+IuqYArVvFvr9tnabJeRrMaQg7vO8W3+SV574xa NETVn5Xijro9JueKHu2uTsOpBfk1hUtSZo8CXSAMbM99927joHF3t+Qa+eQOmz5i+dknvm UWE3jb3cgYXU3Hlbgrz9rjZvwE1zP7U= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029666; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X9uwQZxc45oL3RzIBCFXeXBOQabgo6olEaP7HqJa8nE=; b=X62gdTbOFGKa2m9nA37Q3vP7QlpMQohsv5xTvus289W6m0Gwi60n6rUknq3N5RfzLUZi4y 4ahEZG47uW3vNkBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 811B3137EB; Mon, 15 Jul 2024 07:47:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id RWL3DSDUlGbLbQAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:47:44 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 02/14] SUNRPC: make various functions static, or not exported. Date: Mon, 15 Jul 2024 17:14:15 +1000 Message-ID: <20240715074657.18174-3-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,imap1.dmz-prg2.suse.org:helo]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Score: -2.80 X-Spam-Level: Various functions are only used within the sunrpc module, and several are only use in the one file. So clean up: These are marked static, and any EXPORT is removed. svc_rcpb_setup() svc_rqst_alloc() svc_rqst_free() - also moved before first use svc_rpcbind_set_version() svc_drop() - also moved to svc.c These are now not EXPORTed, but are not static. svc_authenticate() svc_sock_update_bufs() Signed-off-by: NeilBrown --- include/linux/sunrpc/svc.h | 9 ------- include/linux/sunrpc/svcauth.h | 1 - include/linux/sunrpc/svcsock.h | 2 -- net/sunrpc/sunrpc.h | 4 +++ net/sunrpc/svc.c | 48 ++++++++++++++++++---------------- net/sunrpc/svc_xprt.c | 9 ------- net/sunrpc/svcauth.c | 1 - net/sunrpc/svcsock.c | 1 - 8 files changed, 29 insertions(+), 46 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index a7d0406b9ef5..e4fa25fafa97 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -401,17 +401,13 @@ struct svc_procedure { */ int sunrpc_set_pool_mode(const char *val); int sunrpc_get_pool_mode(char *val, size_t size); -int svc_rpcb_setup(struct svc_serv *serv, struct net *net); void svc_rpcb_cleanup(struct svc_serv *serv, struct net *net); int svc_bind(struct svc_serv *serv, struct net *net); struct svc_serv *svc_create(struct svc_program *, unsigned int, int (*threadfn)(void *data)); -struct svc_rqst *svc_rqst_alloc(struct svc_serv *serv, - struct svc_pool *pool, int node); bool svc_rqst_replace_page(struct svc_rqst *rqstp, struct page *page); void svc_rqst_release_pages(struct svc_rqst *rqstp); -void svc_rqst_free(struct svc_rqst *); void svc_exit_thread(struct svc_rqst *); struct svc_serv * svc_create_pooled(struct svc_program *prog, struct svc_stat *stats, @@ -446,11 +442,6 @@ int svc_generic_rpcbind_set(struct net *net, u32 version, int family, unsigned short proto, unsigned short port); -int svc_rpcbind_set_version(struct net *net, - const struct svc_program *progp, - u32 version, int family, - unsigned short proto, - unsigned short port); #define RPC_MAX_ADDRBUFLEN (63U) diff --git a/include/linux/sunrpc/svcauth.h b/include/linux/sunrpc/svcauth.h index 61c455f1e1f5..63cf6fb26dcc 100644 --- a/include/linux/sunrpc/svcauth.h +++ b/include/linux/sunrpc/svcauth.h @@ -151,7 +151,6 @@ struct auth_ops { struct svc_xprt; -extern enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp); extern rpc_authflavor_t svc_auth_flavor(struct svc_rqst *rqstp); extern int svc_authorise(struct svc_rqst *rqstp); extern enum svc_auth_status svc_set_client(struct svc_rqst *rqstp); diff --git a/include/linux/sunrpc/svcsock.h b/include/linux/sunrpc/svcsock.h index 7c78ec6356b9..bf45d9e8492a 100644 --- a/include/linux/sunrpc/svcsock.h +++ b/include/linux/sunrpc/svcsock.h @@ -58,8 +58,6 @@ static inline u32 svc_sock_final_rec(struct svc_sock *svsk) */ void svc_recv(struct svc_rqst *rqstp); void svc_send(struct svc_rqst *rqstp); -void svc_drop(struct svc_rqst *); -void svc_sock_update_bufs(struct svc_serv *serv); int svc_addsock(struct svc_serv *serv, struct net *net, const int fd, char *name_return, const size_t len, const struct cred *cred); diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h index d4a362c9e4b3..e3c6e3b63f0b 100644 --- a/net/sunrpc/sunrpc.h +++ b/net/sunrpc/sunrpc.h @@ -36,7 +36,11 @@ static inline int sock_is_loopback(struct sock *sk) return loopback; } +struct svc_serv; +struct svc_rqst; int rpc_clients_notifier_register(void); void rpc_clients_notifier_unregister(void); void auth_domain_cleanup(void); +void svc_sock_update_bufs(struct svc_serv *serv); +enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp); #endif /* _NET_SUNRPC_SUNRPC_H */ diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index e03f14024e47..072ad115ae3d 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -32,6 +32,7 @@ #include #include "fail.h" +#include "sunrpc.h" #define RPCDBG_FACILITY RPCDBG_SVCDSP @@ -417,7 +418,7 @@ struct svc_pool *svc_pool_for_cpu(struct svc_serv *serv) return &serv->sv_pools[pidx % serv->sv_nrpools]; } -int svc_rpcb_setup(struct svc_serv *serv, struct net *net) +static int svc_rpcb_setup(struct svc_serv *serv, struct net *net) { int err; @@ -429,7 +430,6 @@ int svc_rpcb_setup(struct svc_serv *serv, struct net *net) svc_unregister(serv, net); return 0; } -EXPORT_SYMBOL_GPL(svc_rpcb_setup); void svc_rpcb_cleanup(struct svc_serv *serv, struct net *net) { @@ -664,7 +664,20 @@ svc_release_buffer(struct svc_rqst *rqstp) put_page(rqstp->rq_pages[i]); } -struct svc_rqst * +static void +svc_rqst_free(struct svc_rqst *rqstp) +{ + folio_batch_release(&rqstp->rq_fbatch); + svc_release_buffer(rqstp); + if (rqstp->rq_scratch_page) + put_page(rqstp->rq_scratch_page); + kfree(rqstp->rq_resp); + kfree(rqstp->rq_argp); + kfree(rqstp->rq_auth_data); + kfree_rcu(rqstp, rq_rcu_head); +} + +static struct svc_rqst * svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node) { struct svc_rqst *rqstp; @@ -698,7 +711,6 @@ svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node) svc_rqst_free(rqstp); return NULL; } -EXPORT_SYMBOL_GPL(svc_rqst_alloc); static struct svc_rqst * svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node) @@ -933,24 +945,6 @@ void svc_rqst_release_pages(struct svc_rqst *rqstp) } } -/* - * Called from a server thread as it's exiting. Caller must hold the "service - * mutex" for the service. - */ -void -svc_rqst_free(struct svc_rqst *rqstp) -{ - folio_batch_release(&rqstp->rq_fbatch); - svc_release_buffer(rqstp); - if (rqstp->rq_scratch_page) - put_page(rqstp->rq_scratch_page); - kfree(rqstp->rq_resp); - kfree(rqstp->rq_argp); - kfree(rqstp->rq_auth_data); - kfree_rcu(rqstp, rq_rcu_head); -} -EXPORT_SYMBOL_GPL(svc_rqst_free); - void svc_exit_thread(struct svc_rqst *rqstp) { @@ -1098,6 +1092,7 @@ static int __svc_register(struct net *net, const char *progname, return error; } +static int svc_rpcbind_set_version(struct net *net, const struct svc_program *progp, u32 version, int family, @@ -1108,7 +1103,6 @@ int svc_rpcbind_set_version(struct net *net, version, family, proto, port); } -EXPORT_SYMBOL_GPL(svc_rpcbind_set_version); int svc_generic_rpcbind_set(struct net *net, const struct svc_program *progp, @@ -1526,6 +1520,14 @@ svc_process_common(struct svc_rqst *rqstp) goto sendit; } +/* + * Drop request + */ +static void svc_drop(struct svc_rqst *rqstp) +{ + trace_svc_drop(rqstp); +} + /** * svc_process - Execute one RPC transaction * @rqstp: RPC transaction context diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index d3735ab3e6d1..53ebc719ff5a 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -905,15 +905,6 @@ void svc_recv(struct svc_rqst *rqstp) } EXPORT_SYMBOL_GPL(svc_recv); -/* - * Drop request - */ -void svc_drop(struct svc_rqst *rqstp) -{ - trace_svc_drop(rqstp); -} -EXPORT_SYMBOL_GPL(svc_drop); - /** * svc_send - Return reply to client * @rqstp: RPC transaction context diff --git a/net/sunrpc/svcauth.c b/net/sunrpc/svcauth.c index 1619211f0960..93d9e949e265 100644 --- a/net/sunrpc/svcauth.c +++ b/net/sunrpc/svcauth.c @@ -98,7 +98,6 @@ enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp) rqstp->rq_authop = aops; return aops->accept(rqstp); } -EXPORT_SYMBOL_GPL(svc_authenticate); /** * svc_set_client - Assign an appropriate 'auth_domain' as the client diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 6b3f01beb294..825ec5357691 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -1378,7 +1378,6 @@ void svc_sock_update_bufs(struct svc_serv *serv) set_bit(XPT_CHNGBUF, &svsk->sk_xprt.xpt_flags); spin_unlock_bh(&serv->sv_lock); } -EXPORT_SYMBOL_GPL(svc_sock_update_bufs); /* * Initialize socket for RPC use and create svc_sock struct From patchwork Mon Jul 15 07:14:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733034 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0EB5171E60 for ; Mon, 15 Jul 2024 07:48:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029684; cv=none; b=STcotiL1PUSzBoXH0ydUTnhoq+7DLMlMW3Giv4s7IvZ0g5nOl4ZBnBvhykwytFRhkYUMIslDCOewYMwEvm1DeKUlJSfe6kXtJ+T99Eo4UMk5p5f1nzycOQX66iXfoLCvLmAtEYxnQH5ZJjnWUqrXaqHdtsYWFpQ6osnEtvPzHMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029684; c=relaxed/simple; bh=HMf8nadfHZrqhSMc0OEmbeqj7bjEsGvDPKJgWV+wY40=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h6+2qxx8OHHSzSe03WUQFbq8a84DsQivQ2QZkZ6OtrXv7/QB/JaHJsinJbvJVnMG5l4TAhdzeatkUc84aCdidDWcgWQMH6NF/G9o075U4gIsNBwYcaDakQlH9FzQeH32r3mdhVGwfCF/n3zgxC2IY+HHp1YX55mupyeasszzwlw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=q1mh5WYg; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=vG/5zg4P; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=q1mh5WYg; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=vG/5zg4P; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="q1mh5WYg"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="vG/5zg4P"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="q1mh5WYg"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="vG/5zg4P" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id BA75A21B94; Mon, 15 Jul 2024 07:48:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029680; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yZt5Mxs7PNooXY2SkHlQyLphtZXwvbQEjr2BEFIyhc8=; b=q1mh5WYgaSmpN/lQn1CXp5sBBxmJtbrPXztC0pEpU4HTc6ORiXnDAa+Xw6W/IyFrfa5xXj uc5WM8jpEoxoNOAD+OiJfDMcs/3cNGZq+utLaJ/YhO+G9UQc6rqLrbXN4Btodygy/Cj3So 6H1dYp+9OWq7F05lAhnVRR9pd15CCvw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029680; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yZt5Mxs7PNooXY2SkHlQyLphtZXwvbQEjr2BEFIyhc8=; b=vG/5zg4PJzKZ/SKYJgI8GxxIsH7r4jtarRLzT9XkLfXFKE9h+lVep+WuWTs3umdEH0qKOT xw+CP0KZr0Q9BdCw== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=q1mh5WYg; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b="vG/5zg4P" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029680; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yZt5Mxs7PNooXY2SkHlQyLphtZXwvbQEjr2BEFIyhc8=; b=q1mh5WYgaSmpN/lQn1CXp5sBBxmJtbrPXztC0pEpU4HTc6ORiXnDAa+Xw6W/IyFrfa5xXj uc5WM8jpEoxoNOAD+OiJfDMcs/3cNGZq+utLaJ/YhO+G9UQc6rqLrbXN4Btodygy/Cj3So 6H1dYp+9OWq7F05lAhnVRR9pd15CCvw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029680; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yZt5Mxs7PNooXY2SkHlQyLphtZXwvbQEjr2BEFIyhc8=; b=vG/5zg4PJzKZ/SKYJgI8GxxIsH7r4jtarRLzT9XkLfXFKE9h+lVep+WuWTs3umdEH0qKOT xw+CP0KZr0Q9BdCw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 54A0F137EB; Mon, 15 Jul 2024 07:47:58 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id cvrPAi7UlGbZbQAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:47:58 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 03/14] nfsd: move nfsd_pool_stats_open into nfsctl.c Date: Mon, 15 Jul 2024 17:14:16 +1000 Message-ID: <20240715074657.18174-4-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.de:+]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_SEVEN(0.00)[7]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; RCVD_VIA_SMTP_AUTH(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,suse.de:dkim,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns] X-Spam-Flag: NO X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Level: X-Rspamd-Queue-Id: BA75A21B94 nfsd_pool_stats_open() is used in nfsctl.c, so move it there. Signed-off-by: NeilBrown --- fs/nfsd/nfsctl.c | 7 +++++++ fs/nfsd/nfsd.h | 2 -- fs/nfsd/nfssvc.c | 7 ------- 3 files changed, 7 insertions(+), 9 deletions(-) diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 9e0ea6fc2aa3..9b47723fc110 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -174,6 +174,13 @@ static int export_features_show(struct seq_file *m, void *v) DEFINE_SHOW_ATTRIBUTE(export_features); +static int nfsd_pool_stats_open(struct inode *inode, struct file *file) +{ + struct nfsd_net *nn = net_generic(inode->i_sb->s_fs_info, nfsd_net_id); + + return svc_pool_stats_open(&nn->nfsd_info, file); +} + static const struct file_operations pool_stats_operations = { .open = nfsd_pool_stats_open, .read = seq_read, diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index cec8697b1cd6..39e109a7d56d 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -111,8 +111,6 @@ int nfsd_nrthreads(struct net *); int nfsd_nrpools(struct net *); int nfsd_get_nrthreads(int n, int *, struct net *); int nfsd_set_nrthreads(int n, int *, struct net *); -int nfsd_pool_stats_open(struct inode *, struct file *); -int nfsd_pool_stats_release(struct inode *, struct file *); void nfsd_shutdown_threads(struct net *net); bool i_am_nfsd(void); diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 0bc8eaa5e009..f25b26bc5670 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -1084,10 +1084,3 @@ bool nfssvc_encode_voidres(struct svc_rqst *rqstp, struct xdr_stream *xdr) { return true; } - -int nfsd_pool_stats_open(struct inode *inode, struct file *file) -{ - struct nfsd_net *nn = net_generic(inode->i_sb->s_fs_info, nfsd_net_id); - - return svc_pool_stats_open(&nn->nfsd_info, file); -} From patchwork Mon Jul 15 07:14:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733035 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64031171E60 for ; Mon, 15 Jul 2024 07:48:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029690; cv=none; b=CIx5CjHWyukKKqFBIZNXarcRESlPcxgg+yfe+ugRAJLRNoA/T7UvI9NlpuCjKBtJ9JXyDdpoVK53tSDLDFe3bGMCBBuGGM6DYsZug3dicddnf1TLdQ2KUaMrn1cQ283vpCLnNvQ2Wp52eUDZpw931vw/xy9YzrYai8YIA26wUDk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029690; c=relaxed/simple; bh=XAIt577rdD1BLzIIoxIA3+BIZ6OcaoafplETOQgO6aE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=m71S9xyKW80OXv+t4y3w2xvjPGhsMFuxuPgVjzRiClmJLC8DaIzHtR3OJVrEvDF6LxBlgR5677bJKAAV++nIVFL85Ji7YKybxA9oPLsLd0NGlcjbPpWaPGHtV4lWJADRoizDVVQc7x3Fp6yOiJsy5PnQ67qszmGUK3hiTtaaQIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=vAhhQ6AM; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=tegVgq8X; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=vAhhQ6AM; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=tegVgq8X; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="vAhhQ6AM"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="tegVgq8X"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="vAhhQ6AM"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="tegVgq8X" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 919F61F7B2; Mon, 15 Jul 2024 07:48:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029686; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LkrXDdPcfKMg2MMMJGfN7MQNt+6lc0RMFe7+No+U/Mg=; b=vAhhQ6AMHS6smOxabLafiRrlhSSMW7U43St95soiOeeIVU4f0dJddHFpiRrcu88XeeqjJk ONxHYuyK6z8evJlvECf4iFZEIHsgW8dtekcXF4s6nMdF8NCJlqEoGKbSalEIcWGvaFDG9B lraNz5Hbv/9cFHY5FT15l0M2tFJiDk4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029686; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LkrXDdPcfKMg2MMMJGfN7MQNt+6lc0RMFe7+No+U/Mg=; b=tegVgq8XwMATkHL1L379btEJ2/6DAOmVd+74pCZarjSyUrtI/GE49MS5V9bPZIVuJn8rW4 EDxdEIq4Cfd3vvAg== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029686; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LkrXDdPcfKMg2MMMJGfN7MQNt+6lc0RMFe7+No+U/Mg=; b=vAhhQ6AMHS6smOxabLafiRrlhSSMW7U43St95soiOeeIVU4f0dJddHFpiRrcu88XeeqjJk ONxHYuyK6z8evJlvECf4iFZEIHsgW8dtekcXF4s6nMdF8NCJlqEoGKbSalEIcWGvaFDG9B lraNz5Hbv/9cFHY5FT15l0M2tFJiDk4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029686; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LkrXDdPcfKMg2MMMJGfN7MQNt+6lc0RMFe7+No+U/Mg=; b=tegVgq8XwMATkHL1L379btEJ2/6DAOmVd+74pCZarjSyUrtI/GE49MS5V9bPZIVuJn8rW4 EDxdEIq4Cfd3vvAg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 2EA3E137EB; Mon, 15 Jul 2024 07:48:03 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id SNBBNTPUlGbibQAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:03 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 04/14] nfsd: don't allocate the versions array. Date: Mon, 15 Jul 2024 17:14:17 +1000 Message-ID: <20240715074657.18174-5-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Flag: NO X-Spam-Score: 1.20 X-Spamd-Result: default: False [1.20 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,imap1.dmz-prg2.suse.org:helo]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-Spam-Level: * Instead of using kmalloc to allocate an array for storing active version info, just declare and array to the max size - it is only 5 or so. Signed-off-by: NeilBrown --- fs/nfs/nfs4state.c | 2 + fs/nfsd/cache.h | 2 +- fs/nfsd/netns.h | 6 +-- fs/nfsd/nfsctl.c | 10 +++-- fs/nfsd/nfsd.h | 9 +++- fs/nfsd/nfssvc.c | 100 ++++++++------------------------------------- 6 files changed, 36 insertions(+), 93 deletions(-) diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index 5b452411e8fd..68c663626480 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -1953,6 +1953,8 @@ static int nfs4_do_reclaim(struct nfs_client *clp, const struct nfs4_state_recov if (lost_locks) pr_warn("NFS: %s: lost %d locks\n", clp->cl_hostname, lost_locks); + nfs4_free_state_owners(&freeme); + set_bit(ops->owner_flag_bit, &sp->so_flags); nfs4_put_state_owner(sp); status = nfs4_recovery_handle_error(clp, status); diff --git a/fs/nfsd/cache.h b/fs/nfsd/cache.h index 66a05fefae98..bb7addef4a31 100644 --- a/fs/nfsd/cache.h +++ b/fs/nfsd/cache.h @@ -10,7 +10,7 @@ #define NFSCACHE_H #include -#include "netns.h" +#include "nfsd.h" /* * Representation of a reply cache entry. diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 14ec15656320..238fc4e56e53 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -152,8 +152,8 @@ struct nfsd_net { /* * Version information */ - bool *nfsd_versions; - bool *nfsd4_minorversions; + bool nfsd_versions[NFSD_MAXVERS + 1]; + bool nfsd4_minorversions[NFSD_SUPPORTED_MINOR_VERSION + 1]; /* * Duplicate reply cache @@ -219,8 +219,6 @@ struct nfsd_net { #define nfsd_netns_ready(nn) ((nn)->sessionid_hashtbl) extern bool nfsd_support_version(int vers); -extern void nfsd_netns_free_versions(struct nfsd_net *nn); - extern unsigned int nfsd_net_id; void nfsd_copy_write_verifier(__be32 verf[2], struct nfsd_net *nn); diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 9b47723fc110..5b0f2e0d7ccf 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -2232,8 +2232,9 @@ int nfsd_nl_pool_mode_get_doit(struct sk_buff *skb, struct genl_info *info) */ static __net_init int nfsd_net_init(struct net *net) { - int retval; struct nfsd_net *nn = net_generic(net, nfsd_net_id); + int retval; + int i; retval = nfsd_export_init(net); if (retval) @@ -2247,8 +2248,10 @@ static __net_init int nfsd_net_init(struct net *net) goto out_repcache_error; memset(&nn->nfsd_svcstats, 0, sizeof(nn->nfsd_svcstats)); nn->nfsd_svcstats.program = &nfsd_program; - nn->nfsd_versions = NULL; - nn->nfsd4_minorversions = NULL; + for (i = 0; i < sizeof(nn->nfsd_versions); i++) + nn->nfsd_versions[i] = nfsd_support_version(i); + for (i = 0; i < sizeof(nn->nfsd4_minorversions); i++) + nn->nfsd4_minorversions[i] = nfsd_support_version(4); nn->nfsd_info.mutex = &nfsd_mutex; nn->nfsd_serv = NULL; nfsd4_init_leases_net(nn); @@ -2279,7 +2282,6 @@ static __net_exit void nfsd_net_exit(struct net *net) percpu_counter_destroy_many(nn->counter, NFSD_STATS_COUNTERS_NUM); nfsd_idmap_shutdown(net); nfsd_export_shutdown(net); - nfsd_netns_free_versions(nn); } static struct pernet_operations nfsd_net_ops = { diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index 39e109a7d56d..369c3b3ce53e 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -23,9 +23,7 @@ #include -#include "netns.h" #include "export.h" -#include "stats.h" #undef ifdebug #ifdef CONFIG_SUNRPC_DEBUG @@ -37,7 +35,14 @@ /* * nfsd version */ +#define NFSD_MINVERS 2 +#define NFSD_MAXVERS 4 #define NFSD_SUPPORTED_MINOR_VERSION 2 +bool nfsd_support_version(int vers); + +#include "netns.h" +#include "stats.h" + /* * Maximum blocksizes supported by daemon under various circumstances. */ diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index f25b26bc5670..4438cdcd4873 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -116,15 +116,12 @@ static const struct svc_version *nfsd_version[] = { #endif }; -#define NFSD_MINVERS 2 -#define NFSD_NRVERS ARRAY_SIZE(nfsd_version) - struct svc_program nfsd_program = { #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL) .pg_next = &nfsd_acl_program, #endif .pg_prog = NFS_PROGRAM, /* program number */ - .pg_nvers = NFSD_NRVERS, /* nr of entries in nfsd_version */ + .pg_nvers = NFSD_MAXVERS+1, /* nr of entries in nfsd_version */ .pg_vers = nfsd_version, /* version table */ .pg_name = "nfsd", /* program name */ .pg_class = "nfsd", /* authentication class */ @@ -135,78 +132,24 @@ struct svc_program nfsd_program = { bool nfsd_support_version(int vers) { - if (vers >= NFSD_MINVERS && vers < NFSD_NRVERS) + if (vers >= NFSD_MINVERS && vers <= NFSD_MAXVERS) return nfsd_version[vers] != NULL; return false; } -static bool * -nfsd_alloc_versions(void) -{ - bool *vers = kmalloc_array(NFSD_NRVERS, sizeof(bool), GFP_KERNEL); - unsigned i; - - if (vers) { - /* All compiled versions are enabled by default */ - for (i = 0; i < NFSD_NRVERS; i++) - vers[i] = nfsd_support_version(i); - } - return vers; -} - -static bool * -nfsd_alloc_minorversions(void) -{ - bool *vers = kmalloc_array(NFSD_SUPPORTED_MINOR_VERSION + 1, - sizeof(bool), GFP_KERNEL); - unsigned i; - - if (vers) { - /* All minor versions are enabled by default */ - for (i = 0; i <= NFSD_SUPPORTED_MINOR_VERSION; i++) - vers[i] = nfsd_support_version(4); - } - return vers; -} - -void -nfsd_netns_free_versions(struct nfsd_net *nn) -{ - kfree(nn->nfsd_versions); - kfree(nn->nfsd4_minorversions); - nn->nfsd_versions = NULL; - nn->nfsd4_minorversions = NULL; -} - -static void -nfsd_netns_init_versions(struct nfsd_net *nn) -{ - if (!nn->nfsd_versions) { - nn->nfsd_versions = nfsd_alloc_versions(); - nn->nfsd4_minorversions = nfsd_alloc_minorversions(); - if (!nn->nfsd_versions || !nn->nfsd4_minorversions) - nfsd_netns_free_versions(nn); - } -} - int nfsd_vers(struct nfsd_net *nn, int vers, enum vers_op change) { - if (vers < NFSD_MINVERS || vers >= NFSD_NRVERS) + if (vers < NFSD_MINVERS || vers > NFSD_MAXVERS) return 0; switch(change) { case NFSD_SET: - if (nn->nfsd_versions) - nn->nfsd_versions[vers] = nfsd_support_version(vers); + nn->nfsd_versions[vers] = nfsd_support_version(vers); break; case NFSD_CLEAR: - nfsd_netns_init_versions(nn); - if (nn->nfsd_versions) - nn->nfsd_versions[vers] = false; + nn->nfsd_versions[vers] = false; break; case NFSD_TEST: - if (nn->nfsd_versions) - return nn->nfsd_versions[vers]; - fallthrough; + return nn->nfsd_versions[vers]; case NFSD_AVAIL: return nfsd_support_version(vers); } @@ -233,23 +176,16 @@ int nfsd_minorversion(struct nfsd_net *nn, u32 minorversion, enum vers_op change switch(change) { case NFSD_SET: - if (nn->nfsd4_minorversions) { - nfsd_vers(nn, 4, NFSD_SET); - nn->nfsd4_minorversions[minorversion] = - nfsd_vers(nn, 4, NFSD_TEST); - } + nfsd_vers(nn, 4, NFSD_SET); + nn->nfsd4_minorversions[minorversion] = + nfsd_vers(nn, 4, NFSD_TEST); break; case NFSD_CLEAR: - nfsd_netns_init_versions(nn); - if (nn->nfsd4_minorversions) { - nn->nfsd4_minorversions[minorversion] = false; - nfsd_adjust_nfsd_versions4(nn); - } + nn->nfsd4_minorversions[minorversion] = false; + nfsd_adjust_nfsd_versions4(nn); break; case NFSD_TEST: - if (nn->nfsd4_minorversions) - return nn->nfsd4_minorversions[minorversion]; - return nfsd_vers(nn, 4, NFSD_TEST); + return nn->nfsd4_minorversions[minorversion]; case NFSD_AVAIL: return minorversion <= NFSD_SUPPORTED_MINOR_VERSION && nfsd_vers(nn, 4, NFSD_AVAIL); @@ -568,11 +504,11 @@ void nfsd_reset_versions(struct nfsd_net *nn) { int i; - for (i = 0; i < NFSD_NRVERS; i++) + for (i = 0; i <= NFSD_MAXVERS; i++) if (nfsd_vers(nn, i, NFSD_TEST)) return; - for (i = 0; i < NFSD_NRVERS; i++) + for (i = 0; i <= NFSD_MAXVERS; i++) if (i != 4) nfsd_vers(nn, i, NFSD_SET); else { @@ -905,17 +841,17 @@ nfsd_init_request(struct svc_rqst *rqstp, if (likely(nfsd_vers(nn, rqstp->rq_vers, NFSD_TEST))) return svc_generic_init_request(rqstp, progp, ret); - ret->mismatch.lovers = NFSD_NRVERS; - for (i = NFSD_MINVERS; i < NFSD_NRVERS; i++) { + ret->mismatch.lovers = NFSD_MAXVERS + 1; + for (i = NFSD_MINVERS; i <= NFSD_MAXVERS; i++) { if (nfsd_vers(nn, i, NFSD_TEST)) { ret->mismatch.lovers = i; break; } } - if (ret->mismatch.lovers == NFSD_NRVERS) + if (ret->mismatch.lovers > NFSD_MAXVERS) return rpc_prog_unavail; ret->mismatch.hivers = NFSD_MINVERS; - for (i = NFSD_NRVERS - 1; i >= NFSD_MINVERS; i--) { + for (i = NFSD_MAXVERS; i >= NFSD_MINVERS; i--) { if (nfsd_vers(nn, i, NFSD_TEST)) { ret->mismatch.hivers = i; break; From patchwork Mon Jul 15 07:14:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733036 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A1A66AB8 for ; Mon, 15 Jul 2024 07:48:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029699; cv=none; b=YFRS4pMWBLslSn0pXha8cCMcGFrbr8z0TeGo2ikaRTslIxz0qo0LI+Ep53ZpAwtpwksSVk23dQRADcrdui3xE9mdU1WBRE+YoXJoBddZEB6e2U/axk2954t931MLBI5zphLGJE8UNdESorAhblmDQ01FraZA+Elvy4Dn4eO/q28= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029699; c=relaxed/simple; bh=99Z/wZz4TmUVBP36hmX82NjN1edwTRO82OONO0MSXmg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KGETT93q0XA/oFlnBnVKcVSvRXMJydR3k3p0NcSLJVREaeD9iPd3AA8dqJOy1t8YfDcDfj2n4sl7dPVvWXhyCE/+1fLRROl/oVJqMeHjDnbBbA0EdnxiZoa5U7yoxqCux2ITpIkUqEz7CHBtxQ95Yg/FQPB4zk/aU3/D+7lvigU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=m15JLQl0; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=m2DnhXSr; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=m15JLQl0; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=m2DnhXSr; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="m15JLQl0"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="m2DnhXSr"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="m15JLQl0"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="m2DnhXSr" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 6BC1A1F7B2; Mon, 15 Jul 2024 07:48:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029696; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=znITT74L4Ep/dK3Tfsb6tWxfoOcXiEs8IHfp00tR+s0=; b=m15JLQl0vsp411jgVhBqjMb7sYUI0RQvOeIb0HP0kPMlSuPIq0Hs8wHcL99Hko1n0NR2NP 9rOicBydsFeKAI7MlJ9Bvh3h2dICKVpJHStZoBnu+nfhyazHhbxvBPzY1Nm2UkoD3Cqt5/ IAPlrMjdawCXnenJ8XzOBB0HuLRte30= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029696; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=znITT74L4Ep/dK3Tfsb6tWxfoOcXiEs8IHfp00tR+s0=; b=m2DnhXSryaLl9Ke/seLwCoFaZ5UbOHRrZLv4OHIbDgdtjAW832YWuuXqWY0r0mH+KlA8b2 pXGGNBfDGwYPRYCw== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=m15JLQl0; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=m2DnhXSr DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029696; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=znITT74L4Ep/dK3Tfsb6tWxfoOcXiEs8IHfp00tR+s0=; b=m15JLQl0vsp411jgVhBqjMb7sYUI0RQvOeIb0HP0kPMlSuPIq0Hs8wHcL99Hko1n0NR2NP 9rOicBydsFeKAI7MlJ9Bvh3h2dICKVpJHStZoBnu+nfhyazHhbxvBPzY1Nm2UkoD3Cqt5/ IAPlrMjdawCXnenJ8XzOBB0HuLRte30= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029696; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=znITT74L4Ep/dK3Tfsb6tWxfoOcXiEs8IHfp00tR+s0=; b=m2DnhXSryaLl9Ke/seLwCoFaZ5UbOHRrZLv4OHIbDgdtjAW832YWuuXqWY0r0mH+KlA8b2 pXGGNBfDGwYPRYCw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 04B31137EB; Mon, 15 Jul 2024 07:48:13 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id WtQLKz3UlGbobQAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:13 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 05/14] sunrpc: change sp_nrthreads from atomic_t to unsigned int. Date: Mon, 15 Jul 2024 17:14:18 +1000 Message-ID: <20240715074657.18174-6-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: 6BC1A1F7B2 X-Spam-Flag: NO X-Spam-Score: 0.99 X-Rspamd-Action: no action X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [0.99 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.de:dkim,suse.de:email]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Spam-Level: X-Spamd-Bar: / sp_nrthreads is only ever accessed under the service mutex nlmsvc_mutex nfs_callback_mutex nfsd_mutex so these is no need for it to be an atomic_t. The fact that all code using it is single-threaded means that we can simplify svc_pool_victim and remove the temporary elevation of sp_nrthreads. Signed-off-by: NeilBrown --- fs/nfsd/nfsctl.c | 2 +- fs/nfsd/nfssvc.c | 2 +- include/linux/sunrpc/svc.h | 4 ++-- net/sunrpc/svc.c | 31 +++++++++++-------------------- 4 files changed, 15 insertions(+), 24 deletions(-) diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 5b0f2e0d7ccf..d85b6d1fa31f 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1769,7 +1769,7 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info) struct svc_pool *sp = &nn->nfsd_serv->sv_pools[i]; err = nla_put_u32(skb, NFSD_A_SERVER_THREADS, - atomic_read(&sp->sp_nrthreads)); + sp->sp_nrthreads); if (err) goto err_unlock; } diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 4438cdcd4873..7377422a34df 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -641,7 +641,7 @@ int nfsd_get_nrthreads(int n, int *nthreads, struct net *net) if (serv) for (i = 0; i < serv->sv_nrpools && i < n; i++) - nthreads[i] = atomic_read(&serv->sv_pools[i].sp_nrthreads); + nthreads[i] = serv->sv_pools[i].sp_nrthreads; return 0; } diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index e4fa25fafa97..99e9345d829e 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -33,9 +33,9 @@ * node traffic on multi-node NUMA NFS servers. */ struct svc_pool { - unsigned int sp_id; /* pool id; also node id on NUMA */ + unsigned int sp_id; /* pool id; also node id on NUMA */ struct lwq sp_xprts; /* pending transports */ - atomic_t sp_nrthreads; /* # of threads in pool */ + unsigned int sp_nrthreads; /* # of threads in pool */ struct list_head sp_all_threads; /* all server threads */ struct llist_head sp_idle_threads; /* idle server threads */ diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 072ad115ae3d..0d8588bc693c 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -725,7 +725,7 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node) serv->sv_nrthreads += 1; spin_unlock_bh(&serv->sv_lock); - atomic_inc(&pool->sp_nrthreads); + pool->sp_nrthreads += 1; /* Protected by whatever lock the service uses when calling * svc_set_num_threads() @@ -780,31 +780,22 @@ svc_pool_victim(struct svc_serv *serv, struct svc_pool *target_pool, struct svc_pool *pool; unsigned int i; -retry: pool = target_pool; - if (pool != NULL) { - if (atomic_inc_not_zero(&pool->sp_nrthreads)) - goto found_pool; - return NULL; - } else { + if (!pool) { for (i = 0; i < serv->sv_nrpools; i++) { pool = &serv->sv_pools[--(*state) % serv->sv_nrpools]; - if (atomic_inc_not_zero(&pool->sp_nrthreads)) - goto found_pool; + if (pool->sp_nrthreads) + break; } - return NULL; } -found_pool: - set_bit(SP_VICTIM_REMAINS, &pool->sp_flags); - set_bit(SP_NEED_VICTIM, &pool->sp_flags); - if (!atomic_dec_and_test(&pool->sp_nrthreads)) + if (pool && pool->sp_nrthreads) { + set_bit(SP_VICTIM_REMAINS, &pool->sp_flags); + set_bit(SP_NEED_VICTIM, &pool->sp_flags); return pool; - /* Nothing left in this pool any more */ - clear_bit(SP_NEED_VICTIM, &pool->sp_flags); - clear_bit(SP_VICTIM_REMAINS, &pool->sp_flags); - goto retry; + } + return NULL; } static int @@ -883,7 +874,7 @@ svc_set_num_threads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) if (!pool) nrservs -= serv->sv_nrthreads; else - nrservs -= atomic_read(&pool->sp_nrthreads); + nrservs -= pool->sp_nrthreads; if (nrservs > 0) return svc_start_kthreads(serv, pool, nrservs); @@ -953,7 +944,7 @@ svc_exit_thread(struct svc_rqst *rqstp) list_del_rcu(&rqstp->rq_all); - atomic_dec(&pool->sp_nrthreads); + pool->sp_nrthreads -= 1; spin_lock_bh(&serv->sv_lock); serv->sv_nrthreads -= 1; From patchwork Mon Jul 15 07:14:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733037 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24649175A6 for ; Mon, 15 Jul 2024 07:48:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029709; cv=none; b=UmB0Hf0iR26oDXnB/bg3bJnJrKqWGwYsxgIOJeTFXG3nskNbXxCp9VgPH14/rhA9NzjYoTZ/bq6cvInIhnuMU8WULf+sLjpSzi6YuhHcDgdvUWPUkwNQlJouobkrFyxSRy17vKM5yIMu69pc2Nt2vKhrXMj1XioVhfzZbmT1VsU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029709; c=relaxed/simple; bh=VvoEjtc3fDoIvHWknjtpZZmysx5tdv49snIFKNA6w6M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uIPReUGl2DpenIM24sqw3qu7JD83xiBHyHlRdGjZSACSp/jiyyI7bLskbmdzQME7TGg2IGBCV7LcWEMv9IdwDcqUiKhuI8v9X+pLy6sHYxciozlQzcfRBWzPwWl3G3umK8JvQXFE+4duZ8PooJIy//Gomp/DSkVqU7SQnT/esb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=TkqNtHi8; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Iia96VLe; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=TkqNtHi8; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Iia96VLe; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="TkqNtHi8"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Iia96VLe"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="TkqNtHi8"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Iia96VLe" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 4C0761F7FA; Mon, 15 Jul 2024 07:48:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029706; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rfxkogkznY/FiY3BUtpurKcdlYdaAoakknz608gEtBw=; b=TkqNtHi8CE+iV56eX9Rer6e269g4N9ynDYFBRXzWURIuLW1P+o56ESmjTnAEvFj/0YB1lf RMQu0RpeTiGxovOU4u6Ce7ywvkmYOSdul3YG5MmngEispvb39sjrNp+Hm2ohGgg3+gaCF3 DL6TKyPdvmbSzhretI9Gxm3tHxI4aJI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029706; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rfxkogkznY/FiY3BUtpurKcdlYdaAoakknz608gEtBw=; b=Iia96VLev8xPz9a4ZdtuoxktVvLUaMeCYRfFaXoYsW67jFsjo5/86EDIGltl1KdvYj/0eW xvoUNSN+MUPe84Bg== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=TkqNtHi8; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=Iia96VLe DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029706; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rfxkogkznY/FiY3BUtpurKcdlYdaAoakknz608gEtBw=; b=TkqNtHi8CE+iV56eX9Rer6e269g4N9ynDYFBRXzWURIuLW1P+o56ESmjTnAEvFj/0YB1lf RMQu0RpeTiGxovOU4u6Ce7ywvkmYOSdul3YG5MmngEispvb39sjrNp+Hm2ohGgg3+gaCF3 DL6TKyPdvmbSzhretI9Gxm3tHxI4aJI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029706; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rfxkogkznY/FiY3BUtpurKcdlYdaAoakknz608gEtBw=; b=Iia96VLev8xPz9a4ZdtuoxktVvLUaMeCYRfFaXoYsW67jFsjo5/86EDIGltl1KdvYj/0eW xvoUNSN+MUPe84Bg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id CF37B137EB; Mon, 15 Jul 2024 07:48:23 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id kAPIIEfUlGb2bQAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:23 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 06/14] sunrpc: don't take ->sv_lock when updating ->sv_nrthreads. Date: Mon, 15 Jul 2024 17:14:19 +1000 Message-ID: <20240715074657.18174-7-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.de:email,suse.de:dkim]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Spam-Flag: NO X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Level: X-Rspamd-Queue-Id: 4C0761F7FA As documented in svc_xprt.c, sv_nrthreads is protected by the service mutex, and it does not need ->sv_lock. (->sv_lock is needed only for sv_permsocks, sv_tempsocks, and sv_tmpcnt). So remove the unnecessary locking. Signed-off-by: NeilBrown --- net/sunrpc/svc.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 0d8588bc693c..f4fc3d82e2bb 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -721,10 +721,7 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node) if (!rqstp) return ERR_PTR(-ENOMEM); - spin_lock_bh(&serv->sv_lock); serv->sv_nrthreads += 1; - spin_unlock_bh(&serv->sv_lock); - pool->sp_nrthreads += 1; /* Protected by whatever lock the service uses when calling @@ -945,10 +942,7 @@ svc_exit_thread(struct svc_rqst *rqstp) list_del_rcu(&rqstp->rq_all); pool->sp_nrthreads -= 1; - - spin_lock_bh(&serv->sv_lock); serv->sv_nrthreads -= 1; - spin_unlock_bh(&serv->sv_lock); svc_sock_update_bufs(serv); svc_rqst_free(rqstp); From patchwork Mon Jul 15 07:14:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733038 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBE696AB8 for ; Mon, 15 Jul 2024 07:48:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029715; cv=none; b=kFapN9ti7HNn4aP+cQiQh/hvAAs7BQu2jEfM6RA9/MDymoXAovKHizkhLNJM5KBpXZCidy3XMYfOy32iRj/AssHbFW3Ltx8pvG4T6AV6veaG/62XsJ9919h0HXF4+/QZYD8gSN+MYX0rvmmY+Ups/tUp6ePPMxcqVU3QvU6WDfw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029715; c=relaxed/simple; bh=CLhX6vpT4BIEobXt15m61jAzWkehqqnZ6KTQUixkOP0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lYvIFTuYZ0MsG2bfhvmnY5tqJ7NepJ/Ox9Su1+heuNJzp37++EUpJV4avZxXiyRNCBrp9TV9InffUZjGseRzQv5B+nfHacVrzDF5pwihOQc7MDan65SdPYUZHpJLnBpjXs24yDikPAyAxqiqd61TGnnJbBYtSPLn89lfrEIWLxU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=pcJkZVZU; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=CVQrUvqc; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=pcJkZVZU; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=CVQrUvqc; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="pcJkZVZU"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="CVQrUvqc"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="pcJkZVZU"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="CVQrUvqc" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 089951F7B2; Mon, 15 Jul 2024 07:48:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029712; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IaMXKNV+oIixwuRWZ4nip45zTBjKxx2jZDfhXmuv9Bo=; b=pcJkZVZUUtOEGMSK2E7+P6GHXqJLpKuAfKtssNm1mMkBQgfPDmPh/yzpVpXPpO6Adna8+K EACR2m3le/nPweaf4BEV0hRwxSjC5pr0YxbgT/FIjrZHUNgHzPv7GwXniS4m0Ojbtl0Ban 7DSmfUAYZgH0TICtcRYX/opXuZc4sJk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029712; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IaMXKNV+oIixwuRWZ4nip45zTBjKxx2jZDfhXmuv9Bo=; b=CVQrUvqctBeLpnGfT5EFrN5wr6a2PeqiucRaZqEPq1LvnLTAmK761ol5r6dgNvT7LzEG31 DEZ32IhCbqDUASBA== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029712; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IaMXKNV+oIixwuRWZ4nip45zTBjKxx2jZDfhXmuv9Bo=; b=pcJkZVZUUtOEGMSK2E7+P6GHXqJLpKuAfKtssNm1mMkBQgfPDmPh/yzpVpXPpO6Adna8+K EACR2m3le/nPweaf4BEV0hRwxSjC5pr0YxbgT/FIjrZHUNgHzPv7GwXniS4m0Ojbtl0Ban 7DSmfUAYZgH0TICtcRYX/opXuZc4sJk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029712; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IaMXKNV+oIixwuRWZ4nip45zTBjKxx2jZDfhXmuv9Bo=; b=CVQrUvqctBeLpnGfT5EFrN5wr6a2PeqiucRaZqEPq1LvnLTAmK761ol5r6dgNvT7LzEG31 DEZ32IhCbqDUASBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id A068E137EB; Mon, 15 Jul 2024 07:48:29 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id LuWcFU3UlGYAbgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:29 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 07/14] Change unshare_fs_struct() to never fail. Date: Mon, 15 Jul 2024 17:14:20 +1000 Message-ID: <20240715074657.18174-8-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Flag: NO X-Spam-Score: 1.20 X-Spamd-Result: default: False [1.20 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:email]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-Spam-Level: * nfsd threads need to not share the init fs_struct as they need to manipulate umask independently. So they call unshare_fs_struct() and are the only user of that function. In the unlikely event that unshare_fs_struct() fails, the thread will exit calling svc_exit_thread() BEFORE svc_thread_should_stop() reports 'true'. This is a problem because svc_exit_thread() assumes that svc_stop_threads() is running and consequently (in the nfsd case) nfsd_mutex is held. This ensures that the list_del_rcu() call in svc_exit_thread() cannot race with any other manipulation of ->sp_all_threads. While it would be possible to add some other exclusion, doing so would introduce unnecessary complexity. unshare_fs_struct() does not fail in practice. So the simplest solution is to make this explicit. i.e. use __GFP_NOFAIL which is safe on such a small allocation - about 64 bytes. Change unshare_fs_struct() to not return any error, and remove the error handling from nfsd(). An alternate approach would be to create a variant of kthread_create_on_node() which didn't set CLONE_FS. Signed-off-by: NeilBrown --- fs/fs_struct.c | 42 ++++++++++++++++++++------------------- fs/nfsd/nfssvc.c | 9 +++------ include/linux/fs_struct.h | 2 +- 3 files changed, 26 insertions(+), 27 deletions(-) diff --git a/fs/fs_struct.c b/fs/fs_struct.c index 64c2d0814ed6..49fba862e408 100644 --- a/fs/fs_struct.c +++ b/fs/fs_struct.c @@ -109,35 +109,39 @@ void exit_fs(struct task_struct *tsk) } } +static void init_fs_struct(struct fs_struct *fs, struct fs_struct *old) +{ + fs->users = 1; + fs->in_exec = 0; + spin_lock_init(&fs->lock); + seqcount_spinlock_init(&fs->seq, &fs->lock); + fs->umask = old->umask; + + spin_lock(&old->lock); + fs->root = old->root; + path_get(&fs->root); + fs->pwd = old->pwd; + path_get(&fs->pwd); + spin_unlock(&old->lock); +} + struct fs_struct *copy_fs_struct(struct fs_struct *old) { struct fs_struct *fs = kmem_cache_alloc(fs_cachep, GFP_KERNEL); /* We don't need to lock fs - think why ;-) */ - if (fs) { - fs->users = 1; - fs->in_exec = 0; - spin_lock_init(&fs->lock); - seqcount_spinlock_init(&fs->seq, &fs->lock); - fs->umask = old->umask; - - spin_lock(&old->lock); - fs->root = old->root; - path_get(&fs->root); - fs->pwd = old->pwd; - path_get(&fs->pwd); - spin_unlock(&old->lock); - } + if (fs) + init_fs_struct(fs, old); return fs; } -int unshare_fs_struct(void) +void unshare_fs_struct(void) { struct fs_struct *fs = current->fs; - struct fs_struct *new_fs = copy_fs_struct(fs); + struct fs_struct *new_fs = kmem_cache_alloc(fs_cachep, + GFP_KERNEL| __GFP_NOFAIL); int kill; - if (!new_fs) - return -ENOMEM; + init_fs_struct(new_fs, fs); task_lock(current); spin_lock(&fs->lock); @@ -148,8 +152,6 @@ int unshare_fs_struct(void) if (kill) free_fs_struct(fs); - - return 0; } EXPORT_SYMBOL_GPL(unshare_fs_struct); diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 7377422a34df..f5de04a63c6f 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -873,11 +873,9 @@ nfsd(void *vrqstp) /* At this point, the thread shares current->fs * with the init process. We need to create files with the - * umask as defined by the client instead of init's umask. */ - if (unshare_fs_struct() < 0) { - printk("Unable to start nfsd thread: out of memory\n"); - goto out; - } + * umask as defined by the client instead of init's umask. + */ + unshare_fs_struct(); current->fs->umask = 0; @@ -899,7 +897,6 @@ nfsd(void *vrqstp) atomic_dec(&nfsd_th_cnt); -out: /* Release the thread */ svc_exit_thread(rqstp); return 0; diff --git a/include/linux/fs_struct.h b/include/linux/fs_struct.h index 783b48dedb72..8282e6c7ff29 100644 --- a/include/linux/fs_struct.h +++ b/include/linux/fs_struct.h @@ -22,7 +22,7 @@ extern void set_fs_root(struct fs_struct *, const struct path *); extern void set_fs_pwd(struct fs_struct *, const struct path *); extern struct fs_struct *copy_fs_struct(struct fs_struct *); extern void free_fs_struct(struct fs_struct *); -extern int unshare_fs_struct(void); +extern void unshare_fs_struct(void); static inline void get_fs_root(struct fs_struct *fs, struct path *root) { From patchwork Mon Jul 15 07:14:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733039 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97671175A6 for ; Mon, 15 Jul 2024 07:48:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029721; cv=none; b=DR7BSA9EsbpL8Pw21q7SOMqFYYvmJ2G1Da4GC+zimUCujNCE2CQbCwyuV+mAIFsDXIYMl1wMkea1NDNOxEfdXRXpemQxptTNov4qi7doXiXgSSVh2fZKLNcneajq/MF4Esly241zcZmiee4lrEt5XEzMfL5rNC6QsbSeTbGiLJY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029721; c=relaxed/simple; bh=jbe3R80TgjNy421fNmKGFS8ZOzHZJkUbxgEj1dygTKg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e44om9A5PqpnzQfkroN9g8FPuJwYVqJLxe/yu1w+QtS1uPRwJxs4cvAc/mrv8jzh0f64E/fcDbiIfpd9S/dEFJRh4ma9u3+bLmwWBDdeNeMmjeyc/Hvgm19ww8UFjR6Duze8IH59mIJWz4TZIsRAH96Sw5oey/erS6S/frXmtPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=QDuLgegs; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=6LGtjzE+; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=QDuLgegs; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=6LGtjzE+; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="QDuLgegs"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="6LGtjzE+"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="QDuLgegs"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="6LGtjzE+" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C17B81F7B2; Mon, 15 Jul 2024 07:48:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029717; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gsecy/yC1BWeMJJWw1HEobEKDsEPySMCLYSyZ8CMAKc=; b=QDuLgegsq6KhIe0rMYyCXmAO9u1FuGCZxx+jDGGB5iqJbS+EXYHIchaxEVj9ZhPWt5Po0D 9xFI46gGeFd8d2WG5Z011kmh6+HRlAPoMd9v/QjaxZUQPxhOtsr4GbXgOiDz2YYHNfBBhx pUFE44CZbvevFidlqI41UwQ7TNVONTo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029717; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gsecy/yC1BWeMJJWw1HEobEKDsEPySMCLYSyZ8CMAKc=; b=6LGtjzE+TxgYfkwntwA41IZRblJYrEdci6maRaXsdUz8x3gRD/8VZ0p1y+rJms7vg8ZIjI DEMPOkt9DIXxq4AA== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=QDuLgegs; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=6LGtjzE+ DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029717; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gsecy/yC1BWeMJJWw1HEobEKDsEPySMCLYSyZ8CMAKc=; b=QDuLgegsq6KhIe0rMYyCXmAO9u1FuGCZxx+jDGGB5iqJbS+EXYHIchaxEVj9ZhPWt5Po0D 9xFI46gGeFd8d2WG5Z011kmh6+HRlAPoMd9v/QjaxZUQPxhOtsr4GbXgOiDz2YYHNfBBhx pUFE44CZbvevFidlqI41UwQ7TNVONTo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029717; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gsecy/yC1BWeMJJWw1HEobEKDsEPySMCLYSyZ8CMAKc=; b=6LGtjzE+TxgYfkwntwA41IZRblJYrEdci6maRaXsdUz8x3gRD/8VZ0p1y+rJms7vg8ZIjI DEMPOkt9DIXxq4AA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 6460C137EB; Mon, 15 Jul 2024 07:48:35 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id ey3wBlPUlGYHbgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:35 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 08/14] SUNRPC: move nrthreads counting to start/stop threads. Date: Mon, 15 Jul 2024 17:14:21 +1000 Message-ID: <20240715074657.18174-9-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,suse.de:dkim,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Spam-Flag: NO X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Level: X-Rspamd-Queue-Id: C17B81F7B2 sp_nrthreads and sv_nrthreads are the number of threads that have been explicitly requested. Future patches will allow extra threads to be created as needed. So move the updating of these fields to code which is for updating configuration rather that code that is for starting/stopping threads. Signed-off-by: NeilBrown --- net/sunrpc/svc.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index f4fc3d82e2bb..d814b2cfa84f 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -721,9 +721,6 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node) if (!rqstp) return ERR_PTR(-ENOMEM); - serv->sv_nrthreads += 1; - pool->sp_nrthreads += 1; - /* Protected by whatever lock the service uses when calling * svc_set_num_threads() */ @@ -818,6 +815,8 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) svc_exit_thread(rqstp); return PTR_ERR(task); } + serv->sv_nrthreads += 1; + chosen_pool->sp_nrthreads += 1; rqstp->rq_task = task; if (serv->sv_nrpools > 1) @@ -840,6 +839,8 @@ svc_stop_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) victim = svc_pool_victim(serv, pool, &state); if (!victim) break; + victim->sp_nrthreads -= 1; + serv->sv_nrthreads -= 1; svc_pool_wake_idle_thread(victim); wait_on_bit(&victim->sp_flags, SP_VICTIM_REMAINS, TASK_IDLE); @@ -941,8 +942,6 @@ svc_exit_thread(struct svc_rqst *rqstp) list_del_rcu(&rqstp->rq_all); - pool->sp_nrthreads -= 1; - serv->sv_nrthreads -= 1; svc_sock_update_bufs(serv); svc_rqst_free(rqstp); From patchwork Mon Jul 15 07:14:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733040 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5404E1836C1 for ; Mon, 15 Jul 2024 07:48:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029726; cv=none; b=fRFrMSLSinNuKby41qqeI6pd5+o+xrjH51+OxCXsX5ifnsflYbnSpy6ZpqXVX9t7OzRNOUzijp2mMddokmp8YfvhZa5B+/xhe7ASIGEp7RHRTCkZHSbSJO7x3Ti/iU37wqR2qrCkJ83pvCW7smvT+BLYxDRr7/fkGTorLIZ/8x4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029726; c=relaxed/simple; bh=f5BbXpNPQnWu/8Nlp5XWOowsugkJkccRpkGNU2TtxWo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ERLLaxuNrjJ69nQR/EtCNX5YRKytr5sfLwuEcuhr+YCG5161KltU/xZ2sy3iwjt5CDWto8Ju4h/R9WT+Z7zyVFYA/xl1sRkBQmqLQ6U0TzxJs8ITKm5oyWUXlig8RJ0y7tqY6l9wNvC3/tXuJmR9SpnVWiJzJ+ayknXKOiboSfo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=At1fCskP; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=+p/x5r5E; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=At1fCskP; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=+p/x5r5E; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="At1fCskP"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="+p/x5r5E"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="At1fCskP"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="+p/x5r5E" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9E3531F7FA; Mon, 15 Jul 2024 07:48:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029723; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S26RsZliLAWJR8QysfTpg9Wsa/I9wXuD3pM8/jzFvH0=; b=At1fCskPwembE8J2PwSqMZC+zFMs/Vo/z+wizWomHckQu9WwKhvj1rSWgNyEOwXfKm3wG3 wqFi85qisVK8S/gcCNhUkt3nKA2aY6YB3mUzjSw+i0y+yGfg2wLH3i90WAYZuMwenKMQAY MG8wFYafVI2ilgJfLpgis2JtTyCuQfM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029723; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S26RsZliLAWJR8QysfTpg9Wsa/I9wXuD3pM8/jzFvH0=; b=+p/x5r5Eyp7l6nfseTgNP4j/t2aMk0HFhnRW1xFXZmOTqnHU7HdliJITVWiHkdW+CSRXMF 9jEYB9XlHQmEOcCQ== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=At1fCskP; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b="+p/x5r5E" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029723; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S26RsZliLAWJR8QysfTpg9Wsa/I9wXuD3pM8/jzFvH0=; b=At1fCskPwembE8J2PwSqMZC+zFMs/Vo/z+wizWomHckQu9WwKhvj1rSWgNyEOwXfKm3wG3 wqFi85qisVK8S/gcCNhUkt3nKA2aY6YB3mUzjSw+i0y+yGfg2wLH3i90WAYZuMwenKMQAY MG8wFYafVI2ilgJfLpgis2JtTyCuQfM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029723; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S26RsZliLAWJR8QysfTpg9Wsa/I9wXuD3pM8/jzFvH0=; b=+p/x5r5Eyp7l6nfseTgNP4j/t2aMk0HFhnRW1xFXZmOTqnHU7HdliJITVWiHkdW+CSRXMF 9jEYB9XlHQmEOcCQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 362AA137EB; Mon, 15 Jul 2024 07:48:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id LZT0NljUlGYhbgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:40 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 09/14] nfsd: return hard failure for OP_SETCLIENTID when there are too many clients. Date: Mon, 15 Jul 2024 17:14:22 +1000 Message-ID: <20240715074657.18174-10-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: 9E3531F7FA X-Spam-Flag: NO X-Spam-Score: 0.99 X-Rspamd-Action: no action X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [0.99 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:dkim,suse.de:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Spam-Level: X-Spamd-Bar: / If there are more non-courteous clients than the calculated limit, we should fail the request rather than report a soft failure and encouraging the client to retry indefinitely. If there a courteous clients which push us over the limit, then expedite their removal. This is not known to have caused a problem is production use, but testing of lots of clients reports repeated NFS4ERR_DELAY responses which doesn't seem helpful. Also remove an outdated comment - we do use a slab cache. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/nfs4state.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index a20c2c9d7d45..88936f3189e1 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -2212,21 +2212,20 @@ STALE_CLIENTID(clientid_t *clid, struct nfsd_net *nn) return 1; } -/* - * XXX Should we use a slab cache ? - * This type of memory management is somewhat inefficient, but we use it - * anyway since SETCLIENTID is not a common operation. - */ static struct nfs4_client *alloc_client(struct xdr_netobj name, struct nfsd_net *nn) { struct nfs4_client *clp; int i; - if (atomic_read(&nn->nfs4_client_count) >= nn->nfs4_max_clients) { + if (atomic_read(&nn->nfs4_client_count) - + atomic_read(&nn->nfsd_courtesy_clients) >= nn->nfs4_max_clients) + return ERR_PTR(-EREMOTEIO); + + if (atomic_read(&nn->nfs4_client_count) >= nn->nfs4_max_clients && + atomic_read(&nn->nfsd_courtesy_clients) > 0) mod_delayed_work(laundry_wq, &nn->laundromat_work, 0); - return NULL; - } + clp = kmem_cache_zalloc(client_slab, GFP_KERNEL); if (clp == NULL) return NULL; @@ -3115,8 +3114,8 @@ static struct nfs4_client *create_client(struct xdr_netobj name, struct dentry *dentries[ARRAY_SIZE(client_files)]; clp = alloc_client(name, nn); - if (clp == NULL) - return NULL; + if (IS_ERR_OR_NULL(clp)) + return clp; ret = copy_cred(&clp->cl_cred, &rqstp->rq_cred); if (ret) { @@ -3498,6 +3497,8 @@ nfsd4_exchange_id(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new = create_client(exid->clname, rqstp, &verf); if (new == NULL) return nfserr_jukebox; + if (IS_ERR(new)) + return nfserr_resource; status = copy_impl_id(new, exid); if (status) goto out_nolock; @@ -4416,6 +4417,8 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new = create_client(clname, rqstp, &clverifier); if (new == NULL) return nfserr_jukebox; + if (IS_ERR(new)) + return nfserr_resource; spin_lock(&nn->client_lock); conf = find_confirmed_client_by_name(&clname, nn); if (conf && client_has_state(conf)) { From patchwork Mon Jul 15 07:14:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733041 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40A376AB8 for ; Mon, 15 Jul 2024 07:48:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029736; cv=none; b=nZAAlIk83h2ln6PhKP+a0nI0Yx/7Cz2gLSgllX1D5viCz6l1GMKIBr4bYs0i3hCdhRNGYvB2i6uD9i8rI13p8w64cgLX2ykAPEzygmo9GxPtgjb7huFle8ETYSKEWJ8HW/MZFRyywlpD6FH+qR6qXftY5BRuYoSULOdVN7Gq1Gg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029736; c=relaxed/simple; bh=TNwG1A1Ug7JYDJcYLHP0dBi6biuInT0z27a5DVDSYfA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IoKLppezoJFrSvSQoJQvSFz+IduBTuRWhKGc/Z2Lpj6CzDLQp7GxoAv1JGJU71XJy8ozR2+C4uNeqZ3XphcOWruOA7vk/oG+XvK92xbjbNlVTEiRGSc65yKNTRedHuMCy3R/2Gf0iFiHUfZVwMtHVK6Csz9QLEg+ZTp1ubflLrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=mABKd4s3; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Af5aFq7N; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=mABKd4s3; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Af5aFq7N; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="mABKd4s3"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Af5aFq7N"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="mABKd4s3"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Af5aFq7N" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7B1911F7FA; Mon, 15 Jul 2024 07:48:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029733; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Cxwmq0yD9JnyqpYfnRRM5fh611Hdmw98vPH3b0+NyM=; b=mABKd4s3QLnjuZO/HrrCRQENG6sUDzJOL/zCauUhuGFVvPv/2259HpR3pZ+OS0U2j65Lx8 N1k0hmQ+qAANVzl41MvGmDgVb8ne39LuSN5eIoYzP5FD2fLEZRV0SUa3mlFZ1hJWSerl+N rLqDi/2aH+zVKyHhw9DCOxxpkr3/17s= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029733; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Cxwmq0yD9JnyqpYfnRRM5fh611Hdmw98vPH3b0+NyM=; b=Af5aFq7No3r/GHieASK8YG4T4VJ3T6sVd4E1s3+/hpl+i+CiE3ko8pO4xikVYiRR6DESap 8YjsDn+naJocT/BQ== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029733; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Cxwmq0yD9JnyqpYfnRRM5fh611Hdmw98vPH3b0+NyM=; b=mABKd4s3QLnjuZO/HrrCRQENG6sUDzJOL/zCauUhuGFVvPv/2259HpR3pZ+OS0U2j65Lx8 N1k0hmQ+qAANVzl41MvGmDgVb8ne39LuSN5eIoYzP5FD2fLEZRV0SUa3mlFZ1hJWSerl+N rLqDi/2aH+zVKyHhw9DCOxxpkr3/17s= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029733; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Cxwmq0yD9JnyqpYfnRRM5fh611Hdmw98vPH3b0+NyM=; b=Af5aFq7No3r/GHieASK8YG4T4VJ3T6sVd4E1s3+/hpl+i+CiE3ko8pO4xikVYiRR6DESap 8YjsDn+naJocT/BQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 11E27137EB; Mon, 15 Jul 2024 07:48:50 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id lUBsLmLUlGYybgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:48:50 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 10/14] nfs: dynamically adjust per-client DRC slot limits. Date: Mon, 15 Jul 2024 17:14:23 +1000 Message-ID: <20240715074657.18174-11-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Flag: NO X-Spam-Score: 1.20 X-Spamd-Result: default: False [1.20 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:email]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-Spam-Level: * Currently per-client DRC slot limits (for v4.1+) are calculated when the client connects and are left unchanged. So earlier clients can get a larger share when memory is tight. The heuristic for choosing a number includes the number of configured server threads. This is problematic for 2 reasons. 1/ sv_nrthreads is different in different net namespaces, but the memory allocation is global across all namespaces. So different namespaces get treated differently without good reason. 2/ a future patch will auto-configure number of threads based on load so that there is no need to preconfigure a number. This will make the current heuristic even more arbitrary. NFSv4.1 allows the number of slots to be varied dynamically - in the reply to each SEQUENCE op. With this patch we provide a provisional upper limit in the EXCHANGE_ID reply which might end up being too big, and adjust it with each SEQUENCE reply. The goal for when memory is tight is to allow each client to have a similar number of slots. So clients that ask for larger slots get more memory. This may not be ideal. It could be changed later. So we track the sum of the slot sizes of all active clients, and share memory out based on the ratio of the slot size for a given client with the total slot size. We never allow more in a SEQUENCE reply than we did in the EXCHANGE_ID reply. Signed-off-by: NeilBrown --- fs/nfsd/nfs4state.c | 81 ++++++++++++++++++++++++--------------------- fs/nfsd/nfs4xdr.c | 2 +- fs/nfsd/nfsd.h | 2 +- fs/nfsd/nfssvc.c | 7 ++-- fs/nfsd/state.h | 2 +- fs/nfsd/xdr4.h | 2 -- 6 files changed, 52 insertions(+), 44 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 88936f3189e1..4dd619e6010f 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -1909,44 +1909,26 @@ static inline u32 slot_bytes(struct nfsd4_channel_attrs *ca) } /* - * XXX: If we run out of reserved DRC memory we could (up to a point) - * re-negotiate active sessions and reduce their slot usage to make - * room for new connections. For now we just fail the create session. + * When a client connects it gets a max_requests number that would allow + * it to use 1/8 of the memory we think can reasonably be used for the DRC. + * Later in response to SEQUENCE operations we further limit that when there + * are more than 8 concurrent clients. */ -static u32 nfsd4_get_drc_mem(struct nfsd4_channel_attrs *ca, struct nfsd_net *nn) +static u32 nfsd4_get_drc_mem(struct nfsd4_channel_attrs *ca) { u32 slotsize = slot_bytes(ca); u32 num = ca->maxreqs; - unsigned long avail, total_avail; - unsigned int scale_factor; + unsigned long avail; spin_lock(&nfsd_drc_lock); - if (nfsd_drc_max_mem > nfsd_drc_mem_used) - total_avail = nfsd_drc_max_mem - nfsd_drc_mem_used; - else - /* We have handed out more space than we chose in - * set_max_drc() to allow. That isn't really a - * problem as long as that doesn't make us think we - * have lots more due to integer overflow. - */ - total_avail = 0; - avail = min((unsigned long)NFSD_MAX_MEM_PER_SESSION, total_avail); - /* - * Never use more than a fraction of the remaining memory, - * unless it's the only way to give this client a slot. - * The chosen fraction is either 1/8 or 1/number of threads, - * whichever is smaller. This ensures there are adequate - * slots to support multiple clients per thread. - * Give the client one slot even if that would require - * over-allocation--it is better than failure. - */ - scale_factor = max_t(unsigned int, 8, nn->nfsd_serv->sv_nrthreads); - avail = clamp_t(unsigned long, avail, slotsize, - total_avail/scale_factor); - num = min_t(int, num, avail / slotsize); - num = max_t(int, num, 1); - nfsd_drc_mem_used += num * slotsize; + avail = min(NFSD_MAX_MEM_PER_SESSION, + nfsd_drc_max_mem / 8); + + num = clamp_t(int, num, 1, avail / slotsize); + + nfsd_drc_slotsize_sum += slotsize; + spin_unlock(&nfsd_drc_lock); return num; @@ -1957,10 +1939,33 @@ static void nfsd4_put_drc_mem(struct nfsd4_channel_attrs *ca) int slotsize = slot_bytes(ca); spin_lock(&nfsd_drc_lock); - nfsd_drc_mem_used -= slotsize * ca->maxreqs; + nfsd_drc_slotsize_sum -= slotsize; spin_unlock(&nfsd_drc_lock); } +/* + * Report the number of slots that we would like the client to limit + * itself to. When the number of clients is large, we start sharing + * memory so all clients get the same number of slots. + */ +static unsigned int nfsd4_get_drc_slots(struct nfsd4_session *session) +{ + u32 slotsize = slot_bytes(&session->se_fchannel); + unsigned long avail; + unsigned long slotsize_sum = READ_ONCE(nfsd_drc_slotsize_sum); + + if (slotsize_sum < slotsize) + slotsize_sum = slotsize; + + /* Find our share of avail mem across all active clients, + * then limit to 1/8 of total, and configured max + */ + avail = min3(nfsd_drc_max_mem * slotsize / nfsd_drc_slotsize_sum, + nfsd_drc_max_mem / 8, + NFSD_MAX_MEM_PER_SESSION); + return max3(1UL, avail / slotsize, session->se_fchannel.maxreqs); +} + static struct nfsd4_session *alloc_session(struct nfsd4_channel_attrs *fattrs, struct nfsd4_channel_attrs *battrs) { @@ -3726,7 +3731,7 @@ static __be32 check_forechannel_attrs(struct nfsd4_channel_attrs *ca, struct nfs * Note that we always allow at least one slot, because our * accounting is soft and provides no guarantees either way. */ - ca->maxreqs = nfsd4_get_drc_mem(ca, nn); + ca->maxreqs = nfsd4_get_drc_mem(ca); return nfs_ok; } @@ -4220,10 +4225,12 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, slot = session->se_slots[seq->slotid]; dprintk("%s: slotid %d\n", __func__, seq->slotid); - /* We do not negotiate the number of slots yet, so set the - * maxslots to the session maxreqs which is used to encode - * sr_highest_slotid and the sr_target_slot id to maxslots */ - seq->maxslots = session->se_fchannel.maxreqs; + /* Negotiate number of slots: set the target, and use the + * same for max as long as it doesn't decrease the max + * (that isn't allowed). + */ + seq->target_maxslots = nfsd4_get_drc_slots(session); + seq->maxslots = max(seq->maxslots, seq->target_maxslots); trace_nfsd_slot_seqid_sequence(clp, seq, slot); status = check_slot_seqid(seq->seqid, slot->sl_seqid, diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c index 42b41d55d4ed..a65812fcdae0 100644 --- a/fs/nfsd/nfs4xdr.c +++ b/fs/nfsd/nfs4xdr.c @@ -4961,7 +4961,7 @@ nfsd4_encode_sequence(struct nfsd4_compoundres *resp, __be32 nfserr, if (nfserr != nfs_ok) return nfserr; /* sr_target_highest_slotid */ - nfserr = nfsd4_encode_slotid4(xdr, seq->maxslots - 1); + nfserr = nfsd4_encode_slotid4(xdr, seq->target_maxslots - 1); if (nfserr != nfs_ok) return nfserr; /* sr_status_flags */ diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index 369c3b3ce53e..e4c643255dc9 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -90,7 +90,7 @@ extern const struct svc_version nfsd_version2, nfsd_version3, nfsd_version4; extern struct mutex nfsd_mutex; extern spinlock_t nfsd_drc_lock; extern unsigned long nfsd_drc_max_mem; -extern unsigned long nfsd_drc_mem_used; +extern unsigned long nfsd_drc_slotsize_sum; extern atomic_t nfsd_th_cnt; /* number of available threads */ extern const struct seq_operations nfs_exports_op; diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index f5de04a63c6f..b005b2e2e6ad 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -78,7 +78,7 @@ DEFINE_MUTEX(nfsd_mutex); */ DEFINE_SPINLOCK(nfsd_drc_lock); unsigned long nfsd_drc_max_mem; -unsigned long nfsd_drc_mem_used; +unsigned long nfsd_drc_slotsize_sum; #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL) static const struct svc_version *nfsd_acl_version[] = { @@ -532,10 +532,13 @@ void nfsd_reset_versions(struct nfsd_net *nn) */ static void set_max_drc(void) { + if (nfsd_drc_max_mem) + return; + #define NFSD_DRC_SIZE_SHIFT 7 nfsd_drc_max_mem = (nr_free_buffer_pages() >> NFSD_DRC_SIZE_SHIFT) * PAGE_SIZE; - nfsd_drc_mem_used = 0; + nfsd_drc_slotsize_sum = 0; dprintk("%s nfsd_drc_max_mem %lu \n", __func__, nfsd_drc_max_mem); } diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index ffc217099d19..2b1d619bc00f 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -213,7 +213,7 @@ static inline struct nfs4_delegation *delegstateid(struct nfs4_stid *s) /* Maximum number of slots per session. 160 is useful for long haul TCP */ #define NFSD_MAX_SLOTS_PER_SESSION 160 /* Maximum session per slot cache size */ -#define NFSD_SLOT_CACHE_SIZE 2048 +#define NFSD_SLOT_CACHE_SIZE 2048UL /* Maximum number of NFSD_SLOT_CACHE_SIZE slots per session */ #define NFSD_CACHE_SIZE_SLOTS_PER_SESSION 32 #define NFSD_MAX_MEM_PER_SESSION \ diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h index fbdd42cde1fa..1c78a09bf63f 100644 --- a/fs/nfsd/xdr4.h +++ b/fs/nfsd/xdr4.h @@ -575,9 +575,7 @@ struct nfsd4_sequence { u32 slotid; /* request/response */ u32 maxslots; /* request/response */ u32 cachethis; /* request */ -#if 0 u32 target_maxslots; /* response */ -#endif /* not yet */ u32 status_flags; /* response */ }; From patchwork Mon Jul 15 07:14:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733042 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 200106AB8 for ; Mon, 15 Jul 2024 07:49:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029746; cv=none; b=SEmMq6WmA/xrZauC0ckWHXMdoD3YzbP2grWtY41JCTzAGEAEldPR84kiW4whIGeBiZI064/sGaGeUdyvuergzO/svuRQEMSxdDNUA3fXzEtJYV8aVRrw3lmQjUzUqMzdIMWmAGRAvzNCRkHW2ojPzPwLmfpc1wIm28VrmEbADcM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029746; c=relaxed/simple; bh=Q2cqC3RFZVLQ6NR9srwocSyY/7b/4/ZMMSJrqV6q5Hc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SSldMSuFXDp9LpKY44X1gSAu2naWZIyTlogATcw/QbrmddmkVSWJrSPca0ZoMvI2zelIXfb4Pe9Q+05nk2BSfHYs7/7WvGf0C0rPE7XF68P5Hk0aS8kFYrHOcs+UkIMtocalBm2C5kUu1w9P7III3p4sC+pDQwQqam26+TkYCl0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=ecAYK8b6; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=73NlEOng; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=ecAYK8b6; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=73NlEOng; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="ecAYK8b6"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="73NlEOng"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="ecAYK8b6"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="73NlEOng" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 3A7DF1F808; Mon, 15 Jul 2024 07:49:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029743; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y/oi/OgRxfQdh5wWF4wVViRTlYMjE++WKv4N6DBP0wk=; b=ecAYK8b6CpmLH72orMCfPb9XJR2VA4wNu5IbSPG4xbWDbZs22AJW6sB5djntTMuf0skVPF OTSnD41Urnzl+7K02h/oCHPGV3dU2oV8XdvifSxSuYQdL2tM5F1is+xV6N1AuG3Mgl+1SE dSejjCMxQrjjbIalBrqy5Bjc6eQ6qOE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029743; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y/oi/OgRxfQdh5wWF4wVViRTlYMjE++WKv4N6DBP0wk=; b=73NlEOng1rcsEkNWvxeShU58KepQt/tyM4tk7pFma0p2Hw9Zn4O2Fdzj26JIPf9O8W4lXM a9qgCx7xSu9O89CQ== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=ecAYK8b6; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=73NlEOng DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029743; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y/oi/OgRxfQdh5wWF4wVViRTlYMjE++WKv4N6DBP0wk=; b=ecAYK8b6CpmLH72orMCfPb9XJR2VA4wNu5IbSPG4xbWDbZs22AJW6sB5djntTMuf0skVPF OTSnD41Urnzl+7K02h/oCHPGV3dU2oV8XdvifSxSuYQdL2tM5F1is+xV6N1AuG3Mgl+1SE dSejjCMxQrjjbIalBrqy5Bjc6eQ6qOE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029743; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y/oi/OgRxfQdh5wWF4wVViRTlYMjE++WKv4N6DBP0wk=; b=73NlEOng1rcsEkNWvxeShU58KepQt/tyM4tk7pFma0p2Hw9Zn4O2Fdzj26JIPf9O8W4lXM a9qgCx7xSu9O89CQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id D17E6137EB; Mon, 15 Jul 2024 07:49:00 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 5OuSIWzUlGY9bgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:49:00 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 11/14] nfsd: don't use sv_nrthreads in connection limiting calculations. Date: Mon, 15 Jul 2024 17:14:24 +1000 Message-ID: <20240715074657.18174-12-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: 3A7DF1F808 X-Spam-Flag: NO X-Spam-Score: 0.99 X-Rspamd-Action: no action X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [0.99 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.de:dkim,suse.de:email]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Spam-Level: X-Spamd-Bar: / The heuristic for limiting the number of incoming connections to nfsd currently uses sv_nrthreads - allowing more connections if more threads were configured. A future patch will allow number of threads to grow dynamically so that there is no need to configure sv_nrthreads. So we need a different solution for limiting connections. It isn't clear what problem is solved by limiting connections (as mentioned in a code comment) but the most likely problem is a connection storm - many connections that are not doing productive work. These will be closed after about 6 minutes already but it might help to slow down a storm. This patch add a per-connection flag XPT_PEER_VALID which indicates that the peer has presented a filehandle for which it has some sort of access. i.e the peer is known to be trusted in some way. We now only count connections which have NOT be determined to be valid. There should be relative few of these at any given time. If the number of non-validated peer exceed as limit - currently 64 - we close the oldest non-validated peer to avoid having too many of these useless connections. Signed-off-by: NeilBrown --- fs/nfsd/netns.h | 4 ++-- fs/nfsd/nfsfh.c | 8 ++++++++ include/linux/sunrpc/svc.h | 2 +- include/linux/sunrpc/svc_xprt.h | 4 ++++ net/sunrpc/svc_xprt.c | 33 +++++++++++++++++---------------- 5 files changed, 32 insertions(+), 19 deletions(-) diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 238fc4e56e53..0d2ac15a5003 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -128,8 +128,8 @@ struct nfsd_net { unsigned char writeverf[8]; /* - * Max number of connections this nfsd container will allow. Defaults - * to '0' which is means that it bases this on the number of threads. + * Max number of non-validated connections this nfsd container + * will allow. Defaults to '0' gets mapped to 64. */ unsigned int max_connections; diff --git a/fs/nfsd/nfsfh.c b/fs/nfsd/nfsfh.c index 0b75305fb5f5..08742bf8de02 100644 --- a/fs/nfsd/nfsfh.c +++ b/fs/nfsd/nfsfh.c @@ -391,6 +391,14 @@ fh_verify(struct svc_rqst *rqstp, struct svc_fh *fhp, umode_t type, int access) goto out; skip_pseudoflavor_check: + if (test_bit(XPT_TEMP, &rqstp->rq_xprt->xpt_flags) && + !test_and_set_bit(XPT_PEER_VALID, &rqstp->rq_xprt->xpt_flags)) { + struct svc_serv *serv = rqstp->rq_server; + spin_lock(&serv->sv_lock); + serv->sv_tmpcnt -= 1; + spin_unlock(&serv->sv_lock); + } + /* Finally, check access permissions. */ error = nfsd_permission(rqstp, exp, dentry, access); out: diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 99e9345d829e..0b414af448e0 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -79,7 +79,7 @@ struct svc_serv { unsigned int sv_xdrsize; /* XDR buffer size */ struct list_head sv_permsocks; /* all permanent sockets */ struct list_head sv_tempsocks; /* all temporary sockets */ - int sv_tmpcnt; /* count of temporary sockets */ + int sv_tmpcnt; /* count of temporary "valid" sockets */ struct timer_list sv_temptimer; /* timer for aging temporary sockets */ char * sv_name; /* service name */ diff --git a/include/linux/sunrpc/svc_xprt.h b/include/linux/sunrpc/svc_xprt.h index 0981e35a9fed..92565133b3b6 100644 --- a/include/linux/sunrpc/svc_xprt.h +++ b/include/linux/sunrpc/svc_xprt.h @@ -99,6 +99,10 @@ enum { XPT_HANDSHAKE, /* xprt requests a handshake */ XPT_TLS_SESSION, /* transport-layer security established */ XPT_PEER_AUTH, /* peer has been authenticated */ + XPT_PEER_VALID, /* peer has presented a filehandle that + * it has access to. It is NOT counted + * in ->sv_tmpcnt. + */ }; static inline void unregister_xpt_user(struct svc_xprt *xpt, struct svc_xpt_user *u) diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 53ebc719ff5a..a9215e1a2f38 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -606,7 +606,8 @@ int svc_port_is_privileged(struct sockaddr *sin) } /* - * Make sure that we don't have too many active connections. If we have, + * Make sure that we don't have too many connections that have not yet + * demonstrated that they have access the the NFS server. If we have, * something must be dropped. It's not clear what will happen if we allow * "too many" connections, but when dealing with network-facing software, * we have to code defensively. Here we do that by imposing hard limits. @@ -625,27 +626,26 @@ int svc_port_is_privileged(struct sockaddr *sin) */ static void svc_check_conn_limits(struct svc_serv *serv) { - unsigned int limit = serv->sv_maxconn ? serv->sv_maxconn : - (serv->sv_nrthreads+3) * 20; + unsigned int limit = serv->sv_maxconn ? serv->sv_maxconn : 64; if (serv->sv_tmpcnt > limit) { - struct svc_xprt *xprt = NULL; + struct svc_xprt *xprt = NULL, *xprti; spin_lock_bh(&serv->sv_lock); if (!list_empty(&serv->sv_tempsocks)) { - /* Try to help the admin */ - net_notice_ratelimited("%s: too many open connections, consider increasing the %s\n", - serv->sv_name, serv->sv_maxconn ? - "max number of connections" : - "number of threads"); /* * Always select the oldest connection. It's not fair, - * but so is life + * but nor is life. */ - xprt = list_entry(serv->sv_tempsocks.prev, - struct svc_xprt, - xpt_list); - set_bit(XPT_CLOSE, &xprt->xpt_flags); - svc_xprt_get(xprt); + list_for_each_entry_reverse(xprti, &serv->sv_tempsocks, + xpt_list) + { + if (!test_bit(XPT_PEER_VALID, &xprti->xpt_flags)) { + xprt = xprti; + set_bit(XPT_CLOSE, &xprt->xpt_flags); + svc_xprt_get(xprt); + break; + } + } } spin_unlock_bh(&serv->sv_lock); @@ -1039,7 +1039,8 @@ static void svc_delete_xprt(struct svc_xprt *xprt) spin_lock_bh(&serv->sv_lock); list_del_init(&xprt->xpt_list); - if (test_bit(XPT_TEMP, &xprt->xpt_flags)) + if (test_bit(XPT_TEMP, &xprt->xpt_flags) && + !test_bit(XPT_PEER_VALID, &xprt->xpt_flags)) serv->sv_tmpcnt--; spin_unlock_bh(&serv->sv_lock); From patchwork Mon Jul 15 07:14:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733043 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDAAB6AB8 for ; Mon, 15 Jul 2024 07:49:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029752; cv=none; b=D4dJOheYwKUm/1lEod5lAPqd122iar/9vsmNCpuvv7x6fwFWxjktNM4a4ldB4E4jpM1t+xR92q7QPowSTL2bd7gyAJsdA4/A7CwLAiCwk64wCZFnubrRnkxH4jnGRuNdWWguJBCZ8/dVqmZuRfBJqVwHJLpWr5ftak791wej1W4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029752; c=relaxed/simple; bh=4CavPxHnvjgwRqk3b5jd7+VUlCYG8HEpNIq2rFjgU/Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Te4HsJIHOdEcsYtZxRHVaOzUh5FqlsmVEzAaayOyc9kcunq1Z7dQbgqVWA7iHX7N7nibi8iGi/A5AJNlwod+nE7UPMJwPLJlG+PQXXCKlLSK6Eyf1VykGwVY/8KHdgTmcutz/JCboQBUfASWCyBOVlglVhR/moddqogMfNh88MM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=OtmEte89; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=arz/tj4K; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=OtmEte89; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=arz/tj4K; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="OtmEte89"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="arz/tj4K"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="OtmEte89"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="arz/tj4K" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 08E1521A03; Mon, 15 Jul 2024 07:49:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029749; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yxItJadAbe2VKEiMTtOyf0T3cnXp9s78Ly9lPd+I4FU=; b=OtmEte89xn/wf1uIUnJE7T08oC4tGTTTJr7UCgIccv1G51M4a6gDusSc1lFR6KsI/Qa2Dr BYI2tv/9WH1lwKnubA1xC2v4taej2UX3PpSIKgCz+hgA5PKjLbRloTBhw/ttRR26hSmVZU ZRCsvzzavyNGY6SQQ7K8Q0ojXgQaw44= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029749; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yxItJadAbe2VKEiMTtOyf0T3cnXp9s78Ly9lPd+I4FU=; b=arz/tj4KCXpC63HVUiHlxJuxWVRcF+FRXx8B5kg5ebwPJC9JsMDMa6gTSrviNkw1P62vgn xw6fdBB94zbNE2Aw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029749; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yxItJadAbe2VKEiMTtOyf0T3cnXp9s78Ly9lPd+I4FU=; b=OtmEte89xn/wf1uIUnJE7T08oC4tGTTTJr7UCgIccv1G51M4a6gDusSc1lFR6KsI/Qa2Dr BYI2tv/9WH1lwKnubA1xC2v4taej2UX3PpSIKgCz+hgA5PKjLbRloTBhw/ttRR26hSmVZU ZRCsvzzavyNGY6SQQ7K8Q0ojXgQaw44= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029749; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yxItJadAbe2VKEiMTtOyf0T3cnXp9s78Ly9lPd+I4FU=; b=arz/tj4KCXpC63HVUiHlxJuxWVRcF+FRXx8B5kg5ebwPJC9JsMDMa6gTSrviNkw1P62vgn xw6fdBB94zbNE2Aw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 94FEE137EB; Mon, 15 Jul 2024 07:49:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id wbHgEnLUlGZJbgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:49:06 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 12/14] sunrpc: introduce possibility that requested number of threads is different from actual Date: Mon, 15 Jul 2024 17:14:25 +1000 Message-ID: <20240715074657.18174-13-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,imap1.dmz-prg2.suse.org:helo]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Score: -2.80 X-Spam-Level: New fields sp_nractual and sv_nractual track how many actual threads are running. sp_nrhtreads and sv_nrthreads will be the number that were explicitly request. Currently nractually == nrthreads. sv_nractual is used for sizing UDP incoming socket space - in the rare case that UDP is used. This is because each thread might need to keep a request in the skbs. Signed-off-by: NeilBrown --- include/linux/sunrpc/svc.h | 4 +++- net/sunrpc/svc.c | 22 +++++++++++++++------- net/sunrpc/svcsock.c | 2 +- 3 files changed, 19 insertions(+), 9 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 0b414af448e0..363105fc6326 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -36,6 +36,7 @@ struct svc_pool { unsigned int sp_id; /* pool id; also node id on NUMA */ struct lwq sp_xprts; /* pending transports */ unsigned int sp_nrthreads; /* # of threads in pool */ + unsigned int sp_nractual; /* # of threads running */ struct list_head sp_all_threads; /* all server threads */ struct llist_head sp_idle_threads; /* idle server threads */ @@ -69,7 +70,8 @@ struct svc_serv { struct svc_program * sv_program; /* RPC program */ struct svc_stat * sv_stats; /* RPC statistics */ spinlock_t sv_lock; - unsigned int sv_nrthreads; /* # of server threads */ + unsigned int sv_nrthreads; /* # of server threads requested*/ + unsigned int sv_nractual; /* # of running threads */ unsigned int sv_maxconn; /* max connections allowed or * '0' causing max to be based * on number of threads. */ diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index d814b2cfa84f..33c1a7793f63 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -785,8 +785,12 @@ svc_pool_victim(struct svc_serv *serv, struct svc_pool *target_pool, } if (pool && pool->sp_nrthreads) { - set_bit(SP_VICTIM_REMAINS, &pool->sp_flags); - set_bit(SP_NEED_VICTIM, &pool->sp_flags); + if (pool->sp_nrthreads <= pool->sp_nractual) { + set_bit(SP_VICTIM_REMAINS, &pool->sp_flags); + set_bit(SP_NEED_VICTIM, &pool->sp_flags); + pool->sp_nractual -= 1; + serv->sv_nractual -= 1; + } return pool; } return NULL; @@ -806,6 +810,12 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) chosen_pool = svc_pool_next(serv, pool, &state); node = svc_pool_map_get_node(chosen_pool->sp_id); + serv->sv_nrthreads += 1; + chosen_pool->sp_nrthreads += 1; + + if (chosen_pool->sp_nrthreads <= chosen_pool->sp_nractual) + continue; + rqstp = svc_prepare_thread(serv, chosen_pool, node); if (IS_ERR(rqstp)) return PTR_ERR(rqstp); @@ -815,8 +825,8 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) svc_exit_thread(rqstp); return PTR_ERR(task); } - serv->sv_nrthreads += 1; - chosen_pool->sp_nrthreads += 1; + serv->sv_nractual += 1; + chosen_pool->sp_nractual += 1; rqstp->rq_task = task; if (serv->sv_nrpools > 1) @@ -846,6 +856,7 @@ svc_stop_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) TASK_IDLE); nrservs++; } while (nrservs < 0); + svc_sock_update_bufs(serv); return 0; } @@ -937,13 +948,10 @@ void svc_rqst_release_pages(struct svc_rqst *rqstp) void svc_exit_thread(struct svc_rqst *rqstp) { - struct svc_serv *serv = rqstp->rq_server; struct svc_pool *pool = rqstp->rq_pool; list_del_rcu(&rqstp->rq_all); - svc_sock_update_bufs(serv); - svc_rqst_free(rqstp); clear_and_wake_up_bit(SP_VICTIM_REMAINS, &pool->sp_flags); diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 825ec5357691..191dbc648bd0 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -588,7 +588,7 @@ static int svc_udp_recvfrom(struct svc_rqst *rqstp) * provides an upper bound on the number of threads * which will access the socket. */ - svc_sock_setbufsize(svsk, serv->sv_nrthreads + 3); + svc_sock_setbufsize(svsk, serv->sv_nractual + 3); clear_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags); err = kernel_recvmsg(svsk->sk_sock, &msg, NULL, From patchwork Mon Jul 15 07:14:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733044 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E657175A6 for ; Mon, 15 Jul 2024 07:49:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029762; cv=none; b=e8ZD10hUFRf1cgqQdVBnB9toIIusJQoDgquYiG78EhHAWDCTnCNgTsUuhx80lvgPGQhG7hp0WNS2wWc5oOcXqgmy3OfRC6ZUsUnxGtNknBhQLouNTGtjkerXweJBK/wtnU8kUzny/pyer3DwyRbA8jn3IGxDMvZvhpbsl/s7wQ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029762; c=relaxed/simple; bh=vscPcf6UYREKHryS/pLl53183fRoe1ba0l8MhyHCm1c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s/0+4lv2sxX1rA1vXzjVeoH6bU8TuIFd/KlfAmUmQgdhDYdpjqwD/H8Y6n2a5mvtuKb8iofAo6jkX79mKJU2C/rGqaHguVC/IcvLvHJfTyjEHtO073QNtvuvhA2pR9ebaV0gFb3NRXlGfr+h+fivtGPNQ4PhBW8G31REuBgCtHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=ZGbxhIwh; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=iJXCk/Mj; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=ZGbxhIwh; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=iJXCk/Mj; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="ZGbxhIwh"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="iJXCk/Mj"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="ZGbxhIwh"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="iJXCk/Mj" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C3E9821A03; Mon, 15 Jul 2024 07:49:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029758; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=06DghijWnCCc3GMBDVkth3Kjwzk52ff3/Ea7+Rb/gzA=; b=ZGbxhIwh4Tg0Ua3LbD4vl1PVEAcaHlNtgd992yiGQTz4sROe0du7/3eBIiqSm3GTr62aBy biObp3MPtBmtOl8mU4MW264a79M8BeIjzJWGzO4lWzIEh1fdROzPlI+7ilq0bIAcW3gciL xiBjjCZgrWsX2Rj0x0scGU038PoK96Q= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029758; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=06DghijWnCCc3GMBDVkth3Kjwzk52ff3/Ea7+Rb/gzA=; b=iJXCk/MjzshBRmHOiogui0WdpvfjWRpKUaXn+JGS4o7hbfhFy3QpCp3bAiQU8I/RWb0p4j shfWwvaoT+tZDjBQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029758; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=06DghijWnCCc3GMBDVkth3Kjwzk52ff3/Ea7+Rb/gzA=; b=ZGbxhIwh4Tg0Ua3LbD4vl1PVEAcaHlNtgd992yiGQTz4sROe0du7/3eBIiqSm3GTr62aBy biObp3MPtBmtOl8mU4MW264a79M8BeIjzJWGzO4lWzIEh1fdROzPlI+7ilq0bIAcW3gciL xiBjjCZgrWsX2Rj0x0scGU038PoK96Q= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029758; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=06DghijWnCCc3GMBDVkth3Kjwzk52ff3/Ea7+Rb/gzA=; b=iJXCk/MjzshBRmHOiogui0WdpvfjWRpKUaXn+JGS4o7hbfhFy3QpCp3bAiQU8I/RWb0p4j shfWwvaoT+tZDjBQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 661A0137EB; Mon, 15 Jul 2024 07:49:16 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 5apjB3zUlGZXbgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:49:16 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 13/14] nfsd: introduce concept of a maximum number of threads. Date: Mon, 15 Jul 2024 17:14:26 +1000 Message-ID: <20240715074657.18174-14-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:email]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Score: -2.80 X-Spam-Level: A future patch will allow the number of threads in each nfsd pool to vary dynamically. The lower bound will be the number explicit requested via /proc/fs/nfsd/threads or /proc/fs/nfsd/pool_threads The upper bound can be set in each net-namespace by writing /proc/fs/nfsd/max_threads. This upper bound applies across all pools, there is no per-pool upper limit. If no upper bound is set, then one is calculated. A global upper limit is chosen based on amount of memory. This limit only affects dynamic changes. Static configuration can always over-ride it. We track how many threads are configured in each net namespace, with the max or the min. We also track how many net namespaces have nfsd configured with only a min, not a max. The difference between the calculated max and the total allocation is available to be shared among those namespaces which don't have a maximum configured. Within a namespace, the available share is distributed equally across all pools. In the common case there is one namespace and one pool. A small number of threads are configured as a minimum and no maximum is set. In this case the effective maximum will be directly based on total memory. Approximately 8 per gigabyte. Signed-off-by: NeilBrown --- fs/nfsd/netns.h | 6 +++++ fs/nfsd/nfsctl.c | 45 +++++++++++++++++++++++++++++++++++ fs/nfsd/nfsd.h | 4 ++++ fs/nfsd/nfssvc.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/nfsd/trace.h | 19 +++++++++++++++ 5 files changed, 135 insertions(+) diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 0d2ac15a5003..329484696a42 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -133,6 +133,12 @@ struct nfsd_net { */ unsigned int max_connections; + /* + * Maximum number of threads to auto-adjust up to. If 0 then a + * share of nfsd_max_threads will be used. + */ + unsigned int max_threads; + u32 clientid_base; u32 clientid_counter; u32 clverifier_counter; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index d85b6d1fa31f..37e9936567e9 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -48,6 +48,7 @@ enum { NFSD_Ports, NFSD_MaxBlkSize, NFSD_MaxConnections, + NFSD_MaxThreads, NFSD_Filecache, NFSD_Leasetime, NFSD_Gracetime, @@ -68,6 +69,7 @@ static ssize_t write_versions(struct file *file, char *buf, size_t size); static ssize_t write_ports(struct file *file, char *buf, size_t size); static ssize_t write_maxblksize(struct file *file, char *buf, size_t size); static ssize_t write_maxconn(struct file *file, char *buf, size_t size); +static ssize_t write_maxthreads(struct file *file, char *buf, size_t size); #ifdef CONFIG_NFSD_V4 static ssize_t write_leasetime(struct file *file, char *buf, size_t size); static ssize_t write_gracetime(struct file *file, char *buf, size_t size); @@ -87,6 +89,7 @@ static ssize_t (*const write_op[])(struct file *, char *, size_t) = { [NFSD_Ports] = write_ports, [NFSD_MaxBlkSize] = write_maxblksize, [NFSD_MaxConnections] = write_maxconn, + [NFSD_MaxThreads] = write_maxthreads, #ifdef CONFIG_NFSD_V4 [NFSD_Leasetime] = write_leasetime, [NFSD_Gracetime] = write_gracetime, @@ -939,6 +942,47 @@ static ssize_t write_maxconn(struct file *file, char *buf, size_t size) return scnprintf(buf, SIMPLE_TRANSACTION_LIMIT, "%u\n", maxconn); } +/* + * write_maxthreads - Set or report the current max number threads + * + * Input: + * buf: ignored + * size: zero + * OR + * + * Input: + * buf: C string containing an unsigned + * integer value representing the new + * max number of threads + * size: non-zero length of C string in @buf + * Output: + * On success: passed-in buffer filled with '\n'-terminated C string + * containing numeric value of max_threads setting + * for this net namespace; + * return code is the size in bytes of the string + * On error: return code is zero or a negative errno value + */ +static ssize_t write_maxthreads(struct file *file, char *buf, size_t size) +{ + char *mesg = buf; + struct nfsd_net *nn = net_generic(netns(file), nfsd_net_id); + unsigned int maxthreads = nn->max_threads; + + if (size > 0) { + int rv = get_uint(&mesg, &maxthreads); + + if (rv) + return rv; + trace_nfsd_ctl_maxthreads(netns(file), maxthreads); + mutex_lock(&nfsd_mutex); + nn->max_threads = maxthreads; + nfsd_update_nets(); + mutex_unlock(&nfsd_mutex); + } + + return scnprintf(buf, SIMPLE_TRANSACTION_LIMIT, "%u\n", maxthreads); +} + #ifdef CONFIG_NFSD_V4 static ssize_t __nfsd4_write_time(struct file *file, char *buf, size_t size, time64_t *time, struct nfsd_net *nn) @@ -1372,6 +1416,7 @@ static int nfsd_fill_super(struct super_block *sb, struct fs_context *fc) [NFSD_Ports] = {"portlist", &transaction_ops, S_IWUSR|S_IRUGO}, [NFSD_MaxBlkSize] = {"max_block_size", &transaction_ops, S_IWUSR|S_IRUGO}, [NFSD_MaxConnections] = {"max_connections", &transaction_ops, S_IWUSR|S_IRUGO}, + [NFSD_MaxThreads] = {"max_threads", &transaction_ops, S_IWUSR|S_IRUGO}, [NFSD_Filecache] = {"filecache", &nfsd_file_cache_stats_fops, S_IRUGO}, #ifdef CONFIG_NFSD_V4 [NFSD_Leasetime] = {"nfsv4leasetime", &transaction_ops, S_IWUSR|S_IRUSR}, diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index e4c643255dc9..6874c2de670b 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -156,6 +156,10 @@ int nfsd_create_serv(struct net *net); void nfsd_destroy_serv(struct net *net); extern int nfsd_max_blksize; +void nfsd_update_nets(void); +extern unsigned int nfsd_max_threads; +extern unsigned long nfsd_net_used; +extern unsigned int nfsd_net_cnt; static inline int nfsd_v4client(struct svc_rqst *rq) { diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index b005b2e2e6ad..75d78c17756f 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -80,6 +80,21 @@ DEFINE_SPINLOCK(nfsd_drc_lock); unsigned long nfsd_drc_max_mem; unsigned long nfsd_drc_slotsize_sum; +/* + * nfsd_max_threads is auto-configured based on available ram. + * Each network namespace can configure a minimum number of threads + * and optionally a maximum. + * nfsd_net_used is the number of the max or min from each net namespace. + * nfsd_new_cnt is the number of net namespaces with a configured minimum + * but no configured maximum. + * When nfsd_max_threads exceeds nfsd_net_used, the different is divided + * by nfsd_net_cnt and this number gives the excess above the configured minimum + * for all net namespaces without a configured maximum. + */ +unsigned int nfsd_max_threads; +unsigned long nfsd_net_used; +unsigned int nfsd_net_cnt; + #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL) static const struct svc_version *nfsd_acl_version[] = { # if defined(CONFIG_NFSD_V2_ACL) @@ -130,6 +145,47 @@ struct svc_program nfsd_program = { .pg_rpcbind_set = nfsd_rpcbind_set, }; +void nfsd_update_nets(void) +{ + struct net *net; + + if (nfsd_max_threads == 0) { + nfsd_max_threads = (nr_free_buffer_pages() >> 7) / + (NFSSVC_MAXBLKSIZE >> PAGE_SHIFT); + } + nfsd_net_used = 0; + nfsd_net_cnt = 0; + down_read(&net_rwsem); + for_each_net(net) { + struct nfsd_net *nn = net_generic(net, nfsd_net_id); + + if (!nn->nfsd_serv) + continue; + if (nn->max_threads > 0) { + nfsd_net_used += nn->max_threads; + } else { + nfsd_net_used += nn->nfsd_serv->sv_nrthreads; + nfsd_net_cnt += 1; + } + } + up_read(&net_rwsem); +} + +static inline int nfsd_max_pool_threads(struct svc_pool *p, struct nfsd_net *nn) +{ + int svthreads = nn->nfsd_serv->sv_nrthreads; + + if (nn->max_threads > 0) + return nn->max_threads; + if (nfsd_net_cnt == 0 || svthreads == 0) + return 0; + if (nfsd_max_threads < nfsd_net_cnt) + return p->sp_nrthreads; + /* Share nfsd_max_threads among all net, then among pools in this net. */ + return p->sp_nrthreads + + nfsd_max_threads / nfsd_net_cnt * p->sp_nrthreads / svthreads; +} + bool nfsd_support_version(int vers) { if (vers >= NFSD_MINVERS && vers <= NFSD_MAXVERS) @@ -474,6 +530,7 @@ void nfsd_destroy_serv(struct net *net) spin_lock(&nfsd_notifier_lock); nn->nfsd_serv = NULL; spin_unlock(&nfsd_notifier_lock); + nfsd_update_nets(); /* check if the notifier still has clients */ if (atomic_dec_return(&nfsd_notifier_refcount) == 0) { @@ -614,6 +671,8 @@ int nfsd_create_serv(struct net *net) nn->nfsd_serv = serv; spin_unlock(&nfsd_notifier_lock); + nfsd_update_nets(); + set_max_drc(); /* check if the notifier is already set */ if (atomic_inc_return(&nfsd_notifier_refcount) == 1) { @@ -720,6 +779,7 @@ int nfsd_set_nrthreads(int n, int *nthreads, struct net *net) goto out; } out: + nfsd_update_nets(); return err; } @@ -759,6 +819,7 @@ nfsd_svc(int n, int *nthreads, struct net *net, const struct cred *cred, const c error = nfsd_set_nrthreads(n, nthreads, net); if (error) goto out_put; + nfsd_update_nets(); error = serv->sv_nrthreads; out_put: if (serv->sv_nrthreads == 0) diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h index 77bbd23aa150..92b888e178e8 100644 --- a/fs/nfsd/trace.h +++ b/fs/nfsd/trace.h @@ -2054,6 +2054,25 @@ TRACE_EVENT(nfsd_ctl_maxconn, ) ); +TRACE_EVENT(nfsd_ctl_maxthreads, + TP_PROTO( + const struct net *net, + int maxthreads + ), + TP_ARGS(net, maxthreads), + TP_STRUCT__entry( + __field(unsigned int, netns_ino) + __field(int, maxthreads) + ), + TP_fast_assign( + __entry->netns_ino = net->ns.inum; + __entry->maxthreads = maxthreads + ), + TP_printk("maxthreads=%d", + __entry->maxthreads + ) +); + TRACE_EVENT(nfsd_ctl_time, TP_PROTO( const struct net *net, From patchwork Mon Jul 15 07:14:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13733045 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 785526AB8 for ; Mon, 15 Jul 2024 07:49:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029772; cv=none; b=kOEK+RKb9hUlu9tJtEECo5y57yiTjeTGx/U+/xwIAQoTHMIz57NiuIVUoFG0e769V9/MwGp1+UJbmd44i0zRqG3iyeTNXfx1s81SnQS2p6rClDT5acLDSJJpqUMjXZRvThJ/OV2saMcM+iNyjiFN5EIOe7j2dApHlp4o3iFivLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721029772; c=relaxed/simple; bh=Erzkq6D6GxGT67AZuBp5xl3cCK4QWTXJBOik6JiiF8U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CZN0UL3SC5ZCzIHkGSd31z+Ng/94FvTVTeu9kvdq6T/5W+GE8am3LzEmhj9m7LKxoKMJ9F6AtQ+LhaJg7V/eCYPdA5Jb/sYrIl9dXRn51tGVxZmoQlWja3PE7IU4xXQASE6K0RIvOkDIVHNHsNMRbK6AsZVRA5oj+jObjPxBQEE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=vps7pJlU; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=2cNjGf96; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=vps7pJlU; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=2cNjGf96; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="vps7pJlU"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="2cNjGf96"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="vps7pJlU"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="2cNjGf96" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9E31621A03; Mon, 15 Jul 2024 07:49:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029768; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZiQ27tJ54p1JmOSv2GfocIQv/5T9sX5doNifIBp4E2o=; b=vps7pJlUgrX52GJqOp8/hcq9zXYVBIXXxosFP0Op6UO7WyOAMIAKbmQ3B8sgSGKwmk3v+Y 5TMw8MCkN5QRZmtBEfv2yLsvaM7ZDU7cL+h2bgVgd1/701Xbwd+mmLletWAjG+czQ9t1t1 YtJoNHazlWeRBo9cHOuaQmMh/P4Y0y4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029768; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZiQ27tJ54p1JmOSv2GfocIQv/5T9sX5doNifIBp4E2o=; b=2cNjGf96oaJJaw6GbGiK3BMnhBU1lWEpp7SQzhGjyYlq7zB6fXyWimgnMv+ybKPCIc1Awv E0mA4uWFc+S7y7CA== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=vps7pJlU; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=2cNjGf96 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1721029768; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZiQ27tJ54p1JmOSv2GfocIQv/5T9sX5doNifIBp4E2o=; b=vps7pJlUgrX52GJqOp8/hcq9zXYVBIXXxosFP0Op6UO7WyOAMIAKbmQ3B8sgSGKwmk3v+Y 5TMw8MCkN5QRZmtBEfv2yLsvaM7ZDU7cL+h2bgVgd1/701Xbwd+mmLletWAjG+czQ9t1t1 YtJoNHazlWeRBo9cHOuaQmMh/P4Y0y4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1721029768; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZiQ27tJ54p1JmOSv2GfocIQv/5T9sX5doNifIBp4E2o=; b=2cNjGf96oaJJaw6GbGiK3BMnhBU1lWEpp7SQzhGjyYlq7zB6fXyWimgnMv+ybKPCIc1Awv E0mA4uWFc+S7y7CA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 34E37137EB; Mon, 15 Jul 2024 07:49:25 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id JMjxNoXUlGZjbgAAD6G6ig (envelope-from ); Mon, 15 Jul 2024 07:49:25 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Steve Dickson Subject: [PATCH 14/14] nfsd: adjust number of running nfsd threads Date: Mon, 15 Jul 2024 17:14:27 +1000 Message-ID: <20240715074657.18174-15-neilb@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240715074657.18174-1-neilb@suse.de> References: <20240715074657.18174-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: 9E31621A03 X-Spam-Flag: NO X-Spam-Score: 0.99 X-Rspamd-Action: no action X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [0.99 / 50.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.de:dkim,suse.de:email]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DWL_DNSWL_BLOCKED(0.00)[suse.de:dkim]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Spam-Level: X-Spamd-Bar: / svc_recv() is changed to return a status. This can be: -ETIMEDOUT - waited for 5 seconds and found nothing to do. This is boring. Also there are more actual threads than really needed. -EBUSY - I did something, but there is more stuff to do and no one idle who I can wake up to do it. BTW I successful set a flag: SP_TASK_STARTING. You better clear it. 0 - just minding my own business, nothing to see here. nfsd() is changed to pay attention to this status. In the case of -ETIMEDOUT, if the service mutex can be taken (trylock), the thread becomes and RQ_VICTIM so that it will exit. In the case of -EBUSY, if the actual number of threads is below the calculated maximum, a new thread is started. SP_TASK_STARTING is cleared. To support the above, some code is split out of svc_start_kthreads() into svc_new_thread(). I think we want memory pressure to be able to push a thread into returning -ETIMEDOUT. That can come later. There are printk's in here. They can be discarded or turned into trace points once we are sure about what we want. Signed-off-by: NeilBrown --- fs/nfsd/nfssvc.c | 32 ++++++++++++++++- fs/nfsd/vfs.c | 1 + include/linux/sunrpc/svc.h | 2 ++ include/linux/sunrpc/svcsock.h | 2 +- net/sunrpc/svc.c | 66 +++++++++++++++++++--------------- net/sunrpc/svc_xprt.c | 46 +++++++++++++++++++----- 6 files changed, 110 insertions(+), 39 deletions(-) diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 75d78c17756f..1c8a7dcbfc42 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -931,9 +931,11 @@ static int nfsd(void *vrqstp) { struct svc_rqst *rqstp = (struct svc_rqst *) vrqstp; + struct svc_pool *pool = rqstp->rq_pool; struct svc_xprt *perm_sock = list_entry(rqstp->rq_server->sv_permsocks.next, typeof(struct svc_xprt), xpt_list); struct net *net = perm_sock->xpt_net; struct nfsd_net *nn = net_generic(net, nfsd_net_id); + bool have_mutex = false; /* At this point, the thread shares current->fs * with the init process. We need to create files with the @@ -954,7 +956,33 @@ nfsd(void *vrqstp) /* Update sv_maxconn if it has changed */ rqstp->rq_server->sv_maxconn = nn->max_connections; - svc_recv(rqstp); + switch (svc_recv(rqstp)) { + case -ETIMEDOUT: /* Nothing to do */ + if (mutex_trylock(&nfsd_mutex)) { + if (pool->sp_nractual > pool->sp_nrthreads) { + set_bit(RQ_VICTIM, &rqstp->rq_flags); + pool->sp_nractual -= 1; + printk("Kill a victim\n"); + have_mutex = true; + } else + mutex_unlock(&nfsd_mutex); + } else printk("trylock failed\n"); + break; + case -EBUSY: /* Too much to do */ + if (pool->sp_nractual < nfsd_max_pool_threads(pool, nn) && + mutex_trylock(&nfsd_mutex)) { + // check no idle threads? + if (pool->sp_nractual < nfsd_max_pool_threads(pool,nn)) { + printk("start new thread\n"); + svc_new_thread(rqstp->rq_server, pool); + } + mutex_unlock(&nfsd_mutex); + } + clear_bit(SP_TASK_STARTING, &pool->sp_flags); + break; + default: + break; + } nfsd_file_net_dispose(nn); } @@ -963,6 +991,8 @@ nfsd(void *vrqstp) /* Release the thread */ svc_exit_thread(rqstp); + if (have_mutex) + mutex_unlock(&nfsd_mutex); return 0; } diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c index 29b1f3613800..92bc7c778411 100644 --- a/fs/nfsd/vfs.c +++ b/fs/nfsd/vfs.c @@ -1203,6 +1203,7 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct nfsd_file *nf, commit_reset_write_verifier(nn, rqstp, host_err); goto out_nfserr; } + msleep(40); *cnt = host_err; nfsd_stats_io_write_add(nn, exp, *cnt); fsnotify_modify(file); diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 363105fc6326..6c9d0e42f5d5 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -53,6 +53,7 @@ enum { SP_TASK_PENDING, /* still work to do even if no xprt is queued */ SP_NEED_VICTIM, /* One thread needs to agree to exit */ SP_VICTIM_REMAINS, /* One thread needs to actually exit */ + SP_TASK_STARTING, /* Task has started but not added to idle yet */ }; @@ -410,6 +411,7 @@ struct svc_serv *svc_create(struct svc_program *, unsigned int, bool svc_rqst_replace_page(struct svc_rqst *rqstp, struct page *page); void svc_rqst_release_pages(struct svc_rqst *rqstp); +int svc_new_thread(struct svc_serv *serv, struct svc_pool *pool); void svc_exit_thread(struct svc_rqst *); struct svc_serv * svc_create_pooled(struct svc_program *prog, struct svc_stat *stats, diff --git a/include/linux/sunrpc/svcsock.h b/include/linux/sunrpc/svcsock.h index bf45d9e8492a..11d43600eabb 100644 --- a/include/linux/sunrpc/svcsock.h +++ b/include/linux/sunrpc/svcsock.h @@ -56,7 +56,7 @@ static inline u32 svc_sock_final_rec(struct svc_sock *svsk) /* * Function prototypes. */ -void svc_recv(struct svc_rqst *rqstp); +int svc_recv(struct svc_rqst *rqstp); void svc_send(struct svc_rqst *rqstp); int svc_addsock(struct svc_serv *serv, struct net *net, const int fd, char *name_return, const size_t len, diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 33c1a7793f63..26b6e73fc0de 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -796,19 +796,46 @@ svc_pool_victim(struct svc_serv *serv, struct svc_pool *target_pool, return NULL; } -static int -svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) +int svc_new_thread(struct svc_serv *serv, struct svc_pool *pool) { struct svc_rqst *rqstp; struct task_struct *task; - struct svc_pool *chosen_pool; - unsigned int state = serv->sv_nrthreads-1; int node; - do { - nrservs--; - chosen_pool = svc_pool_next(serv, pool, &state); - node = svc_pool_map_get_node(chosen_pool->sp_id); + node = svc_pool_map_get_node(pool->sp_id); + + rqstp = svc_prepare_thread(serv, pool, node); + if (IS_ERR(rqstp)) + return PTR_ERR(rqstp); + set_bit(SP_TASK_STARTING, &pool->sp_flags); + task = kthread_create_on_node(serv->sv_threadfn, rqstp, + node, "%s", serv->sv_name); + if (IS_ERR(task)) { + clear_bit(SP_TASK_STARTING, &pool->sp_flags); + svc_exit_thread(rqstp); + return PTR_ERR(task); + } + serv->sv_nractual += 1; + pool->sp_nractual += 1; + + rqstp->rq_task = task; + if (serv->sv_nrpools > 1) + svc_pool_map_set_cpumask(task, pool->sp_id); + + svc_sock_update_bufs(serv); + wake_up_process(task); + return 0; +} +EXPORT_SYMBOL_GPL(svc_new_thread); + +static int +svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) +{ + unsigned int state = serv->sv_nrthreads-1; + int err = 0; + + while (!err && nrservs--) { + struct svc_pool *chosen_pool = svc_pool_next(serv, pool, &state); serv->sv_nrthreads += 1; chosen_pool->sp_nrthreads += 1; @@ -816,27 +843,10 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) if (chosen_pool->sp_nrthreads <= chosen_pool->sp_nractual) continue; - rqstp = svc_prepare_thread(serv, chosen_pool, node); - if (IS_ERR(rqstp)) - return PTR_ERR(rqstp); - task = kthread_create_on_node(serv->sv_threadfn, rqstp, - node, "%s", serv->sv_name); - if (IS_ERR(task)) { - svc_exit_thread(rqstp); - return PTR_ERR(task); - } - serv->sv_nractual += 1; - chosen_pool->sp_nractual += 1; - - rqstp->rq_task = task; - if (serv->sv_nrpools > 1) - svc_pool_map_set_cpumask(task, chosen_pool->sp_id); - - svc_sock_update_bufs(serv); - wake_up_process(task); - } while (nrservs > 0); + err = svc_new_thread(serv, chosen_pool); + } - return 0; + return err; } static int diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index a9215e1a2f38..b382bc690670 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -729,15 +729,19 @@ svc_thread_should_sleep(struct svc_rqst *rqstp) return true; } -static void svc_thread_wait_for_work(struct svc_rqst *rqstp) +static bool svc_thread_wait_for_work(struct svc_rqst *rqstp) { struct svc_pool *pool = rqstp->rq_pool; + bool did_wait = false; if (svc_thread_should_sleep(rqstp)) { set_current_state(TASK_IDLE | TASK_FREEZABLE); llist_add(&rqstp->rq_idle, &pool->sp_idle_threads); - if (likely(svc_thread_should_sleep(rqstp))) - schedule(); + clear_bit(SP_TASK_STARTING, &pool->sp_flags); + if (likely(svc_thread_should_sleep(rqstp))) { + schedule_timeout(5*HZ); + did_wait = true; + } while (!llist_del_first_this(&pool->sp_idle_threads, &rqstp->rq_idle)) { @@ -749,7 +753,12 @@ static void svc_thread_wait_for_work(struct svc_rqst *rqstp) * for this new work. This thread can safely sleep * until woken again. */ - schedule(); + if (did_wait) { + schedule_timeout(HZ); + } else { + schedule_timeout(5*HZ); + did_wait = true; + } set_current_state(TASK_IDLE | TASK_FREEZABLE); } __set_current_state(TASK_RUNNING); @@ -757,6 +766,7 @@ static void svc_thread_wait_for_work(struct svc_rqst *rqstp) cond_resched(); } try_to_freeze(); + return did_wait; } static void svc_add_new_temp_xprt(struct svc_serv *serv, struct svc_xprt *newxpt) @@ -840,6 +850,8 @@ static void svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt) static void svc_thread_wake_next(struct svc_rqst *rqstp) { + clear_bit(SP_TASK_STARTING, &rqstp->rq_pool->sp_flags); + if (!svc_thread_should_sleep(rqstp)) /* More work pending after I dequeued some, * wake another worker @@ -854,21 +866,31 @@ static void svc_thread_wake_next(struct svc_rqst *rqstp) * This code is carefully organised not to touch any cachelines in * the shared svc_serv structure, only cachelines in the local * svc_pool. + * + * Returns -ETIMEDOUT if idle for an extended period + * -EBUSY is there is more work to do than available threads + * 0 otherwise. */ -void svc_recv(struct svc_rqst *rqstp) +int svc_recv(struct svc_rqst *rqstp) { struct svc_pool *pool = rqstp->rq_pool; + bool did_wait; + int ret = 0; if (!svc_alloc_arg(rqstp)) - return; + return ret; - svc_thread_wait_for_work(rqstp); + did_wait = svc_thread_wait_for_work(rqstp); + + if (did_wait && svc_thread_should_sleep(rqstp) && + pool->sp_nractual > pool->sp_nrthreads) + ret = -ETIMEDOUT; clear_bit(SP_TASK_PENDING, &pool->sp_flags); if (svc_thread_should_stop(rqstp)) { svc_thread_wake_next(rqstp); - return; + return ret; } rqstp->rq_xprt = svc_xprt_dequeue(pool); @@ -882,8 +904,13 @@ void svc_recv(struct svc_rqst *rqstp) */ if (pool->sp_idle_threads.first) rqstp->rq_chandle.thread_wait = 5 * HZ; - else + else { rqstp->rq_chandle.thread_wait = 1 * HZ; + if (!did_wait && + !test_and_set_bit(SP_TASK_STARTING, + &pool->sp_flags)) + ret = -EBUSY; + } trace_svc_xprt_dequeue(rqstp); svc_handle_xprt(rqstp, xprt); @@ -902,6 +929,7 @@ void svc_recv(struct svc_rqst *rqstp) } } #endif + return ret; } EXPORT_SYMBOL_GPL(svc_recv);