From patchwork Mon Jan 27 01:20:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950923 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7692717C61 for ; Mon, 27 Jan 2025 01:23:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941011; cv=none; b=oril8jhgdJDfOdNGF1uSU7Gjwa4QMm1GE/B1ZTGIKp2qRlXagALtXsmqNY8kMsUXopUIKLEXZb33A73Xl+luubjlwSYtw0VsDlyG54Lw8Tk0Aj0Myv0Oc3XEnCLjtt1ZdQ4llnuyY7YP7sJZI4oM5vK0aWFd0EbEAm3/2ypbe38= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941011; c=relaxed/simple; bh=3DNYhuzHNSR3CsewQyYh8Q6egmAkKnly1dssUXC+dGQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hCVAhjq4l8WnABmwePBrkZ5HhASL7DJvI5RwUsDGhpMPpp1zbTb9+53591bRL83Iaon91YNXgoPT3p9CAH3ZBCrdWy3zCcI3kTvEIVNbdhf0W5HyuSAIRkcSq4EZ8LVL9CF5+rqymUpsZ53gtXHgZC3/jg651bzCx5PO+OGDIF4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=OOMlKs/6; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=KwMECPvT; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=OOMlKs/6; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=KwMECPvT; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="OOMlKs/6"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="KwMECPvT"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="OOMlKs/6"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="KwMECPvT" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 74AEC2115F; Mon, 27 Jan 2025 01:23:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941007; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RA8f+jALc4WeYOXCgaXMs89tCcT4PCwOc89tR4FZ6Tw=; b=OOMlKs/6QbKJRqrH46CuLgVboyzg8SYph7vwGh8oTZfUKoftmf1gXFZ1D/E9ejDoOSUyWv 6NHywwpzAPRzU7C9wRuI+E+hij8t7aCJDeUdZp0lm9UM7GPg7upeiNkA82AokY+HAWDd7J A2yVEyZywUX/VTILThIs0QSdXWnTV7c= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941007; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RA8f+jALc4WeYOXCgaXMs89tCcT4PCwOc89tR4FZ6Tw=; b=KwMECPvTcHgBytmYFrnc8GfGclImy9/fHvs6uqIjG8gtvdQIY7v9GPdmMQPi782MvktRYe 77NVnUgP8WhlmeBA== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b="OOMlKs/6"; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=KwMECPvT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941007; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RA8f+jALc4WeYOXCgaXMs89tCcT4PCwOc89tR4FZ6Tw=; b=OOMlKs/6QbKJRqrH46CuLgVboyzg8SYph7vwGh8oTZfUKoftmf1gXFZ1D/E9ejDoOSUyWv 6NHywwpzAPRzU7C9wRuI+E+hij8t7aCJDeUdZp0lm9UM7GPg7upeiNkA82AokY+HAWDd7J A2yVEyZywUX/VTILThIs0QSdXWnTV7c= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941007; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RA8f+jALc4WeYOXCgaXMs89tCcT4PCwOc89tR4FZ6Tw=; b=KwMECPvTcHgBytmYFrnc8GfGclImy9/fHvs6uqIjG8gtvdQIY7v9GPdmMQPi782MvktRYe 77NVnUgP8WhlmeBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id EDBCD13782; Mon, 27 Jan 2025 01:23:24 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id MW7qJwzglmdwfAAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:23:24 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 1/7] nfsd: filecache: remove race handling. Date: Mon, 27 Jan 2025 12:20:32 +1100 Message-ID: <20250127012257.1803314-2-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: 74AEC2115F X-Spam-Score: -3.01 X-Rspamd-Action: no action X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.de:+]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:rdns,imap1.dmz-prg2.suse.org:helo] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spam-Flag: NO X-Spam-Level: The race that this code tries to protect against is not interesting. The code is problematic as we access the "nf" after we have given our reference to the lru system. While that take 2+ seconds to free things it is still poor form. The only interesting race I can find would be with nfsd_file_close_inode_sync(); This is the only place that really doesn't want the file to stay on the LRU when unhashed (which is the direct consequence of the race). However for the race to happen, some other thread must own a reference to a file and be putting in while nfsd_file_close_inode_sync() is trying to close all files for an inode. If this is possible, that other thread could simply call nfsd_file_put() a little bit later and the result would be the same: not all files are closed when nfsd_file_close_inode_sync() completes. If this was really a problem, we would need to wait in close_inode_sync for the other references to be dropped. We probably don't want to do that. So it is best to simply remove this code. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/filecache.c | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index e342b2e76882..f5a92ac3f16f 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -371,20 +371,10 @@ nfsd_file_put(struct nfsd_file *nf) /* Try to add it to the LRU. If that fails, decrement. */ if (nfsd_file_lru_add(nf)) { - /* If it's still hashed, we're done */ - if (test_bit(NFSD_FILE_HASHED, &nf->nf_flags)) { - nfsd_file_schedule_laundrette(); - return; - } - - /* - * We're racing with unhashing, so try to remove it from - * the LRU. If removal fails, then someone else already - * has our reference. - */ - if (!nfsd_file_lru_remove(nf)) - return; + nfsd_file_schedule_laundrette(); + return; } + } if (refcount_dec_and_test(&nf->nf_ref)) nfsd_file_free(nf); From patchwork Mon Jan 27 01:20:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950924 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2219F4C6E for ; Mon, 27 Jan 2025 01:23:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941016; cv=none; b=L92vX6zY73RijoHztgJFhnMcX0OWwRB0Wyl+wqV3RpCwNcqlDqwmN4270lrot/BKYDXC7jV+choHtpmbjyG0WfwIcRmTxMCblMUZkIETfH/vPzCjXhMi8crrSx7VMN7tk8jglI9tCvFghQ3mBGUMDJiOA3nDjiRfPz72NWdS7Y0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941016; c=relaxed/simple; bh=LxeM8iYQ7KlloFc0NBpnaQBIQ3sh4XIFyg35f7Psqzc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oYA3DbXMj5IdEcER4RFi9Dp3Ajzi0qKV3X+RYRAJrCf6eHACVHfAwXqegqSa3kgQSnv1fUKIhuQ2IthVEMtklIZBAA9QrakiQKzPRdRiXr7AXJGvQbdCzOZJH5QGPV0t1GDgArmuXlIa622IIcRWkrkpf5W4nOo+bkyo8TNSmGo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=NQfKAbeg; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=QNYO7W7J; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=NQfKAbeg; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=QNYO7W7J; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="NQfKAbeg"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="QNYO7W7J"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="NQfKAbeg"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="QNYO7W7J" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 509032115F; Mon, 27 Jan 2025 01:23:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941013; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuwfrUPGCHQYnEmI0HOezSBu6W/N58f0uYCU2IBYhd4=; b=NQfKAbegS1t0J4nrPyK2ZOlmaFQ8qKdfbZh+rcAYRkL8If3tDAxJHnQPzyW8afKslb5vdj 7jNcGhUEp19bG91JH3eu04+8ZFDnzgI4jeo4nWJJ+vuoslrFbpAzjjQ3Qs0XgSs2ONT/jI PdNQ7q3cOGCjaXTLT7txwj4rwQy9GXM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941013; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuwfrUPGCHQYnEmI0HOezSBu6W/N58f0uYCU2IBYhd4=; b=QNYO7W7Js+82LeBNbEdc91k3VnYlyIwgnPFt0d1ZTXTjVXYzZ4YxdugBT+1/wXkx41AnmU KT+GDNAClFm2VGBg== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941013; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuwfrUPGCHQYnEmI0HOezSBu6W/N58f0uYCU2IBYhd4=; b=NQfKAbegS1t0J4nrPyK2ZOlmaFQ8qKdfbZh+rcAYRkL8If3tDAxJHnQPzyW8afKslb5vdj 7jNcGhUEp19bG91JH3eu04+8ZFDnzgI4jeo4nWJJ+vuoslrFbpAzjjQ3Qs0XgSs2ONT/jI PdNQ7q3cOGCjaXTLT7txwj4rwQy9GXM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941013; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuwfrUPGCHQYnEmI0HOezSBu6W/N58f0uYCU2IBYhd4=; b=QNYO7W7Js+82LeBNbEdc91k3VnYlyIwgnPFt0d1ZTXTjVXYzZ4YxdugBT+1/wXkx41AnmU KT+GDNAClFm2VGBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id D8C8313782; Mon, 27 Jan 2025 01:23:30 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id TBPeIhLglmc6fgAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:23:30 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 2/7] nfsd: filecache: use nfsd_file_dispose_list() in nfsd_file_close_inode_sync() Date: Mon, 27 Jan 2025 12:20:33 +1100 Message-ID: <20250127012257.1803314-3-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo]; RCVD_TLS_ALL(0.00)[] X-Spam-Score: -2.80 X-Spam-Flag: NO nfsd_file_close_inode_sync() contains an exactly copy of nfsd_file_dispose_list(). This patch removes it and calls nfsd_file_dispose_list() instead. Reviewed-by: Jeff Layton Signed-off-by: NeilBrown --- fs/nfsd/filecache.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index f5a92ac3f16f..549969d4aa7c 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -681,17 +681,12 @@ nfsd_file_close_inode(struct inode *inode) void nfsd_file_close_inode_sync(struct inode *inode) { - struct nfsd_file *nf; LIST_HEAD(dispose); trace_nfsd_file_close(inode); nfsd_file_queue_for_close(inode, &dispose); - while (!list_empty(&dispose)) { - nf = list_first_entry(&dispose, struct nfsd_file, nf_gc); - list_del_init(&nf->nf_gc); - nfsd_file_free(nf); - } + nfsd_file_dispose_list(&dispose); } static int From patchwork Mon Jan 27 01:20:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950925 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D3214C6E for ; Mon, 27 Jan 2025 01:23:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941023; cv=none; b=GA0gBGPCTH/vz3tw0gB2FN4HUrXIci+JeKYd+aQ7W43Sn99NFBObBh8hsT5Ovvf38oHuG3s9x5Wey1+A5d3LGQcplOT/JdOrpuq4BFey9khfwqhM5l11OIr9AaX1tskWS6tsczVHMY001Vri+xf/WZYZvN/b4YisooZw7wyhPF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941023; c=relaxed/simple; bh=4w6tOB4tRGCAHLn377kIRAuXJYu/l61pBpio+KATOsk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dUUzfg27MJgCkI/qtV2dE3gZdz/nEHwDqFLd+KE1fUeoil9FfoXZqUOA9YCw+VWPx6vohZNfKaDrUJs+cIyY2M/xzcgMdBJ6+Qs0gE08dXvecIWJfe2eftgvwvvDhY0hFqWEC0yG8OCrd0/7TEtYJFZx6vvmnTVNnW/Vo9pn+Fg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=PXkIOpcm; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Nhc/qeTs; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=PXkIOpcm; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Nhc/qeTs; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="PXkIOpcm"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Nhc/qeTs"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="PXkIOpcm"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Nhc/qeTs" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9A2791F38F; Mon, 27 Jan 2025 01:23:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941019; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0o7e2GUBtE9TJ5xQZyDGgS87nL6Ky3E6vZGTESNcucQ=; b=PXkIOpcmQPd6hQCG8t49aEru0GsIBDQFgLnmGdKbFKYp3Cx+THQsMcMyLtyCafy9qcnZYI 9nrIzv8euyRkWgzzrrp/o0VMk+j3CwnEfoquK2VKXdPb6Cwp/ZEDMZrS57Y9zvBLephgfh YR3UngF+GxwMBmfc+Ag1xpyswF2KMz4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941019; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0o7e2GUBtE9TJ5xQZyDGgS87nL6Ky3E6vZGTESNcucQ=; b=Nhc/qeTs5OcHlKbHXoDBYWWOCRB3EBBTRXEH3u1eySm7GM0k7n2Fs0jcEkk1bILGCV7QZp j1FB/DLoE1nN/RDQ== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941019; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0o7e2GUBtE9TJ5xQZyDGgS87nL6Ky3E6vZGTESNcucQ=; b=PXkIOpcmQPd6hQCG8t49aEru0GsIBDQFgLnmGdKbFKYp3Cx+THQsMcMyLtyCafy9qcnZYI 9nrIzv8euyRkWgzzrrp/o0VMk+j3CwnEfoquK2VKXdPb6Cwp/ZEDMZrS57Y9zvBLephgfh YR3UngF+GxwMBmfc+Ag1xpyswF2KMz4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941019; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0o7e2GUBtE9TJ5xQZyDGgS87nL6Ky3E6vZGTESNcucQ=; b=Nhc/qeTs5OcHlKbHXoDBYWWOCRB3EBBTRXEH3u1eySm7GM0k7n2Fs0jcEkk1bILGCV7QZp j1FB/DLoE1nN/RDQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id D8DB013782; Mon, 27 Jan 2025 01:23:36 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id bVjrIhjglmc4AQAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:23:36 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 3/7] nfsd: filecache: move globals nfsd_file_lru and nfsd_file_shrinker to be per-net Date: Mon, 27 Jan 2025 12:20:34 +1100 Message-ID: <20250127012257.1803314-4-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Level: The final freeing of nfsd files is done by per-net nfsd threads (which call nfsd_file_net_dispose()) so it makes some sense to make more of the freeing infrastructure to be per-net - in struct nfsd_fcache_disposal. This is a step towards replacing the list_lru with simple lists which each share the per-net lock in nfsd_fcache_disposal and will require less list walking. As the net is always shutdown before there is any chance that the rest of the filecache is shut down we can removed the tests on NFSD_FILE_CACHE_UP. For the filecache stats file, which assumes a global lru, we keep a separate counter which includes all files in all netns lrus. Reviewed-by: Jeff Layton Signed-off-by: NeilBrown --- fs/nfsd/filecache.c | 128 +++++++++++++++++++++++++------------------- 1 file changed, 72 insertions(+), 56 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 549969d4aa7c..55f69bcde500 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -60,18 +60,20 @@ static DEFINE_PER_CPU(unsigned long, nfsd_file_allocations); static DEFINE_PER_CPU(unsigned long, nfsd_file_releases); static DEFINE_PER_CPU(unsigned long, nfsd_file_total_age); static DEFINE_PER_CPU(unsigned long, nfsd_file_evictions); +static DEFINE_PER_CPU(unsigned long, nfsd_file_lru_total); struct nfsd_fcache_disposal { spinlock_t lock; + struct list_lru file_lru; struct list_head freeme; + struct delayed_work filecache_laundrette; + struct shrinker *file_shrinker; }; static struct kmem_cache *nfsd_file_slab; static struct kmem_cache *nfsd_file_mark_slab; -static struct list_lru nfsd_file_lru; static unsigned long nfsd_file_flags; static struct fsnotify_group *nfsd_file_fsnotify_group; -static struct delayed_work nfsd_filecache_laundrette; static struct rhltable nfsd_file_rhltable ____cacheline_aligned_in_smp; @@ -109,11 +111,18 @@ static const struct rhashtable_params nfsd_file_rhash_params = { }; static void -nfsd_file_schedule_laundrette(void) +nfsd_file_schedule_laundrette(struct nfsd_fcache_disposal *l) { - if (test_bit(NFSD_FILE_CACHE_UP, &nfsd_file_flags)) - queue_delayed_work(system_unbound_wq, &nfsd_filecache_laundrette, - NFSD_LAUNDRETTE_DELAY); + queue_delayed_work(system_unbound_wq, &l->filecache_laundrette, + NFSD_LAUNDRETTE_DELAY); +} + +static void +nfsd_file_schedule_laundrette_net(struct net *net) +{ + struct nfsd_net *nn = net_generic(net, nfsd_net_id); + + nfsd_file_schedule_laundrette(nn->fcache_disposal); } static void @@ -318,11 +327,14 @@ nfsd_file_check_writeback(struct nfsd_file *nf) mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK); } - static bool nfsd_file_lru_add(struct nfsd_file *nf) { + struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id); + struct nfsd_fcache_disposal *l = nn->fcache_disposal; + set_bit(NFSD_FILE_REFERENCED, &nf->nf_flags); - if (list_lru_add_obj(&nfsd_file_lru, &nf->nf_lru)) { + if (list_lru_add_obj(&l->file_lru, &nf->nf_lru)) { + this_cpu_inc(nfsd_file_lru_total); trace_nfsd_file_lru_add(nf); return true; } @@ -331,7 +343,11 @@ static bool nfsd_file_lru_add(struct nfsd_file *nf) static bool nfsd_file_lru_remove(struct nfsd_file *nf) { - if (list_lru_del_obj(&nfsd_file_lru, &nf->nf_lru)) { + struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id); + struct nfsd_fcache_disposal *l = nn->fcache_disposal; + + if (list_lru_del_obj(&l->file_lru, &nf->nf_lru)) { + this_cpu_dec(nfsd_file_lru_total); trace_nfsd_file_lru_del(nf); return true; } @@ -371,7 +387,7 @@ nfsd_file_put(struct nfsd_file *nf) /* Try to add it to the LRU. If that fails, decrement. */ if (nfsd_file_lru_add(nf)) { - nfsd_file_schedule_laundrette(); + nfsd_file_schedule_laundrette_net(nf->nf_net); return; } @@ -525,12 +541,14 @@ nfsd_file_lru_cb(struct list_head *item, struct list_lru_one *lru, if (!refcount_dec_and_test(&nf->nf_ref)) { trace_nfsd_file_gc_in_use(nf); list_lru_isolate(lru, &nf->nf_lru); + this_cpu_dec(nfsd_file_lru_total); return LRU_REMOVED; } /* Refcount went to zero. Unhash it and queue it to the dispose list */ nfsd_file_unhash(nf); list_lru_isolate(lru, &nf->nf_lru); + this_cpu_dec(nfsd_file_lru_total); list_add(&nf->nf_gc, head); this_cpu_inc(nfsd_file_evictions); trace_nfsd_file_gc_disposed(nf); @@ -538,18 +556,18 @@ nfsd_file_lru_cb(struct list_head *item, struct list_lru_one *lru, } static void -nfsd_file_gc(void) +nfsd_file_gc(struct nfsd_fcache_disposal *l) { - unsigned long remaining = list_lru_count(&nfsd_file_lru); + unsigned long remaining = list_lru_count(&l->file_lru); LIST_HEAD(dispose); unsigned long ret; while (remaining > 0) { unsigned long num_to_scan = min(remaining, NFSD_FILE_GC_BATCH); - ret = list_lru_walk(&nfsd_file_lru, nfsd_file_lru_cb, + ret = list_lru_walk(&l->file_lru, nfsd_file_lru_cb, &dispose, num_to_scan); - trace_nfsd_file_gc_removed(ret, list_lru_count(&nfsd_file_lru)); + trace_nfsd_file_gc_removed(ret, list_lru_count(&l->file_lru)); nfsd_file_dispose_list_delayed(&dispose); remaining -= num_to_scan; } @@ -558,32 +576,36 @@ nfsd_file_gc(void) static void nfsd_file_gc_worker(struct work_struct *work) { - nfsd_file_gc(); - if (list_lru_count(&nfsd_file_lru)) - nfsd_file_schedule_laundrette(); + struct nfsd_fcache_disposal *l = container_of( + work, struct nfsd_fcache_disposal, filecache_laundrette.work); + nfsd_file_gc(l); + if (list_lru_count(&l->file_lru)) + nfsd_file_schedule_laundrette(l); } static unsigned long nfsd_file_lru_count(struct shrinker *s, struct shrink_control *sc) { - return list_lru_count(&nfsd_file_lru); + struct nfsd_fcache_disposal *l = s->private_data; + + return list_lru_count(&l->file_lru); } static unsigned long nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc) { + struct nfsd_fcache_disposal *l = s->private_data; + LIST_HEAD(dispose); unsigned long ret; - ret = list_lru_shrink_walk(&nfsd_file_lru, sc, + ret = list_lru_shrink_walk(&l->file_lru, sc, nfsd_file_lru_cb, &dispose); - trace_nfsd_file_shrinker_removed(ret, list_lru_count(&nfsd_file_lru)); + trace_nfsd_file_shrinker_removed(ret, list_lru_count(&l->file_lru)); nfsd_file_dispose_list_delayed(&dispose); return ret; } -static struct shrinker *nfsd_file_shrinker; - /** * nfsd_file_cond_queue - conditionally unhash and queue a nfsd_file * @nf: nfsd_file to attempt to queue @@ -763,29 +785,10 @@ nfsd_file_cache_init(void) goto out_err; } - ret = list_lru_init(&nfsd_file_lru); - if (ret) { - pr_err("nfsd: failed to init nfsd_file_lru: %d\n", ret); - goto out_err; - } - - nfsd_file_shrinker = shrinker_alloc(0, "nfsd-filecache"); - if (!nfsd_file_shrinker) { - ret = -ENOMEM; - pr_err("nfsd: failed to allocate nfsd_file_shrinker\n"); - goto out_lru; - } - - nfsd_file_shrinker->count_objects = nfsd_file_lru_count; - nfsd_file_shrinker->scan_objects = nfsd_file_lru_scan; - nfsd_file_shrinker->seeks = 1; - - shrinker_register(nfsd_file_shrinker); - ret = lease_register_notifier(&nfsd_file_lease_notifier); if (ret) { pr_err("nfsd: unable to register lease notifier: %d\n", ret); - goto out_shrinker; + goto out_err; } nfsd_file_fsnotify_group = fsnotify_alloc_group(&nfsd_file_fsnotify_ops, @@ -798,17 +801,12 @@ nfsd_file_cache_init(void) goto out_notifier; } - INIT_DELAYED_WORK(&nfsd_filecache_laundrette, nfsd_file_gc_worker); out: if (ret) clear_bit(NFSD_FILE_CACHE_UP, &nfsd_file_flags); return ret; out_notifier: lease_unregister_notifier(&nfsd_file_lease_notifier); -out_shrinker: - shrinker_free(nfsd_file_shrinker); -out_lru: - list_lru_destroy(&nfsd_file_lru); out_err: kmem_cache_destroy(nfsd_file_slab); nfsd_file_slab = NULL; @@ -860,13 +858,38 @@ nfsd_alloc_fcache_disposal(void) if (!l) return NULL; spin_lock_init(&l->lock); + INIT_DELAYED_WORK(&l->filecache_laundrette, nfsd_file_gc_worker); INIT_LIST_HEAD(&l->freeme); + l->file_shrinker = shrinker_alloc(0, "nfsd-filecache"); + if (!l->file_shrinker) { + pr_err("nfsd: failed to allocate nfsd_file_shrinker\n"); + kfree(l); + return NULL; + } + l->file_shrinker->count_objects = nfsd_file_lru_count; + l->file_shrinker->scan_objects = nfsd_file_lru_scan; + l->file_shrinker->seeks = 1; + l->file_shrinker->private_data = l; + + /* if file_lru is not zeroed it can trigger a bug: ->key is the problem */ + memset(&l->file_lru, 0, sizeof(l->file_lru)); + if (list_lru_init(&l->file_lru)) { + pr_err("nfsd: failed to init nfsd_file_lru\n"); + shrinker_free(l->file_shrinker); + kfree(l); + return NULL; + } + + shrinker_register(l->file_shrinker); return l; } static void nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l) { + cancel_delayed_work_sync(&l->filecache_laundrette); + shrinker_free(l->file_shrinker); + list_lru_destroy(&l->file_lru); nfsd_file_dispose_list(&l->freeme); kfree(l); } @@ -919,14 +942,7 @@ nfsd_file_cache_shutdown(void) return; lease_unregister_notifier(&nfsd_file_lease_notifier); - shrinker_free(nfsd_file_shrinker); - /* - * make sure all callers of nfsd_file_lru_cb are done before - * calling nfsd_file_cache_purge - */ - cancel_delayed_work_sync(&nfsd_filecache_laundrette); __nfsd_file_cache_purge(NULL); - list_lru_destroy(&nfsd_file_lru); rcu_barrier(); fsnotify_put_group(nfsd_file_fsnotify_group); nfsd_file_fsnotify_group = NULL; @@ -944,6 +960,7 @@ nfsd_file_cache_shutdown(void) per_cpu(nfsd_file_releases, i) = 0; per_cpu(nfsd_file_total_age, i) = 0; per_cpu(nfsd_file_evictions, i) = 0; + per_cpu(nfsd_file_lru_total, i) = 0; } } @@ -1297,8 +1314,6 @@ int nfsd_file_cache_stats_show(struct seq_file *m, void *v) struct bucket_table *tbl; struct rhashtable *ht; - lru = list_lru_count(&nfsd_file_lru); - rcu_read_lock(); ht = &nfsd_file_rhltable.ht; count = atomic_read(&ht->nelems); @@ -1309,6 +1324,7 @@ int nfsd_file_cache_stats_show(struct seq_file *m, void *v) mutex_unlock(&nfsd_mutex); for_each_possible_cpu(i) { + lru += per_cpu(nfsd_file_lru_total, i); hits += per_cpu(nfsd_file_cache_hits, i); acquisitions += per_cpu(nfsd_file_acquisitions, i); allocations += per_cpu(nfsd_file_allocations, i); From patchwork Mon Jan 27 01:20:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950926 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9E194C6E for ; Mon, 27 Jan 2025 01:23:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941033; cv=none; b=moBtDrBIKk22skkF9G3U2vVoBrpvMIbABnslXc5Ry/8IofcBDCJ7KrTdXxK+jhDwpspY9kVr1NeWGwAqfxj0HA8YwdsDLpX8ArzcHwvTlSnkwbkG7k1FfwcKEsAk5FYW5WCShnayITOBuZI8BG0mHx0Sj8T+t8CqgRnyXK5HvWw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941033; c=relaxed/simple; bh=vvO3m+ekDCL0a+WDqWumYofw1EWEBjuggwhXTZ9ImaM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=peo0GpB2j8Jw/WKIAKdDu7Bb31vfhYp+YgOEzsLxj/SR+AK7uzmrune+KG4qIj2q/qswaKRZvGkmcnyvsWwPm0Zl8l0ziR+ulU60qwM9Fdfbrxu/uPkL09kPoPNcg6jwKIVzGmJvEqakOCJAK/pPD5jHDHNVLTVUOGtWo1b5Odw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=XhvZHGCq; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=hALEfaTA; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=QKTnLMfi; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=F8bY0IHx; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="XhvZHGCq"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="hALEfaTA"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="QKTnLMfi"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="F8bY0IHx" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id EF30A2115C; Mon, 27 Jan 2025 01:23:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941030; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2eM9RdFV1eLrwEHUtFQR1t/a/Kca3QscmnLfs6Cjz7Y=; b=XhvZHGCqAbsIyfnswD0+ZtfpwdPdJwpzmHmZspmNKALwdUR6rEw8g5etmNqPbwQeNL1KPd 5D/PguIj0Je2EycSnJ+wZi/9yVZmvwsinnFnXxLJ5GTfDFZIqXGkKBYbH1c/PFBqSgEzgv t8O+z/VuL5O8wiCtjH2nFXJIVTsuTq0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941030; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2eM9RdFV1eLrwEHUtFQR1t/a/Kca3QscmnLfs6Cjz7Y=; b=hALEfaTAAXkzZEnM2DdZ//aJyuKZY5vAMMzgxYf3OuRZGquv2PKcp9rKQXxYR50IGqwr0j KLkDaXAzceVF6hBw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941029; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2eM9RdFV1eLrwEHUtFQR1t/a/Kca3QscmnLfs6Cjz7Y=; b=QKTnLMfiZV/6xgLV1goMCk0ba+CrgGCRHoM0BVnW4zbbsv4VD+f4Ktju//QlZvfdOhLk3R aMMfQu0NLniNwb66zoMEZ5cW0nTl/tdJN8Qr/SUCmVYr0Vxmng4nuRDkak+YWGY/5N366F wi3SZAzba3OOnoDRC+nq3lpoS6rzLy0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941029; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2eM9RdFV1eLrwEHUtFQR1t/a/Kca3QscmnLfs6Cjz7Y=; b=F8bY0IHxwB8tN0D8F9m55BCMJXmqHjzcRE1U6DKvncUDOkwm4nDtCqhbhuXHyVn2Q50U0Q eKVAsxtfZqyr7aCQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 396A713782; Mon, 27 Jan 2025 01:23:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id emaYNyLglmeIBAAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:23:46 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 4/7] nfsd: filecache: change garbage collection list management. Date: Mon, 27 Jan 2025 12:20:35 +1100 Message-ID: <20250127012257.1803314-5-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Level: The nfsd filecache currently uses list_lru for tracking files recently used in NFSv3 requests which need to be "garbage collected" when they have becoming idle - unused for 2-4 seconds. I do not believe list_lru is a good tool for this. It does not allow the timeout which filecache requires so we have to add a timeout mechanism which holds the list_lru lock while the whole list is scanned looking for entries that haven't been recently accessed. When the list is largish (even a few hundred) this can block new requests which need the lock to remove a file to access it. This patch removes the list_lru and instead uses 2 simple linked lists. When a file is accessed it is removed from whichever list it is on, then added to the tail of the first list. Every 2 seconds the second list is moved to the "freeme" list and the first list is moved to the second list. This avoids any need to walk a list to find old entries. Previously a file would be unhashed before being moved to the freeme list. We don't do that any more. The freeme list is much like the other two lists (recent and older) in that they all hold a reference to the file and the file is still hashed. When the nfsd thread processes the freeme list it now uses the new nfsd_file_release_list() which uses nfsd_file_cond_queue() to unhash and drop the refcount. We no longer have a precise count of the size of the lru (recent + older) as we don't know how big "older" is when it is moved to "freeme". However the shrinker can cope with an approximation. So we keep a count of number in the lru and when "older" is moved to "freeme" we divide that count by 2. When we remove anything from the lru we decrement that counter but ensure it never goes negative. Naturally when we add to the lru we increase the counter. For the filecache stats file, which assumes a global lru, we keep a separate counter which includes all files in all netns in recent or older or freeme. We discard the nf_gc linkage in an nfsd_file and only use nf_lru. We discard NFSD_FILE_REFERENCED. This patch drops the nfsd_file_gc_removed() trace point. I couldn't think of useful information to provide. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/filecache.c | 218 +++++++++++++++++++++----------------------- fs/nfsd/filecache.h | 4 +- fs/nfsd/trace.h | 4 - 3 files changed, 107 insertions(+), 119 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 55f69bcde500..1e90da507152 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -34,7 +34,6 @@ #include #include #include -#include #include #include #include @@ -64,10 +63,13 @@ static DEFINE_PER_CPU(unsigned long, nfsd_file_lru_total); struct nfsd_fcache_disposal { spinlock_t lock; - struct list_lru file_lru; - struct list_head freeme; + struct list_head recent; /* have been used in last 0-2 seconds */ + struct list_head older; /* haven't been used in last 0-2 seconds */ + struct list_head freeme; /* ready to be discarded */ + unsigned long num_gc; /* Approximate size of recent plus older */ struct delayed_work filecache_laundrette; struct shrinker *file_shrinker; + struct nfsd_net *nn; }; static struct kmem_cache *nfsd_file_slab; @@ -227,7 +229,6 @@ nfsd_file_alloc(struct net *net, struct inode *inode, unsigned char need, this_cpu_inc(nfsd_file_allocations); INIT_LIST_HEAD(&nf->nf_lru); - INIT_LIST_HEAD(&nf->nf_gc); nf->nf_birthtime = ktime_get(); nf->nf_file = NULL; nf->nf_cred = get_current_cred(); @@ -332,12 +333,16 @@ static bool nfsd_file_lru_add(struct nfsd_file *nf) struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id); struct nfsd_fcache_disposal *l = nn->fcache_disposal; - set_bit(NFSD_FILE_REFERENCED, &nf->nf_flags); - if (list_lru_add_obj(&l->file_lru, &nf->nf_lru)) { + spin_lock(&l->lock); + if (list_empty(&nf->nf_lru)) { + list_add_tail(&nf->nf_lru, &l->recent); + l->num_gc += 1; this_cpu_inc(nfsd_file_lru_total); trace_nfsd_file_lru_add(nf); + spin_unlock(&l->lock); return true; } + spin_unlock(&l->lock); return false; } @@ -346,11 +351,17 @@ static bool nfsd_file_lru_remove(struct nfsd_file *nf) struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id); struct nfsd_fcache_disposal *l = nn->fcache_disposal; - if (list_lru_del_obj(&l->file_lru, &nf->nf_lru)) { + spin_lock(&l->lock); + if (!list_empty(&nf->nf_lru)) { + list_del_init(&nf->nf_lru); this_cpu_dec(nfsd_file_lru_total); + if (l->num_gc > 0) + l->num_gc -= 1; trace_nfsd_file_lru_del(nf); + spin_unlock(&l->lock); return true; } + spin_unlock(&l->lock); return false; } @@ -430,12 +441,26 @@ nfsd_file_dispose_list(struct list_head *dispose) struct nfsd_file *nf; while (!list_empty(dispose)) { - nf = list_first_entry(dispose, struct nfsd_file, nf_gc); - list_del_init(&nf->nf_gc); + nf = list_first_entry(dispose, struct nfsd_file, nf_lru); + list_del_init(&nf->nf_lru); nfsd_file_free(nf); } } +static void +nfsd_file_cond_queue(struct nfsd_file *nf, struct list_head *dispose); + +static void +nfsd_file_release_list(struct list_head *dispose) +{ + LIST_HEAD(dispose2); + struct nfsd_file *nf, *nf2; + + list_for_each_entry_safe(nf, nf2, dispose, nf_lru) + nfsd_file_cond_queue(nf, &dispose2); + nfsd_file_dispose_list(&dispose2); +} + /** * nfsd_file_dispose_list_delayed - move list of dead files to net's freeme list * @dispose: list of nfsd_files to be disposed @@ -448,13 +473,13 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose) { while(!list_empty(dispose)) { struct nfsd_file *nf = list_first_entry(dispose, - struct nfsd_file, nf_gc); + struct nfsd_file, nf_lru); struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id); struct nfsd_fcache_disposal *l = nn->fcache_disposal; struct svc_serv *serv; spin_lock(&l->lock); - list_move_tail(&nf->nf_gc, &l->freeme); + list_move_tail(&nf->nf_lru, &l->freeme); spin_unlock(&l->lock); /* @@ -486,90 +511,32 @@ void nfsd_file_net_dispose(struct nfsd_net *nn) int i; spin_lock(&l->lock); - for (i = 0; i < 8 && !list_empty(&l->freeme); i++) - list_move(l->freeme.next, &dispose); + for (i = 0; i < 8 && !list_empty(&l->freeme); i++) { + struct nfsd_file *nf = list_first_entry( + &l->freeme, struct nfsd_file, nf_lru); + + /* + * Don't throw out files that are still + * undergoing I/O or that have uncleared errors + * pending. + */ + if (nfsd_file_check_writeback(nf)) { + trace_nfsd_file_gc_writeback(nf); + list_move(&nf->nf_lru, &l->recent); + l->num_gc += 1; + } else { + trace_nfsd_file_gc_disposed(nf); + list_move(&nf->nf_lru, &dispose); + this_cpu_inc(nfsd_file_evictions); + } + } spin_unlock(&l->lock); if (!list_empty(&l->freeme)) /* Wake up another thread to share the work * *before* doing any actual disposing. */ svc_wake_up(nn->nfsd_serv); - nfsd_file_dispose_list(&dispose); - } -} - -/** - * nfsd_file_lru_cb - Examine an entry on the LRU list - * @item: LRU entry to examine - * @lru: controlling LRU - * @arg: dispose list - * - * Return values: - * %LRU_REMOVED: @item was removed from the LRU - * %LRU_ROTATE: @item is to be moved to the LRU tail - * %LRU_SKIP: @item cannot be evicted - */ -static enum lru_status -nfsd_file_lru_cb(struct list_head *item, struct list_lru_one *lru, - void *arg) -{ - struct list_head *head = arg; - struct nfsd_file *nf = list_entry(item, struct nfsd_file, nf_lru); - - /* We should only be dealing with GC entries here */ - WARN_ON_ONCE(!test_bit(NFSD_FILE_GC, &nf->nf_flags)); - - /* - * Don't throw out files that are still undergoing I/O or - * that have uncleared errors pending. - */ - if (nfsd_file_check_writeback(nf)) { - trace_nfsd_file_gc_writeback(nf); - return LRU_SKIP; - } - - /* If it was recently added to the list, skip it */ - if (test_and_clear_bit(NFSD_FILE_REFERENCED, &nf->nf_flags)) { - trace_nfsd_file_gc_referenced(nf); - return LRU_ROTATE; - } - - /* - * Put the reference held on behalf of the LRU. If it wasn't the last - * one, then just remove it from the LRU and ignore it. - */ - if (!refcount_dec_and_test(&nf->nf_ref)) { - trace_nfsd_file_gc_in_use(nf); - list_lru_isolate(lru, &nf->nf_lru); - this_cpu_dec(nfsd_file_lru_total); - return LRU_REMOVED; - } - - /* Refcount went to zero. Unhash it and queue it to the dispose list */ - nfsd_file_unhash(nf); - list_lru_isolate(lru, &nf->nf_lru); - this_cpu_dec(nfsd_file_lru_total); - list_add(&nf->nf_gc, head); - this_cpu_inc(nfsd_file_evictions); - trace_nfsd_file_gc_disposed(nf); - return LRU_REMOVED; -} - -static void -nfsd_file_gc(struct nfsd_fcache_disposal *l) -{ - unsigned long remaining = list_lru_count(&l->file_lru); - LIST_HEAD(dispose); - unsigned long ret; - - while (remaining > 0) { - unsigned long num_to_scan = min(remaining, NFSD_FILE_GC_BATCH); - - ret = list_lru_walk(&l->file_lru, nfsd_file_lru_cb, - &dispose, num_to_scan); - trace_nfsd_file_gc_removed(ret, list_lru_count(&l->file_lru)); - nfsd_file_dispose_list_delayed(&dispose); - remaining -= num_to_scan; + nfsd_file_release_list(&dispose); } } @@ -578,9 +545,19 @@ nfsd_file_gc_worker(struct work_struct *work) { struct nfsd_fcache_disposal *l = container_of( work, struct nfsd_fcache_disposal, filecache_laundrette.work); - nfsd_file_gc(l); - if (list_lru_count(&l->file_lru)) + + spin_lock(&l->lock); + list_splice_init(&l->older, &l->freeme); + list_splice_init(&l->recent, &l->older); + /* We don't know how many were moved to 'freeme' and don't want + * to waste time counting - guess a half. + */ + l->num_gc /= 2; + if (!list_empty(&l->freeme)) + svc_wake_up(l->nn->nfsd_serv); + if (!list_empty(&l->older) || !list_empty(&l->recent)) nfsd_file_schedule_laundrette(l); + spin_unlock(&l->lock); } static unsigned long @@ -588,22 +565,40 @@ nfsd_file_lru_count(struct shrinker *s, struct shrink_control *sc) { struct nfsd_fcache_disposal *l = s->private_data; - return list_lru_count(&l->file_lru); + return l->num_gc; } static unsigned long nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc) { struct nfsd_fcache_disposal *l = s->private_data; + struct nfsd_file *nf; + int scanned = 0; + + spin_lock(&l->lock); + while (scanned < sc->nr_to_scan && + (nf = list_first_entry_or_null(&l->older, + struct nfsd_file, nf_lru)) != NULL) { + list_del_init(&nf->nf_lru); + list_add_tail(&nf->nf_lru, &l->freeme); + if (l->num_gc > 0) + l->num_gc -= 1; + scanned += 1; + } + if (scanned > 0) + svc_wake_up(l->nn->nfsd_serv); - LIST_HEAD(dispose); - unsigned long ret; + trace_nfsd_file_shrinker_removed(scanned, l->num_gc); - ret = list_lru_shrink_walk(&l->file_lru, sc, - nfsd_file_lru_cb, &dispose); - trace_nfsd_file_shrinker_removed(ret, list_lru_count(&l->file_lru)); - nfsd_file_dispose_list_delayed(&dispose); - return ret; + while (scanned < sc->nr_to_scan && + (nf = list_first_entry_or_null(&l->recent, + struct nfsd_file, nf_lru)) != NULL) { + list_del_init(&nf->nf_lru); + list_add_tail(&nf->nf_lru, &l->older); + scanned += 1; + } + spin_unlock(&l->lock); + return scanned; } /** @@ -616,7 +611,6 @@ nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc) */ static void nfsd_file_cond_queue(struct nfsd_file *nf, struct list_head *dispose) - __must_hold(RCU) { int decrement = 1; @@ -634,7 +628,7 @@ nfsd_file_cond_queue(struct nfsd_file *nf, struct list_head *dispose) /* If refcount goes to 0, then put on the dispose list */ if (refcount_sub_and_test(decrement, &nf->nf_ref)) { - list_add(&nf->nf_gc, dispose); + list_add(&nf->nf_lru, dispose); trace_nfsd_file_closing(nf); } } @@ -859,7 +853,10 @@ nfsd_alloc_fcache_disposal(void) return NULL; spin_lock_init(&l->lock); INIT_DELAYED_WORK(&l->filecache_laundrette, nfsd_file_gc_worker); + INIT_LIST_HEAD(&l->recent); + INIT_LIST_HEAD(&l->older); INIT_LIST_HEAD(&l->freeme); + l->num_gc = 0; l->file_shrinker = shrinker_alloc(0, "nfsd-filecache"); if (!l->file_shrinker) { pr_err("nfsd: failed to allocate nfsd_file_shrinker\n"); @@ -871,15 +868,6 @@ nfsd_alloc_fcache_disposal(void) l->file_shrinker->seeks = 1; l->file_shrinker->private_data = l; - /* if file_lru is not zeroed it can trigger a bug: ->key is the problem */ - memset(&l->file_lru, 0, sizeof(l->file_lru)); - if (list_lru_init(&l->file_lru)) { - pr_err("nfsd: failed to init nfsd_file_lru\n"); - shrinker_free(l->file_shrinker); - kfree(l); - return NULL; - } - shrinker_register(l->file_shrinker); return l; } @@ -889,8 +877,12 @@ nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l) { cancel_delayed_work_sync(&l->filecache_laundrette); shrinker_free(l->file_shrinker); - list_lru_destroy(&l->file_lru); - nfsd_file_dispose_list(&l->freeme); + nfsd_file_release_list(&l->recent); + WARN_ON_ONCE(!list_empty(&l->recent)); + nfsd_file_release_list(&l->older); + WARN_ON_ONCE(!list_empty(&l->older)); + nfsd_file_release_list(&l->freeme); + WARN_ON_ONCE(!list_empty(&l->freeme)); kfree(l); } @@ -909,6 +901,8 @@ nfsd_file_cache_start_net(struct net *net) struct nfsd_net *nn = net_generic(net, nfsd_net_id); nn->fcache_disposal = nfsd_alloc_fcache_disposal(); + if (nn->fcache_disposal) + nn->fcache_disposal->nn = nn; return nn->fcache_disposal ? 0 : -ENOMEM; } diff --git a/fs/nfsd/filecache.h b/fs/nfsd/filecache.h index a09a851d510e..02059b6c5e9a 100644 --- a/fs/nfsd/filecache.h +++ b/fs/nfsd/filecache.h @@ -42,15 +42,13 @@ struct nfsd_file { struct net *nf_net; #define NFSD_FILE_HASHED (0) #define NFSD_FILE_PENDING (1) -#define NFSD_FILE_REFERENCED (2) -#define NFSD_FILE_GC (3) +#define NFSD_FILE_GC (2) unsigned long nf_flags; refcount_t nf_ref; unsigned char nf_may; struct nfsd_file_mark *nf_mark; struct list_head nf_lru; - struct list_head nf_gc; struct rcu_head nf_rcu; ktime_t nf_birthtime; }; diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h index ad2c0c432d08..efa683541ed5 100644 --- a/fs/nfsd/trace.h +++ b/fs/nfsd/trace.h @@ -1038,7 +1038,6 @@ DEFINE_CLID_EVENT(confirmed_r); __print_flags(val, "|", \ { 1 << NFSD_FILE_HASHED, "HASHED" }, \ { 1 << NFSD_FILE_PENDING, "PENDING" }, \ - { 1 << NFSD_FILE_REFERENCED, "REFERENCED" }, \ { 1 << NFSD_FILE_GC, "GC" }) DECLARE_EVENT_CLASS(nfsd_file_class, @@ -1314,9 +1313,7 @@ DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_lru_add); DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_lru_add_disposed); DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_lru_del); DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_lru_del_disposed); -DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_in_use); DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_writeback); -DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_referenced); DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_disposed); DECLARE_EVENT_CLASS(nfsd_file_lruwalk_class, @@ -1345,7 +1342,6 @@ DEFINE_EVENT(nfsd_file_lruwalk_class, name, \ ), \ TP_ARGS(removed, remaining)) -DEFINE_NFSD_FILE_LRUWALK_EVENT(nfsd_file_gc_removed); DEFINE_NFSD_FILE_LRUWALK_EVENT(nfsd_file_shrinker_removed); TRACE_EVENT(nfsd_file_close, From patchwork Mon Jan 27 01:20:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950927 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C434E63B9 for ; Mon, 27 Jan 2025 01:23:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941039; cv=none; b=JZrzBn6r9mRHNaQUQeDEtO8c9jl07GvAOeRZI0qCua4P4ra0cNrZXrzwDDIWNBIXZYepbjiYGIKzMFRpg//8aRreLN7Tu/Oi03AWAk9DnbgdvZ3O+CK3D55FPsboVxltdMpxcYmVZRZ7cgx9MrFvyAHEg/xJ8Fb6QWoumUxH8To= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941039; c=relaxed/simple; bh=Q6gMICUKxtn76g9OSvu0xrLxkp9vwwrxxUB4dbDtNNI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=joAO1sTiAseXJVf1KQBFVToo4aZ/Rr9RZJST/4nqg/lsBczpHw+pe70JTlz3aoE1H2HqSG3IS8HJRFDjqxskwoHqECbkZfM+O6n1JOkRC8l8YsYmEOfXslOUAou5oDMqaAQa0fkqMh/O2cJli/wUc9kKzBvzzWHohegqotqIuqM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=b04gqQ09; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=co6VceD+; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=tQMltoNX; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=40aP6bN1; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="b04gqQ09"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="co6VceD+"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="tQMltoNX"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="40aP6bN1" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id F1C852115F; Mon, 27 Jan 2025 01:23:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941036; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FZ1A1OPOfoNZqH8/KSvUxQHtXalDCFAPQqPtj4zIwJk=; b=b04gqQ09/VHqict9pPS+e0gDn54kUVib+oUaCf4lOWVEZBGHtOcauR7RjRlh0kU0YIhJgu bzRiBmzUlMxoF9UtmWnQCYmTRX0mCtlD3eGuDD7l7NHJZBCSG5GHaA+6QSB9uVk2nMT1Fg 4svketSguAd/0tLi1jJd7AsW+PZBEag= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941036; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FZ1A1OPOfoNZqH8/KSvUxQHtXalDCFAPQqPtj4zIwJk=; b=co6VceD+mIpKlaI/gOnF4R+XY9Vt9BZJvUXoMWmVrJrmBGBGjsbsCaY86C8sQDs+Rgzjnu DOZ3q/o7+t93LSDQ== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=tQMltoNX; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=40aP6bN1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941035; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FZ1A1OPOfoNZqH8/KSvUxQHtXalDCFAPQqPtj4zIwJk=; b=tQMltoNXmDgH8hRRrN1bmunuq3+jN89yFWD/8QAeCeIQreXEBiGMm+GTDG919+6DnbCjxf tV598y+osNbbedquMsFRDh1F0uDIjpY68d2L6+z/LV/WqdWLBwR7Wcc2JTkvpZeAUzhgM3 fNn0OpJDLSSe5P9qiHX2C1xdV7RXjzw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941035; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FZ1A1OPOfoNZqH8/KSvUxQHtXalDCFAPQqPtj4zIwJk=; b=40aP6bN1ir5/38s+MKCHWuLbo6k3L4h8XCMwghMTjViirpQ4HvNNgWEHiKS3o1Kdditd6w rdeA69oapoWolcAQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8141613782; Mon, 27 Jan 2025 01:23:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id QeBrDSnglmeEBgAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:23:53 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 5/7] nfsd: filecache: document the arbitrary limit on file-disposes-per-loop Date: Mon, 27 Jan 2025 12:20:36 +1100 Message-ID: <20250127012257.1803314-6-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: F1C852115F X-Spam-Level: X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; DKIM_TRACE(0.00)[suse.de:+] X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Flag: NO Rather than having the bare number "8" use a named constant and explain the tradeoffs that lead to the choice. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/filecache.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 1e90da507152..7264faa57280 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -493,6 +493,21 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose) } } +/* + * Disposing of files can involve non-trivial work, and they + * can appear in batches. So we don't want to try handling them + * all in one thread - if there are lots it would be better to allow + * several nfsd threads to handle them in parallel. + * On average one RPC request can create at most 1 file to be disposed + * so handling one each time around the nfsd loop should keep the list + * under control. However there are often benefits of batching so + * 2 at a time will likely be more efficient than 1. 4 more so. + * We need to choose a number which will often handle all the files, + * but will allow other threads to help when the list gets long. + * The current choice is: + */ +#define NFSD_FILE_DISPOSE_BATCH 8 + /** * nfsd_file_net_dispose - deal with nfsd_files waiting to be disposed. * @nn: nfsd_net in which to find files to be disposed. @@ -511,7 +526,8 @@ void nfsd_file_net_dispose(struct nfsd_net *nn) int i; spin_lock(&l->lock); - for (i = 0; i < 8 && !list_empty(&l->freeme); i++) { + for (i = 0; i < NFSD_FILE_DISPOSE_BATCH && + !list_empty(&l->freeme); i++) { struct nfsd_file *nf = list_first_entry( &l->freeme, struct nfsd_file, nf_lru); From patchwork Mon Jan 27 01:20:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950928 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEDDB748F for ; Mon, 27 Jan 2025 01:24:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941045; cv=none; b=mSqEl4S5YqFVJ37AmTLh4/4Z405dIC424z/tLAqqytEWKxjWntaZC+H1s133tEIt5yOWUVwWj4E0UvOM8PmoSSXJmSrwHDFUG1aGrTonPQIcPtWZxJQC9ScK+GN1nG0DVV0uhp88tnFMqBX/Lg+4XhzO72KHE5NAHJV83JbU9wY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941045; c=relaxed/simple; bh=R7m2owio9kZQqxShzhzbGlntX6WT7Jyy2pMKJ/3lrq0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sqsctwFTaRb+huLHBLgAxeV7SOFeeoeyDBVPXmzOD4EcynzG85iz81SGlc7InaK2Djz/d/8RimhKXL+Hz+KFEis93IDWAy0eec0m7E3/GAQtm1CgVELH1Cc29gspL+EArXNFc7q6tQ7Zl3I44qKo1ex0K7yuzuQwT/OK+jlH4Bw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=1bbUW81J; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=O5TcVYE9; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=1bbUW81J; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=IMX40MhZ; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="1bbUW81J"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="O5TcVYE9"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="1bbUW81J"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="IMX40MhZ" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E1AB621160; Mon, 27 Jan 2025 01:24:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941041; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gTX9N9tDS+afoY64V0+pIMkwbjDy/6zwdtprJN5rJB0=; b=1bbUW81JVYZWud4v8ANKD1CC/janU7Cn+WMf6Z9SnAlp9hmu1xzdHnwFpBFxUYD9JlxUWJ znc5mga6bpxJO24uZuUNoYcKhlXAXFaw3BLcnZG7ucICrDSPrTTXSv+gkFgcnCaOBspVHa D3ifTXIhUDbr+ET6CWaXfEX5PuyKf4w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941042; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gTX9N9tDS+afoY64V0+pIMkwbjDy/6zwdtprJN5rJB0=; b=O5TcVYE9wi0VTe64rkPI0vGK1mTfAzUX/tt5wD5V0TzkuBmjhuxXMg6XuaCHYxlmATToRf az49OQLYt7krwZAA== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=1bbUW81J; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=IMX40MhZ DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941041; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gTX9N9tDS+afoY64V0+pIMkwbjDy/6zwdtprJN5rJB0=; b=1bbUW81JVYZWud4v8ANKD1CC/janU7Cn+WMf6Z9SnAlp9hmu1xzdHnwFpBFxUYD9JlxUWJ znc5mga6bpxJO24uZuUNoYcKhlXAXFaw3BLcnZG7ucICrDSPrTTXSv+gkFgcnCaOBspVHa D3ifTXIhUDbr+ET6CWaXfEX5PuyKf4w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941041; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gTX9N9tDS+afoY64V0+pIMkwbjDy/6zwdtprJN5rJB0=; b=IMX40MhZLxdR3bpW92JPAGX6Ba6k9fzGKEX2mNOj8d7xqJVmfc/AvuEbXgsoshouzV/uxE No3wdoneMVS8tyBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 7548513782; Mon, 27 Jan 2025 01:23:59 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id MIKnCi/glmeqCAAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:23:59 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 6/7] nfsd: filecache: change garbage collection to a timer. Date: Mon, 27 Jan 2025 12:20:37 +1100 Message-ID: <20250127012257.1803314-7-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Rspamd-Queue-Id: E1AB621160 X-Spam-Score: -3.01 X-Rspamd-Action: no action X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:rdns,imap1.dmz-prg2.suse.org:helo]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; DKIM_TRACE(0.00)[suse.de:+] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spam-Flag: NO X-Spam-Level: garbage collection never sleeps and no longer walks a list so it runs quickly only requiring a spinlock. This means we don't need to use a workqueue, we can use a simple timer instead. Rather than taking the lock in the timer callback, which would require using _bh locking, simply test a flag and wake an nfsd thread. That thread checks the flag and ages the lists when needed. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/filecache.c | 43 ++++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 7264faa57280..eb95a53f806f 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -67,10 +67,12 @@ struct nfsd_fcache_disposal { struct list_head older; /* haven't been used in last 0-2 seconds */ struct list_head freeme; /* ready to be discarded */ unsigned long num_gc; /* Approximate size of recent plus older */ - struct delayed_work filecache_laundrette; + struct timer_list timer; struct shrinker *file_shrinker; struct nfsd_net *nn; + unsigned long flags; }; +#define NFSD_FCACHE_DO_AGE BIT(0) /* time to age the lists */ static struct kmem_cache *nfsd_file_slab; static struct kmem_cache *nfsd_file_mark_slab; @@ -115,8 +117,8 @@ static const struct rhashtable_params nfsd_file_rhash_params = { static void nfsd_file_schedule_laundrette(struct nfsd_fcache_disposal *l) { - queue_delayed_work(system_unbound_wq, &l->filecache_laundrette, - NFSD_LAUNDRETTE_DELAY); + if (!timer_pending(&l->timer)) + mod_timer(&l->timer, jiffies + NFSD_LAUNDRETTE_DELAY); } static void @@ -521,6 +523,19 @@ void nfsd_file_net_dispose(struct nfsd_net *nn) { struct nfsd_fcache_disposal *l = nn->fcache_disposal; + if (test_and_clear_bit(NFSD_FCACHE_DO_AGE, &l->flags)) { + spin_lock(&l->lock); + list_splice_init(&l->older, &l->freeme); + list_splice_init(&l->recent, &l->older); + /* We don't know how many were moved to 'freeme' and don't want + * to waste time counting - guess a half. This is only used + * for the shrinker which doesn't need complete precision. + */ + l->num_gc /= 2; + if (!list_empty(&l->older) || !list_empty(&l->recent)) + mod_timer(&l->timer, jiffies + NFSD_LAUNDRETTE_DELAY); + spin_unlock(&l->lock); + } if (!list_empty(&l->freeme)) { LIST_HEAD(dispose); int i; @@ -557,23 +572,13 @@ void nfsd_file_net_dispose(struct nfsd_net *nn) } static void -nfsd_file_gc_worker(struct work_struct *work) +nfsd_file_gc_worker(struct timer_list *t) { struct nfsd_fcache_disposal *l = container_of( - work, struct nfsd_fcache_disposal, filecache_laundrette.work); + t, struct nfsd_fcache_disposal, timer); - spin_lock(&l->lock); - list_splice_init(&l->older, &l->freeme); - list_splice_init(&l->recent, &l->older); - /* We don't know how many were moved to 'freeme' and don't want - * to waste time counting - guess a half. - */ - l->num_gc /= 2; - if (!list_empty(&l->freeme)) - svc_wake_up(l->nn->nfsd_serv); - if (!list_empty(&l->older) || !list_empty(&l->recent)) - nfsd_file_schedule_laundrette(l); - spin_unlock(&l->lock); + set_bit(NFSD_FCACHE_DO_AGE, &l->flags); + svc_wake_up(l->nn->nfsd_serv); } static unsigned long @@ -868,7 +873,7 @@ nfsd_alloc_fcache_disposal(void) if (!l) return NULL; spin_lock_init(&l->lock); - INIT_DELAYED_WORK(&l->filecache_laundrette, nfsd_file_gc_worker); + timer_setup(&l->timer, nfsd_file_gc_worker, 0); INIT_LIST_HEAD(&l->recent); INIT_LIST_HEAD(&l->older); INIT_LIST_HEAD(&l->freeme); @@ -891,7 +896,7 @@ nfsd_alloc_fcache_disposal(void) static void nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l) { - cancel_delayed_work_sync(&l->filecache_laundrette); + del_timer_sync(&l->timer); shrinker_free(l->file_shrinker); nfsd_file_release_list(&l->recent); WARN_ON_ONCE(!list_empty(&l->recent)); From patchwork Mon Jan 27 01:20:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 13950929 X-Patchwork-Delegate: cel@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A381A846F for ; Mon, 27 Jan 2025 01:24:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941055; cv=none; b=hgEA/mhXR8zYwTGukpbt7wpOJ3IW5t6K+h/qJfXM3/2DKAYt2tVtspHvFnVJBXhgmLTNOqjjtApsJ5jCeYmNXFqsAWiFIrbx7R9y0UGPKNnpNHr3cL8BeAUAzU6goKhfswh2kRslsd8E89tA37f1fiRHN/I7GADKzQvRoNXZ/e0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737941055; c=relaxed/simple; bh=hpVFIxNkTwoADFO3VTp3uOPYEl9E8pocDx3JeqjBxJ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mo1kfhxwMFIlaih2Vu8OsHCmyozz+93lSrag7rqw/bCOfXukiJtglWT7f31/5Uhj7mzZdD3Vu4xucb/7JbYilOoehDLYwnJSNvwBVmL0Q+q0tviHuCT/s6PJt3GcCZB19J3GUIZr7l2a0vFbWdcybs1gYKM1fTazfheg1Upx/Hg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=uI3VAnVu; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=AmJceLAo; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=uI3VAnVu; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=AmJceLAo; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="uI3VAnVu"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="AmJceLAo"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="uI3VAnVu"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="AmJceLAo" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id CFF202115C; Mon, 27 Jan 2025 01:24:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941051; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gyPkhG69o++iLuxVpbL32v/xCxj2DJj65/hX9+DWokk=; b=uI3VAnVuIFizI/IAcZ4BXvrp8BXUKCjOLZvo6UeN0bX+wPzd2TiQkdblghfZ6Py1bwXRq+ KsaoemFM6ytuch4vIsso5RpJaZvn9qqLCsy7LhYMJ6GoTrzCr6wTk09BmDNqasWz4Cx0Oe dhbFgfzYtONjI2t5ETeIUhIAc0+IKQs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941051; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gyPkhG69o++iLuxVpbL32v/xCxj2DJj65/hX9+DWokk=; b=AmJceLAow0OIAQPzKLolIlIl7+kqHzDlbUWWb/YHPQiEY2zD3WOsH0Vw7Y+TJKe+299Faj 2EHGFoLFhGYljIDw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737941051; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gyPkhG69o++iLuxVpbL32v/xCxj2DJj65/hX9+DWokk=; b=uI3VAnVuIFizI/IAcZ4BXvrp8BXUKCjOLZvo6UeN0bX+wPzd2TiQkdblghfZ6Py1bwXRq+ KsaoemFM6ytuch4vIsso5RpJaZvn9qqLCsy7LhYMJ6GoTrzCr6wTk09BmDNqasWz4Cx0Oe dhbFgfzYtONjI2t5ETeIUhIAc0+IKQs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737941051; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gyPkhG69o++iLuxVpbL32v/xCxj2DJj65/hX9+DWokk=; b=AmJceLAow0OIAQPzKLolIlIl7+kqHzDlbUWWb/YHPQiEY2zD3WOsH0Vw7Y+TJKe+299Faj 2EHGFoLFhGYljIDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 62EE413782; Mon, 27 Jan 2025 01:24:09 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id ez8iBjnglmegCwAAD6G6ig (envelope-from ); Mon, 27 Jan 2025 01:24:09 +0000 From: NeilBrown To: Chuck Lever , Jeff Layton Cc: linux-nfs@vger.kernel.org, Olga Kornievskaia , Dai Ngo , Tom Talpey , Dave Chinner Subject: [PATCH 7/7] nfsd: filecache: give disposal lock a unique class name. Date: Mon, 27 Jan 2025 12:20:38 +1100 Message-ID: <20250127012257.1803314-8-neilb@suse.de> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250127012257.1803314-1-neilb@suse.de> References: <20250127012257.1803314-1-neilb@suse.de> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_RATELIMIT(0.00)[from(RLewrxuus8mos16izbn)]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Level: There are at least three locks in the kernel which are initialised as spin_Lock_init(&l->lock); This makes them hard to differential in /proc/lock_stat. For the lock in nfsd/filecache.c introduce a variable with a more descriptve name so we can: spin_lock_init(&nfsd_fcache_disposal->lock); and easily identify this in /proc/lock_stat. Signed-off-by: NeilBrown Reviewed-by: Jeff Layton --- fs/nfsd/filecache.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index eb95a53f806f..af95bc381753 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -867,12 +867,13 @@ __nfsd_file_cache_purge(struct net *net) static struct nfsd_fcache_disposal * nfsd_alloc_fcache_disposal(void) { - struct nfsd_fcache_disposal *l; + struct nfsd_fcache_disposal *l, *nfsd_fcache_disposal; l = kmalloc(sizeof(*l), GFP_KERNEL); if (!l) return NULL; - spin_lock_init(&l->lock); + nfsd_fcache_disposal = l; + spin_lock_init(&nfsd_fcache_disposal->lock); timer_setup(&l->timer, nfsd_file_gc_worker, 0); INIT_LIST_HEAD(&l->recent); INIT_LIST_HEAD(&l->older);