From patchwork Wed Aug 7 10:31:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756114 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A275A1DD398; Wed, 7 Aug 2024 10:31:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; cv=none; b=Dqi7p6sh+Q+vAbYcAbjxXrTajDnIb+VLvnSxjeVeSS+P42zJsUrbDfGXgxCOufD+qEw6srwXSHB27ETtOBchIKH8KGRU4SLVSGny+vTaBJuGaa7Uz6iNiKCzq3TOf9q8TswmDrdfywcrWWZjA+OFUtlGdeeh537BE08P54xIAIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; c=relaxed/simple; bh=1SjpLUxInJUsm5hyKUAPgUIElsshMJHymB63wFaYeyM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=f2Sg+GApnCrOkB661yNjiK0PKgRfrZ+GEf+0s1QzZ5t1SypOESf4rLL+ToJW8Q6vdBQ0/djm8uzENmFYP13htjPc0JZRNlNntBYtZk1ScPiERDYTaWdEDkRXd0EFtKyf5f5+eVrwU1uvMl9v7YiqEklKpfiUtVRqrSkO6WThkKk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=H+6QUiPp; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=eEuKmuDB; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=fOdhnuMq; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=1OHbFK1E; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="H+6QUiPp"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="eEuKmuDB"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="fOdhnuMq"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="1OHbFK1E" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C1E7121B34; Wed, 7 Aug 2024 10:31:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZCscfrhLRUfMCCWf1VM+44Ik8h8Xr8RVOlGx1djGvg=; b=H+6QUiPpPhQpEm6CkhIQP14Nxt2nm39M0qa/fJzn4/8NcdPkEUbNCp5ApLUluDayMmbxsI Ogz3u05H/9SvrmD4bmitU98bMZCZWVObe1YdKZX1LpMN91aWxqVT48m9urAnIRc/CD5RO8 1R58YVokSRFEo2xz5mpXUZIoyjK1GL0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZCscfrhLRUfMCCWf1VM+44Ik8h8Xr8RVOlGx1djGvg=; b=eEuKmuDBbPGuw/CzZG72wgUoZmxzsyiQjilyImHwCV8XORHQc5MhvIPrYkRCPLPWl+3ktT C3xDmnek8T5zSoDw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026693; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZCscfrhLRUfMCCWf1VM+44Ik8h8Xr8RVOlGx1djGvg=; b=fOdhnuMq7Epl6L70RKYzdyPXf7JSKSyqw8P8qa/e+I+vTB9ms9RMNNq8iguFye8Z6GQNMy KX61+KuWN4iKeQlnMUql60dxMyVrbLMIRCvCD5QyE/TsOEkeCgSI4F0BWo6MX0Tzl0j3RU hCGB5k4ZNsoMAoVbDGPKVBLFg6XktVc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026693; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZCscfrhLRUfMCCWf1VM+44Ik8h8Xr8RVOlGx1djGvg=; b=1OHbFK1EKbGQNsgpWy4MW+DURb63K9CBOD0BAlrZCWjH1MoaX1crKULopCP6ajBRGJfEZs CfKnGAJqy5lLSYBQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 96CED13B03; Wed, 7 Aug 2024 10:31:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id ePOIJAVNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:33 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:14 +0200 Subject: [PATCH v2 1/7] mm, slab: dissolve shutdown_cache() into its caller Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-1-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; R_RATELIMIT(0.00)[to_ip_from(RLtsk3gtac773whqka7ht6mdi4)] X-Spam-Score: -2.80 X-Spam-Flag: NO X-Spam-Level: There's only one caller of shutdown_cache() so move the code into it. Preparatory patch for further changes, no functional change. Signed-off-by: Vlastimil Babka Reviewed-by: Jann Horn --- mm/slab_common.c | 40 ++++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 40b582a014b8..b76d65d7fe33 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -540,27 +540,6 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) } } -static int shutdown_cache(struct kmem_cache *s) -{ - /* free asan quarantined objects */ - kasan_cache_shutdown(s); - - if (__kmem_cache_shutdown(s) != 0) - return -EBUSY; - - list_del(&s->list); - - if (s->flags & SLAB_TYPESAFE_BY_RCU) { - list_add_tail(&s->list, &slab_caches_to_rcu_destroy); - schedule_work(&slab_caches_to_rcu_destroy_work); - } else { - kfence_shutdown_cache(s); - debugfs_slab_release(s); - } - - return 0; -} - void slab_kmem_cache_release(struct kmem_cache *s) { __kmem_cache_release(s); @@ -585,9 +564,26 @@ void kmem_cache_destroy(struct kmem_cache *s) if (s->refcount) goto out_unlock; - err = shutdown_cache(s); + /* free asan quarantined objects */ + kasan_cache_shutdown(s); + + err = __kmem_cache_shutdown(s); WARN(err, "%s %s: Slab cache still has objects when called from %pS", __func__, s->name, (void *)_RET_IP_); + + if (err) + goto out_unlock; + + list_del(&s->list); + + if (rcu_set) { + list_add_tail(&s->list, &slab_caches_to_rcu_destroy); + schedule_work(&slab_caches_to_rcu_destroy_work); + } else { + kfence_shutdown_cache(s); + debugfs_slab_release(s); + } + out_unlock: mutex_unlock(&slab_mutex); cpus_read_unlock(); From patchwork Wed Aug 7 10:31:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756116 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C9041D364C; Wed, 7 Aug 2024 10:31:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; cv=none; b=WZVWkM4pHTNZZtc5l8heeZYggnMsjT5ll9+JBW4CsjhTY3xJvCy0wg9jM71CYwozNDOcbV1f+VsKqzgq3D/+4qg9a1etg6JqCca50cN1eaFCRN9QsVDEHSXHyiQi2c87OWNY9k4s4uOmfy0vgoKlN4/Oc4eOjdmQDyLDIzCHKgk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; c=relaxed/simple; bh=IKzNm6IhIvGRZL206iBD5aH9mRPWOtJEttlogjtjEAU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=XXdge/Q/F4NqMM/qlYluJPi1Cr3OtpRuZncfaA3LmC96L8zfodKlT5ANH/byi60HyNkG0M7BLjEKrHwcT7e0ynLJjnBcJ+HMH/5hoCwINrnPq2d9c5jxCkjTun7i1uPzIaHScJnFkIMhjjysVhQR1TZKvI/79WLyn1AcgD1KE0U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=YwEVT8Zy; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=nRFhcHRO; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=UlFErIrL; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=ljOrAaOB; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="YwEVT8Zy"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="nRFhcHRO"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="UlFErIrL"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="ljOrAaOB" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E65B621B51; Wed, 7 Aug 2024 10:31:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3R4VEs0s4uyTtqaeQIxLvgBnoD49RLjNoURIedFN5O4=; b=YwEVT8ZywArwGZA+nX+2r6ZqSUHyB34SZZ+gcNuyQbXQNjl7UorcCTTahscphEP4uM0uRl 0sq+avJB8JMUV0MKExx6ltEUFA8zrnUKKme7E+G5hUk6yLUXfaa9hFzX7fbBCCJTehhJ1Q NHHX0xxgKe6yBpJQGGVjA6srVHlQ2w0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3R4VEs0s4uyTtqaeQIxLvgBnoD49RLjNoURIedFN5O4=; b=nRFhcHROYA7mPEx9T9wJ3K/ieeUPGDvJc0Ur2giQhqT6GxIAjw4g9c+Cxl2BIQxqdDJCxW bc7vpKTTnDX3OcDg== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UlFErIrL; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=ljOrAaOB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026693; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3R4VEs0s4uyTtqaeQIxLvgBnoD49RLjNoURIedFN5O4=; b=UlFErIrLpnllen+2sGfZKyfqLpuG/tXTzEN7tHbxb13lhuPvv1jnjXxKgzGK/2QZvhAtyp Ozf+yqRz4R/g2sq7kLK6V4lguB4Gd4FSGiXinAgl2nxUD1UxGD/iazj78yt48yJP+7d5VS iw/5YJggf7GKVaBqXozLjSX+19GkJTo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026693; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3R4VEs0s4uyTtqaeQIxLvgBnoD49RLjNoURIedFN5O4=; b=ljOrAaOBHnj/XoxD6FsSLvTyJgxcUxrXrtisHPcjUxCiZpfBZmOM/o33djb00n3ZY4EbCj 7j07+ZnY+3912vCg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id BC82F13B05; Wed, 7 Aug 2024 10:31:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id ELi5LQVNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:33 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:15 +0200 Subject: [PATCH v2 2/7] mm, slab: unlink slabinfo, sysfs and debugfs immediately Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-2-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spam-Level: X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[]; DKIM_TRACE(0.00)[suse.cz:+]; R_RATELIMIT(0.00)[to_ip_from(RLsm9p66qmnckghmjmpccdnq6s)]; TO_DN_SOME(0.00)[] X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -3.01 X-Rspamd-Queue-Id: E65B621B51 kmem_cache_destroy() includes removing the associated sysfs and debugfs directories, and the cache from the list of caches that appears in /proc/slabinfo. Currently this might not happen immediately when: - the cache is SLAB_TYPESAFE_BY_RCU and the cleanup is delayed, including the directores removal - __kmem_cache_shutdown() fails due to outstanding objects - the directories remain indefinitely When a cache is recreated with the same name, such as due to module unload followed by a load, the directories will fail to be recreated for the new instance of the cache due to the old directories being present. The cache will also appear twice in /proc/slabinfo. While we want to convert the SLAB_TYPESAFE_BY_RCU cleanup to be synchronous again, the second point remains. So let's fix this first and have the directories and slabinfo removed immediately in kmem_cache_destroy() and regardless of __kmem_cache_shutdown() success. This should not make debugging harder if __kmem_cache_shutdown() fails, because a detailed report of outstanding objects is printed into dmesg already due to the failure. Also simplify kmem_cache_release() sysfs handling by using __is_defined(SLAB_SUPPORTS_SYSFS). Note the resulting code in kmem_cache_destroy() is a bit ugly but will be further simplified - this is in order to make small bisectable steps. Signed-off-by: Vlastimil Babka Reviewed-by: Jann Horn --- mm/slab_common.c | 57 ++++++++++++++++++++++++++------------------------------ 1 file changed, 26 insertions(+), 31 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index b76d65d7fe33..db61df3b4282 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -484,31 +484,19 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, } EXPORT_SYMBOL(kmem_buckets_create); -#ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called * once or there will be a use-after-free problem. The actual deletion * and release of the kobject does not need slab_mutex or cpu_hotplug_lock * protection. So they are now done without holding those locks. - * - * Note that there will be a slight delay in the deletion of sysfs files - * if kmem_cache_release() is called indrectly from a work function. */ static void kmem_cache_release(struct kmem_cache *s) { - if (slab_state >= FULL) { - sysfs_slab_unlink(s); + if (__is_defined(SLAB_SUPPORTS_SYSFS) && slab_state >= FULL) sysfs_slab_release(s); - } else { + else slab_kmem_cache_release(s); - } } -#else -static void kmem_cache_release(struct kmem_cache *s) -{ - slab_kmem_cache_release(s); -} -#endif static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) { @@ -534,7 +522,6 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) rcu_barrier(); list_for_each_entry_safe(s, s2, &to_destroy, list) { - debugfs_slab_release(s); kfence_shutdown_cache(s); kmem_cache_release(s); } @@ -549,8 +536,8 @@ void slab_kmem_cache_release(struct kmem_cache *s) void kmem_cache_destroy(struct kmem_cache *s) { - int err = -EBUSY; bool rcu_set; + int err; if (unlikely(!s) || !kasan_check_byte(s)) return; @@ -558,11 +545,14 @@ void kmem_cache_destroy(struct kmem_cache *s) cpus_read_lock(); mutex_lock(&slab_mutex); - rcu_set = s->flags & SLAB_TYPESAFE_BY_RCU; - s->refcount--; - if (s->refcount) - goto out_unlock; + if (s->refcount) { + mutex_unlock(&slab_mutex); + cpus_read_unlock(); + return; + } + + rcu_set = s->flags & SLAB_TYPESAFE_BY_RCU; /* free asan quarantined objects */ kasan_cache_shutdown(s); @@ -571,24 +561,29 @@ void kmem_cache_destroy(struct kmem_cache *s) WARN(err, "%s %s: Slab cache still has objects when called from %pS", __func__, s->name, (void *)_RET_IP_); - if (err) - goto out_unlock; - list_del(&s->list); - if (rcu_set) { - list_add_tail(&s->list, &slab_caches_to_rcu_destroy); - schedule_work(&slab_caches_to_rcu_destroy_work); - } else { + if (!err && !rcu_set) kfence_shutdown_cache(s); - debugfs_slab_release(s); - } -out_unlock: mutex_unlock(&slab_mutex); cpus_read_unlock(); - if (!err && !rcu_set) + + if (slab_state >= FULL) + sysfs_slab_unlink(s); + debugfs_slab_release(s); + + if (err) + return; + + if (rcu_set) { + mutex_lock(&slab_mutex); + list_add_tail(&s->list, &slab_caches_to_rcu_destroy); + schedule_work(&slab_caches_to_rcu_destroy_work); + mutex_unlock(&slab_mutex); + } else { kmem_cache_release(s); + } } EXPORT_SYMBOL(kmem_cache_destroy); From patchwork Wed Aug 7 10:31:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756113 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F67A1D365C; Wed, 7 Aug 2024 10:31:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; cv=none; b=F+f6ntAOmecBulyG8bQAaqZ0NLo1MvfUqdquF9IVChKCLJbuZUDHQNYGWwFSPWr23r87Fo2O/83IFZnH2aCQjzGscw/XFuo2PKFHrakyyA+PhoYKskpSUZ4WdXL4rOVnzv+nqNtH7t33DpAD9eFi0j+JSdGrpx1Qf/SJGkdpHvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; c=relaxed/simple; bh=evxzCTWCDQfcpF6frzqbhgzHo8yDiLX25dyXYcfc6kU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=giYg6xF9xE+wbkgxei0KA9vE/yyFj1JHRGEnsz4YGfuMpgqcyiDjWFgczJn3zBrksOHQXhwS2hi5Qy4+JlRaMTA6JEas0rrXuvByWGFgZLf4P5+HkqPT1gM562sMGoeyZBXMs02B1gtZiKmgBC80Nxb5Cq0L8awH+obkTXYHKpc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=wJzF/g6n; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=XJKck9Ri; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=wJzF/g6n; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=XJKck9Ri; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="wJzF/g6n"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="XJKck9Ri"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="wJzF/g6n"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="XJKck9Ri" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 1D32E21CF0; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Upn+l7aIdIa1NYqSvgAPp2lNr0Z+34OGHoqNziNBnOM=; b=wJzF/g6nBkUp2U1+03yNOdlGyz2ExiWnsH3D/0Wwg/RXxLwUipTPUlBysih9eTFbPT95HJ KnlcMXhUgiqjP73r6WibjR/7FEVkpwZXxuTlvlrwUfJ0/M8vHk9ErP/rWQOgmf/WRzOyDB 8bIw7+Qj4Lac6rQngcwV16wpIcnjSdg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Upn+l7aIdIa1NYqSvgAPp2lNr0Z+34OGHoqNziNBnOM=; b=XJKck9RiujgMYrmJ/eDJpmgVR9itnsNx6AAUnOBb58PCpTCwKezxC2ANlomHFHr15pB6uj THCMdGLt2CwDHBBA== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Upn+l7aIdIa1NYqSvgAPp2lNr0Z+34OGHoqNziNBnOM=; b=wJzF/g6nBkUp2U1+03yNOdlGyz2ExiWnsH3D/0Wwg/RXxLwUipTPUlBysih9eTFbPT95HJ KnlcMXhUgiqjP73r6WibjR/7FEVkpwZXxuTlvlrwUfJ0/M8vHk9ErP/rWQOgmf/WRzOyDB 8bIw7+Qj4Lac6rQngcwV16wpIcnjSdg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Upn+l7aIdIa1NYqSvgAPp2lNr0Z+34OGHoqNziNBnOM=; b=XJKck9RiujgMYrmJ/eDJpmgVR9itnsNx6AAUnOBb58PCpTCwKezxC2ANlomHFHr15pB6uj THCMdGLt2CwDHBBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id E333F13297; Wed, 7 Aug 2024 10:31:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id sNEoNwVNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:33 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:16 +0200 Subject: [PATCH v2 3/7] mm, slab: move kfence_shutdown_cache() outside slab_mutex Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-3-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spam-Level: X-Spamd-Result: default: False [0.20 / 50.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TAGGED_RCPT(0.00)[]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; MID_RHS_MATCH_FROM(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[] X-Spam-Flag: NO X-Spam-Score: 0.20 kfence_shutdown_cache() is called under slab_mutex when the cache is destroyed synchronously, and outside slab_mutex during the delayed destruction of SLAB_TYPESAFE_BY_RCU caches. It seems it should always be safe to call it outside of slab_mutex so we can just move the call to kmem_cache_release(), which is called outside. Signed-off-by: Vlastimil Babka Reviewed-by: Jann Horn --- mm/slab_common.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index db61df3b4282..a079b8540334 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -492,6 +492,7 @@ EXPORT_SYMBOL(kmem_buckets_create); */ static void kmem_cache_release(struct kmem_cache *s) { + kfence_shutdown_cache(s); if (__is_defined(SLAB_SUPPORTS_SYSFS) && slab_state >= FULL) sysfs_slab_release(s); else @@ -521,10 +522,8 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) rcu_barrier(); - list_for_each_entry_safe(s, s2, &to_destroy, list) { - kfence_shutdown_cache(s); + list_for_each_entry_safe(s, s2, &to_destroy, list) kmem_cache_release(s); - } } void slab_kmem_cache_release(struct kmem_cache *s) @@ -563,9 +562,6 @@ void kmem_cache_destroy(struct kmem_cache *s) list_del(&s->list); - if (!err && !rcu_set) - kfence_shutdown_cache(s); - mutex_unlock(&slab_mutex); cpus_read_unlock(); From patchwork Wed Aug 7 10:31:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756115 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A108F1DD391; Wed, 7 Aug 2024 10:31:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; cv=none; b=cV6x2+gDJVP1AmhR2YFao+NYg+lJwuNOCt68IWyZx3CclwzKcpTz0IzkK/7Y0WiVAYt5G+bGrowQyHRbE6yjHG5CsA/b72SWk34tTe8NHnRHAf294MGE+1LuhjRpQBdBtXCneVprct6QpFvYltcy72hFusV/OG1Q29b0kOrNehM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026704; c=relaxed/simple; bh=SXToQPwiarLAnhHwZ8FG3NHd6oGUZrTgfGJR3Xw2jdw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=td6d2AuBRxOF5eiUlLnEJ3GM3hUAaIlxJLRFI2JJn0zhJ9Sm2zPqO2U2uw0E6zLMUxn4loWAPHr5aPn+PKWMZR3NhGfv0jkEE6xAiPWeoRbVX8t7OOQzzfi46omrsGHq7Ojxshlk1NhAcfot0fziw/CjSWtRm+9DG+ZtaazCDi8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=sXT1azEr; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=hd64V33C; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=sXT1azEr; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=hd64V33C; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="sXT1azEr"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="hd64V33C"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="sXT1azEr"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="hd64V33C" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 4316B21CF3; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h1qL27IVdY8UFaeKEjvymEDe/A+3RtrkI79ewzI2zdg=; b=sXT1azErC5SItYAr7d2Iuk/s5nav7pHXgOhxwwfhZpF9uISaP+xjVtcjuKPwr5m81SNCO2 Z3/lsR4fIfLGYKPUFliujQZjaxSIk1BjhT/VIsyrovU8hwbOOAzt8ufqKbTQzbS/LU+n4W wGYJeSleY7ZJi4sQL0CMZq9JNxNL9Yg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h1qL27IVdY8UFaeKEjvymEDe/A+3RtrkI79ewzI2zdg=; b=hd64V33CKjwjSliPi0q8xdy7QfWL/ZqIaEGlov4D0Oi4o/JMxrydeuwFKww5/WHL3vuic3 wshtwX3dxGS1QuBA== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h1qL27IVdY8UFaeKEjvymEDe/A+3RtrkI79ewzI2zdg=; b=sXT1azErC5SItYAr7d2Iuk/s5nav7pHXgOhxwwfhZpF9uISaP+xjVtcjuKPwr5m81SNCO2 Z3/lsR4fIfLGYKPUFliujQZjaxSIk1BjhT/VIsyrovU8hwbOOAzt8ufqKbTQzbS/LU+n4W wGYJeSleY7ZJi4sQL0CMZq9JNxNL9Yg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h1qL27IVdY8UFaeKEjvymEDe/A+3RtrkI79ewzI2zdg=; b=hd64V33CKjwjSliPi0q8xdy7QfWL/ZqIaEGlov4D0Oi4o/JMxrydeuwFKww5/WHL3vuic3 wshtwX3dxGS1QuBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1631513B06; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id uH8iBQZNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:34 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:17 +0200 Subject: [PATCH v2 4/7] mm, slab: reintroduce rcu_barrier() into kmem_cache_destroy() Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-4-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; R_RATELIMIT(0.00)[to_ip_from(RLtsk3gtac773whqka7ht6mdi4)] X-Spam-Flag: NO X-Spam-Score: -2.80 There used to be a rcu_barrier() for SLAB_TYPESAFE_BY_RCU caches in kmem_cache_destroy() until commit 657dc2f97220 ("slab: remove synchronous rcu_barrier() call in memcg cache release path") moved it to an asynchronous work that finishes the destroying of such caches. The motivation for that commit was the MEMCG_KMEM integration that at the time created and removed clones of the global slab caches together with their cgroups, and blocking cgroups removal was unwelcome. The implementation later changed to per-object memcg tracking using a single cache, so there should be no more need for a fast non-blocking kmem_cache_destroy(), which is typically only done when a module is unloaded etc. Going back to synchronous barrier has the following advantages: - simpler implementation - it's easier to test the result of kmem_cache_destroy() in a kunit test Thus effectively revert commit 657dc2f97220. It is not a 1:1 revert as the code has changed since. The main part is that kmem_cache_release(s) is always called from kmem_cache_destroy(), but for SLAB_TYPESAFE_BY_RCU caches there's a rcu_barrier() first. Suggested-by: Mateusz Guzik Signed-off-by: Vlastimil Babka Reviewed-by: Jann Horn --- mm/slab_common.c | 47 ++++------------------------------------------- 1 file changed, 4 insertions(+), 43 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index a079b8540334..c40227d5fa07 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -40,11 +40,6 @@ LIST_HEAD(slab_caches); DEFINE_MUTEX(slab_mutex); struct kmem_cache *kmem_cache; -static LIST_HEAD(slab_caches_to_rcu_destroy); -static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work); -static DECLARE_WORK(slab_caches_to_rcu_destroy_work, - slab_caches_to_rcu_destroy_workfn); - /* * Set of flags that will prevent slab merging */ @@ -499,33 +494,6 @@ static void kmem_cache_release(struct kmem_cache *s) slab_kmem_cache_release(s); } -static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) -{ - LIST_HEAD(to_destroy); - struct kmem_cache *s, *s2; - - /* - * On destruction, SLAB_TYPESAFE_BY_RCU kmem_caches are put on the - * @slab_caches_to_rcu_destroy list. The slab pages are freed - * through RCU and the associated kmem_cache are dereferenced - * while freeing the pages, so the kmem_caches should be freed only - * after the pending RCU operations are finished. As rcu_barrier() - * is a pretty slow operation, we batch all pending destructions - * asynchronously. - */ - mutex_lock(&slab_mutex); - list_splice_init(&slab_caches_to_rcu_destroy, &to_destroy); - mutex_unlock(&slab_mutex); - - if (list_empty(&to_destroy)) - return; - - rcu_barrier(); - - list_for_each_entry_safe(s, s2, &to_destroy, list) - kmem_cache_release(s); -} - void slab_kmem_cache_release(struct kmem_cache *s) { __kmem_cache_release(s); @@ -535,7 +503,6 @@ void slab_kmem_cache_release(struct kmem_cache *s) void kmem_cache_destroy(struct kmem_cache *s) { - bool rcu_set; int err; if (unlikely(!s) || !kasan_check_byte(s)) @@ -551,8 +518,6 @@ void kmem_cache_destroy(struct kmem_cache *s) return; } - rcu_set = s->flags & SLAB_TYPESAFE_BY_RCU; - /* free asan quarantined objects */ kasan_cache_shutdown(s); @@ -572,14 +537,10 @@ void kmem_cache_destroy(struct kmem_cache *s) if (err) return; - if (rcu_set) { - mutex_lock(&slab_mutex); - list_add_tail(&s->list, &slab_caches_to_rcu_destroy); - schedule_work(&slab_caches_to_rcu_destroy_work); - mutex_unlock(&slab_mutex); - } else { - kmem_cache_release(s); - } + if (s->flags & SLAB_TYPESAFE_BY_RCU) + rcu_barrier(); + + kmem_cache_release(s); } EXPORT_SYMBOL(kmem_cache_destroy); From patchwork Wed Aug 7 10:31:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756118 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 797D91DF680; Wed, 7 Aug 2024 10:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026707; cv=none; b=Xrs6/j3nzUZU24kYCNVsZ7ZlG9MZMqNjBMW7mQvJOrUam/WpLm/tCJOj59mR1p1ELJWeNrfuHcsayGeO6HFEl5yGvG909oEmxyHf1W7sZTcDRl/nrBfcMq29dU6HOAw95bcFcKXi6Ft9KVyj7YcvzcYgBE5ZTC/CZPfvhocvdvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026707; c=relaxed/simple; bh=qqSyR3F36zg6ItnZnkJHnykEHqCUkuxH66SdYCyKjmM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=td4+7ei5JP0wYca84cF9p0YDI+TfUjYAxt/MF5rTm+n4AZpJU4gsbdI9rVl5GNOrSRZYqmqgj72kzRdttDDX6/3+MlmwvMdttPN8FhH+rlfr9MXkOTjDAV+NpfrJPviwqutIbmbEcXqKOzQoTPMumqCVv9erlZqBNd+4IeK4D1o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=jpwuLCpP; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=FR81OezO; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=jpwuLCpP; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=FR81OezO; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="jpwuLCpP"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="FR81OezO"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="jpwuLCpP"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="FR81OezO" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6A06C21CF4; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j2GOiA9T7Jb2ihHQdRICUWrhL/KNdHwLnom7wJXeQZ0=; b=jpwuLCpPuvSyiqyVN47EiRV73g9FPVh5mrTmQ+obDDW+us1CYub59oQma/q10TbxCfFVXc TGxKtRPpf5FiiLkhBYov+vP8ijkpHtOvsxJaqq/MT3gRodsm9+vm6fCa+vEhEBOL2k5Qa5 Xv2h7HzZgIDk7h1nE3KCRiNdfdGZ31E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j2GOiA9T7Jb2ihHQdRICUWrhL/KNdHwLnom7wJXeQZ0=; b=FR81OezOYQfW7NBMp5RYhI9AcpsQcoUxO6KI+QsVLzVsKMxv/hh8vLbnigIQJHnif1mmXK QemlZIN1J5GkqrDA== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=jpwuLCpP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=FR81OezO DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j2GOiA9T7Jb2ihHQdRICUWrhL/KNdHwLnom7wJXeQZ0=; b=jpwuLCpPuvSyiqyVN47EiRV73g9FPVh5mrTmQ+obDDW+us1CYub59oQma/q10TbxCfFVXc TGxKtRPpf5FiiLkhBYov+vP8ijkpHtOvsxJaqq/MT3gRodsm9+vm6fCa+vEhEBOL2k5Qa5 Xv2h7HzZgIDk7h1nE3KCRiNdfdGZ31E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j2GOiA9T7Jb2ihHQdRICUWrhL/KNdHwLnom7wJXeQZ0=; b=FR81OezOYQfW7NBMp5RYhI9AcpsQcoUxO6KI+QsVLzVsKMxv/hh8vLbnigIQJHnif1mmXK QemlZIN1J5GkqrDA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 3CBD413B08; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id IOqMDgZNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:34 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:18 +0200 Subject: [PATCH v2 5/7] rcu/kvfree: Add kvfree_rcu_barrier() API Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-5-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spam-Level: X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Flag: NO X-Rspamd-Queue-Id: 6A06C21CF4 X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.cz:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_RATELIMIT(0.00)[to_ip_from(RLsm9p66qmnckghmjmpccdnq6s)]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org From: "Uladzislau Rezki (Sony)" Add a kvfree_rcu_barrier() function. It waits until all in-flight pointers are freed over RCU machinery. It does not wait any GP completion and it is within its right to return immediately if there are no outstanding pointers. This function is useful when there is a need to guarantee that a memory is fully freed before destroying memory caches. For example, during unloading a kernel module. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Vlastimil Babka --- include/linux/rcutiny.h | 5 +++ include/linux/rcutree.h | 1 + kernel/rcu/tree.c | 103 ++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 101 insertions(+), 8 deletions(-) diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index d9ac7b136aea..522123050ff8 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -111,6 +111,11 @@ static inline void __kvfree_call_rcu(struct rcu_head *head, void *ptr) kvfree(ptr); } +static inline void kvfree_rcu_barrier(void) +{ + rcu_barrier(); +} + #ifdef CONFIG_KASAN_GENERIC void kvfree_call_rcu(struct rcu_head *head, void *ptr); #else diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 254244202ea9..58e7db80f3a8 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -35,6 +35,7 @@ static inline void rcu_virt_note_context_switch(void) void synchronize_rcu_expedited(void); void kvfree_call_rcu(struct rcu_head *head, void *ptr); +void kvfree_rcu_barrier(void); void rcu_barrier(void); void rcu_momentary_dyntick_idle(void); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e641cc681901..ebcfed9b570e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3584,18 +3584,15 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp) } /* - * This function is invoked after the KFREE_DRAIN_JIFFIES timeout. + * Return: %true if a work is queued, %false otherwise. */ -static void kfree_rcu_monitor(struct work_struct *work) +static bool +kvfree_rcu_queue_batch(struct kfree_rcu_cpu *krcp) { - struct kfree_rcu_cpu *krcp = container_of(work, - struct kfree_rcu_cpu, monitor_work.work); unsigned long flags; + bool queued = false; int i, j; - // Drain ready for reclaim. - kvfree_rcu_drain_ready(krcp); - raw_spin_lock_irqsave(&krcp->lock, flags); // Attempt to start a new batch. @@ -3634,11 +3631,27 @@ static void kfree_rcu_monitor(struct work_struct *work) // be that the work is in the pending state when // channels have been detached following by each // other. - queue_rcu_work(system_wq, &krwp->rcu_work); + queued = queue_rcu_work(system_wq, &krwp->rcu_work); } } raw_spin_unlock_irqrestore(&krcp->lock, flags); + return queued; +} + +/* + * This function is invoked after the KFREE_DRAIN_JIFFIES timeout. + */ +static void kfree_rcu_monitor(struct work_struct *work) +{ + struct kfree_rcu_cpu *krcp = container_of(work, + struct kfree_rcu_cpu, monitor_work.work); + + // Drain ready for reclaim. + kvfree_rcu_drain_ready(krcp); + + // Queue a batch for a rest. + kvfree_rcu_queue_batch(krcp); // If there is nothing to detach, it means that our job is // successfully done here. In case of having at least one @@ -3859,6 +3872,80 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) } EXPORT_SYMBOL_GPL(kvfree_call_rcu); +/** + * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete. + * + * Note that a single argument of kvfree_rcu() call has a slow path that + * triggers synchronize_rcu() following by freeing a pointer. It is done + * before the return from the function. Therefore for any single-argument + * call that will result in a kfree() to a cache that is to be destroyed + * during module exit, it is developer's responsibility to ensure that all + * such calls have returned before the call to kmem_cache_destroy(). + */ +void kvfree_rcu_barrier(void) +{ + struct kfree_rcu_cpu_work *krwp; + struct kfree_rcu_cpu *krcp; + bool queued; + int i, cpu; + + /* + * Firstly we detach objects and queue them over an RCU-batch + * for all CPUs. Finally queued works are flushed for each CPU. + * + * Please note. If there are outstanding batches for a particular + * CPU, those have to be finished first following by queuing a new. + */ + for_each_possible_cpu(cpu) { + krcp = per_cpu_ptr(&krc, cpu); + + /* + * Check if this CPU has any objects which have been queued for a + * new GP completion. If not(means nothing to detach), we are done + * with it. If any batch is pending/running for this "krcp", below + * per-cpu flush_rcu_work() waits its completion(see last step). + */ + if (!need_offload_krc(krcp)) + continue; + + while (1) { + /* + * If we are not able to queue a new RCU work it means: + * - batches for this CPU are still in flight which should + * be flushed first and then repeat; + * - no objects to detach, because of concurrency. + */ + queued = kvfree_rcu_queue_batch(krcp); + + /* + * Bail out, if there is no need to offload this "krcp" + * anymore. As noted earlier it can run concurrently. + */ + if (queued || !need_offload_krc(krcp)) + break; + + /* There are ongoing batches. */ + for (i = 0; i < KFREE_N_BATCHES; i++) { + krwp = &(krcp->krw_arr[i]); + flush_rcu_work(&krwp->rcu_work); + } + } + } + + /* + * Now we guarantee that all objects are flushed. + */ + for_each_possible_cpu(cpu) { + krcp = per_cpu_ptr(&krc, cpu); + + for (i = 0; i < KFREE_N_BATCHES; i++) { + krwp = &(krcp->krw_arr[i]); + flush_rcu_work(&krwp->rcu_work); + } + } +} +EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); + static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { From patchwork Wed Aug 7 10:31:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756119 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B3AB1E2103; Wed, 7 Aug 2024 10:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026708; cv=none; b=YtDuceL/KlvVKwl/ECUw/19Qyd3QXknDDONZS9xYWtNTsxPqRoLmR+LPACZNJ+QlFv4YtmgwtBidn1fscT1PRF/9M4JPkd965xSzot91e7Zav5tjb5RMgbboRXwztSVbO6OHaE8c/s5Yje4pn33j9mLlCCfovwX4h4jAB7vOkfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026708; c=relaxed/simple; bh=odcOPsl7Np+e6W0q9bc8BGObw3TeQehB272xRz/S2bw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=SHip7bJFc35t22Dh3cVT5pHmUGk3aJIjLkwmuFA1uq4NZI2O530dB/mRLQx/zxR77qu7MkPh6mwljjtu22/Pqg2DyPLMZUEmFgy9b5XZ9CA+ypZgI06xZ3qAjsbPuKovl93pWdXNcV8e6//WG62xubZFRAaFNVOyXR4HFEw/whI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=G9cWwfVr; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=K+AeEX2G; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=G9cWwfVr; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=K+AeEX2G; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="G9cWwfVr"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="K+AeEX2G"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="G9cWwfVr"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="K+AeEX2G" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 98D2D21CF7; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=foIdXKngemBlbTDIsFcIDdiOSQLvnJzh8edzRJqHuXI=; b=G9cWwfVrm6JkmFYpoZ79hf1LRv8wfwCYrhj5x8KhYvHgPmR/Q8EoYbtVH8/cCiskxZYvvd E5icluLF+y0nv2ZVZL45wecJWESrbLKXcAvpFFq/iioLssYkIZYfITiIc/vSSC+Eb9IE1R +5ATK74aL1BUYbZYiE9eWHfEhcPz4xY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=foIdXKngemBlbTDIsFcIDdiOSQLvnJzh8edzRJqHuXI=; b=K+AeEX2G13kP7EZPoVKdShS1QkNC45172PSLvkDCLjgwVmS0cE6LRJjxTbf7bshybHTxvm PYcwtsP7pO28nBCQ== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=G9cWwfVr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=K+AeEX2G DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=foIdXKngemBlbTDIsFcIDdiOSQLvnJzh8edzRJqHuXI=; b=G9cWwfVrm6JkmFYpoZ79hf1LRv8wfwCYrhj5x8KhYvHgPmR/Q8EoYbtVH8/cCiskxZYvvd E5icluLF+y0nv2ZVZL45wecJWESrbLKXcAvpFFq/iioLssYkIZYfITiIc/vSSC+Eb9IE1R +5ATK74aL1BUYbZYiE9eWHfEhcPz4xY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=foIdXKngemBlbTDIsFcIDdiOSQLvnJzh8edzRJqHuXI=; b=K+AeEX2G13kP7EZPoVKdShS1QkNC45172PSLvkDCLjgwVmS0cE6LRJjxTbf7bshybHTxvm PYcwtsP7pO28nBCQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 629DA13B09; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id GD/NFwZNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:34 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:19 +0200 Subject: [PATCH v2 6/7] mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy() Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-6-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spam-Level: X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Flag: NO X-Rspamd-Queue-Id: 98D2D21CF7 X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.cz:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_RATELIMIT(0.00)[to_ip_from(RLsm9p66qmnckghmjmpccdnq6s)]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org We would like to replace call_rcu() users with kfree_rcu() where the existing callback is just a kmem_cache_free(). However this causes issues when the cache can be destroyed (such as due to module unload). Currently such modules should be issuing rcu_barrier() before kmem_cache_destroy() to have their call_rcu() callbacks processed first. This barrier is however not sufficient for kfree_rcu() in flight due to the batching introduced by a35d16905efc ("rcu: Add basic support for kfree_rcu() batching"). This is not a problem for kmalloc caches which are never destroyed, but since removing SLOB, kfree_rcu() is allowed also for any other cache, that might be destroyed. In order not to complicate the API, put the responsibility for handling outstanding kfree_rcu() in kmem_cache_destroy() itself. Use the newly introduced kvfree_rcu_barrier() to wait before destroying the cache. This is similar to how we issue rcu_barrier() for SLAB_TYPESAFE_BY_RCU caches, but has to be done earlier, as the latter only needs to wait for the empty slab pages to finish freeing, and not objects from the slab. Users of call_rcu() with arbitrary callbacks should still issue rcu_barrier() before destroying the cache and unloading the module, as kvfree_rcu_barrier() is not a superset of rcu_barrier() and the callbacks may be invoking module code or performing other actions that are necessary for a successful unload. Signed-off-by: Vlastimil Babka --- mm/slab_common.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/slab_common.c b/mm/slab_common.c index c40227d5fa07..1a2873293f5d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -508,6 +508,9 @@ void kmem_cache_destroy(struct kmem_cache *s) if (unlikely(!s) || !kasan_check_byte(s)) return; + /* in-flight kfree_rcu()'s may include objects from our cache */ + kvfree_rcu_barrier(); + cpus_read_lock(); mutex_lock(&slab_mutex); From patchwork Wed Aug 7 10:31:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13756117 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 798511DF685; Wed, 7 Aug 2024 10:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026707; cv=none; b=m3DtGsdiN4bl1QciXcMx6gXTLS6fNsh5CXXFnOX0Tgkex2sixYOodNquOKx4lNYG6o4x7CQ9i0F32ysxBEkSsCK89DO5dNxMWkebhUzyA2SAvsfq0c/NV5Yj8i/SG/li1MQa4bkQvXg1IATYs71RYYMeYLnRV1EPA8LbWDpqpAQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723026707; c=relaxed/simple; bh=2NlTIsOxItqB2jzIp/5ureCC+QPO7m/8hI7FlxJyvaQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=IFanHHsC+pNU0Et8LWji7c3apGCD0i7zG+wCQGjJM6v0a7k640CaBsW77DYbV5WQHHPAJ8ZVcG/ZUclFcG5LqDw+GM86XmZETctoTAR6oJ1/5N/05mTwbYZQ/aQgqma+pgIt5npwAmoIVDgM8+SplQonYezMps0/S+fazIvLaFA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=YxdH1qXX; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=vcYRtDmt; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=YxdH1qXX; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=vcYRtDmt; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="YxdH1qXX"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="vcYRtDmt"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="YxdH1qXX"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="vcYRtDmt" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E058D21CF8; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuVEO95mFBgy7A/J87dAp3t2Ac0+1I7QV5k30Rj6i0c=; b=YxdH1qXXixOjsSeR7XWT4mVBgYw4MIKJRlJPn8+9DEcB6Id7ZeJsf7jAwmIrUe7B8qslDX bthEONOthVa1Xs1o94b4bcxHgWYltEAlagcBW8TgQ3hAxh7q9f1O6z7+zQPBKwnmVERAN+ hGQP+DOXciuO6o3OurjuOca8b7YiLNw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuVEO95mFBgy7A/J87dAp3t2Ac0+1I7QV5k30Rj6i0c=; b=vcYRtDmtYJYHVbuFcRFhhOq63pzn8Due2/K0avtWel9EtUI2XCebywtWT+rafw8fajSUJz 29BVY9NHSqgED0DQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuVEO95mFBgy7A/J87dAp3t2Ac0+1I7QV5k30Rj6i0c=; b=YxdH1qXXixOjsSeR7XWT4mVBgYw4MIKJRlJPn8+9DEcB6Id7ZeJsf7jAwmIrUe7B8qslDX bthEONOthVa1Xs1o94b4bcxHgWYltEAlagcBW8TgQ3hAxh7q9f1O6z7+zQPBKwnmVERAN+ hGQP+DOXciuO6o3OurjuOca8b7YiLNw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1723026694; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuVEO95mFBgy7A/J87dAp3t2Ac0+1I7QV5k30Rj6i0c=; b=vcYRtDmtYJYHVbuFcRFhhOq63pzn8Due2/K0avtWel9EtUI2XCebywtWT+rafw8fajSUJz 29BVY9NHSqgED0DQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8837F13B0B; Wed, 7 Aug 2024 10:31:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id kGj1IAZNs2YsHwAAD6G6ig (envelope-from ); Wed, 07 Aug 2024 10:31:34 +0000 From: Vlastimil Babka Date: Wed, 07 Aug 2024 12:31:20 +0200 Subject: [PATCH v2 7/7] kunit, slub: add test_kfree_rcu() and test_leak_destroy() Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240807-b4-slab-kfree_rcu-destroy-v2-7-ea79102f428c@suse.cz> References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> To: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes Cc: Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , Vlastimil Babka X-Mailer: b4 0.14.1 X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_TWELVE(0.00)[27]; FREEMAIL_TO(0.00)[kernel.org,joelfernandes.org,joshtriplett.org,gmail.com,linux.com,google.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[goodmis.org,efficios.com,gmail.com,inria.fr,kernel.org,zx2c4.com,linux-foundation.org,linux.dev,kvack.org,vger.kernel.org,google.com,googlegroups.com,suse.cz]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; R_RATELIMIT(0.00)[to_ip_from(RLtsk3gtac773whqka7ht6mdi4)] X-Spam-Flag: NO X-Spam-Score: -2.80 Add a test that will create cache, allocate one object, kfree_rcu() it and attempt to destroy it. As long as the usage of kvfree_rcu_barrier() in kmem_cache_destroy() works correctly, there should be no warnings in dmesg and the test should pass. Additionally add a test_leak_destroy() test that leaks an object on purpose and verifies that kmem_cache_destroy() catches it. Signed-off-by: Vlastimil Babka --- lib/slub_kunit.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/lib/slub_kunit.c b/lib/slub_kunit.c index e6667a28c014..6e3a1e5a7142 100644 --- a/lib/slub_kunit.c +++ b/lib/slub_kunit.c @@ -5,6 +5,7 @@ #include #include #include +#include #include "../mm/slab.h" static struct kunit_resource resource; @@ -157,6 +158,34 @@ static void test_kmalloc_redzone_access(struct kunit *test) kmem_cache_destroy(s); } +struct test_kfree_rcu_struct { + struct rcu_head rcu; +}; + +static void test_kfree_rcu(struct kunit *test) +{ + struct kmem_cache *s = test_kmem_cache_create("TestSlub_kfree_rcu", + sizeof(struct test_kfree_rcu_struct), + SLAB_NO_MERGE); + struct test_kfree_rcu_struct *p = kmem_cache_alloc(s, GFP_KERNEL); + + kfree_rcu(p, rcu); + kmem_cache_destroy(s); + + KUNIT_EXPECT_EQ(test, 0, slab_errors); +} + +static void test_leak_destroy(struct kunit *test) +{ + struct kmem_cache *s = test_kmem_cache_create("TestSlub_kfree_rcu", + 64, SLAB_NO_MERGE); + kmem_cache_alloc(s, GFP_KERNEL); + + kmem_cache_destroy(s); + + KUNIT_EXPECT_EQ(test, 1, slab_errors); +} + static int test_init(struct kunit *test) { slab_errors = 0; @@ -177,6 +206,8 @@ static struct kunit_case test_cases[] = { KUNIT_CASE(test_clobber_redzone_free), KUNIT_CASE(test_kmalloc_redzone_access), + KUNIT_CASE(test_kfree_rcu), + KUNIT_CASE(test_leak_destroy), {} };