From patchwork Fri Feb 14 16:27:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13975225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E06DDC02198 for ; Fri, 14 Feb 2025 16:28:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D75C6B0089; Fri, 14 Feb 2025 11:28:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 785606B008A; Fri, 14 Feb 2025 11:28:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 516B5280002; Fri, 14 Feb 2025 11:28:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2F0CE6B0089 for ; Fri, 14 Feb 2025 11:28:00 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D3D7FC0674 for ; Fri, 14 Feb 2025 16:27:59 +0000 (UTC) X-FDA: 83119081878.14.BB73230 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf29.hostedemail.com (Postfix) with ESMTP id AF846120008 for ; Fri, 14 Feb 2025 16:27:57 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739550478; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7D/TFBKOt3H025n7eeSZr2QojKC+CoF2FQqDZxhLNXs=; b=PQwhDX5v77GF6mqkpet30fqGDx0NHVuwK437rTNI/IweG2XwTBX6O2+4iuk5NaDUo5ZNTV eu/Ehn9Udwe04/lC3BIp7vJ3FngcbMkGIwi4SULVTt/XeHWIvN6XTxi9puSae0IPP56YFd c2chqVaGZAjnpJnL2SCLy7QU4MS13yc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739550478; a=rsa-sha256; cv=none; b=nnggLEMUH5iTb/KfNXa3iL27tsNazBQ8Uxhxi+VV39s3YcqRgl0+xd7h4SHwl3fPL/5vFu LQDF9Vn/rMNVUBYtntdk/1lCJEFcY/xygj9HNe1ZMRucpZBNbiekMhVGcHYVIIpHn7aQ0a s05+q3T93Jk8FAnLnfY4XjxMXJ0wRIo= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id F20B31F397; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id DE1DC13285; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2Mf3Nf5ur2eHSAAAD6G6ig (envelope-from ); Fri, 14 Feb 2025 16:27:42 +0000 From: Vlastimil Babka Date: Fri, 14 Feb 2025 17:27:43 +0100 Subject: [PATCH RFC v2 07/10] slab: determine barn status racily outside of lock MIME-Version: 1.0 Message-Id: <20250214-slub-percpu-caches-v2-7-88592ee0966a@suse.cz> References: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> In-Reply-To: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspam-User: X-Stat-Signature: spxuzacfyb414bsupsrtomqb6zqdzh9n X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AF846120008 X-HE-Tag: 1739550477-918897 X-HE-Meta: U2FsdGVkX1+OyCpBo+Yn9ApnOafxALPWQWCp5YdtplU1jxXV7cprjD0bySv07Cj5sj4w98thD5gYO5r4thxcofdDjx7bSY5LcxbniCpyiiv8DqAEsBUNsEOk0x4nSVjFSrWPhNgH7XePXbLSgxrRsB3wvC9l3RAzaLSa09fN31tRK+96uX0D4vXSexWnx5eeAAKTctpOO53C0akTkfB3dnqL7e0U1IGmYKvZYs/TvFJtXGbgyaBLJMZ64BJvOTNVrMYnMFeOG/iAMvocnbPoZyHuzj70/TCr96YUw2Jrk+b4AGTWmKrr6dJS6v5IBGYqRZDCRfPHxHjfuNxqz8N3xqpw7heXqL2T6DnOfpOqcyhYeTr/FJ2k75kMl9qsPEjFf6TLG90pIVfBCASfWEQhcNE13VxPkvfStabIQ1jKeKVmaRfUgEIpyuAGJaN79dnP3vrOMAJ2GishYdZjF93A81V0DS1BPiWiBBQINmkAr8vwwIkfMp/IM/0HVoWgzeFeI4NVgOfZ9vw5trV6R6t1VHQ4G4skLFZVCgMNPGmIo/O95ZiRkoPtGomxd/RKuF0G6Nb2A0FpR9BAsods/Qwm8GZRnxkcVNpq/6Gabr3c7ySjvDdt80GBxX+L5UhzOmBzyOm6NPs4h6rLZTyWZWrOrl3P/bSw6zipAwIELOvR3hxPlxjmuj7ItrEFNYtl8VT7qLSqpGClVmKG8Y/d0M1RsDlSS9kQYe4daNO7+4kMc9/Ru9DXQCcubiFQR6HCmRV7HLsD9W52cWTjpwWSAqD+r6DCtZAJ5pfCqpFAM+EGqVoMl53ZNVYd1XPaILxgN3iWCGPNytk/wLgLAb9prUbPhtV+L7YMPQolZi+KzYpUGd6pkQmylcloIX5+/f84o2YaPRTQYozZA/+hlRSqcrLBEEW++WoV94hC8sLw2f78UT74vJl60Qk+PSv9yeN44IEjqQbyNLTLOW5vlIQ5aLt cHjre3u6 gpjoAwflTRHSIw7dFrO5avQPBQmQq/CqdKxUOg/08XNyPcFsBa5MCFz1F4s6HS+Pg9vbrbs/t9tmaSTWB7dTkMcFXbGPVK18Tv58M7/hr/aVb44JFpDvV12dSdsWkpd1/KR5+dOgks72BaVMMM9lvPYSnMxoz5iwFQ95tHzyl9x3RfvCkCOkOcpAxBVStuvW2yYSK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The possibility of many barn operations is determined by the current number of full or empty sheaves. Taking the barn->lock just to find out that e.g. there are no empty sheaves results in unnecessary overhead and lock contention. Thus perform these checks outside of the lock with a data_race() annotated variable read and fail quickly without taking the lock. Checks for sheaf availability that racily succeed have to be obviously repeated under the lock for correctness, but we can skip repeating checks if there are too many sheaves on the given list as the limits don't need to be strict. Signed-off-by: Vlastimil Babka --- mm/slub.c | 57 ++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index c1df7cf22267f28f743404531bef921e25fac086..72e6437f1d74bfacbb1cd7642af42929c48cc66a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2685,9 +2685,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn) struct slab_sheaf *empty = NULL; unsigned long flags; + if (!data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_empty) { + if (likely(barn->nr_empty)) { empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); @@ -2703,38 +2706,36 @@ static int barn_put_empty_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf, bool ignore_limit) { unsigned long flags; - int ret = 0; + + /* we don't repeat the check under barn->lock as it's not critical */ + if (!ignore_limit && data_race(barn->nr_empty) >= MAX_EMPTY_SHEAVES) + return -E2BIG; spin_lock_irqsave(&barn->lock, flags); - if (!ignore_limit && barn->nr_empty >= MAX_EMPTY_SHEAVES) { - ret = -E2BIG; - } else { - list_add(&sheaf->barn_list, &barn->sheaves_empty); - barn->nr_empty++; - } + list_add(&sheaf->barn_list, &barn->sheaves_empty); + barn->nr_empty++; spin_unlock_irqrestore(&barn->lock, flags); - return ret; + return 0; } static int barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf, bool ignore_limit) { unsigned long flags; - int ret = 0; + + /* we don't repeat the check under barn->lock as it's not critical */ + if (!ignore_limit && data_race(barn->nr_full) >= MAX_FULL_SHEAVES) + return -E2BIG; spin_lock_irqsave(&barn->lock, flags); - if (!ignore_limit && barn->nr_full >= MAX_FULL_SHEAVES) { - ret = -E2BIG; - } else { - list_add(&sheaf->barn_list, &barn->sheaves_full); - barn->nr_full++; - } + list_add(&sheaf->barn_list, &barn->sheaves_full); + barn->nr_full++; spin_unlock_irqrestore(&barn->lock, flags); - return ret; + return 0; } static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn) @@ -2742,6 +2743,9 @@ static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn) struct slab_sheaf *sheaf = NULL; unsigned long flags; + if (!data_race(barn->nr_full) && !data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); if (barn->nr_full) { @@ -2772,9 +2776,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty) struct slab_sheaf *full = NULL; unsigned long flags; + if (!data_race(barn->nr_full)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_full) { + if (likely(barn->nr_full)) { full = list_first_entry(&barn->sheaves_full, struct slab_sheaf, barn_list); list_del(&full->barn_list); @@ -2797,19 +2804,23 @@ barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full) struct slab_sheaf *empty; unsigned long flags; + /* we don't repeat this check under barn->lock as it's not critical */ + if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES) + return ERR_PTR(-E2BIG); + if (!data_race(barn->nr_empty)) + return ERR_PTR(-ENOMEM); + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_full >= MAX_FULL_SHEAVES) { - empty = ERR_PTR(-E2BIG); - } else if (!barn->nr_empty) { - empty = ERR_PTR(-ENOMEM); - } else { + if (likely(barn->nr_empty)) { empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); list_add(&full->barn_list, &barn->sheaves_full); barn->nr_empty--; barn->nr_full++; + } else { + empty = ERR_PTR(-ENOMEM); } spin_unlock_irqrestore(&barn->lock, flags);