From patchwork Tue Oct 31 14:07:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13441577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29ADAC4332F for ; Tue, 31 Oct 2023 14:09:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B68A86B0303; Tue, 31 Oct 2023 10:09:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF0176B0304; Tue, 31 Oct 2023 10:09:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 991736B0305; Tue, 31 Oct 2023 10:09:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7DF316B0303 for ; Tue, 31 Oct 2023 10:09:10 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 54BDD140328 for ; Tue, 31 Oct 2023 14:09:10 +0000 (UTC) X-FDA: 81405938460.29.499CE55 Received: from out-185.mta1.migadu.com (out-185.mta1.migadu.com [95.215.58.185]) by imf21.hostedemail.com (Postfix) with ESMTP id 25CAB1C002E for ; Tue, 31 Oct 2023 14:09:07 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=e0oJj347; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.185 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698761348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A++jOVGS7LIP72VwRB52HksoP76pYwnYqPVPh8XlVaY=; b=o+KuPsWDf8rssweeZa7mdzU1RK1+aENRmGTwcHtOWJs+RHqz5TB4Ep+hMHQLdrzULqQXQG IozOwpSa//m/Ny0o4z8mt/w3z8c5BxtU5JEyd5RNlvAjcMz1TV0zVe3yafMojPRpGZv8ne tUGf7sLCbRwGoRL2r9j9+46lZ/wv4hs= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=e0oJj347; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.185 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698761348; a=rsa-sha256; cv=none; b=HWZlxor2hRQW/i3S8nX9tJtoC7YB6xt0lmMaZ/HMj4OscYdLJwWXhDvPUJzvcZEWrZZlKa a7l8lA0cH/GX/tXWyNMcEu9JEEQKgF7fms5/cXxjkFmD1O8NWSmlgLAwsgLCBDDVmHRxFj eFSO4spFdEmhyZFYTEbEnBTNxLJkoLk= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A++jOVGS7LIP72VwRB52HksoP76pYwnYqPVPh8XlVaY=; b=e0oJj347aPZSwbHKIh3vXLsSkKARGwrOKNnsPyGmHd3T7kZSTXQX3ogXWfbcaQarI3N9T+ K6a8+/g2w/Lvo1/fNOwaYaNltKZM0McSUApZ+72CI1R6rJaPoGntDGGb/xG8G5fgvNBo8e 6XB8zAf1SLuL1e087s8VurnX6PJPY5s= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 3/9] slub: Keep track of whether slub is on the per-node partial list Date: Tue, 31 Oct 2023 14:07:35 +0000 Message-Id: <20231031140741.79387-4-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 25CAB1C002E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: xcxdphdui7p64yueegg41kudq4ts1f3t X-HE-Tag: 1698761347-225172 X-HE-Meta: U2FsdGVkX19gTvKlasa6WEsLJGEctEtCdiENveTi5mdy7aPs30R77e94CJ7CxjtJrYdmz93nD2+RF4PAYgM/QHrgtpqJCBTASGHI05ac9FLdO0Idl0YI4OWwhZQ5/Zh7EbCaXVrvhcmX8vSvgLW2OwFfzrXjQ757qY5ydiSzIsvHAaXwi9TTiek1QiXwhwzmBeDCUZOVEgmlbLthxgP8rr9Hh/Y+kNXwHEnv2jprOsrWcoV/RgnHg3u7hZNi88Rh3Hy1gSaz25igJAup49nNdKxrSNTRonRkiXic5Bqg/BdzTn/s2yYtmK/rckPnlpQ9EMGBZdGNbc3wzag8UEqOOd58aO+BRQvh/PX+kqlWQMzIAkNfdz5t6CXxFOAORbzU7ewH+XVBItf1BjL6rLdEz5zlRWIa/0ILWEQjGVGP08Yey9hOu54bcKoK0GEf4SKL6ZX9hHd8HrIOi0/f+fLRtXlZVrQgLAnE/sTL8lA21wGIiZj//fn4VX5NkyNondd+pbHeLcT4FQa4yA6KEA2qOYAoOrXcnNBGw+s+xeZsUVWYJAQRZ8G4byBTqGL5FzG22Z2ilpyLep6ASfj0hU9QX/xs+5ZK51+p6JCKYPxtKdE7TV0DOeDQWEC/yNJ5uWnDcfJwRCQoYcWVkp3dmnbZKJQmcSPCUe/FnD8DMlHugnlR/Li9G98KIOTFExuVDEQS4B1TnwDEUbvk9d/klNheiiU2tYqFUUfJkbMGcJSCxxeKDyFqz3Ym3G0pXBLBLrw4VCP3IY1ucQg7giyPor1+Wtw3aQf7fBcAfqsGdTYBm3OgwBER55R6lHIU/sGS7IsHiC3noG+TVli2RVk8YMkdCvggaqPwwgKpLsC9YNQlWlSNduMbHyNJNlPon5dV30+j7ydtaa2GSQF5UuP7zjtoPpkB+5U0dmqGLcTi1HFvDIeiJ6Jk5DOmkEFmQVToTeZW0/47rohLwZ/dCyfKf7D MeGl2F1W tj6dnJ4mkNnQqZNnsit/5swgXtzxhg3G318rBg8W3ENnfw+3FOVUegOcnvQG/Ob8iaytnG8Oa2sE0Q2G4ynwwmy34ofN2RDXGr/9ryLXGH0af1wJ2qbfHVT9JqUJ4HizGrvY/IhzuHk68lqthpVEYvPNrqTzKtiepuAYMwzAicJyfjirB+4mJjuKgUf6aA/8Sh8bVS5Va5jMWkL4K48Ko0z6XLi53Smp9leLwLR7/Km7fARWj0Kfjb/Wo87Tm4N5SD9MWfM52mMViWE4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chengming Zhou Now we rely on the "frozen" bit to see if we should manipulate the slab->slab_list, which will be changed in the following patch. Instead we introduce another way to keep track of whether slub is on the per-node partial list, here we reuse the PG_workingset bit. We use __set_bit and __clear_bit directly instead of the atomic version for better performance and it's safe since it's protected by the slub node list_lock. Suggested-by: Matthew Wilcox Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 03384cd965c5..eed8ae0dbaf9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2116,6 +2116,25 @@ static void discard_slab(struct kmem_cache *s, struct slab *slab) free_slab(s, slab); } +/* + * SLUB reuses PG_workingset bit to keep track of whether it's on + * the per-node partial list. + */ +static inline bool slab_test_node_partial(const struct slab *slab) +{ + return folio_test_workingset((struct folio *)slab_folio(slab)); +} + +static inline void slab_set_node_partial(struct slab *slab) +{ + __set_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + +static inline void slab_clear_node_partial(struct slab *slab) +{ + __clear_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + /* * Management of partially allocated slabs. */ @@ -2127,6 +2146,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, int tail) list_add_tail(&slab->slab_list, &n->partial); else list_add(&slab->slab_list, &n->partial); + slab_set_node_partial(slab); } static inline void add_partial(struct kmem_cache_node *n, @@ -2141,6 +2161,7 @@ static inline void remove_partial(struct kmem_cache_node *n, { lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); + slab_clear_node_partial(slab); n->nr_partial--; } @@ -4833,6 +4854,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s) if (free == slab->objects) { list_move(&slab->slab_list, &discard); + slab_clear_node_partial(slab); n->nr_partial--; dec_slabs_node(s, node, slab->objects); } else if (free <= SHRINK_PROMOTE_MAX)