From patchwork Thu Nov 2 03:23:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13443349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12134C4167D for ; Thu, 2 Nov 2023 03:24:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7C8F8E0016; Wed, 1 Nov 2023 23:24:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A078F8E0009; Wed, 1 Nov 2023 23:24:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87FA58E0016; Wed, 1 Nov 2023 23:24:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 785F28E0009 for ; Wed, 1 Nov 2023 23:24:43 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5646880E6F for ; Thu, 2 Nov 2023 03:24:43 +0000 (UTC) X-FDA: 81411572046.29.6198220 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) by imf25.hostedemail.com (Postfix) with ESMTP id A10C0A0004 for ; Thu, 2 Nov 2023 03:24:41 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=KQjfhe1X; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698895481; a=rsa-sha256; cv=none; b=K3jAasAlAXWGEul/9UWB5NdtMKiivYP69OmU1FxmNh+bRJewG4rVGn5yZUB0gwDtvA4C3k 0CSItSQ9tzs0wFMoCdsReo6unr+oluPG2W1lAgGxUc0lyKsHxB8lKpXoed1trTV9lY7mRT J4cEZiLm3/D3q4fkt2y5VjoZN79XvWU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=KQjfhe1X; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698895481; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SxN7Gc7oPUWEeX+rwHiBiXl514BfxOyW9F4Ey0p4ljE=; b=lTLF7BAiN7Snnq5i6XnF1zctuXqHQ40NmA8EV/717hV2NqzmrP/+kq1spjGDmnHVCmyr1M uCUSGpeSL7ZrrmPWQ9FLAbFgeKW0BPn3EhN2hQqapWMReXNed1ngABI68xKDcKlZV4GUq1 jFQyuCUZ82M2D8Z4w28sSUgXnzEVPwk= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895480; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SxN7Gc7oPUWEeX+rwHiBiXl514BfxOyW9F4Ey0p4ljE=; b=KQjfhe1X0vzh80JaLfhIpxkVvacWOXmJrdGiki+nTBnVHkJnDiiAVRsK2kELO7fDNxvQI6 G6ybzeZ5YjP1xah95OqvCvgzwmI2JTnc6KOhO8JBYI5WOuwCfe9ULPpNMpW0LdOZ/2n96O //Wvc+JEmR8lCOYJBPeBl/5I81NJdYw= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 1/9] slub: Reflow ___slab_alloc() Date: Thu, 2 Nov 2023 03:23:22 +0000 Message-Id: <20231102032330.1036151-2-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A10C0A0004 X-Stat-Signature: f8kdp1bjfdg1shuuumc415rnabpapqzy X-HE-Tag: 1698895481-53424 X-HE-Meta: U2FsdGVkX182y7oRFjUKNu/vOAPJ5Dy4FDTRursOYxzRgjJzlQiqWSYF3LrFVFlwhWR0JZR/zPoPHJEC78m0WIG1K+LQE2EaBf5YcAVOQ1SQWAZS7lxePVE2pp8f0EjjV6V5LiHcoChskPvAOftoPuNnePfrY7/9wI4U2O+xBNEvt+nHpZFsRCh2XErGSaHTFR54PPyHfzvvjBUR1FmAmMUAbUk/IRYKbmL8ZfCgPdcIGEtDDbbVO1qgvTY6kDIJkb54ZbYow7RxhMuM28bXTAZMjMyaR4Qi9BzyqIb525kmEDa6ax2l0+dp/vKQRLaX3WVPxGKO3kIok6Ea6AsDLovURXiGge+PebP3R1dWi9+aXBbrMJtWFWS1mqiKV0iQMtSWt7SmYzMUD++XEwrk3Q04IorXDcrktYtQ5OpUi/qF75QjnJgchs7iC6og8N13dMwJyH3wpGRXdo6N3z/mOnZlOH3IMVZ0uxhc5jZH1kKXNuSSppDyHm0Z0FYRd+1yjLyJKJHwRI24MjZOxoVuB0l3NbVhfVxfQkawmVah1NB9Vne4v+bNlNVxUDQIPncq6aHy6vl8W/Xe8rEKz3jMPTVu7aYTjuyw8V/76Ff8wkPmSxQa+VFdpzJvZITajPIUx3gzntVaAS0RYckcVcKarnZpRCIqWML5M0UjIAG6BIaDQsybMbBeiSJjEFiGn6jdocjuC4c2Zvh8XTV7v844Cg/6bL5C6W6B6JFIZ7HKwXjOQwfs/NUKaDWTDHcmWhLFjwcioG3jl6L53+UI1o1th7PtbURgNSGFVJIpUhR/9V3NN6zohz2d/uiSI02M/QJnacd5rb8Atm2xwiBFRM3trMHMjHbHpFp3t0AGILTfdn6sMAYJdmCPfQAHvLRITdfUwupr4KqgH3hykj2E8GxTkwKqG9h0rRQPKWgPCm4lmIq8H3s3ot6t4knRyiJ+v9YV/H7/jPsL24V5xSdGvWH v0sTrsfk Q5C4F6Ijgi/rOqyJ1T0wtn1/tl3RO1Tt+2ZR/FhSIx9OVfvFft3BPABMnw+s8BVZ7wT77dNO7v7XYZYV3Ngy8zh4m8vt0QEMGl062cLQIMSmwyVl9bcIkXUGu9fwpquhBLUtncqJ5Mla/cB8AQPeX9yck2txPkSsdL5KnV21u/QHjDLph56w2/aeoDA79JKyhnEXOBRJ9Ua2VwXb0/OY85kae6zG/zUwzMpF2DFz43FQ8wt4U0fHqjvT+rde+U0N91oxS/eN7Yfm1+n6V6pG1ccm3gg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chengming Zhou The get_partial() interface used in ___slab_alloc() may return a single object in the "kmem_cache_debug(s)" case, in which we will just return the "freelist" object. Move this handling up to prepare for later changes. And the "pfmemalloc_match()" part is not needed for node partial slab, since we already check this in the get_partial_node(). Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 63d281dfacdb..0b0fdc8c189f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3216,8 +3216,21 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, pc.slab = &slab; pc.orig_size = orig_size; freelist = get_partial(s, node, &pc); - if (freelist) - goto check_new_slab; + if (freelist) { + if (kmem_cache_debug(s)) { + /* + * For debug caches here we had to go through + * alloc_single_from_partial() so just store the + * tracking info and return the object. + */ + if (s->flags & SLAB_STORE_USER) + set_track(s, freelist, TRACK_ALLOC, addr); + + return freelist; + } + + goto retry_load_slab; + } slub_put_cpu_ptr(s->cpu_slab); slab = new_slab(s, gfpflags, node); @@ -3253,20 +3266,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, inc_slabs_node(s, slab_nid(slab), slab->objects); -check_new_slab: - - if (kmem_cache_debug(s)) { - /* - * For debug caches here we had to go through - * alloc_single_from_partial() so just store the tracking info - * and return the object - */ - if (s->flags & SLAB_STORE_USER) - set_track(s, freelist, TRACK_ALLOC, addr); - - return freelist; - } - if (unlikely(!pfmemalloc_match(slab, gfpflags))) { /* * For !pfmemalloc_match() case we don't load freelist so that