From patchwork Wed Apr 17 18:59:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianfeng Wang X-Patchwork-Id: 13633767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51401C4345F for ; Wed, 17 Apr 2024 18:59:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDBDE6B0098; Wed, 17 Apr 2024 14:59:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8B9F6B009A; Wed, 17 Apr 2024 14:59:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C83B56B009B; Wed, 17 Apr 2024 14:59:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A9C596B0098 for ; Wed, 17 Apr 2024 14:59:50 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 22EC880FB7 for ; Wed, 17 Apr 2024 18:59:50 +0000 (UTC) X-FDA: 82019938140.04.838EEE2 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf22.hostedemail.com (Postfix) with ESMTP id DB404C0012 for ; Wed, 17 Apr 2024 18:59:46 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=cVuaG39E; dmarc=pass (policy=quarantine) header.from=oracle.com; spf=pass (imf22.hostedemail.com: domain of jianfeng.w.wang@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=jianfeng.w.wang@oracle.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713380387; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vz/MWxuYKbCjC1t4MkPCHPpvqr4U9WONMklYCJ0rdMU=; b=J4jEPp0Bqy71NoYqY/mCu/9iVqiijkL1ETGnzGv1K/eeyD6r1Lmzwshk7nGP7FTwUTM/N7 95J4mLv/X5kqFf1F111cqONWOloFRpLB9v5Y2JtSJb9CjUBx9DVHA2d/bIUnt2vSeFMGDY 2a+hg1cOI/QQgmtyYwRKxWm16chwALE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=cVuaG39E; dmarc=pass (policy=quarantine) header.from=oracle.com; spf=pass (imf22.hostedemail.com: domain of jianfeng.w.wang@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=jianfeng.w.wang@oracle.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713380387; a=rsa-sha256; cv=none; b=sMqn3m9yIgyZ3fwsXsGdbQ/5miDjGYmfOkGehmK5bIRHHe6+1AilRRkMkHhOTkwVC5KRXf kS4jiMYLN+da8BtRwCKPRpIyh7FWntQ5pxb0x1ZI9++o1AydxNCIZ14dFhK6Q01GDx+8hr sZzz8yTvMq/UcPhps8ICPJlBhM95iPc= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 43HHiViW019610; Wed, 17 Apr 2024 18:59:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=vz/MWxuYKbCjC1t4MkPCHPpvqr4U9WONMklYCJ0rdMU=; b=cVuaG39EMK+RqJdvzopoz5AuVvT4sTmM0ah1F9NfMEonqEAFNxThirWFkqA88guAyOji wNi+1mkwEaYXv2hysb6eB59PzDDiWZZXSRK/cL/SvoghmVUDM4Sk2eBlz4swfP9zE/aM i9mxRQtOWbqR4Ye6FwWMj1xplLk2zU37GDcdiTuG9S/9LndLUhTMCV22ceY7YsKGl9B6 +/DOe2p9UCSg/tasX21Wr2ZsoR7NC8v7U0PRrT/69jYdoc0viBa97Smt32r4uEhBK0H6 6WLcCI82VMX7pono3HGDx/6o4mZk9oPIyZBfjD/dtl3DqzCY/wmORsr33fli3lI598z0 uw== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3xfjkv8mmf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 17 Apr 2024 18:59:41 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 43HI0GIn012573; Wed, 17 Apr 2024 18:59:41 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3xgkwhbxjv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 17 Apr 2024 18:59:41 +0000 Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 43HIxdmr026239; Wed, 17 Apr 2024 18:59:40 GMT Received: from jfwang-mac.us.oracle.com (dhcp-10-65-140-165.vpn.oracle.com [10.65.140.165]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3xgkwhbxhc-2; Wed, 17 Apr 2024 18:59:40 +0000 From: Jianfeng Wang To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, cl@linux.com, vbabka@suse.cz, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, junxiao.bi@oracle.com Subject: [PATCH v2 1/1] slub: limit number of slabs to scan in count_partial() Date: Wed, 17 Apr 2024 11:59:38 -0700 Message-ID: <20240417185938.5237-2-jianfeng.w.wang@oracle.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20240417185938.5237-1-jianfeng.w.wang@oracle.com> References: <20240417185938.5237-1-jianfeng.w.wang@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-17_16,2024-04-17_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 phishscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000 definitions=main-2404170136 X-Proofpoint-ORIG-GUID: jdvXbnSCRn9YRfbIdXYvMB6viwMeTQjb X-Proofpoint-GUID: jdvXbnSCRn9YRfbIdXYvMB6viwMeTQjb X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DB404C0012 X-Rspam-User: X-Stat-Signature: 4k3h7eno3sghwm45nspq3raewq9g4m7a X-HE-Tag: 1713380386-124983 X-HE-Meta: U2FsdGVkX1+5ZTTuerLDypN1hYHtTS16Dg+XYbaFVmE/p6qny069VOtrFpCFh86EBRYD2FN/XWIF7nCLtfAdTOvuBpU8+t5ru0SE70QrhlQCAVKMo+LAD6o95J1RnIqgmx97OFeLUU2d5eyrqFfnhKzpszWElzeuq3OHO2Qo85C7O+huQOGh6PKUj/lIzlV4dWMy6Tz+36FWyQ/jGe2AhukkShy6sGdtJK6wDIyVDNR9F1WnXu7Vscbl1Wpj6nONNz2MApRt1Go1XPkvu6caK4MAlpmon5saMu7nBGVmKg3Puad4SVNLbExxz+yjgTW2wInUGd7zGtMJL+Z2Xo5Q4lcKakEBK+A/M6anrcJX1Met7bZZzAPXY3thkS3zfFmqJnkXZ0hef72BKfZ0hvNTy4HZAWUE+Gpdl+GZkCJrZ/JPK1C6l0LEixFvKhiPbSuZlsRgXeBSPllpJsDOauBRc6ol79bC2Pw5DJlX0VvKOsgIGV+XKcg7dnZ6ISPxH1GOEvSopXRXt6c3QVVpXfD5mqYKOUIZgWlzq33vLCLZS0QShEQMsJDPrT+JAzlPdzUZhj1unJMqMqcBX1A6TScXJb95rmHgP4zKBsnqFeSn5JlCsBgrlLqKuldc9Yqvmp6jqehoqa0riVd/w+f7dIeItqNBu/5DoRvA64HEP2iF5vGW/AmPQnZ3hWL2QS4TVh6MDfxtyVBBgx0OrslIkd9pZUlo3EyAEicAyzmPdN19H36eZ1uhvZxPXckMx1ODGYovEJ8SFBLuA3hvStmEcMH7zJxUnjopRR2XnOzg1GCMETgtpP9t15Me5MaA8cI2QomzrGE6XuD79CqClB4MuBnjwSllA1FwM+7ZCNeZY7aEix/7GXV/jFgql99u5ZvC5MWa56l4RtT0gzEE56fINBoN/VRxaSAyGYh+4xtU1Sr3DUxIbwLRpOrNfJqoNm7m+Gl/tfPFdDMdNO2ct3q8bZO P0MMTp0z iLLRCpIglN675H9leAFyh85LTBy8VL4qrPYCMJfy81+zQ8IIA0dqX+iaI6N5j+6Ls19lXtdVunwd//5MpAXU7CMTFYl1cpcXN7RshB0MaNFtHoHgC3/IPmOFU2fsfZew17JDN0xINoww9U6Z449lCU9l507vQqT1L7dijXEsnP1Dm0LzIYv6SovELRwpRff1UBGubopxtQ6LcSLjbWzpSLcgcUwvhYColEexKd225UoS2AmWQmFePMqlO3vjWDuNGgUoQYA5J3dEuOd42pdGl+dINegcnYt2lxCb0mryuMFUDF1oEraVkRNHgrmLL3FTk/orZJXUQ3ZQC1iVXCzTDzgdrNapiom0QSLTVr4jRVPTFyNfRXbDcJwq85EvVr1W1H6N5+uvhHUb7iuEWjqZUarGqKoUvHmah3dCZn9mSRmDAyR9wrBL8iJg5Lg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When reading "/proc/slabinfo", the kernel needs to report the number of free objects for each kmem_cache. The current implementation uses count_partial() to count the number of free objects by scanning each kmem_cache_node's list of partial slabs and summing free objects from every partial slab in the list. This process must hold per kmem_cache_node spinlock and disable IRQ, and may take a long time. Consequently, it can block slab allocations on other CPU cores and cause timeouts for network devices and so on, when the partial list is long. In production, even NMI watchdog can be triggered due to this matter: e.g., for "buffer_head", the number of partial slabs was observed to be ~1M in one kmem_cache_node. This problem was also confirmed by several others [1-3]. Iterating a partial list to get the exact count of objects can cause soft lockups for a long list with or without the lock (e.g., if preemption is disabled), and is not very useful too: the object count can change right after the lock is released. The approach of maintaining free-object counters requires atomic operations on the fast path [3]. So, the fix is to limit the number of slabs to scan in count_partial(). Suppose the limit is N. If the list's length is not greater than N, output the exact count by traversing the whole list; if its length is greater than N, then output an approximated count by traversing a subset of the list. The proposed method is to scan N/2 slabs from the list's head and the other N/2 slabs from the tail. For a partial list with ~280K slabs, benchmarks show that this approach performs better than just counting from the list's head, after slabs get sorted by kmem_cache_shrink(). Default the limit to 10000, as it produces an approximation within 1% of the exact count for both scenarios. Benchmarks: Diff = (exact - approximated) / exact * Normal case (w/o kmem_cache_shrink()): | MAX_TO_SCAN | Diff (count from head)| Diff (count head+tail)| | 1000 | 0.43 % | 1.09 % | | 5000 | 0.06 % | 0.37 % | | 10000 | 0.02 % | 0.16 % | | 20000 | 0.009 % | -0.003 % | * Skewed case (w/ kmem_cache_shrink()): | MAX_TO_SCAN | Diff (count from head)| Diff (count head+tail)| | 1000 | 12.46 % | 6.75 % | | 5000 | 5.38 % | 1.27 % | | 10000 | 4.99 % | 0.22 % | | 20000 | 4.86 % | -0.06 % | [1] https://lore.kernel.org/linux-mm/ alpine.DEB.2.21.2003031602460.1537@www.lameter.com/T/ [2] https://lore.kernel.org/lkml/ alpine.DEB.2.22.394.2008071258020.55871@www.lameter.com/T/ [3] https://lore.kernel.org/lkml/ 1e01092b-140d-2bab-aeba-321a74a194ee@linux.com/T/ Signed-off-by: Jianfeng Wang --- mm/slub.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1bb2a93cf7b6..7e34f2f0ba85 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3213,6 +3213,8 @@ static inline bool free_debug_processing(struct kmem_cache *s, #endif /* CONFIG_SLUB_DEBUG */ #if defined(CONFIG_SLUB_DEBUG) || defined(SLAB_SUPPORTS_SYSFS) +#define MAX_PARTIAL_TO_SCAN 10000 + static unsigned long count_partial(struct kmem_cache_node *n, int (*get_count)(struct slab *)) { @@ -3221,8 +3223,30 @@ static unsigned long count_partial(struct kmem_cache_node *n, struct slab *slab; spin_lock_irqsave(&n->list_lock, flags); - list_for_each_entry(slab, &n->partial, slab_list) - x += get_count(slab); + if (n->nr_partial <= MAX_PARTIAL_TO_SCAN) { + list_for_each_entry(slab, &n->partial, slab_list) + x += get_count(slab); + } else { + /* + * For a long list, approximate the total count of objects in + * it to meet the limit on the number of slabs to scan. + * Scan from both the list's head and tail for better accuracy. + */ + unsigned long scanned = 0; + + list_for_each_entry(slab, &n->partial, slab_list) { + x += get_count(slab); + if (++scanned == MAX_PARTIAL_TO_SCAN / 2) + break; + } + list_for_each_entry_reverse(slab, &n->partial, slab_list) { + x += get_count(slab); + if (++scanned == MAX_PARTIAL_TO_SCAN) + break; + } + x = mult_frac(x, n->nr_partial, scanned); + x = min(x, node_nr_objs(n)); + } spin_unlock_irqrestore(&n->list_lock, flags); return x; }