From patchwork Tue Sep 5 14:13:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 13374492 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 064C9C83F3E for ; Tue, 5 Sep 2023 14:08:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74DAA8D0005; Tue, 5 Sep 2023 10:08:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D5CA8D0001; Tue, 5 Sep 2023 10:08:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 576E68D0005; Tue, 5 Sep 2023 10:08:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 39C1C8D0001 for ; Tue, 5 Sep 2023 10:08:52 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E1AA416086B for ; Tue, 5 Sep 2023 14:08:51 +0000 (UTC) X-FDA: 81202724862.15.05876B0 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by imf14.hostedemail.com (Postfix) with ESMTP id 2C49B1002EF for ; Tue, 5 Sep 2023 14:07:40 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Chp4sjZp; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf14.hostedemail.com: domain of feng.tang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=feng.tang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693922861; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QGTOghc+gA04fugpli15BRz2MjimwwZ2E3P/yXbdOM4=; b=8A6CcbsBhYNq7mTIdfHteUPs+sDu7cDspzK3NEFxPxKN6KzKQ0+VE/Uu28e5JlEMzMSc9U qVzE5bhfEDjEP0V2ht6CkrG4IzansLpZs37Y/3SzOa4PpMqyPI1YTmkJW1+pCOHYK/NcKU 6gbDaGmFyOuXXuGCl3RY9n0qx0IsiCo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Chp4sjZp; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf14.hostedemail.com: domain of feng.tang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=feng.tang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693922861; a=rsa-sha256; cv=none; b=tjWvFK0rWlquSf0Y0tzOkluwcjzesA9xFme7C4KetSnJT9uNUVkfXiyFpiV/qtv1NYCu7d yUGL+f5rEhdiNdmU35wwOqqny8U+NnmkjXzNwDcXXoZLZzMAm834TrEVhEQYoEvx1LbN7j Qbw6ptLZzjVrBg6aL51xNF8dGNrNZ70= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1693922861; x=1725458861; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dvT4bjQOLiJP78MPuTog+CQfsmhI/8vfzdHD6sqXSeU=; b=Chp4sjZpwKYYI5/DAqL/ForcPSy5o4Q/37vkevCcYWSTYh53Sh17IBx6 el9U9/aT2ZjeFaEQVpozVOWjonHygFi7FCkvtfil9JdUoeH2FOyUWnSFO xzX2QyJvgICiIzH7ZGsBLtRc+1AxKUpG9vHXmTSI6OJdmmKmHnIZVEjZE ynJz77l2ix39K69pO3i+oC7iE3JvELEDlj8wcmqeqIa0Vblsk5OcNhXhR iBNs6W//h/+3cR/8gmW56HRGS6CUQqEPj2uPa0Pkrx2usfXxcroNxgRK5 UcTiwxzFMtKaWz9Hw4qvT4nhrV/4ltCnQ0vU+EaZZCgbjHkvgKiqoX0Va A==; X-IronPort-AV: E=McAfee;i="6600,9927,10824"; a="380609619" X-IronPort-AV: E=Sophos;i="6.02,229,1688454000"; d="scan'208";a="380609619" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Sep 2023 07:06:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10824"; a="811242163" X-IronPort-AV: E=Sophos;i="6.02,229,1688454000"; d="scan'208";a="811242163" Received: from shbuild999.sh.intel.com ([10.239.146.107]) by fmsmga004.fm.intel.com with ESMTP; 05 Sep 2023 07:06:25 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [RFC Patch 3/3] mm/slub: setup maxim per-node partial according to cpu numbers Date: Tue, 5 Sep 2023 22:13:48 +0800 Message-Id: <20230905141348.32946-4-feng.tang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230905141348.32946-1-feng.tang@intel.com> References: <20230905141348.32946-1-feng.tang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2C49B1002EF X-Stat-Signature: izqebr1bqa5k4fe59b1x9crjk4xm8hh7 X-HE-Tag: 1693922860-508866 X-HE-Meta: U2FsdGVkX18upu3Q5kHgD5iU9WUoQCxl+XC2SA5WTz3vH1rbsrqbJkAAMhcFnX6NDRt3dzT6oe9smHJIwX9FCrkPr6NKXJAcNjn0tr+EImc+vpObxYYvEdotqScvSSpntKxTskT+tqHFXZWJOCeg/bCYXKVbnDicdq3qbkEG9RS2Z8i1vKRrp8zfZW6MfjyQW9gfcSuUGTLzhF10OBjqrAnY9bLyLa9Qb+TrIL+U1nWonCyLBf9jwwnmjdtAbxJKQOtwtBWXYqyaRMOT8NL9fQTa5B1QJleRBUBe5cWcv8phK6eQhg0vMch+YkrgV9vED4ssRVtiuh9Z3g4p9Nnd0U42WXRx8gIN5KvZBFj49HcIIv15xHuuIJrB/NFo0uSwySTZw1Hsvh034Wvi3dAsDnTMAftgvoio86+gnr9+yFvS7KrcgDXnesuRuX8MBgED/GgX8mTOGIYUzRUTa+aWCKjhpdWL/h9PMjNr4BNybXppcsfylfaNvbyJIyFg2JCvCb7ck5BrGrMc210I8zUUfpz80hif5hHHK2MGxtaYjI0faLGSYLxHSgXa+RKF4bcKILd07Wt6t5BOQOTy/k0nWSMEHelQYVahgpfg3dHfxZnzZNNYiVd3HjvAPlCrX/ey9nUujlOVYY1wlxv2KIYkW5mIKfda7ro/LGtwlyIERko7pDwbqfccWA7EGg9RmfJ3YljKazqekqMmj5qLfv/aEu3jJzj7VkCOp9T2U8joqrdd2pxou49qKFYYlMymp4mNP5j4oDVqhjqW7vKBdcpI4twr0Jr27VT1aYSo6GKyEDSccpGppohxbAUo8ZDuhNcNXzsdFOVxIi5JbMEuXK8efCRT6bvpjXOl6lgII0C7eJH17pORE6py8KF6xVV3OPg77RMDv44gV4UVDr4WERBdFsC5pm5wvupC0+ssJAW/tW6bg0kCmMTvmDyjffdGlJlztUCX/E0PUZoSlkuu1tu YKfSQ/bW NYA6mDU3Bw5ioB+i9gpRM9LDe0WhR6NdUECjrJ8KVrDnDfyxtc2GQtGXw+bjuYWGFfUAUAkZAE3AuqkjosNCtcUjJYqjcLc2pzVcmXIHTyMXqZ8cHYD7+uRHUy1rk75EJ+gCT/fzZU3Chci6OluJM2Yq2MxXKcRvfLKryLBk6XCxORD3iIT6ScSn7+hrES/36El4zJUSJEKd2XSvcfaw32ISxcioXLwhyKRh2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently most of the slab's min_partial is set to 5 (as MIN_PARTIAL is 5). This is fine for older or small systesms, and could be too small for a large system with hundreds of CPUs, when per-node 'list_lock' is contended for allocating from and freeing to per-node partial list. So enlarge it based on the CPU numbers per node. Signed-off-by: Feng Tang --- include/linux/nodemask.h | 1 + mm/slub.c | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h index 8d07116caaf1..6e22caab186d 100644 --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -530,6 +530,7 @@ static inline int node_random(const nodemask_t *maskp) #define num_online_nodes() num_node_state(N_ONLINE) #define num_possible_nodes() num_node_state(N_POSSIBLE) +#define num_cpu_nodes() num_node_state(N_CPU) #define node_online(node) node_state((node), N_ONLINE) #define node_possible(node) node_state((node), N_POSSIBLE) diff --git a/mm/slub.c b/mm/slub.c index 09ae1ed642b7..984e012d7bbc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4533,6 +4533,7 @@ static int calculate_sizes(struct kmem_cache *s) static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) { + unsigned long min_partial; s->flags = kmem_cache_flags(s->size, flags, s->name); #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); @@ -4564,8 +4565,12 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); + + min_partial = rounddown_pow_of_two(num_cpus() / num_cpu_nodes()); + min_partial = max_t(unsigned long, MIN_PARTIAL, min_partial); + + s->min_partial = min_t(unsigned long, min_partial * 2, ilog2(s->size) / 2); + s->min_partial = max_t(unsigned long, min_partial, s->min_partial); set_cpu_partial(s);