From patchwork Mon Oct 16 05:29:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13422488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46DACCDB465 for ; Mon, 16 Oct 2023 05:30:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFE4B8D0030; Mon, 16 Oct 2023 01:30:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CAE5D8D0001; Mon, 16 Oct 2023 01:30:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B27828D0030; Mon, 16 Oct 2023 01:30:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9FE1F8D0001 for ; Mon, 16 Oct 2023 01:30:27 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 78EB6B5C2F for ; Mon, 16 Oct 2023 05:30:27 +0000 (UTC) X-FDA: 81350199294.28.4C2662E Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by imf03.hostedemail.com (Postfix) with ESMTP id 555BE20004 for ; Mon, 16 Oct 2023 05:30:24 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=i8llK6Hh; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697434224; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5/6gOK5aHsw1B56hGCkN1OotrD6M3Gmw5q2H3PeSKYc=; b=IkN9aaXHYhr9wQMIbCCtYQhEaE/fUagybu48ILIofvT7Q7ZeBSE717sPfJ03b3SgEOXqW3 Hevc+SSh0FF3LAnyY1Y9ADOrCCx9LaxNKjgGpXHEcp8WXMhva/m/6ldKqDXV2Uch9BbWiK 6mO72dOtJAtFawvccFzCUX3iBfZRm+w= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=i8llK6Hh; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697434224; a=rsa-sha256; cv=none; b=UKnYGFhZbEJPa8c8G+lVaBGyy5BrLgOVKbk2R/stGndE2Lb5Bsgwl0tIQt57k0m7rbmnK7 yUtip1Q1JBZmkUJY0VZIvj2KH1wWE8gK++mkk4VCEqu/HV21azJOSQogjXrWgD51RLvN1M SiBS+qBiPCgFSNzrhiINXS+sG9WY/p4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697434224; x=1728970224; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yIFg5arfuvqezYrEgzPwE443naXU4SI+Zr6n5bfusVQ=; b=i8llK6HhDeb79U4PjMBZQmGtqIOrpSOUhcyalKKIGZ7/c37XUns65LKy 6xChU80/rOyKnTvjEiv0Brr+5FTTboLiEgGOEJbNQ8fuHleePKcsx6h/2 Z6vESh9QqVnJ2/dP3HOvWWiakttDi4DuW+VtIjvfCG/KRbYxG7DGF/eNS TSOuHjayIelDgALprKq3NtxUHL5DNR4Di88KPxK9DWW1VM9Hl5OZHVjNA u3hkixXBfjkEIpb1e0m80zcaAshtiMZnRcnuf6COPOw5iSo8Xm1l2pioK eJ6FsrpjBj8qgoXZepa2B5KZDxfuK3eC8Mvrx8AAUyGEztO7VNdsCmZSg w==; X-IronPort-AV: E=McAfee;i="6600,9927,10863"; a="389307942" X-IronPort-AV: E=Sophos;i="6.03,228,1694761200"; d="scan'208";a="389307942" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2023 22:30:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10863"; a="899356632" X-IronPort-AV: E=Sophos;i="6.03,228,1694761200"; d="scan'208";a="899356632" Received: from yhuang6-mobl2.sh.intel.com ([10.238.6.133]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2023 22:28:22 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arjan Van De Ven , Huang Ying , Mel Gorman , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: [PATCH -V3 1/9] mm, pcp: avoid to drain PCP when process exit Date: Mon, 16 Oct 2023 13:29:54 +0800 Message-Id: <20231016053002.756205-2-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231016053002.756205-1-ying.huang@intel.com> References: <20231016053002.756205-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 555BE20004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: hms3oneq5aizujdyqbnfwi3p69yp3gnq X-HE-Tag: 1697434224-874532 X-HE-Meta: U2FsdGVkX18lQUNYEydqEy2HSiGYTsHoBSaKbhxykHtKdxoK/juX9HTvNflxF2bWwBYOf+vkZN6aFWvLjtFO+3kJLAhd05G3/RjIJ2x6+wUfYMr1V+gqT/MyUl5ULl7GVMXd0KZxrNiSIY+rt56oz4IQJlDMbB+dZ+1uH6uxbcxkgbMrwd4YgXh7O+JrRluDN7us0hEShpLjvEia+0UgMYOxjNDXMU4bAWPK+CEvw8GbhnAoKNHpnkPchtL15t8owhLJLbYi1e4fRcq5DAU3xoYkjAvFPZ9/bdWoDSPzF4y/mtQd6cmQ81PF03SEBhBsFGfpxAfEfdZfEmHaCifxbPKZcgEH7EIwOrlGIlzD79h3kVjNA/8ppCVC6QMxTb9nwbkN7h7CeII6SJKNp4veU5+vNGYAsTD5jNzm77Yzdx5vJziYbBXfwSCJiqUkg1x+Q8+wmS8xjCBMEdsFwMGaAyTSiyPfqBz4ojFgPOKvgbldqArm3aNUGDpwSxeJzKKxnBrWXXf2TcwbrYnRIXvaWBZH43lQ07WuZH7rExEXHNWj2gGIfW4sPrx/MFraxDIGeqnF0QOGLK73qnUC4mVcFngnP2Pcz7dVtuebUaas8aaKV725K7vnCdJUcEMff2oFm4dhYOqAnC8Ou6N88INoz2BW1qkpSh1nQEmdbDZNL2CcAaKImwJUhjnBfWXaIGhbu3Yq6c6qWARCBKD9Omc5CkcG4mcZ9pEsJDnT+z4YyXnRoUwzPnlfHUElOI/pEOt5JcxyvUPyg6DQp6JRBUVgzOLlZrL49fY1lASd43+EUZMRpcY5oouKNyPslRgAfz0utjmrT/oAjID3ix4tmAXE75XBihuLGPV7L5YEqDoaudtBaICBu/wohXnMhKe7jDMOoHDkWhfRP6nB/QZYNG1xsUnsp6J16zf1ye+C/TuX8HnvKQZA3WaBSKzeSEY4zgq4GZogpa8ShIMDWiQzjb2 MpN3AddI bmd1PeLJTunuNzocyMbWfIva7XRUDCvvziVgTjAI/9/P5yc+JMPC07NewJxySy07pKd+1Eodupkzo8qKzzqNG0ZiS2OGXsZvJraauLFBVq6UzLKBHZNJUM4xC9G2IUq0qsF2xGiAInJvs0ga7cGn5C5jGVK6gSg6a1xFA9kBBltRStXZPXZKG9tc7PD46oAQJ71IO4nSZZRr3DSsRD8Iwt0drpeubUvppXHZVF0MrGhk26s7dOBnsuyqO2CRInwLO6iC51f4lyaBEUM5BXlZ4Tzvtrw/K+9ImyVTtSCKrKmcDHLUvy/gHALbGjfzkDICo8FFEgKnLHrIUCoct1XQYZkVV9FDlhXs8r+teOGRBpwY7NKRWuKilUvTez3bC9tr8hE0TbvwKb1NTyuWyuYbcnzFSPWpmqfWogijZr6UIPNfcX24ZJbes6OIachP3LCIHbXz0xYNo/foveOHYumtG3Lv6HId81GyJ1VAdj549bUEbwxrgF4k+lYsLKqAdbsL/HsMijfiOVZ5LnboV+gauxWmniVjH/ML/K+Oo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In commit f26b3fa04611 ("mm/page_alloc: limit number of high-order pages on PCP during bulk free"), the PCP (Per-CPU Pageset) will be drained when PCP is mostly used for high-order pages freeing to improve the cache-hot pages reusing between page allocation and freeing CPUs. But, the PCP draining mechanism may be triggered unexpectedly when process exits. With some customized trace point, it was found that PCP draining (free_high == true) was triggered with the order-1 page freeing with the following call stack, => free_unref_page_commit => free_unref_page => __mmdrop => exit_mm => do_exit => do_group_exit => __x64_sys_exit_group => do_syscall_64 Checking the source code, this is the page table PGD freeing (mm_free_pgd()). It's a order-1 page freeing if CONFIG_PAGE_TABLE_ISOLATION=y. Which is a common configuration for security. Just before that, page freeing with the following call stack was found, => free_unref_page_commit => free_unref_page_list => release_pages => tlb_batch_pages_flush => tlb_finish_mmu => exit_mmap => __mmput => exit_mm => do_exit => do_group_exit => __x64_sys_exit_group => do_syscall_64 So, when a process exits, - a large number of user pages of the process will be freed without page allocation, it's highly possible that pcp->free_factor becomes > 0. In fact, this is expected behavior to improve process exit performance. - after freeing all user pages, the PGD will be freed, which is a order-1 page freeing, PCP will be drained. All in all, when a process exits, it's high possible that the PCP will be drained. This is an unexpected behavior. To avoid this, in the patch, the PCP draining will only be triggered for 2 consecutive high-order page freeing. On a 2-socket Intel server with 224 logical CPU, we run 8 kbuild instances in parallel (each with `make -j 28`) in 8 cgroup. This simulates the kbuild server that is used by 0-Day kbuild service. With the patch, the cycles% of the spinlock contention (mostly for zone lock) decreases from 14.0% to 12.8% (with PCP size == 367). The number of PCP draining for high order pages freeing (free_high) decreases 80.5%. This helps network workload too for reduced zone lock contention. On a 2-socket Intel server with 128 logical CPU, with the patch, the network bandwidth of the UNIX (AF_UNIX) test case of lmbench test suite with 16-pair processes increase 16.8%. The cycles% of the spinlock contention (mostly for zone lock) decreases from 51.4% to 46.1%. The number of PCP draining for high order pages freeing (free_high) decreases 30.5%. The cache miss rate keeps 0.2%. Signed-off-by: "Huang, Ying" Acked-by: Mel Gorman Cc: Andrew Morton Cc: Vlastimil Babka Cc: David Hildenbrand Cc: Johannes Weiner Cc: Dave Hansen Cc: Michal Hocko Cc: Pavel Tatashin Cc: Matthew Wilcox Cc: Christoph Lameter --- include/linux/mmzone.h | 12 +++++++++++- mm/page_alloc.c | 11 ++++++++--- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 4106fbc5b4b3..19c40a6f7e45 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -676,12 +676,22 @@ enum zone_watermarks { #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost) #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost) +/* + * Flags used in pcp->flags field. + * + * PCPF_PREV_FREE_HIGH_ORDER: a high-order page is freed in the + * previous page freeing. To avoid to drain PCP for an accident + * high-order page freeing. + */ +#define PCPF_PREV_FREE_HIGH_ORDER BIT(0) + struct per_cpu_pages { spinlock_t lock; /* Protects lists field */ int count; /* number of pages in the list */ int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ - short free_factor; /* batch scaling factor during free */ + u8 flags; /* protected by pcp->lock */ + u8 free_factor; /* batch scaling factor during free */ #ifdef CONFIG_NUMA short expire; /* When 0, remote pagesets are drained */ #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 95546f376302..295e61f0c49d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2370,7 +2370,7 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, { int high; int pindex; - bool free_high; + bool free_high = false; __count_vm_events(PGFREE, 1 << order); pindex = order_to_pindex(migratetype, order); @@ -2383,8 +2383,13 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, * freeing without allocation. The remainder after bulk freeing * stops will be drained from vmstat refresh context. */ - free_high = (pcp->free_factor && order && order <= PAGE_ALLOC_COSTLY_ORDER); - + if (order && order <= PAGE_ALLOC_COSTLY_ORDER) { + free_high = (pcp->free_factor && + (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER)); + pcp->flags |= PCPF_PREV_FREE_HIGH_ORDER; + } else if (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) { + pcp->flags &= ~PCPF_PREV_FREE_HIGH_ORDER; + } high = nr_pcp_high(pcp, zone, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, high, free_high), pcp, pindex);