From patchwork Fri Jun 21 07:54:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13706965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 434A6C27C4F for ; Fri, 21 Jun 2024 07:55:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3B9E6B012A; Fri, 21 Jun 2024 03:55:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC4B16B0132; Fri, 21 Jun 2024 03:55:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A170B6B0134; Fri, 21 Jun 2024 03:55:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 76B506B012A for ; Fri, 21 Jun 2024 03:55:13 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2B7301C194E for ; Fri, 21 Jun 2024 07:55:13 +0000 (UTC) X-FDA: 82254135306.10.0AA95CA Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) by imf08.hostedemail.com (Postfix) with ESMTP id E049B160014 for ; Fri, 21 Jun 2024 07:55:10 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="Mfzx4/5U"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.174 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718956503; a=rsa-sha256; cv=none; b=XGTj5fO0hrvuIgEHLwNhi7IEvb8SqzX8Zg2xaCHsTtmKO7wW7tFfSvWMmQVljJrhYWkTzi 4KU35zWRZDvnXmLdKaCbzuJmKZrdcxRUbECxenq1p3cAY03WGqpfJc+RHPOqKhIzDjO0pc 4eJy8JxoDIda0NjrL22YAOwR7erXwbM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="Mfzx4/5U"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.174 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718956503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cj0RSXpfKb2TCjjea56ybQ9iCdzPc+7EtlYBpcmbVzc=; b=vMHfpNA0bjKgjrnYHV0sDABjuvWoK2efw1O89S9Aof95vwtvZHmRn7oonAg6Ygtm2WP5/0 RNO8V6noKWKE1yGl7j+8D9aTuR/PUwPTECAUQbSSlkW9BzQbPk+tBbCMn7FcBOTvy8QXmy sY9IQj47BNRToCQEY21qk+3927pK6lw= X-Envelope-To: david@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1718956509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cj0RSXpfKb2TCjjea56ybQ9iCdzPc+7EtlYBpcmbVzc=; b=Mfzx4/5UkS4NcNG3exq4rh84KtS2OLmQAWPu7S7npdQtUGhQ/0Sbc21MlcdaIMZPVTwjMD ae2xHB0aTBiqDZZiONUhHFm/5RbR4jEIxq+9BDmuPoMzSoxYN+Dx1iX5S9/F7uyiQm/qLv AjVuWvtAwVyDuDnkwUXGouq15SeK+fU= X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: hughd@google.com X-Envelope-To: chengming.zhou@linux.dev X-Envelope-To: zhouchengming@bytedance.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: aarcange@redhat.com X-Envelope-To: shr@devkernel.io X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Fri, 21 Jun 2024 15:54:29 +0800 Subject: [PATCH v2 1/3] mm/ksm: refactor out try_to_merge_with_zero_page() MIME-Version: 1.0 Message-Id: <20240621-b4-ksm-scan-optimize-v2-1-1c328aa9e30b@linux.dev> References: <20240621-b4-ksm-scan-optimize-v2-0-1c328aa9e30b@linux.dev> In-Reply-To: <20240621-b4-ksm-scan-optimize-v2-0-1c328aa9e30b@linux.dev> To: Andrew Morton , david@redhat.com, aarcange@redhat.com, hughd@google.com, shr@devkernel.io Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1718956503; l=3224; i=chengming.zhou@linux.dev; s=20240617; h=from:subject:message-id; bh=XgUEfBzhfRP/ORf+1yxH34MeMyVcyA4uQSa5dx7WhmQ=; b=MeI/7pIXiWUSInmK68RxchUNLgqfVLCS2dSmEoaOE6psxVMHnvFbf/CCP6m1r9tkrbULcrm3S jcdjyQiAELQCDtQuG372QdBsbsz+DlLVK2VBPccqx3FDGVC5Ymv55y3 X-Developer-Key: i=chengming.zhou@linux.dev; a=ed25519; pk=/XPhIutBo+zyUeQyf4Ni5JYk/PEIWxIeUQqy2DYjmhI= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: E049B160014 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: itny9ozay8r7uonqxmw1igz3to4iqnkh X-HE-Tag: 1718956510-984510 X-HE-Meta: U2FsdGVkX1/k7WAP6syEXfvDA+jEz9yurjqgCl1UivJGPojO59qQmEQzDdxL364A6ssWyaEAdTRwmixTQEft7GhTsV0mq1wiEzXg3mtAdrtWIYs5+ckS/BRimjYjzhYUOHKi0LWkZ1ig3QrAfrjbStLdWRElFLvD7lGTtrU3TL1H+Ud3ZusAW8opIzsSYz2Vgg5myFG7Z7CjMiAwOfKF82Xo+nM8EHkp8agw/cLBr2E5noIV7A/gbgv59iZFgop1f9E+kUPvmM4u4v9Zd687QglaRQpFw+MAmIkiETBkfAvybFL2f2Og9BM7ijsCUvdiTO6vI62rGc72yhKxPiSZ9H4cWEZFiPSfyKmYA6vQZKufsGDl9M+QmSxYnN8dyaxNa63aYlGDb/aOzbBTGbCmKthDONSP6lv3SidE0L0zznz2pgSp4bgt17+kEuHOntaor76Mc61N54N9JPkA5/PKEGl/5B/zFysV05rfxuhAIEqjcc6nlj9t2rf7UVu7vFzyHkHzt9k0rLKdxbFYG/Y7UcngMPfoT7fwbdZ8dyoA2BXptcqq/Na4JbvnQHmeWrDAVAb55LK8xHAjmk4E1Ym+EaF+B5FnpGVGe0rpwlAYwrEWvqc9aVRoW9KXpIjNdwwpmlmEIsA+hf3Q1yGWGGhOT07J+88FMExRpnkaDvlogNQtHbchWIOaJFxPfuI/0FSb5d9tNJfjoRe8v8yZ9djtr9KF6SpkOVTy0bS95FWLqratVvJDSqf2PKs0L5yHgm7tl944mKqNGgDQdBVQ+g0g8Ywi7FafKeISPj2UugXylrpLRlSXPirHcciVpsITYWF+4DAMuWRtVeJBzA7fX3nGAapNHk09pOSUzwJW5Qiq6k8ahqv17+Z0gMO/0PI2SnFOB3ym7uueod3+PWa5LBhIfhNKRMcI+14NG2AtIHdCoDKuMnG/fTUYYNJPUc7lJpOhpRYudCUerE5v/TFIz4V 9gYiUy1w vXbBn9UGIWT6WYAvHWqQXla8tUcVs8gmk/EUv2XS4sPzoDlP3IXBcZseRu+q6EEdjR49w7MlhWhsj7NCuOiLd+CBYvmpIX9GKDxlFyRd23hZZWDWFx/Sxis2TJoGy9YLqkFWXN5NQHW6r7xeo2qpuJMgcxfvJPQPW5zKaof1GbixGtSWWLQOY7ZLhXYIjSxW+xy7TVbqPvNTyPSYjN8mUVq7VdY+si8iYe/3/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for later changes, refactor out a new function called try_to_merge_with_zero_page(), which tries to merge with zero page. Signed-off-by: Chengming Zhou --- mm/ksm.c | 70 ++++++++++++++++++++++++++++++++++++---------------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 34c4820e0d3d..1427abd18627 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1531,6 +1531,44 @@ static int try_to_merge_one_page(struct vm_area_struct *vma, return err; } +/* + * This function returns 0 if the pages were merged or if they are + * no longer merging candidates (e.g., VMA stale), -EFAULT otherwise. + */ +static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item, + struct page *page) +{ + struct mm_struct *mm = rmap_item->mm; + int err = -EFAULT; + + /* + * Same checksum as an empty page. We attempt to merge it with the + * appropriate zero page if the user enabled this via sysfs. + */ + if (ksm_use_zero_pages && (rmap_item->oldchecksum == zero_checksum)) { + struct vm_area_struct *vma; + + mmap_read_lock(mm); + vma = find_mergeable_vma(mm, rmap_item->address); + if (vma) { + err = try_to_merge_one_page(vma, page, + ZERO_PAGE(rmap_item->address)); + trace_ksm_merge_one_page( + page_to_pfn(ZERO_PAGE(rmap_item->address)), + rmap_item, mm, err); + } else { + /* + * If the vma is out of date, we do not need to + * continue. + */ + err = 0; + } + mmap_read_unlock(mm); + } + + return err; +} + /* * try_to_merge_with_ksm_page - like try_to_merge_two_pages, * but no new kernel page is allocated: kpage must already be a ksm page. @@ -2306,7 +2344,6 @@ static void stable_tree_append(struct ksm_rmap_item *rmap_item, */ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_item) { - struct mm_struct *mm = rmap_item->mm; struct ksm_rmap_item *tree_rmap_item; struct page *tree_page = NULL; struct ksm_stable_node *stable_node; @@ -2375,36 +2412,9 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite return; } - /* - * Same checksum as an empty page. We attempt to merge it with the - * appropriate zero page if the user enabled this via sysfs. - */ - if (ksm_use_zero_pages && (checksum == zero_checksum)) { - struct vm_area_struct *vma; + if (!try_to_merge_with_zero_page(rmap_item, page)) + return; - mmap_read_lock(mm); - vma = find_mergeable_vma(mm, rmap_item->address); - if (vma) { - err = try_to_merge_one_page(vma, page, - ZERO_PAGE(rmap_item->address)); - trace_ksm_merge_one_page( - page_to_pfn(ZERO_PAGE(rmap_item->address)), - rmap_item, mm, err); - } else { - /* - * If the vma is out of date, we do not need to - * continue. - */ - err = 0; - } - mmap_read_unlock(mm); - /* - * In case of failure, the page was not really empty, so we - * need to continue. Otherwise we're done. - */ - if (!err) - return; - } tree_rmap_item = unstable_tree_search_insert(rmap_item, page, &tree_page); if (tree_rmap_item) { From patchwork Fri Jun 21 07:54:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13706966 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31D68C27C4F for ; Fri, 21 Jun 2024 07:55:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F20306B0195; Fri, 21 Jun 2024 03:55:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CFCDD6B0190; Fri, 21 Jun 2024 03:55:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A173D6B018B; Fri, 21 Jun 2024 03:55:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6E8096B0179 for ; Fri, 21 Jun 2024 03:55:16 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 22E9D140A5A for ; Fri, 21 Jun 2024 07:55:16 +0000 (UTC) X-FDA: 82254135432.20.77DD57A Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf03.hostedemail.com (Postfix) with ESMTP id DE8C82000C for ; Fri, 21 Jun 2024 07:55:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=UsFO+6jv; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf03.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718956506; a=rsa-sha256; cv=none; b=p0sAYI+XWEviq/2O787Vu05Z0ocQACkGbGcW+nxBNJDpz7t270wgdjMZD4L4ytRyBL7B00 Dd2HgknGaHyjmaKP1OK+JNZDQS5u9B1o8oYmlpnajH/chvge3n34jbmtVCUElzTJhNH2gt LCz4YXy1D9am2vvUB8HRQwXo2l/71N8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=UsFO+6jv; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf03.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718956506; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Sw5yRtWHNEncebzoxSE8G3qpCoVdCgLAzOgxBxk0ok=; b=JGAG89fRJkL2z2fymFN2FXIOc4ODToUpm8AYnqN8FUWi5hTwpCmdvHkE2XfLK/swICQE4p zYMSbXmFhB2U79bBMHrRJir1WGl4JCol7TJ6dY1+RSrg8vdoBgAwzQPa5S9WoNXWwwtk2U yO01dbfmy23mUL3G2D06jtCDmoh1gYQ= X-Envelope-To: david@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1718956512; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Sw5yRtWHNEncebzoxSE8G3qpCoVdCgLAzOgxBxk0ok=; b=UsFO+6jvoTUIca3cM8LE6Q/FLwwPh2O7GrjHMbrKFS456cQzZL2wll0WrqklJKIbpOcaSh Od4d/qgiEbIfG4KtxQUenDjLLejLnjX7OCF5iw6KSalGLSin40c+pRTS6KtuLwPcNe4IRB MOoc6qc2MR7WB2d7s1utSWzdz6ivOHU= X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: hughd@google.com X-Envelope-To: chengming.zhou@linux.dev X-Envelope-To: zhouchengming@bytedance.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: aarcange@redhat.com X-Envelope-To: shr@devkernel.io X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Fri, 21 Jun 2024 15:54:30 +0800 Subject: [PATCH v2 2/3] mm/ksm: don't waste time searching stable tree for fast changing page MIME-Version: 1.0 Message-Id: <20240621-b4-ksm-scan-optimize-v2-2-1c328aa9e30b@linux.dev> References: <20240621-b4-ksm-scan-optimize-v2-0-1c328aa9e30b@linux.dev> In-Reply-To: <20240621-b4-ksm-scan-optimize-v2-0-1c328aa9e30b@linux.dev> To: Andrew Morton , david@redhat.com, aarcange@redhat.com, hughd@google.com, shr@devkernel.io Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1718956503; l=2616; i=chengming.zhou@linux.dev; s=20240617; h=from:subject:message-id; bh=CNJoKmOJG8f4fgj5rPkIJlw0KkteLIjSIgyHyFopqbI=; b=pKzOj1vYEVS8cLcJdoc/uLdiP+G/dkr/63SxIyokg2j6ABqeJkQHe1BneY9VkMaHaMJbezD2Y UKiUVY+NP24AbVV8kQrFJvtlLG8q10a1K7l3UEu74G5jaVQWrvAxiQR X-Developer-Key: i=chengming.zhou@linux.dev; a=ed25519; pk=/XPhIutBo+zyUeQyf4Ni5JYk/PEIWxIeUQqy2DYjmhI= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: DE8C82000C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5rrk4mmzetn8hu9rffjnaqbj8hquxopy X-HE-Tag: 1718956513-291102 X-HE-Meta: U2FsdGVkX1/akw2akBJ3e8GxtEkl7ooa08NzVIXYaVS9WYnEaB0r/MMpkc/kCCpPp7LYT+Y1JBX5glYCzxIt/guUiUMIqrqbV1SYO+TdNeU0yi0DYUXZp3m1jqJcp81aXHtB6ipMFN7V+GMxt5xrawFxgjvO/J6oupm7ak7GQH5x7s5QxXi671lI2vLkeIPYioAIQo4LtUns4Cez39i3VeHuSUy3mUz2wIwv72UUSr3ZOU825andW0LIGB+EfFiu1cpa22O24KaNhD01O9SPP/I0K/bSt9FDWD0A9JkIGon2xs1h6yd6M1Sf96XQ6bsPJbP9V/yCOzVlCArXldLowdRAwTUQnL7fiR2/7LvLzYe8qF+niL6GBlK1EWBGY7g5noISA7SSgjw5UnCs4bY0SlvN/N0T7js9kmCQeihhR7D3+SEbPp9U1uiO+u/QQJJ9glGSIhQCc9dsIcIFiYH7CE0yFsnjT+40SbEETaEUEPbCS0GFZBeCQFvb5swMmGZwoktIjn3PXPm/0uZD6ZFGodJAHvc62M89Ns6Q2bVdBRRBJ/IXi8OBHus/XWMQkomJ2RIYWmcSpwYH2q8fWiuwnXpsVP/EORJomkIaVQCiku8ippRwkozB7UAbztqvNq9HTBKBJL5AssHg+2bLjj3LtLK1xQ3tOzYQntFRhh/dmsCNh24kWoZC8we8rF1VM3h5pNyLVyiO3kdsHAJ8LEqU1hwME9QHanHzK4nDrZz8K8MgwCUg38DV9t3XuLI90aT9Dy1hXnvTxazejoXS4Vrh5gmCUNQd32JymRV/KfS5HDJtr86ST06DjTLKDiFIuAFYJ6fg9lvtN7f1qm2vXCv+4MGeeQOZ+o4erkeoXE7at5Fl+d8FyjI+HZK7BL5MDO2nuOCEBWGf3uGcf3AocxENNgRgUj6Cb5RXQ4RyEMYHPS0RWAKsRtbPWaM+d3Zhv1MI8cGWOtBPZ9g2bo+trOu 6qTGn0qD kJMp9O4w7S4rkXbnetzyGwywOvB+SNMjnVqmnmWMS5/BCINNAF6N8300oBPQ5FHjZS9IOU4325KajAfQe4LcAZZfTGt/f18W2gOCmFF5Tzse4lzaKosBvTVZiv1oGNDi+Ywd1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000067, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The code flow in cmp_and_merge_page() is suboptimal for handling the ksm page and non-ksm page at the same time. For example: - ksm page 1. Mostly just return if this ksm page is not migrated and this rmap_item has been on the rmap hlist. Or we have to fix this rmap_item mapping. 2. But we absolutely don't need to checksum for this ksm page, since it can't change. - non-ksm page 1. First don't need to waste time searching stable tree if fast changing. 2. Should try to merge with zero page before search the stable tree. 3. Then search stable tree to find mergeable ksm page. This patch optimizes the code flow so the handling differences between ksm page and non-ksm page become clearer and more efficient too. Signed-off-by: Chengming Zhou --- mm/ksm.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 1427abd18627..2cf836fb1367 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2370,6 +2370,23 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite */ if (!is_page_sharing_candidate(stable_node)) max_page_sharing_bypass = true; + } else { + remove_rmap_item_from_tree(rmap_item); + + /* + * If the hash value of the page has changed from the last time + * we calculated it, this page is changing frequently: therefore we + * don't want to insert it in the unstable tree, and we don't want + * to waste our time searching for something identical to it there. + */ + checksum = calc_checksum(page); + if (rmap_item->oldchecksum != checksum) { + rmap_item->oldchecksum = checksum; + return; + } + + if (!try_to_merge_with_zero_page(rmap_item, page)) + return; } /* We first start with searching the page inside the stable tree */ @@ -2400,21 +2417,6 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite return; } - /* - * If the hash value of the page has changed from the last time - * we calculated it, this page is changing frequently: therefore we - * don't want to insert it in the unstable tree, and we don't want - * to waste our time searching for something identical to it there. - */ - checksum = calc_checksum(page); - if (rmap_item->oldchecksum != checksum) { - rmap_item->oldchecksum = checksum; - return; - } - - if (!try_to_merge_with_zero_page(rmap_item, page)) - return; - tree_rmap_item = unstable_tree_search_insert(rmap_item, page, &tree_page); if (tree_rmap_item) { From patchwork Fri Jun 21 07:54:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13706967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAAF6C2BA1A for ; Fri, 21 Jun 2024 07:55:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8C418D0143; Fri, 21 Jun 2024 03:55:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A18F78D0138; Fri, 21 Jun 2024 03:55:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 780888D0143; Fri, 21 Jun 2024 03:55:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4DC728D0138 for ; Fri, 21 Jun 2024 03:55:19 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E153080AEB for ; Fri, 21 Jun 2024 07:55:18 +0000 (UTC) X-FDA: 82254135516.20.5152834 Received: from out-188.mta0.migadu.com (out-188.mta0.migadu.com [91.218.175.188]) by imf16.hostedemail.com (Postfix) with ESMTP id 8BF4018000E for ; Fri, 21 Jun 2024 07:55:16 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=TnGrkbwW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718956509; a=rsa-sha256; cv=none; b=TCFexkeJghXh0JIHYZO3t1hPohpA4FjcWKJtsVli3Ok+l/ErzgZfsiMqNXSeHZgwoRVE6G GSMBX3d4L56vRpERR3vXotooXHx0G0aKdzQ1uCo+sWFUPy9rB37uxiK+DegtRdVRW0sVZa vqbFEIQR+6N5XHAw/YnLOjzOmJWz++U= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=TnGrkbwW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718956509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aBTb4+v7Qjp9h/YfZziY90UbHgzMj5tgQ+gOTK4dIwk=; b=Azr8UUfsD0EM1m7dv7Yd7TyXdSYm/2y5k9yzn/haCevVr7b41ZEbhCP+Tsi/nlwsrcxdTu lnryUkHXcFB1NNV2Iwfo8QoUTh4VfzSCEXF5e5KLTjqeCIlukndxxckzQDy4E1Y2hPWKCO gTdY6+z9nb7UroHqAbHXXFCbKpNUPdk= X-Envelope-To: david@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1718956515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aBTb4+v7Qjp9h/YfZziY90UbHgzMj5tgQ+gOTK4dIwk=; b=TnGrkbwW436/Yap5Ue4H98Atcpl4VB9QMsg8iPSTtY0CJAXvj4UHSLInQ2RlqUfxsKFoF0 y2NKHI+3D+4ZH7Ax3Q5YK+ZakWJ7xkP7qExCAAugC9UzcRh4vSq5ZVQGqDArJoWkafB66U VFv5AmeaGMnhhI7TBLLiPwkFjgCsiQ0= X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: hughd@google.com X-Envelope-To: chengming.zhou@linux.dev X-Envelope-To: zhouchengming@bytedance.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: aarcange@redhat.com X-Envelope-To: shr@devkernel.io X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Fri, 21 Jun 2024 15:54:31 +0800 Subject: [PATCH v2 3/3] mm/ksm: optimize the chain()/chain_prune() interfaces MIME-Version: 1.0 Message-Id: <20240621-b4-ksm-scan-optimize-v2-3-1c328aa9e30b@linux.dev> References: <20240621-b4-ksm-scan-optimize-v2-0-1c328aa9e30b@linux.dev> In-Reply-To: <20240621-b4-ksm-scan-optimize-v2-0-1c328aa9e30b@linux.dev> To: Andrew Morton , david@redhat.com, aarcange@redhat.com, hughd@google.com, shr@devkernel.io Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1718956503; l=9550; i=chengming.zhou@linux.dev; s=20240617; h=from:subject:message-id; bh=aNZtikh89uZPUxDF7c2Rld2Ly+4aFi7EXpdx4+SXheY=; b=bOdA9BYysMkUH2+JdBV45OcwVMOTbDIiKLccdbJ5jZ1yl2TbQ/Jxd6AGsGx7IIzzdL+Vo41Jy foNRzuyFJIABRcXbo9+eILkdIBTZgC116f0rFUtBWPJbukeeK2Go5wH X-Developer-Key: i=chengming.zhou@linux.dev; a=ed25519; pk=/XPhIutBo+zyUeQyf4Ni5JYk/PEIWxIeUQqy2DYjmhI= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8BF4018000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: e7d7xnzood11efw5direwq7bk6gq1s1f X-HE-Tag: 1718956516-977254 X-HE-Meta: U2FsdGVkX1+/MT59YpC4ywCqZwyzRL6d74gYQy4T58hn+Bn5WwEkCUlgB5mIdnezmkWH8trZpXiomQsz/2FqNkD72T79QW49Axytj7YC8AxFlmkNOVZtOwMtx01IxSs8cJPa+hFkHpJhPhN75jHEFg8ZZKxujxHwrptzBVZEi0EHfEjEOVuU1uJHQScWeBuRTQQItow4P2x5duOLSyjLSJHW21Pzhe3TTKOZNRhigyyO2yZeohXqlyvLTflpFasYVjS3riWToh30WgJlWifCVyetTw3iqEvvJn836JGSFim7so+aLGclNR41u6S6X+40Q7ge4aVBqfk1FdquPscgvY8nrWh1WczRR3dH+Ls6sE1SFZjylZ8EBlOgkiw0dIUYZgr1A+6iQ8bN5JuSdAnTeh1arYYH9+tCPbI1qE4yfVBXotEBqW0NesHzDNfwseZpVH3mUHCpeCWbb+OwMxpM3DIV2hoBHqPZm9C04z3o0DtH7v7qhf1vtnwANhoHPV5F7dWtTQ1kWALJNyov1Ky/GDvsQlg29vWxXjXY8zM/Zwk788lgeMp7cWA8Aps3L2BmSDzD6SzIxzXKI1tMJiXVlTbz7DYP4r/koZkeDgRbive9XDG+X/ZFRumM/s1HD4S9/kr3LTgx1QYs214NrPAtP6RpRQpAWIgl+eWuubwEpluDzKt1wuIWC+roKdsfVJhqENHWF+8uIxPGwK7H0p5jSE5NQ3gUuhtsqswXn1NzNXPIuDOoRuPY4KJKf2GOJDxPiGxhplnIo9tcQqQePbNgOUnKRW5acj9W4aCAvbUUPsDLTgE+AG+FZcRLDIQR8xnxRmFa69CWqElT5Pwv9BbQCwXkAV55pfqVQCVfyD0D8iVfNzoH5tuUKeOx2Ova1kkfna0cY0zsBWup0bOLQ4jOd+zy4glo+1CVsGOc8Y6vNgEcw26+adp2SbZk9BJqHVhkE9E0Ybtt0QHtt3cai35 6HS6B1om UQMZrDgdswf0uGkdZIcw7SN6pQxHFmq6MSXpKPE2kBhDQz3hEMe95I1KIqlFd8w9bn+Voj/yebTwXPVpz4GFTMARPLkc4DQL8TlSVeM4wrqg2oZAHqid5Wh5baYQQUXBpWYSW2n9ZF33lEceCIz7ao1tpsA57P4bC9mbxie1jDnylVag= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now the implementation of stable_node_dup() causes chain()/chain_prune() interfaces and usages are overcomplicated. Why? stable_node_dup() only find and return a candidate stable_node for sharing, so the users have to recheck using stable_node_dup_any() if any non-candidate stable_node exist. And try to ksm_get_folio() from it again. Actually, stable_node_dup() can just return a best stable_node as it can, then the users can check if it's a candidate for sharing or not. The code is simplified too and fewer corner cases: such as stable_node and stable_node_dup can't be NULL if returned tree_folio is not NULL. Signed-off-by: Chengming Zhou --- mm/ksm.c | 152 ++++++++++++--------------------------------------------------- 1 file changed, 27 insertions(+), 125 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 2cf836fb1367..8a5d88472223 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1663,7 +1663,6 @@ static struct folio *stable_node_dup(struct ksm_stable_node **_stable_node_dup, struct ksm_stable_node *dup, *found = NULL, *stable_node = *_stable_node; struct hlist_node *hlist_safe; struct folio *folio, *tree_folio = NULL; - int nr = 0; int found_rmap_hlist_len; if (!prune_stale_stable_nodes || @@ -1690,33 +1689,26 @@ static struct folio *stable_node_dup(struct ksm_stable_node **_stable_node_dup, folio = ksm_get_folio(dup, KSM_GET_FOLIO_NOLOCK); if (!folio) continue; - nr += 1; - if (is_page_sharing_candidate(dup)) { - if (!found || - dup->rmap_hlist_len > found_rmap_hlist_len) { - if (found) - folio_put(tree_folio); - found = dup; - found_rmap_hlist_len = found->rmap_hlist_len; - tree_folio = folio; - - /* skip put_page for found dup */ - if (!prune_stale_stable_nodes) - break; - continue; - } + /* Pick the best candidate if possible. */ + if (!found || (is_page_sharing_candidate(dup) && + (!is_page_sharing_candidate(found) || + dup->rmap_hlist_len > found_rmap_hlist_len))) { + if (found) + folio_put(tree_folio); + found = dup; + found_rmap_hlist_len = found->rmap_hlist_len; + tree_folio = folio; + /* skip put_page for found candidate */ + if (!prune_stale_stable_nodes && + is_page_sharing_candidate(found)) + break; + continue; } folio_put(folio); } if (found) { - /* - * nr is counting all dups in the chain only if - * prune_stale_stable_nodes is true, otherwise we may - * break the loop at nr == 1 even if there are - * multiple entries. - */ - if (prune_stale_stable_nodes && nr == 1) { + if (hlist_is_singular_node(&found->hlist_dup, &stable_node->hlist)) { /* * If there's not just one entry it would * corrupt memory, better BUG_ON. In KSM @@ -1768,25 +1760,15 @@ static struct folio *stable_node_dup(struct ksm_stable_node **_stable_node_dup, hlist_add_head(&found->hlist_dup, &stable_node->hlist); } + } else { + /* Its hlist must be empty if no one found. */ + free_stable_node_chain(stable_node, root); } *_stable_node_dup = found; return tree_folio; } -static struct ksm_stable_node *stable_node_dup_any(struct ksm_stable_node *stable_node, - struct rb_root *root) -{ - if (!is_stable_node_chain(stable_node)) - return stable_node; - if (hlist_empty(&stable_node->hlist)) { - free_stable_node_chain(stable_node, root); - return NULL; - } - return hlist_entry(stable_node->hlist.first, - typeof(*stable_node), hlist_dup); -} - /* * Like for ksm_get_folio, this function can free the *_stable_node and * *_stable_node_dup if the returned tree_page is NULL. @@ -1807,17 +1789,10 @@ static struct folio *__stable_node_chain(struct ksm_stable_node **_stable_node_d bool prune_stale_stable_nodes) { struct ksm_stable_node *stable_node = *_stable_node; + if (!is_stable_node_chain(stable_node)) { - if (is_page_sharing_candidate(stable_node)) { - *_stable_node_dup = stable_node; - return ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK); - } - /* - * _stable_node_dup set to NULL means the stable_node - * reached the ksm_max_page_sharing limit. - */ - *_stable_node_dup = NULL; - return NULL; + *_stable_node_dup = stable_node; + return ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK); } return stable_node_dup(_stable_node_dup, _stable_node, root, prune_stale_stable_nodes); @@ -1831,16 +1806,10 @@ static __always_inline struct folio *chain_prune(struct ksm_stable_node **s_n_d, } static __always_inline struct folio *chain(struct ksm_stable_node **s_n_d, - struct ksm_stable_node *s_n, + struct ksm_stable_node **s_n, struct rb_root *root) { - struct ksm_stable_node *old_stable_node = s_n; - struct folio *tree_folio; - - tree_folio = __stable_node_chain(s_n_d, &s_n, root, false); - /* not pruning dups so s_n cannot have changed */ - VM_BUG_ON(s_n != old_stable_node); - return tree_folio; + return __stable_node_chain(s_n_d, s_n, root, false); } /* @@ -1858,7 +1827,7 @@ static struct page *stable_tree_search(struct page *page) struct rb_root *root; struct rb_node **new; struct rb_node *parent; - struct ksm_stable_node *stable_node, *stable_node_dup, *stable_node_any; + struct ksm_stable_node *stable_node, *stable_node_dup; struct ksm_stable_node *page_node; struct folio *folio; @@ -1882,45 +1851,7 @@ static struct page *stable_tree_search(struct page *page) cond_resched(); stable_node = rb_entry(*new, struct ksm_stable_node, node); - stable_node_any = NULL; tree_folio = chain_prune(&stable_node_dup, &stable_node, root); - /* - * NOTE: stable_node may have been freed by - * chain_prune() if the returned stable_node_dup is - * not NULL. stable_node_dup may have been inserted in - * the rbtree instead as a regular stable_node (in - * order to collapse the stable_node chain if a single - * stable_node dup was found in it). In such case the - * stable_node is overwritten by the callee to point - * to the stable_node_dup that was collapsed in the - * stable rbtree and stable_node will be equal to - * stable_node_dup like if the chain never existed. - */ - if (!stable_node_dup) { - /* - * Either all stable_node dups were full in - * this stable_node chain, or this chain was - * empty and should be rb_erased. - */ - stable_node_any = stable_node_dup_any(stable_node, - root); - if (!stable_node_any) { - /* rb_erase just run */ - goto again; - } - /* - * Take any of the stable_node dups page of - * this stable_node chain to let the tree walk - * continue. All KSM pages belonging to the - * stable_node dups in a stable_node chain - * have the same content and they're - * write protected at all times. Any will work - * fine to continue the walk. - */ - tree_folio = ksm_get_folio(stable_node_any, - KSM_GET_FOLIO_NOLOCK); - } - VM_BUG_ON(!stable_node_dup ^ !!stable_node_any); if (!tree_folio) { /* * If we walked over a stale stable_node, @@ -1958,7 +1889,7 @@ static struct page *stable_tree_search(struct page *page) goto chain_append; } - if (!stable_node_dup) { + if (!is_page_sharing_candidate(stable_node_dup)) { /* * If the stable_node is a chain and * we got a payload match in memcmp @@ -2067,9 +1998,6 @@ static struct page *stable_tree_search(struct page *page) return &folio->page; chain_append: - /* stable_node_dup could be null if it reached the limit */ - if (!stable_node_dup) - stable_node_dup = stable_node_any; /* * If stable_node was a chain and chain_prune collapsed it, * stable_node has been updated to be the new regular @@ -2114,7 +2042,7 @@ static struct ksm_stable_node *stable_tree_insert(struct folio *kfolio) struct rb_root *root; struct rb_node **new; struct rb_node *parent; - struct ksm_stable_node *stable_node, *stable_node_dup, *stable_node_any; + struct ksm_stable_node *stable_node, *stable_node_dup; bool need_chain = false; kpfn = folio_pfn(kfolio); @@ -2130,33 +2058,7 @@ static struct ksm_stable_node *stable_tree_insert(struct folio *kfolio) cond_resched(); stable_node = rb_entry(*new, struct ksm_stable_node, node); - stable_node_any = NULL; - tree_folio = chain(&stable_node_dup, stable_node, root); - if (!stable_node_dup) { - /* - * Either all stable_node dups were full in - * this stable_node chain, or this chain was - * empty and should be rb_erased. - */ - stable_node_any = stable_node_dup_any(stable_node, - root); - if (!stable_node_any) { - /* rb_erase just run */ - goto again; - } - /* - * Take any of the stable_node dups page of - * this stable_node chain to let the tree walk - * continue. All KSM pages belonging to the - * stable_node dups in a stable_node chain - * have the same content and they're - * write protected at all times. Any will work - * fine to continue the walk. - */ - tree_folio = ksm_get_folio(stable_node_any, - KSM_GET_FOLIO_NOLOCK); - } - VM_BUG_ON(!stable_node_dup ^ !!stable_node_any); + tree_folio = chain(&stable_node_dup, &stable_node, root); if (!tree_folio) { /* * If we walked over a stale stable_node,