From patchwork Mon Apr 14 22:05:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 14051083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1220EC369B5 for ; Mon, 14 Apr 2025 22:07:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8399A28009D; Mon, 14 Apr 2025 18:07:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BF3A28009C; Mon, 14 Apr 2025 18:07:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63BD128009D; Mon, 14 Apr 2025 18:07:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4043E28009C for ; Mon, 14 Apr 2025 18:07:33 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0FB22140746 for ; Mon, 14 Apr 2025 22:07:34 +0000 (UTC) X-FDA: 83334036828.05.4C2FAE5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 54383A0003 for ; Mon, 14 Apr 2025 22:07:32 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EPqDLzkk; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744668452; a=rsa-sha256; cv=none; b=3sfZNSMgaV31FPrXjsdeEFPQVsrHpimfeYCXhmL5H0QQtizltsjy0cs01aKyeicHArO+XG aDxoKaxniBzo55Q20L4C+ahl3g22ZbUEo95QU14xWJoiJfE2L9MtNHWj0vEmm5uRtC/2Z4 dc76BV8ehq1jzGrrXuLADuqD4wrL1vw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EPqDLzkk; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744668452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mFXVbXpUlZP3jDzCFrP5W8zsGmKJ7GyJXlGGdBRRcJ8=; b=2Yij+KTlMKzMJFOJXhSZEYBJN6SY2RAdSitIXO+VpeYNLHD54cO+8IdYpn9+JZ8iLiW6ye uoLEI0HIB0vMu2H67+Hy8J6ng5PXX4JX64yWjd1Yfi+Hz9EVYhgwk/CbhSjOR2GLAMYRjA gedoMHNxVwulm4kFI89NrxgCR70VTsw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1744668451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mFXVbXpUlZP3jDzCFrP5W8zsGmKJ7GyJXlGGdBRRcJ8=; b=EPqDLzkkQax1vkVs3VZzmhemWwq2XhloK+q3IVdCTDCeuArlCtdo+3USvLLKKqcHh6A8aD UxUSSzuEVI3U5NPhY1lY+l6lmA8+aqSPAROc5nMMdkHDFb6bp+OevA7QckASuBp5yg238f Ep9qu6KoKMdPjh5u8dQPyJ+pgzREDic= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-345-lIUx88k_MvqQzURPmO4p-g-1; Mon, 14 Apr 2025 18:07:27 -0400 X-MC-Unique: lIUx88k_MvqQzURPmO4p-g-1 X-Mimecast-MFC-AGG-ID: lIUx88k_MvqQzURPmO4p-g_1744668443 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 76B1B1800EC5; Mon, 14 Apr 2025 22:07:23 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.64.91]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DB93E1955BC2; Mon, 14 Apr 2025 22:07:15 +0000 (UTC) From: Nico Pache To: linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, corbet@lwn.net, rostedt@goodmis.org, mhiramat@kernel.org, mathieu.desnoyers@efficios.com, david@redhat.com, baohua@kernel.org, baolin.wang@linux.alibaba.com, ryan.roberts@arm.com, willy@infradead.org, peterx@redhat.com, ziy@nvidia.com, wangkefeng.wang@huawei.com, usamaarif642@gmail.com, sunnanyong@huawei.com, vishal.moola@gmail.com, thomas.hellstrom@linux.intel.com, yang@os.amperecomputing.com, kirill.shutemov@linux.intel.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, tiwai@suse.de, will@kernel.org, dave.hansen@linux.intel.com, jack@suse.cz, cl@gentwo.org, jglisse@google.com, surenb@google.com, zokeefe@google.com, hannes@cmpxchg.org, rientjes@google.com, mhocko@suse.com Subject: [PATCH v3 06/12] khugepaged: introduce khugepaged_scan_bitmap for mTHP support Date: Mon, 14 Apr 2025 16:05:51 -0600 Message-ID: <20250414220557.35388-7-npache@redhat.com> In-Reply-To: <20250414220557.35388-1-npache@redhat.com> References: <20250414220557.35388-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 54383A0003 X-Stat-Signature: n6jbkqkwn1n7ytgdrwopm33nw18udre1 X-HE-Tag: 1744668452-884708 X-HE-Meta: U2FsdGVkX1/UTkh/gLv02RVBPZMCaA3p98K/XVihUYB3tVnVs/xvBqNOKF52zlgWTsM2ogO5YrcZRfNoODaUWGPHMOIHCF2QjkdgxgJNH2aU7clftQ2IH+OxDpTXwsYwiKCcpBbA91LKRYEf8qOA+/R3Z6ytQdRYAHoG6nldS7Lqgf1viNnUc2z36hhxdzI642vHWiN7NuUR+1+8CsYUWEOmWsurliLUg/sOcMsSL3gKhGmZKoE8ukDquuErL/dOqXe0t/v27Lak3vDZWOoc3o17MPqNyKyJ0ngRgJTkD4qFmkGbLxFYraE4yMW1wlG0agwF8FO+nr8l+qLC1a94d3ZM+3ZME7rkxWchrYbpwOD6GcygNnPT4I2AvmV1B/zytGQrXCAPL0jSrrBKyLjiMxx96on8oZH32/g6Lzxg6yLrlFqdaUMZ6OpMsEjWF7clnP/lDmwymqi3+qYFKTigZ1o7Zyyi2MB1D+MSfPOLkYIe5IzdwA9v71C359qBCEIhAZ9YfcDIjMHv8hj5vtf3Ea54uq7980aGaIP4SQCxhFfWQTzkPioSaO1uuOFxxxr4R86sRTgNCNHlo5znakRuqfNwd4NzG0N7vjz/+54SMDgRd/CV69iAz7OQv5832zshjH8R1O9Uo48Xaj6ed6B43iQIJ02poicHrQZGwDF3n/tA7Wx30MqfJVscZyAbBEfAP77HRgQVJM+lZNMY+Ygh+Z1PpRK2+a1O03IbYQqD2qi7ZCBjhd1AOq0rZg25vKyGEGaiLHbfhqlbQ9rIzXmSvGpbPbDjcEBsIvW625IBS34Ml5zMYiFVHFj9BpmotHuDW+bQP1pGXLWUReQ2oocKVUrnynxZKcQUxSdciKbNKcNz/sne0I1wbOlUdifJI0mmddb9hg/FDm0Bkue41oyZuLWTGRPMRbw/+KyfMlfOhJKT5mhmGLXCsxFQzJR5XxnkdZ+FUaD0zF9xFNq3fP7 mNlzuQUG sKkSqgTREsBKvONh7aWos9oXRToqf8pwWjfkYaXR9hKGzr94O9xumwCoyd/2WOCqZfgfkrnwTVcvrbbKPSnGMZabWCQiuDQv9/fd4OvelIrbiuR/G0ksRUog77wxDJYH4nUdyfW+NI5CHrQ2dcfbZR1MITuW76vEXIPXTt0gmXL0eXG2R4fgE7x8c06grP1DLOiY5XzMdiP1Gt/fQL6vZ12vhpKMkt6ytjni7TEMF50ZJBv1XkFux3LEhxJ9LKMRjlpkYIER9uC/ZVvoFT1GohWVm9CSAKFSas7e0UMYs+o/u8fUgtRlrQZ8mKkov4ZC9AH/Hg0qi+8ho7AgC0GA2CKWldTeszZwaiyOtBZboMbGwmpDl0/A6GoRksIEmpM3w22l42vZJlrBOwmiQw2wOcu11/5/BdI6edoNWYgDCoOs9RlbSdOUVqF6i3Voj0RdlbWHiidDsw1V7RqY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: khugepaged scans PMD ranges for potential collapse to a hugepage. To add mTHP support we use this scan to instead record chunks of fully utilized sections of the PMD. create a bitmap to represent a PMD in order MIN_MTHP_ORDER chunks. by default we will set this to order 3. The reasoning is that for 4K 512 PMD size this results in a 64 bit bitmap which has some optimizations. For other arches like ARM64 64K, we can set a larger order if needed. khugepaged_scan_bitmap uses a stack struct to recursively scan a bitmap that represents chunks of utilized regions. We can then determine what mTHP size fits best and in the following patch, we set this bitmap while scanning the PMD. max_ptes_none is used as a scale to determine how "full" an order must be before being considered for collapse. If a order is set to "always" lets always collapse to that order in a greedy manner. Signed-off-by: Nico Pache --- include/linux/khugepaged.h | 4 ++ mm/khugepaged.c | 94 ++++++++++++++++++++++++++++++++++---- 2 files changed, 89 insertions(+), 9 deletions(-) diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 1f46046080f5..60d41215bc1a 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -1,6 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H #define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 3 +#define KHUGEPAGED_MIN_MTHP_NR (1<mthp_bitmap_stack[++top] = (struct scan_bit_state) + { HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER, 0 }; + + while (top >= 0) { + state = cc->mthp_bitmap_stack[top--]; + order = state.order + KHUGEPAGED_MIN_MTHP_ORDER; + offset = state.offset; + num_chunks = 1 << (state.order); + // Skip mTHP orders that are not enabled + if (!test_bit(order, &enabled_orders)) + goto next; + + // copy the relavant section to a new bitmap + bitmap_shift_right(cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, + MTHP_BITMAP_SIZE); + + bits_set = bitmap_weight(cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none - 1) + >> (HPAGE_PMD_ORDER - state.order); + + //Check if the region is "almost full" based on the threshold + if (bits_set > threshold_bits || is_pmd_only + || test_bit(order, &huge_anon_orders_always)) { + ret = collapse_huge_page(mm, address, referenced, unmapped, cc, + mmap_locked, order, offset * KHUGEPAGED_MIN_MTHP_NR); + if (ret == SCAN_SUCCEED) { + collapsed += (1 << order); + continue; + } + } + +next: + if (state.order > 0) { + next_order = state.order - 1; + mid_offset = offset + (num_chunks / 2); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state) + { next_order, offset }; + } + } + return collapsed; +} + static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked, @@ -1445,9 +1523,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { result = collapse_huge_page(mm, address, referenced, - unmapped, cc); - /* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + unmapped, cc, mmap_locked, HPAGE_PMD_ORDER, 0); } out: trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced,