From patchwork Tue Feb 11 00:30:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Pache X-Patchwork-Id: 13968640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C63B6C02198 for ; Tue, 11 Feb 2025 00:32:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5347128000F; Mon, 10 Feb 2025 19:32:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E3A3280008; Mon, 10 Feb 2025 19:32:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 384C628000F; Mon, 10 Feb 2025 19:32:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1683B280008 for ; Mon, 10 Feb 2025 19:32:16 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C36A01C7DD2 for ; Tue, 11 Feb 2025 00:32:15 +0000 (UTC) X-FDA: 83105787030.12.2EAC112 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id E99AE20003 for ; Tue, 11 Feb 2025 00:32:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PF4jJhrD; spf=pass (imf03.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739233934; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7fUONtMIJRjZ+fpEsfBAOSqHGTOAUFKQgLDIE5UlCPg=; b=X7hozsdUEgVVvMci1wPQ5r+5KLjE145hacknJfgCZwF06hzyY1ePYotOaRAZXqIKncxWUx WU0AXvXZSVhfHXkr+rGqOQmXEIvwYb13bmPhOrME4LHFLSwO3fJQGNV2Y+Ob99yxVCBH9N D0Z3jCDg6vNbL1ev8QFG0mSjXgqZQic= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PF4jJhrD; spf=pass (imf03.hostedemail.com: domain of npache@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739233934; a=rsa-sha256; cv=none; b=hRlMPMccK4rgRtYii+hHK0FV3YKdI9OevkogikE4x9f61KKtm9bBIB0z73gjIqgtC45ev2 3MXTDjK+5NKqarxfM7tmvP1rGbGJQ3NrMB8xAN3unNttuLGbmoHwa/KGKtUhvv8O9i/hJR ebvofogoM0R2VtQwEbLlbBxLSL4Wh/Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739233933; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7fUONtMIJRjZ+fpEsfBAOSqHGTOAUFKQgLDIE5UlCPg=; b=PF4jJhrDB4yq6VoQwjBMiN6G3qaNWxyUCSEIRx5Ord7/dhfxOltYBEAsJZHBOwP0exdzUs VYJcm9aaI/KNwcymyjJSSJEwy4L8GgNDeMRergTB21YOCG0kpMK3Iqs0KUMUdzkI5F+x/j V+PoQSZ39uKwr+7CVe6u+1HTQjwIuMA= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-607-AKmiACISPX-LKpWM4wDUDg-1; Mon, 10 Feb 2025 19:32:09 -0500 X-MC-Unique: AKmiACISPX-LKpWM4wDUDg-1 X-Mimecast-MFC-AGG-ID: AKmiACISPX-LKpWM4wDUDg Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 669A519560AA; Tue, 11 Feb 2025 00:32:05 +0000 (UTC) Received: from h1.redhat.com (unknown [10.22.88.129]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 80DDA1800358; Tue, 11 Feb 2025 00:31:55 +0000 (UTC) From: Nico Pache To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, willy@infradead.org, kirill.shutemov@linux.intel.com, david@redhat.com, aarcange@redhat.com, raquini@redhat.com, dev.jain@arm.com, sunnanyong@huawei.com, usamaarif642@gmail.com, audra@redhat.com, akpm@linux-foundation.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, tiwai@suse.de Subject: [RFC v2 6/9] khugepaged: introduce khugepaged_scan_bitmap for mTHP support Date: Mon, 10 Feb 2025 17:30:25 -0700 Message-ID: <20250211003028.213461-7-npache@redhat.com> In-Reply-To: <20250211003028.213461-1-npache@redhat.com> References: <20250211003028.213461-1-npache@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E99AE20003 X-Stat-Signature: 1ns16dndfbfsmwwgjjgty3cbn4fbqhre X-HE-Tag: 1739233933-235743 X-HE-Meta: U2FsdGVkX1/9gA0hY6pfdGcElYwZDzLeke9JRm5vZfj0Z6NlZ6I8wv66n0HKQciLALH8JIAUbfNbvZM2N4SI3dpYXXrx4vHRsUQYTnQl0EqyWYL44FWq3uNzoc4zXLQwf52mUKCWXA1clk5jzbB7jzfVKUsszsJGqB0cWMEckUYJNQ2oE8d0/gRDefJyHxxeZFiRjhnDszXlzWdnkxR5zadrN7Fk2CM15bRZYacNFhfoA311nzVtjg3bGzVZf1Xo8qBV+NaqhjRY1ihay4YBW68DYAg3aGxdcBKE+gYN0SQvppk2REhxHXh4OFQldZUctP6BI+IxD8UUpBbKkXiZfB05e5O3nt78E1wOXz1qhAX3AiDuOWSOxohEGqwn7ch5BGmAeAJeDCEouJRUkVnqg1Jbk5n/9JJTN47Ygqe3HCXmsQgtCjBMtiB8KJ/3yEM90jKwvNzoCnyW4XMm6AlgmtEkRoVTpN1Yc3/Y1HmeQ/djYc0LzjsU95Z2YEEwRRV40zFUZQknnBKlyqbbOqcxXNLJcYxwgbI685dr/MqCB/QrJ8vVy41QOE+POw0vbPxiMjFjuZMcpjZ7vPyYfOaFtO8S0n60izwpsP0CG4q/Mq1FeFUdQLpL9AUVb7l0EPnnwy0loYk9VINp4wjez4ZXb9R9GaPrk7kh3i+FMNxVWMikLPYApew9ilBjOiqYkRoyvIo8wirEeIVM15J0tN74EMsfpQGu1x0eewW5bwen4ajFJBfrs92tBr7XI3OZUTK8kadMBXX+BA85qIx8gsINzwaAAjK3tpTMEGWiOKF8P9icLKVLj2/xZMZrALGfC9gPltq+UGD5Wr0TiugfxbaTy4aiy4tpe60akmXVAk8PajWjB8L/qqF1dliQxosXqomisedY/vOOXHM+ZKjLRudktuMAgBwNf4TufRuC2Yn81Kdx4E2XGuZ1TJP2sj9je1e8pDkB3ABvrF8mW7rgfjb IimIyFO3 I1YQDEewXGO8fRjvXF4B0p92GOb6msZ3c9gz4V/JfJEfmWhg/MYyJnvxFYe8Z3WUj7y7d18N6c7xDhxmwMKajUcOyGZT/zZp0SdgpXlxuHZelrpRek31vzOtLIiBBBDOrDNfeMuy//+YkBti8YVTd2spPRAV6G2snZ8Zl8gSNjLCWxpd7wjYEjvueBUQl31Mey6J2gbRTQ60NPNEvADfLARxegqJSf16lpBJcN7/NPLtmzcDCrO08VhbBM66dx9cceV4PsP2fw24m6YWiBPynA4g/Jzy4KexUSFVpC8jOY9wpaQFZLUTfWD+T9GvNalo6JaDqywHcn1U5DuwxbpB62nts+1djOEf9sP4jm5KVvCpnu+36ME3L2TG2x+thMNxCJlKkiaoT8/RnKIjQcQ9od5d8sqs2zTjBta2ZM0plRom2jdD8uvCgpbIAbxcM2ZglcqeV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: khugepaged scans PMD ranges for potential collapse to a hugepage. To add mTHP support we use this scan to instead record chunks of fully utilized sections of the PMD. create a bitmap to represent a PMD in order MTHP_MIN_ORDER chunks. by default we will set this to order 3. The reasoning is that for 4K 512 PMD size this results in a 64 bit bitmap which has some optimizations. For other arches like ARM64 64K, we can set a larger order if needed. khugepaged_scan_bitmap uses a stack struct to recursively scan a bitmap that represents chunks of utilized regions. We can then determine what mTHP size fits best and in the following patch, we set this bitmap while scanning the PMD. max_ptes_none is used as a scale to determine how "full" an order must be before being considered for collapse. If a order is set to "always" lets always collapse to that order in a greedy manner. Signed-off-by: Nico Pache --- include/linux/khugepaged.h | 4 ++ mm/khugepaged.c | 89 +++++++++++++++++++++++++++++++++++--- 2 files changed, 86 insertions(+), 7 deletions(-) diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 1f46046080f5..1fe0c4fc9d37 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -1,6 +1,10 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H #define _LINUX_KHUGEPAGED_H +#define MIN_MTHP_ORDER 3 +#define MIN_MTHP_NR (1<mthp_bitmap_stack[++top] = (struct scan_bit_state) + { HPAGE_PMD_ORDER - MIN_MTHP_ORDER, 0 }; + + while (top >= 0) { + state = cc->mthp_bitmap_stack[top--]; + order = state.order + MIN_MTHP_ORDER; + offset = state.offset; + num_chunks = 1 << (state.order); + // Skip mTHP orders that are not enabled + if (!test_bit(order, &enabled_orders)) + goto next; + + // copy the relavant section to a new bitmap + bitmap_shift_right(cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, + MTHP_BITMAP_SIZE); + + bits_set = bitmap_weight(cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none - 1) + >> (HPAGE_PMD_ORDER - state.order); + + //Check if the region is "almost full" based on the threshold + if (bits_set > threshold_bits + || test_bit(order, &huge_anon_orders_always)) { + ret = collapse_huge_page(mm, address, referenced, unmapped, cc, + mmap_locked, order, offset * MIN_MTHP_NR); + if (ret == SCAN_SUCCEED) { + collapsed += (1 << order); + continue; + } + } + +next: + if (state.order > 0) { + next_order = state.order - 1; + mid_offset = offset + (num_chunks / 2); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state) + { next_order, offset }; + } + } + return collapsed; +} + static int khugepaged_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked, @@ -1440,7 +1514,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { result = collapse_huge_page(mm, address, referenced, - unmapped, cc); + unmapped, cc, mmap_locked, HPAGE_PMD_ORDER, 0); /* collapse_huge_page will return with the mmap_lock released */ *mmap_locked = false; } @@ -2856,6 +2930,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, mmdrop(mm); kfree(cc); + return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 : madvise_collapse_errno(last_fail); }