From patchwork Wed Nov 22 16:29:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C940DC61D9B for ; Wed, 22 Nov 2023 16:30:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DF8D8D0037; Wed, 22 Nov 2023 11:30:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48F0D8D0008; Wed, 22 Nov 2023 11:30:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 357368D0037; Wed, 22 Nov 2023 11:30:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 22AFA8D0008 for ; Wed, 22 Nov 2023 11:30:13 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EC7CE160FE4 for ; Wed, 22 Nov 2023 16:30:12 +0000 (UTC) X-FDA: 81486127464.22.0E272E7 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf27.hostedemail.com (Postfix) with ESMTP id B973440021 for ; Wed, 22 Nov 2023 16:30:09 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670609; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uLiZitmjG+3foZ1+dsOMU1U8TEvwumUeFwfgKAb7Uhk=; b=CMTyM6JbwUUeZwraVDKtdx3ILxhdvLFjBXLTZMBv+JVEydP+qjJs9q1fJAcJrPRaG80uEq bnywIMVtM3/dqiym1sp2owWRyo7smusPZy9rGpmMWICRfg2LC8Yu7QqJjfmYmk4Pnz/gNr sn1Hn6JwmPghLHVJydiZLJoKNjx+Uyo= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670609; a=rsa-sha256; cv=none; b=pK0m3QkcofEBvLv9Dzr/5F4eV/nZX2Qcnu1x/6OfVkt2PjVyA5tWi487uQTqf6SVasHi1l 0H9p4BWYRCToqTbfg7dtoLwW0TD3/GJkRWDxOHSor8CWi9kT/IXdPV5PAQMGNbL+8/t99Z Ypr+fBzwRCvQbqiClFyjB8RT3OtpepQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 75A2E1595; Wed, 22 Nov 2023 08:30:55 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 085443F73F; Wed, 22 Nov 2023 08:30:05 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 01/10] mm: Allow deferred splitting of arbitrary anon large folios Date: Wed, 22 Nov 2023 16:29:41 +0000 Message-Id: <20231122162950.3854897-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B973440021 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: mdarrnn91n9x788a8ssb47teymkoxbj3 X-HE-Tag: 1700670609-877150 X-HE-Meta: U2FsdGVkX1+f7IBgpDNlt/iGxq9GcewzfIboeh80ZhveAmejIHb6Vh1728OOaxxUSdNhZBgWzxHrL03neS38S30pC+THcyeK6CtFVkkTqhEa9+Ul/t2QsZMQ5tjdLUVoyWBHbr8YVQ+91E57/kENcFFqz1vQmVWck39T+mevqJxwJOYvakl6ADtURU9OTxFcXyzU1sCv6DvX+pM0OYvDPQT8PSxGFFjDrrTGsoVggc9NEdpllT40qILdp3bfCTlC3AAPNF81F5IyqpakNfSG57TT/5XC5o8yBk9wl6TtRSOn0ra1bmPjrJrXKzrfB9WC3sfNs8+w6XLmbKTvmy5yZjYFLpDxM3GczhDfHVWccsY841ZVhuEuptqv6JWM7g2jir0HwE0e5Amw3HjWtFuTd93fBKMXfd+ZDQjyEdd0C6LFZoyS1mzWz2aQYnJK3hrWigNhLtYXv9US3u5z61FaeFj6/btR56j+esOeTaJgfkp09XlXqj5QBX5wqnASP28donLfkXdeY6xdFbC4CAPbc0nUsWaQYWtPuPzsrzWIcADncBn2paldzme4IRvSd2tSygtryoYbU005I9nT2Pav6FWgpmozD0bjTZlKAM14KID8JZipTR2zDLrL6DO4At9UIPOgXRbUrsHa2ZLkxwZRmCuX1QDC+dSsM9ta0KZ1M/jbjXnc0AzxEx+COcQ8XFpfdUNR3de9bdZ1rofBUeRl9s8AYFgJUy9Qifis8i7ZGN+RlDIK0uSUoHkjifYpMsgymefSpUwDbkBh50lenMOzXLNWGAPXoXJRd4tSgJZ+TXgyCn9nHaajn398RnlJUTgqAbqQ63napQXr3XfxJFvp9eZgejnmivM8Hx4XvymXkLebWLI9oh85sRwQRjL2EBRMWGAf2Fd9CSYm8pcB+vGsTFZ+7U5R1Tm/xET0cV3CkV7MNE9Hgi8Iw9WZhoUJ4go9/il6wjM7JOoluAuwrym m37VHPsO ckeEKHewYTjWYsnDCmG94ucOGxIQ6kfx3K80GRZmaLPgIpYKvvOQCGBLUFHl+/PLQv5WIgL3NJ2VxYL08GnM1rl9AajPpLJtUyi+pz4r7g9mXxZeRORWVxVK+jpWFCe5nzIBc/qFCv0hVLvpRn3Q0/6b12l8XURCtk7YkjfJ365C/1yM/YVHqB5+sP6P6eBgeyd4wv5F+Q7PdOhanOO73qMWUU6qloPTlHUDVghqivYeTJ1J3pxJP+slV2MnwmVBMLCbF7cN5p3qoUjUGE+qEcuiri5tEe6ZwmGlZSodC63zZiymIpcEbHTJDrnf9KONfB1V6d+i3yxUz5md1ExrDIj0AjaWsEg0JUlYc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for the introduction of anonymous small-sized THP, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized. Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: David Hildenbrand Signed-off-by: Ryan Roberts Reviewed-by: Barry Song --- mm/rmap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.25.1 diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..49e4d86a4f70 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1488,11 +1488,11 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, __lruvec_stat_mod_folio(folio, idx, -nr); /* - * Queue anon THP for deferred split if at least one + * Queue anon large folio for deferred split if at least one * page of the folio is unmapped and at least one page * is still mapped. */ - if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) + if (folio_test_large(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) deferred_split_folio(folio); } From patchwork Wed Nov 22 16:29:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62759C61D9D for ; Wed, 22 Nov 2023 16:30:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7C778D0038; Wed, 22 Nov 2023 11:30:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DDE6B8D0008; Wed, 22 Nov 2023 11:30:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE36B8D0038; Wed, 22 Nov 2023 11:30:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B10828D0008 for ; Wed, 22 Nov 2023 11:30:14 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 623E91CBE42 for ; Wed, 22 Nov 2023 16:30:14 +0000 (UTC) X-FDA: 81486127548.08.A24ADDB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id 4CF59C0021 for ; Wed, 22 Nov 2023 16:30:12 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670612; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v5seL2le1Mw6GIfT/Jkn3HeHK2JNJN/qsVUzKZyoMG0=; b=IPUNt6U2jwJhFX746QEQo+/G2+cG2KhY4KsnrUityrZdLcK8BI/zJEdpEQ3z2V/KEkLzIS meB5IEfljnuNkubAEJsERu0FyKq+nVCBjxLLmXt66/trFox4irpd82QnVuEP8oxLmbtg2e QaVFF9Br5gprPfdQsIvcVl9d4t71oNo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670612; a=rsa-sha256; cv=none; b=Zn8qCwGUDOgXVY8KmvaMvcaKiQeI89H6Sf4afzBzhBJNPRZTnzn2DqFyta1M5rtXcEMZhT +okWcY72tSmiU+UiTk6vH29tLkc8AEa5YxoT+oTWBlvMDzQSKiC6s1iVLdhlR5vyFsKk7U 8D6qs3WALpk7D16l/uQwqWPpp+EtbeI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 47A561650; Wed, 22 Nov 2023 08:30:58 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CF0523F73F; Wed, 22 Nov 2023 08:30:08 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() Date: Wed, 22 Nov 2023 16:29:42 +0000 Message-Id: <20231122162950.3854897-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4CF59C0021 X-Rspam-User: X-Stat-Signature: sw3io67nbgtm4cbfsgg8tykqnuopo7sw X-Rspamd-Server: rspam03 X-HE-Tag: 1700670612-80244 X-HE-Meta: U2FsdGVkX1/MpePK6NVZz3ewxM3YNffH7vlak+VxdAakSf+MooNGWf0fcrvIWna1UwlxBDO4ne6pttPj5GgNnUSK8JKVa+akKIXyosTr3tk80ZDR9doBkMztQvqDkH3QN6Ew8O9RhuQWFh5DaviCPM3W8WR88XfssFN5f0upIwTaSYtRz+dbktvGl3QvZK6fCtOCkdt7g6Zbm3QtAErFsa9krVgUiZCTY7xx8fW5zvnoimSIc50fEGPZlEyjl15A9tcNYxOf+mKc7NZ7Dd4SwSq+HkVbifZQgmslqSLcQkySeLMOEFaHXym+ZDlUJVejC1NXnFGPKez2HKmTDcd7OnXUQNxj+ugC8uApCr1d/XyfhdBpYCcCpch02whoDxXLZHBOh/SODa5QUXQonrY+u7p0Koj+V34kOXVvcoCYrs/pH0n0MgsS4go/V4Q77uZzn9oBOeGQ6VPht/QhY7wqZByqzwvZWbjcEgIZGquL/OeLR7+Letj33xJC2uAx2pIdmod37aWoalA6m+j+SqoNozrzSFNMqUqTTviYIAyJOsftq3gK82nyZsWNYETztb8jNa6iJycI2hKXLWu7uyIrkCaF/AuLJp3BkTxfvdcM7xBZ4BiISPdtADS2+cN/m1ToPMWSQ3Jb7m5kX3m1eA1R0k+kS5f+Xss4XEFBS5T8iq7rlMuXGMGi8HcS7BICvZbHyuWGc9mRQGcqwUGU1C/QmO9xp259Jz2DmeNhYlfmrT3tC+jW9gTdR6TDC9+WyPMT7lOQDYlrCW0SuXkMmm5ap521SgIXOUbG8xFuX5r642CweoZMP/ep6X7lT0weO5uzx8yUeGJnmCyD/o80ra513pTuWQVtCWLnuKW3Gimcw3jQ9GRk0R0y9IhRw5JCEpC7zdKqp3pSkAMW4A8frBMnp7W9ChlMcI+GSit1/BTMn3C/EM2WbIWwimR49q0ChKCbKdrcFLEXxlnWCmey2Aq OrxCv2Hq zXJUb0jLWSFrMRl4bd5/RNIpGeASrjND9/0Q2eSIdKZkj/b+RxaWClP11cENpi3kn0izq4BCcIJzwEwD9JTSYvvY46aFu8SDMr6MtIm3Xl/Vw+q5Nw/P7iP2GJ79CxxYexEUbpo+WTwKU0EN3u/Bd1y2eRHuySFD2FHG4q/G2TMGOJUf5PtauMbkZGfEpQghwdORdNY+Hv9EC5W1cXTcqyfotDJu1sSXr3Pe3DeJ/AOTpN8vk/0Fjl1KBDgJtDFdRB79VgFsmkRS4Vrf4r+S0Mag8Cp299F1f9K+IrVSV1HK0kb05AF4XuzqDcceMbpkyUBw9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for supporting anonymous small-sized THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme. Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei Signed-off-by: Ryan Roberts Reviewed-by: David Hildenbrand --- mm/rmap.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) -- 2.25.1 diff --git a/mm/rmap.c b/mm/rmap.c index 49e4d86a4f70..b086dc957b0c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1305,32 +1305,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * - * If the folio is large, it is accounted as a THP. As the folio + * If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { - int nr; + int nr = folio_nr_pages(folio); - VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); + __folio_set_anon(folio, vma, address, true); - if (likely(!folio_test_pmd_mappable(folio))) { + if (likely(!folio_test_large(folio))) { /* increment count (starts at -1) */ atomic_set(&folio->_mapcount, 0); - nr = 1; + SetPageAnonExclusive(&folio->page); + } else if (!folio_test_pmd_mappable(folio)) { + int i; + + for (i = 0; i < nr; i++) { + struct page *page = folio_page(folio, i); + + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + SetPageAnonExclusive(page); + } + + atomic_set(&folio->_nr_pages_mapped, nr); } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); - nr = folio_nr_pages(folio); + SetPageAnonExclusive(&folio->page); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - __folio_set_anon(folio, vma, address, true); - SetPageAnonExclusive(&folio->page); } /** From patchwork Wed Nov 22 16:29:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FD00C61D97 for ; Wed, 22 Nov 2023 16:30:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 862CB8D003B; Wed, 22 Nov 2023 11:30:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EBE88D0008; Wed, 22 Nov 2023 11:30:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A1A88D003B; Wed, 22 Nov 2023 11:30:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 43F7A8D0008 for ; Wed, 22 Nov 2023 11:30:17 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 18A2CB6077 for ; Wed, 22 Nov 2023 16:30:17 +0000 (UTC) X-FDA: 81486127674.20.33077FC Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 3E173140028 for ; Wed, 22 Nov 2023 16:30:14 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670615; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0512/HTGZpV4EDnLnZQIg+htVitY8R0+zwTbrDyppsA=; b=PI58D1X8BIOwOflw5SWnJFSUtFB8X1s8zDiOJWhh5h5wOb6yTWGW9zYunnVTRMyjuZTa/2 Lg8jFrsFT0gTgajOuCWr41PG+YZP8jTYGcYBssC14dZSula3cfCIr+b3+78tg+TVIkKu/x h/1Cxmzjd6184u6xaLi/XyiJZxRrQq4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670615; a=rsa-sha256; cv=none; b=hgqT2lEieCpMNaE+KoaO7bD0jE+gA0g1+Aw5+F2jkrJ34FytEU631/rRYLdlyQjIYS1Wfy Vpe8ipCzVR781QFHZL5bxBdHJE+Tdt7cI5TFhmCB3oppmDzHUVn3OCbJ/J0e32+1y8ufFN xIidcVq3vHCAktuyP08Cdvgwd+xCIUI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 35A2A176B; Wed, 22 Nov 2023 08:31:01 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A217C3F73F; Wed, 22 Nov 2023 08:30:11 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 03/10] mm: thp: Introduce per-size thp sysfs interface Date: Wed, 22 Nov 2023 16:29:43 +0000 Message-Id: <20231122162950.3854897-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: dxgsrjrofwf13dshy6ro5oimfb78b1qb X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3E173140028 X-Rspam-User: X-HE-Tag: 1700670614-83064 X-HE-Meta: U2FsdGVkX18mY3zUtu1sm6js2KcKWrxf6UkeqATEyTUThIgtq+MzM4H3afNAMUfwlvBSD5Ms9C+ddGvOOmY8EUdVuRkanHMlhp0O73CJOE++uBf5GV5pgcjgZa4lmEXQz6pWCsb3WFcONRbpCjmcKhaqchcq/TO1rJUXv61SRARXOAlI+fBnpfZhRlHv0+qJi7sYyb6InuFuXH+SLkAv6tMiF7PjPEF2NvoZs3KnpJywkuPrgcvu/cBRIhGuv00Nqxa3eAeOMNb7JuSHRGox+36c5H+ubv2VeV+7JeC3ojbCoLNdV8lmIIPv3LxaU/JSZbkNMOTN6l88ZmAyRMTh/enkngS2rcS6FYKG3ERHYuohwHwkYuLuh5hPmUg1XHvikEqbjC4TDgkeqAm1RN7X7iC8QGLrAoheRnFUS1gsvhQcgKq5B251sdfyrqmlnxsiqdnIpkiPaJbfUe6UShvQHPRNiwMy8PztKYeaK78WNP8I5i4LVGOouw86fN0y2mwHBlPCAdCyGAt2Vc5nbmITfc/piXr0L0SrWO0UnB6I7XS1fKkfZ1UDbEQ73JURa0cc0d1HxxzvEA4A9x5mEurE0TxcR+lrlnRjfXwlbTSogqPlvbTE8qaKFUPloIKX7oMgOxnOXSKK0wRXa65XAmX1yA/d8Xe4pK3RtIZdGOO9h7GZve0T66O/unFKvQgQup8UG7MCFMZtLMKBr6leJZG5FQDzsL2gzXhPLfm19HrlUaxSeVbRZTHkJD4o+AX/Ji1lFW69rUxDkBXf+QqQmrPXTEDoXFResoNg/k35JAVga0/6V1uak/NQsjwLh74k8SgVSS0S6qoB2zj6I+swsSHuFp9eLDa+6Sxe0eL1QKlnAG2UnG9cPeb6LNnclmfHA1tj2FhNQYgZjql5gsyLvDPwRsJp0OuOoV0GDvbotVdJqN+31+a4bysH2Hgrk+Y2b0VGKAHynztk29J2Vn4+nsb +V6sV+BQ CQ7gZrvuQyqyXiO6JGvYrb+u+rfolvhf3wBJ5qFoAptS6zq5ICCIWD7PW5YCIsa9gZ0jC//pHt/i6pWmk2SrRXz/Ijhk5J9ZwnNjxhcpT8ucvLIaRF/cEBz1TbZKCKYlX2Qwm1BDTphE341j62m1dW0BoT3DCCwerW0Lee1OkUYR2+dgambZYiXSs5lgZr7qn1O4MMUdirIYIcuH1kJ2/qd2gP5rwXMRdIcJeWAOIGlXvQeE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for adding support for anonymous small-sized THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an `enabled` file, which can be set to "global" (to inherrit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated. The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. If there is only 1 order set in the input then the output can continue to be treated like a boolean; this is the case for most call sites. The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a `struct thpsize`, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the `enabled` state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|global] bitfields since this reduces the amount of work required in transhuge_vma_suitable() which is called for every page fault. The remainder is copied from Documentation/admin-guide/mm/transhuge.rst, as modified by this commit. See that file for further details. Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled system wide. This can be achieved per-supported-THP-size with one of:: echo always >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled echo never >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled where is the hugepage size being addressed, the available sizes for which vary by system. Alternatively it is possible to specify that a given hugepage size will inherrit the global enabled setting:: echo global >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled The global (legacy) enabled setting can be set as follows:: echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled By default, PMD-sized hugepages have enabled="global" and all other hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation. Signed-off-by: Ryan Roberts --- Documentation/admin-guide/mm/transhuge.rst | 74 ++++-- Documentation/filesystems/proc.rst | 6 +- fs/proc/task_mmu.c | 3 +- include/linux/huge_mm.h | 100 +++++--- mm/huge_memory.c | 263 +++++++++++++++++++-- mm/khugepaged.c | 16 +- mm/memory.c | 6 +- mm/page_vma_mapped.c | 3 +- 8 files changed, 387 insertions(+), 84 deletions(-) -- 2.25.1 diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index b0cc8243e093..52565e0bd074 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -45,10 +45,23 @@ components: the two is using hugepages just because of the fact the TLB miss is going to run faster. +As well as PMD-sized THP described above, it is also possible to +configure the system to allocate "small-sized THP" to back anonymous +memory (for example 16K, 32K, 64K, etc). These THPs continue to be +PTE-mapped, but in many cases can still provide similar benefits to +those outlined above: Page faults are significantly reduced (by a +factor of e.g. 4, 8, 16, etc), but latency spikes are much less +prominent because the size of each page isn't as huge as the PMD-sized +variant and there is less memory to clear in each page fault. Some +architectures also employ TLB compression mechanisms to squeeze more +entries in when a set of PTEs are virtually and physically contiguous +and approporiately aligned. In this case, TLB misses will occur less +often. + THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task's address space. Unless THP is completely disabled, there is ``khugepaged`` daemon that scans memory and -collapses sequences of basic pages into huge pages. +collapses sequences of basic pages into PMD-sized huge pages. The THP behaviour is controlled via :ref:`sysfs ` interface and using madvise(2) and prctl(2) system calls. @@ -95,12 +108,29 @@ Global THP controls Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled -system wide. This can be achieved with one of:: +system wide. This can be achieved per-supported-THP-size with one of:: + + echo always >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled + echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled + echo never >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled + +where is the hugepage size being addressed, the available sizes +for which vary by system. Alternatively it is possible to specify that +a given hugepage size will inherrit the global enabled setting:: + + echo global >/sys/kernel/mm/transparent_hugepage/hugepages-kB/enabled + +The global (legacy) enabled setting can be set as follows:: echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled +By default, PMD-sized hugepages have enabled="global" and all other +hugepage sizes have enabled="never". If enabling multiple hugepage +sizes, the kernel will select the most appropriate enabled size for a +given allocation. + It's also possible to limit defrag efforts in the VM to generate anonymous hugepages in case they're not immediately free to madvise regions or to never try to defrag memory and simply fallback to regular @@ -146,25 +176,34 @@ madvise never should be self-explanatory. -By default kernel tries to use huge zero page on read page fault to -anonymous mapping. It's possible to disable huge zero page by writing 0 -or enable it back by writing 1:: +By default kernel tries to use huge, PMD-mappable zero page on read +page fault to anonymous mapping. It's possible to disable huge zero +page by writing 0 or enable it back by writing 1:: echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page -Some userspace (such as a test program, or an optimized memory allocation -library) may want to know the size (in bytes) of a transparent hugepage:: +Some userspace (such as a test program, or an optimized memory +allocation library) may want to know the size (in bytes) of a +PMD-mappable transparent hugepage:: cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size -khugepaged will be automatically started when -transparent_hugepage/enabled is set to "always" or "madvise, and it'll -be automatically shutdown if it's set to "never". +khugepaged will be automatically started when one or more hugepage +sizes are enabled (either by directly setting "always" or "madvise", +or by setting "global" while the global enabled is set to "always" or +"madvise"), and it'll be automatically shutdown when the last hugepage +size is disabled (either by directly setting "never", or by setting +"global" while the global enabled is set to "never"). Khugepaged controls ------------------- +.. note:: + khugepaged currently only searches for opportunities to collapse to + PMD-sized THP and no attempt is made to collapse to small-sized + THP. + khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's @@ -282,10 +321,11 @@ force Need of application restart =========================== -The transparent_hugepage/enabled values and tmpfs mount option only affect -future behavior. So to make them effective you need to restart any -application that could have been using hugepages. This also applies to the -regions registered in khugepaged. +The transparent_hugepage/enabled and +transparent_hugepage/hugepages-kB/enabled values and tmpfs mount +option only affect future behavior. So to make them effective you need +to restart any application that could have been using hugepages. This +also applies to the regions registered in khugepaged. Monitoring usage ================ @@ -308,6 +348,10 @@ frequently will incur overhead. There are a number of counters in ``/proc/vmstat`` that may be used to monitor how successfully the system is providing huge pages for use. +.. note:: + Currently the below counters only record events relating to + PMD-sized THP. Events relating to small-sized THP are not included. + thp_fault_alloc is incremented every time a huge page is successfully allocated to handle a page fault. @@ -413,7 +457,7 @@ for huge pages. Optimizing the applications =========================== -To be guaranteed that the kernel will map a 2M page immediately in any +To be guaranteed that the kernel will map a THP immediately in any memory region, the mmap region has to be hugepage naturally aligned. posix_memalign() can provide that guarantee. diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 49ef12df631b..f8e8dd1fd148 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -528,9 +528,9 @@ replaced by copy-on-write) part of the underlying shmem object out on swap. does not take into account swapped out page of underlying shmem objects. "Locked" indicates whether the mapping is locked in memory or not. -"THPeligible" indicates whether the mapping is eligible for allocating THP -pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise. -It just shows the current status. +"THPeligible" indicates whether the mapping is eligible for allocating +naturally aligned THP pages of any currently enabled size. 1 if true, 0 +otherwise. It just shows the current status. "VmFlags" field deserves a separate description. This member represents the kernel flags associated with the particular virtual memory area in two letter diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 51e0ec658457..2e25362ca9fa 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -865,7 +865,8 @@ static int show_smap(struct seq_file *m, void *v) __show_smap(m, &mss, false); seq_printf(m, "THPeligible: %8u\n", - hugepage_vma_check(vma, vma->vm_flags, true, false, true)); + !!hugepage_vma_check(vma, vma->vm_flags, true, false, true, + THP_ORDERS_ALL)); if (arch_pkeys_enabled()) seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..7d6f7d96b039 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -67,6 +67,21 @@ extern struct kobj_attribute shmem_enabled_attr; #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) #define HPAGE_PMD_NR (1<vm_start >> PAGE_SHIFT) - vma->vm_pgoff, - HPAGE_PMD_NR)) - return false; +static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma, + unsigned long addr, unsigned long orders) +{ + int order; + + /* + * Iterate over orders, highest to lowest, removing orders that don't + * meet alignment requirements from the set. Exit loop at first order + * that meets requirements, since all lower orders must also meet + * requirements. + */ + + order = first_order(orders); + + while (orders) { + unsigned long hpage_size = PAGE_SIZE << order; + unsigned long haddr = ALIGN_DOWN(addr, hpage_size); + + if (haddr >= vma->vm_start && + haddr + hpage_size <= vma->vm_end) { + if (!vma_is_anonymous(vma)) { + if (IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - + vma->vm_pgoff, + hpage_size >> PAGE_SHIFT)) + break; + } else + break; + } + + order = next_order(&orders, order); } - haddr = addr & HPAGE_PMD_MASK; - - if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end) - return false; - return true; + return orders; } static inline bool file_thp_enabled(struct vm_area_struct *vma) @@ -130,8 +166,9 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode); } -bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps, bool in_pf, bool enforce_sysfs); +unsigned long hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, bool smaps, bool in_pf, + bool enforce_sysfs, unsigned long orders); #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ @@ -267,17 +304,18 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) return false; } -static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, - unsigned long addr) +static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma, + unsigned long addr, unsigned long orders) { - return false; + return 0; } -static inline bool hugepage_vma_check(struct vm_area_struct *vma, - unsigned long vm_flags, bool smaps, - bool in_pf, bool enforce_sysfs) +static inline unsigned long hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, bool smaps, + bool in_pf, bool enforce_sysfs, + unsigned long orders) { - return false; + return 0; } static inline void folio_prep_large_rmappable(struct folio *folio) {} diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6eb55f97a3d2..0b545d56420c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -74,12 +74,60 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, static atomic_t huge_zero_refcount; struct page *huge_zero_page __read_mostly; unsigned long huge_zero_pfn __read_mostly = ~0UL; +static unsigned long huge_anon_orders_always __read_mostly; +static unsigned long huge_anon_orders_madvise __read_mostly; +static unsigned long huge_anon_orders_global __read_mostly; -bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps, bool in_pf, bool enforce_sysfs) +#define hugepage_global_enabled() \ + (transparent_hugepage_flags & \ + ((1<vm_flags + * @smaps: whether answer will be used for smaps file + * @in_pf: whether answer will be used by page fault handler + * @enforce_sysfs: whether sysfs config should be taken into account + * @orders: bitfield of all orders to consider + * + * Calculates the intersection of the requested hugepage orders and the allowed + * hugepage orders for the provided vma. Permitted orders are encoded as a set + * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3 + * corresponds to order-3, etc). Order-0 is never considered a hugepage order. + * + * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage + * orders are allowed. + */ +unsigned long hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, bool smaps, bool in_pf, + bool enforce_sysfs, unsigned long orders) +{ + /* Check the intersection of requested and supported orders. */ + orders &= vma_is_anonymous(vma) ? + THP_ORDERS_ALL_ANON : THP_ORDERS_ALL_FILE; + if (!orders) + return 0; + if (!vma->vm_mm) /* vdso */ - return false; + return 0; /* * Explicitly disabled through madvise or prctl, or some @@ -88,16 +136,16 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, * */ if ((vm_flags & VM_NOHUGEPAGE) || test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)) - return false; + return 0; /* * If the hardware/firmware marked hugepage support disabled. */ if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED)) - return false; + return 0; /* khugepaged doesn't collapse DAX vma, but page fault is fine. */ if (vma_is_dax(vma)) - return in_pf; + return in_pf ? orders : 0; /* * khugepaged special VMA and hugetlb VMA. @@ -105,17 +153,29 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, * VM_MIXEDMAP set. */ if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED)) - return false; + return 0; /* - * Check alignment for file vma and size for both file and anon vma. + * Check alignment for file vma and size for both file and anon vma by + * filtering out the unsuitable orders. * * Skip the check for page fault. Huge fault does the check in fault - * handlers. And this check is not suitable for huge PUD fault. + * handlers. */ - if (!in_pf && - !transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) - return false; + if (!in_pf) { + int order = first_order(orders); + unsigned long addr; + + while (orders) { + addr = vma->vm_end - (PAGE_SIZE << order); + if (transhuge_vma_suitable(vma, addr, BIT(order))) + break; + order = next_order(&orders, order); + } + + if (!orders) + return 0; + } /* * Enabled via shmem mount options or sysfs settings. @@ -124,13 +184,27 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, */ if (!in_pf && shmem_file(vma->vm_file)) return shmem_is_huge(file_inode(vma->vm_file), vma->vm_pgoff, - !enforce_sysfs, vma->vm_mm, vm_flags); + !enforce_sysfs, vma->vm_mm, vm_flags) + ? orders : 0; /* Enforce sysfs THP requirements as necessary */ - if (enforce_sysfs && - (!hugepage_flags_enabled() || (!(vm_flags & VM_HUGEPAGE) && - !hugepage_flags_always()))) - return false; + if (enforce_sysfs) { + if (vma_is_anonymous(vma)) { + unsigned long mask = READ_ONCE(huge_anon_orders_always); + + if (vm_flags & VM_HUGEPAGE) + mask |= READ_ONCE(huge_anon_orders_madvise); + if (hugepage_global_always() || + ((vm_flags & VM_HUGEPAGE) && hugepage_global_enabled())) + mask |= READ_ONCE(huge_anon_orders_global); + + orders &= mask; + if (!orders) + return 0; + } else if (!hugepage_global_enabled() || + (!(vm_flags & VM_HUGEPAGE) && !hugepage_global_always())) + return 0; + } if (!vma_is_anonymous(vma)) { /* @@ -138,15 +212,15 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, * in fault path. */ if (((in_pf || smaps)) && vma->vm_ops->huge_fault) - return true; + return orders; /* Only regular file is valid in collapse path */ if (((!in_pf || smaps)) && file_thp_enabled(vma)) - return true; - return false; + return orders; + return 0; } if (vma_is_temporary_stack(vma)) - return false; + return 0; /* * THPeligible bit of smaps should show 1 for proper VMAs even @@ -156,9 +230,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, * the first page fault. */ if (!vma->anon_vma) - return (smaps || in_pf); + return (smaps || in_pf) ? orders : 0; - return true; + return orders; } static bool get_huge_zero_page(void) @@ -412,9 +486,127 @@ static const struct attribute_group hugepage_attr_group = { .attrs = hugepage_attr, }; +static void hugepage_exit_sysfs(struct kobject *hugepage_kobj); +static void thpsize_release(struct kobject *kobj); +static LIST_HEAD(thpsize_list); + +struct thpsize { + struct kobject kobj; + struct list_head node; + int order; +}; + +#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj) + +static ssize_t thpsize_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + int order = to_thpsize(kobj)->order; + const char *output; + + if (test_bit(order, &huge_anon_orders_always)) + output = "[always] global madvise never"; + else if (test_bit(order, &huge_anon_orders_global)) + output = "always [global] madvise never"; + else if (test_bit(order, &huge_anon_orders_madvise)) + output = "always global [madvise] never"; + else + output = "always global madvise [never]"; + + return sysfs_emit(buf, "%s\n", output); +} + +static ssize_t thpsize_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int order = to_thpsize(kobj)->order; + ssize_t ret = count; + + if (sysfs_streq(buf, "always")) { + set_bit(order, &huge_anon_orders_always); + clear_bit(order, &huge_anon_orders_global); + clear_bit(order, &huge_anon_orders_madvise); + } else if (sysfs_streq(buf, "global")) { + set_bit(order, &huge_anon_orders_global); + clear_bit(order, &huge_anon_orders_always); + clear_bit(order, &huge_anon_orders_madvise); + } else if (sysfs_streq(buf, "madvise")) { + set_bit(order, &huge_anon_orders_madvise); + clear_bit(order, &huge_anon_orders_always); + clear_bit(order, &huge_anon_orders_global); + } else if (sysfs_streq(buf, "never")) { + clear_bit(order, &huge_anon_orders_always); + clear_bit(order, &huge_anon_orders_global); + clear_bit(order, &huge_anon_orders_madvise); + } else + ret = -EINVAL; + + return ret; +} + +static struct kobj_attribute thpsize_enabled_attr = + __ATTR(enabled, 0644, thpsize_enabled_show, thpsize_enabled_store); + +static struct attribute *thpsize_attrs[] = { + &thpsize_enabled_attr.attr, + NULL, +}; + +static const struct attribute_group thpsize_attr_group = { + .attrs = thpsize_attrs, +}; + +static const struct kobj_type thpsize_ktype = { + .release = &thpsize_release, + .sysfs_ops = &kobj_sysfs_ops, +}; + +static struct thpsize *thpsize_create(int order, struct kobject *parent) +{ + unsigned long size = (PAGE_SIZE << order) / SZ_1K; + struct thpsize *thpsize; + int ret; + + thpsize = kzalloc(sizeof(*thpsize), GFP_KERNEL); + if (!thpsize) + return ERR_PTR(-ENOMEM); + + ret = kobject_init_and_add(&thpsize->kobj, &thpsize_ktype, parent, + "hugepages-%lukB", size); + if (ret) { + kfree(thpsize); + return ERR_PTR(ret); + } + + ret = sysfs_create_group(&thpsize->kobj, &thpsize_attr_group); + if (ret) { + kobject_put(&thpsize->kobj); + return ERR_PTR(ret); + } + + thpsize->order = order; + return thpsize; +} + +static void thpsize_release(struct kobject *kobj) +{ + kfree(to_thpsize(kobj)); +} + static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) { int err; + struct thpsize *thpsize; + unsigned long orders; + int order; + + /* + * Default to setting PMD-sized THP to follow the global setting and + * disable all other sizes. powerpc's PMD_ORDER isn't a compile-time + * constant so we have to do this here. + */ + huge_anon_orders_global = BIT(PMD_ORDER); *hugepage_kobj = kobject_create_and_add("transparent_hugepage", mm_kobj); if (unlikely(!*hugepage_kobj)) { @@ -434,8 +626,24 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) goto remove_hp_group; } + orders = THP_ORDERS_ALL_ANON; + order = first_order(orders); + while (orders) { + thpsize = thpsize_create(order, *hugepage_kobj); + if (IS_ERR(thpsize)) { + pr_err("failed to create thpsize for order %d\n", order); + err = PTR_ERR(thpsize); + goto remove_all; + } + list_add(&thpsize->node, &thpsize_list); + order = next_order(&orders, order); + } + return 0; +remove_all: + hugepage_exit_sysfs(*hugepage_kobj); + return err; remove_hp_group: sysfs_remove_group(*hugepage_kobj, &hugepage_attr_group); delete_obj: @@ -445,6 +653,13 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) static void __init hugepage_exit_sysfs(struct kobject *hugepage_kobj) { + struct thpsize *thpsize, *tmp; + + list_for_each_entry_safe(thpsize, tmp, &thpsize_list, node) { + list_del(&thpsize->node); + kobject_put(&thpsize->kobj); + } + sysfs_remove_group(hugepage_kobj, &khugepaged_attr_group); sysfs_remove_group(hugepage_kobj, &hugepage_attr_group); kobject_put(hugepage_kobj); @@ -811,7 +1026,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) struct folio *folio; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; - if (!transhuge_vma_suitable(vma, haddr)) + if (!transhuge_vma_suitable(vma, haddr, BIT(PMD_ORDER))) return VM_FAULT_FALLBACK; if (unlikely(anon_vma_prepare(vma))) return VM_FAULT_OOM; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 064654717843..c3a874c0d601 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -446,7 +446,8 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && hugepage_flags_enabled()) { - if (hugepage_vma_check(vma, vm_flags, false, false, true)) + if (hugepage_vma_check(vma, vm_flags, false, false, true, + BIT(PMD_ORDER))) __khugepaged_enter(vma->vm_mm); } } @@ -922,10 +923,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!vma) return SCAN_VMA_NULL; - if (!transhuge_vma_suitable(vma, address)) + if (!transhuge_vma_suitable(vma, address, BIT(PMD_ORDER))) return SCAN_ADDRESS_RANGE; if (!hugepage_vma_check(vma, vma->vm_flags, false, false, - cc->is_khugepaged)) + cc->is_khugepaged, BIT(PMD_ORDER))) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1503,7 +1504,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * and map it by a PMD, regardless of sysfs THP settings. As such, let's * analogously elide sysfs THP settings here. */ - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) + if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false, + BIT(PMD_ORDER))) return SCAN_VMA_CHECK; /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ @@ -2368,7 +2370,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++; break; } - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, true)) { + if (!hugepage_vma_check(vma, vma->vm_flags, false, false, true, + BIT(PMD_ORDER))) { skip: progress++; continue; @@ -2705,7 +2708,8 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, *prev = vma; - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) + if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false, + BIT(PMD_ORDER))) return -EINVAL; cc = kmalloc(sizeof(*cc), GFP_KERNEL); diff --git a/mm/memory.c b/mm/memory.c index c32954e16b28..9d5e61d6d859 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4319,7 +4319,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) pmd_t entry; vm_fault_t ret = VM_FAULT_FALLBACK; - if (!transhuge_vma_suitable(vma, haddr)) + if (!transhuge_vma_suitable(vma, haddr, BIT(PMD_ORDER))) return ret; page = compound_head(page); @@ -5117,7 +5117,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, return VM_FAULT_OOM; retry_pud: if (pud_none(*vmf.pud) && - hugepage_vma_check(vma, vm_flags, false, true, true)) { + hugepage_vma_check(vma, vm_flags, false, true, true, BIT(PUD_ORDER))) { ret = create_huge_pud(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; @@ -5151,7 +5151,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, goto retry_pud; if (pmd_none(*vmf.pmd) && - hugepage_vma_check(vma, vm_flags, false, true, true)) { + hugepage_vma_check(vma, vm_flags, false, true, true, BIT(PMD_ORDER))) { ret = create_huge_pmd(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index e0b368e545ed..5f7e89c5b595 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -268,7 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) * cleared *pmd but not decremented compound_mapcount(). */ if ((pvmw->flags & PVMW_SYNC) && - transhuge_vma_suitable(vma, pvmw->address) && + transhuge_vma_suitable(vma, pvmw->address, + BIT(PMD_ORDER)) && (pvmw->nr_pages >= HPAGE_PMD_NR)) { spinlock_t *ptl = pmd_lock(mm, pvmw->pmd); From patchwork Wed Nov 22 16:29:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA0A7C61D99 for ; Wed, 22 Nov 2023 16:30:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C2618D0040; Wed, 22 Nov 2023 11:30:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 071D38D0008; Wed, 22 Nov 2023 11:30:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDEBB8D0040; Wed, 22 Nov 2023 11:30:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CEB708D0008 for ; Wed, 22 Nov 2023 11:30:19 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 88168120E5F for ; Wed, 22 Nov 2023 16:30:19 +0000 (UTC) X-FDA: 81486127758.13.21E228C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id BE5CF80024 for ; Wed, 22 Nov 2023 16:30:17 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670617; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GaRguLCBK9uUawAeJ/HZyd0iBmJtPBRZwhu2sCovmzI=; b=qQhS96jxD9VgdpnGU9lAqHj+3MeNujxD+VivYZyyz40h9sE5KVYxC2WayqjtFqXVcYIrdA aT1y9HVi6gsws9v+V34LPc6aQ7bnkQb4PUzpLaIpa8tmnvHFl3mzBA03z4o9RpC9afmwli 28ikwfory3ejivzPYgRE8C9CaDesqwc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670617; a=rsa-sha256; cv=none; b=doaU39GSYHE+f5f2E83/TaK1HHiKkThC/lfr0qFlky8oz8smoFbVbAaQ7UfrN7YMtBjuxu 3S/6/aBgYWpcD8+o/t97LFYtI3IH7A/h1/TrMdvLcEfWzOuGMECTuRoRUTs3zCW4c9JGXt ZCras9EPXxPDa2Nsoe2Te7jSjnPX1UM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 07AFE176C; Wed, 22 Nov 2023 08:31:04 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8F1F93F73F; Wed, 22 Nov 2023 08:30:14 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 04/10] mm: thp: Support allocation of anonymous small-sized THP Date: Wed, 22 Nov 2023 16:29:44 +0000 Message-Id: <20231122162950.3854897-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BE5CF80024 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 3j8wx1xpisf5ye5zhzzjc16rhkoqjrie X-HE-Tag: 1700670617-141161 X-HE-Meta: U2FsdGVkX1+jr8+3Bb2BPgMRe7HMyHfXG95cFfOTost1AsOzCEh3fb66nnEX/mUrePT5IGj3/WzOOkV8XxqnpBkwge77dR4goDRLtd3qA6jkLrZOwVkUcNmhxATNx+kleEhtiMZy8UzChXBjTd0YEPjwMtmyMXhMVQMbW4fC2a25DhbdEt29BLFX7okJoT6LoFQzR5jPToHTRNvKjKIIcx/itADwmvudvWR58X0e+oTfQ0a/mtFQjTPcPhZYY/2vum8uwkDrgk3c4ViqtwtnsULJ0EsZR00ZPpfrDkQXLxJ9ZHT4obscQuggwaReJeeMdcgbOlDW8n37VZD+9A4Br76tQl6GKzmx5xAZyJyYNbhKnDk0oQkzizDn4eanv1dDmmZfmZ9rjYMxfxjTNDCjoLu87GFTjflGHXcrFhoxxbr+TowckT9P9/A5b8WjZFGBdYM7VJM7TRTUQ1/jw46JoCgUSYePjCe0ILxpRPZWScVKNQyU59NTyLTKUqbxBTWPxXVLsPCzi3H1x/q7hSzaFbV65X6MLifRjbOe8I8O6CsSt0xRYyrnloxLPABEUqjMbs1HxOUN/USQKKo2vY9PZhwtsw+K9ghqNQSIATMUZwn28IyxG/Zzr0odrP/0IZR2apwOGhzxGKaPHSGwJDm/UrmDb7LkiWk7PRyeqsHeK75nWjQOZXl91Tb74WyM2/1gFFv++/ceESAwwGo66xfwKw0DbvmLkz1KXxZ+ePDbrK4t7e8ml59oMaKaeTnIWrPmbSJH9bY2Z7D14QAfta1PESLgdjW990I/8Ck0ka6+jRTe1h17Ca5GYjpyHaUd/W3U16Rd+QC5bdd9rs6/2LzCH2ZMK3FNXnQum4TfSOkZ2kct1uekqu3ZjAXKQqYHoeY4MJuk3hVJKfDA/j/nU7rxlgGJHp4hvRfblBdkXTjLd6+Ij9dBO+4oAPDNkVqsDv2//QRw+piMimSjyHb5jj+ /bV5rpcZ HT0sFXsaTsxQjCMiaIt9HBYuhbR5IkNB06wqemQrqsqv+wfR72yteSKtzYf3Wq4UykOe5fcQfTTTbpXx1YxoT+ZRxSOyi70KXQkwhjK93NQQLTvwBrhitKcIu0Ie7gMOpXZP72aSPI/Z1U9zib4b7rG/yHFcJdwvh46vw55K/O2+vmQ6G1a4LLSX3lsGqrLshbG8eUluBWthmu4HV+6U1cM+i8tVPItqMYGEkag/IH0kBnGI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce the logic to allow THP to be configured (through the new sysfs interface we just added) to allocate large folios to back anonymous memory, which are smaller than PMD-size. We call this new THP type "small-sized THP". These small-sized THPs continue to be PTE-mapped, but in many cases can still provide similar benefits to traditional PMD-sized THP: Page faults are significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending on the configured order), but latency spikes are much less prominent because the size of each page isn't as huge as the PMD-sized variant and there is less memory to clear in each page fault. The number of per-page operations (e.g. ref counting, rmap management, lru list management) are also significantly reduced since those ops now become per-folio. Some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and approporiately aligned. In this case, TLB misses will occur less often. The new behaviour is disabled by default, but can be enabled at runtime by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see documentation in previous commit). The long term aim is to change the default to include suitable lower orders, but there are some risks around internal fragmentation that need to be better understood first. Signed-off-by: Ryan Roberts --- include/linux/huge_mm.h | 6 ++- mm/memory.c | 106 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 101 insertions(+), 11 deletions(-) -- 2.25.1 diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7d6f7d96b039..edc302351971 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr; #define HPAGE_PMD_NR (1<vma; + unsigned long orders; + int order; + + /* + * If uffd is active for the vma we need per-page fault fidelity to + * maintain the uffd semantics. + */ + if (userfaultfd_armed(vma)) + goto fallback; + + /* + * Get a list of all the (large) orders below PMD_ORDER that are enabled + * for this vma. Then filter out the orders that can't be allocated over + * the faulting address and still be fully contained in the vma. + */ + orders = hugepage_vma_check(vma, vma->vm_flags, false, true, true, + BIT(PMD_ORDER) - 1); + orders = transhuge_vma_suitable(vma, vmf->address, orders); + + if (!orders) + goto fallback; + + pte = pte_offset_map(vmf->pmd, vmf->address & PMD_MASK); + if (!pte) + return ERR_PTR(-EAGAIN); + + order = first_order(orders); + while (orders) { + addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order); + vmf->pte = pte + pte_index(addr); + if (pte_range_none(vmf->pte, 1 << order)) + break; + order = next_order(&orders, order); + } + + vmf->pte = NULL; + pte_unmap(pte); + + gfp = vma_thp_gfp_mask(vma); + + while (orders) { + addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order); + folio = vma_alloc_folio(gfp, order, vma, addr, true); + if (folio) { + clear_huge_page(&folio->page, addr, 1 << order); + return folio; + } + order = next_order(&orders, order); + } + +fallback: + return vma_alloc_zeroed_movable_folio(vma, vmf->address); +} +#else +#define alloc_anon_folio(vmf) \ + vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address) +#endif + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4129,6 +4207,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) { + int i; + int nr_pages = 1; + unsigned long addr = vmf->address; bool uffd_wp = vmf_orig_pte_uffd_wp(vmf); struct vm_area_struct *vma = vmf->vma; struct folio *folio; @@ -4173,10 +4254,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* Allocate our own private page. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + folio = alloc_anon_folio(vmf); + if (IS_ERR(folio)) + return 0; if (!folio) goto oom; + nr_pages = folio_nr_pages(folio); + addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; folio_throttle_swaprate(folio, GFP_KERNEL); @@ -4193,12 +4279,13 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry), vma); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); if (!vmf->pte) goto release; - if (vmf_pte_changed(vmf)) { - update_mmu_tlb(vma, vmf->address, vmf->pte); + if ((nr_pages == 1 && vmf_pte_changed(vmf)) || + (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages))) { + for (i = 0; i < nr_pages; i++) + update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i); goto release; } @@ -4213,16 +4300,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); setpte: if (uffd_wp) entry = pte_mkuffd_wp(entry); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr_pages); /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, addr, vmf->pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Wed Nov 22 16:29:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEC23C61D99 for ; Wed, 22 Nov 2023 16:30:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61A828D0041; Wed, 22 Nov 2023 11:30:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C9598D0008; Wed, 22 Nov 2023 11:30:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 469568D0041; Wed, 22 Nov 2023 11:30:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2C1B08D0008 for ; Wed, 22 Nov 2023 11:30:23 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EDA2C1A100D for ; Wed, 22 Nov 2023 16:30:22 +0000 (UTC) X-FDA: 81486127884.23.71C8D7A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id E347FA001D for ; Wed, 22 Nov 2023 16:30:20 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670621; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/EYrnf/ZbB63lyuSv0fKQ4km7AIKVUkTF9kM/Ly2yKI=; b=i3kEVo0UNqNrhOC7cbzyrKk72f2V5Sqyo/VmgfHA6sYxXA+IX6VN91SJhl2jVzQXNRcQq/ BBlOpfMta0qnd6NJ4wkQTE2UxfIvgGqHp9fRvInEgNehJjdglxQuzp/fwWYzySKaHwZP4c 1kaUqfrxh1C1DkOhlpPpQnDFHdCBybQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670621; a=rsa-sha256; cv=none; b=GeGUXZ7dpZKdkTEQq8ztkY/JhpGRABm8FJErrILFRz6Et/1x6+v7o2IrDDuaSV85xrAVrS k4QpwyDf9pJboMTK1lUbnCYGiTviaym+J8+igtzGKyITQNDIHZdFCRFWan/OMKk3StZfAF yjemSY0I8bzUML1fi8oEwTEd1z2q8X0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CF476C15; Wed, 22 Nov 2023 08:31:06 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 60F583F73F; Wed, 22 Nov 2023 08:30:17 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 05/10] selftests/mm/kugepaged: Restore thp settings at exit Date: Wed, 22 Nov 2023 16:29:45 +0000 Message-Id: <20231122162950.3854897-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E347FA001D X-Rspam-User: X-Stat-Signature: uia8pn58o3krj6ciza6dt9c433oqi1hn X-Rspamd-Server: rspam01 X-HE-Tag: 1700670620-211930 X-HE-Meta: U2FsdGVkX19gHjY7Zqnc/MvVaFHynhIWUTphiWnk+pbHT/j49HkJvp5VI7C1B4O2bnBaX8AEcxZqJ+HJ/HU2h0iMT3zAuNMVMcUMghE8hsLzaCc8J4y7Yucs/a0y7ZG7ALuth5LZVarMIVy8Ov9iH8A2QxNiz46NsLXYhZzEGJaX0mrO0yDgsI199cXwd3Cd+sQN1kYH9In2bsyI4FUKEI9Z/WWAiqSTs2NNJtwLQE0wwzks9jfenYo1KXLSwvDcQv8iWdlK7TQ9+mHPeaLN0FcVK3wKHLVwdZLBT3PXt2wZ4YdmjHO6imVHOEBr3EkENTIFtQlJCnSn2N7RdRj0kXRJe158eT/IOjfWdJUduTUvguz2r4SMNt8UuXRCjeWe0kyQokstR/keS1CxQsOqs51hoXKcfuwpMhESigDgXGPU/Bf8Zbdfil4oxgGIMuPKmFQda4Fex8+E8HLBl51N/6D7Ea3EkQTjnWpUQFQWvDF2N94SPuymuSfsjjXv+NbzwsmFdjyJFus4ZNKLO83g7ETulLgyfDLfLLtfq4khWmD+XU/39zTnSKtE0dgmRwpcNKD8QILtaBHiHobD+XHZdTYZ9vqp0CKe+qs8LntmWOiO1K9v3z4wBfWKOcZGkqNbmGxRmrAH6LXpfMDSRYxy1PhtEs5uWNAvRyfwldSQhXQ2AEq8R4CfsGdHEliQlMMDhQ33qp6cuyk6JJ6vYeua4M2tUkQ8r7HC3sv1YLzbC95Rlce+F4ZRh65uhrArH0kc9JaiAhMq/BjIE0kgNz0dANYc71mcTaniuPJFKc5+nmdBNuZW4BvYk5yCp2ucnUsMp9wZg5hriPnznDkETGaA6UgWhcCAErbG0eMLAK9Aut6AZlE6bF8qmRlVqwAY3qcUDLaP9SH4HG45oxr8YMSNjG0cwdXJyYhmFrVvP+UUue/nWqqrlxYiKjElkLufjVikE+mL6zz9hbfiCiVAodS Vs33Y0pU cMa/5juBi/Coi6WWdt71TEKP8aIegAaf/dYMLUvgpz6sbgQylzBWNmC3ThCX9u4tOl+buVusXztFq9wuRs23upBx62wUZmHJ14NheYyUNJACujEhn+Tg5YM1CrFedDChVmJOg1Ro5zO5IWwmnAeJklpADPt52f+RIYK+jqjKodMqpOcvXBhfcPY6eCDF3AmNpM9+8YxyM20VbhDURkCcfScCZdTDFM1AQ3qL+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously, the saved thp settings would be restored upon a signal or at the natural end of the test suite. But there are some tests that directly call exit() upon failure. In this case, the thp settings were not being restored, which could then influence other tests. Fix this by installing an atexit() handler to do the actual restore. The signal handler can now just call exit() and the atexit handler is invoked. Signed-off-by: Ryan Roberts Reviewed-by: Alistair Popple --- tools/testing/selftests/mm/khugepaged.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) -- 2.25.1 diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 030667cb5533..fc47a1c4944c 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -374,18 +374,22 @@ static void pop_settings(void) write_settings(current_settings()); } -static void restore_settings(int sig) +static void restore_settings_atexit(void) { if (skip_settings_restore) - goto out; + return; printf("Restore THP and khugepaged settings..."); write_settings(&saved_settings); success("OK"); - if (sig) - exit(EXIT_FAILURE); -out: - exit(exit_status); + + skip_settings_restore = true; +} + +static void restore_settings(int sig) +{ + /* exit() will invoke the restore_settings_atexit handler. */ + exit(sig ? EXIT_FAILURE : exit_status); } static void save_settings(void) @@ -415,6 +419,7 @@ static void save_settings(void) success("OK"); + atexit(restore_settings_atexit); signal(SIGTERM, restore_settings); signal(SIGINT, restore_settings); signal(SIGHUP, restore_settings); From patchwork Wed Nov 22 16:29:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF0F6C61D99 for ; Wed, 22 Nov 2023 16:30:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BAD18D0047; Wed, 22 Nov 2023 11:30:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4697B8D0008; Wed, 22 Nov 2023 11:30:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BB7E8D0047; Wed, 22 Nov 2023 11:30:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 17FF08D0008 for ; Wed, 22 Nov 2023 11:30:26 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B388AB557C for ; Wed, 22 Nov 2023 16:30:25 +0000 (UTC) X-FDA: 81486128010.05.53E0FC2 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id BEF52A0030 for ; Wed, 22 Nov 2023 16:30:23 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670623; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=++A/GjgI5odLCYj+I8EUZ7TNOXj/lOsMg4n9pD9vfbA=; b=TibaZYDjmpz7Vni3gHk2HU4Uw3ymcDReKJl1HAIwgDMwma2Q8+v01A82ysGdQbWU+Pys2l Ye8IzyWODWMJkiRx3U3UoVeASUlGLOfFCpk5ikJmv4Ypx7M5tJPPNwjc2zu1XAd9eLYvkL OK7uAurMyAmBdmYV90fxqv07ROjArm4= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670623; a=rsa-sha256; cv=none; b=uD1UC4SYyAvoxI8UC0Kbr5kNkQT6P4Qez9TiTPNHsBbLubOEk1tddQwJPH5wmrIHPWDUzd 26B+V4M8d4SibQ80t4diMqK+uChQoaEk/aSlzTsNq7NFk0i4q3bnW5hg9yvmzIIXXEuGJ6 iBz7oheEwk7/9xD/lQe2HawDaiyLbvM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BCE561650; Wed, 22 Nov 2023 08:31:09 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 33D6B3F73F; Wed, 22 Nov 2023 08:30:20 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 06/10] selftests/mm: Factor out thp settings management Date: Wed, 22 Nov 2023 16:29:46 +0000 Message-Id: <20231122162950.3854897-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BEF52A0030 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1zitxog7ipkk5zgtn51ew49855ubcdog X-HE-Tag: 1700670623-974975 X-HE-Meta: U2FsdGVkX19H5dlVGoDuDSoOV5nwuvVi8AzA1gnk5qdRjRkojOsOEMB+r/ejfsE62iak7bqW3OV4UHT7WRpxKlXPEh/9Lm8EzQNaJ4wnHeaZR7YRYW/BsJJ7jtnkved6tpuXhynso9pYZ8do5FcXsTdlrTXrxZY/x8E3a5V0ceHYSmxXth9t9pM75ljQPDeLdpD+KcrFNFbb2Kpn/iMPjJT7SHklaUFnNA5awIUwWTjVeKIv4IKYb8vCZvH4Ri+x1dH+ed+rJqib47UX9CTq8Y/sO6rVnQ0IPGyZQ3am/V0jZoixkUFCSgnJQrF7dCjaE/Q4xrLrWpoWozmj/IwtgdiJNywuBRBzbGrjOKho90vyZVShipmn8/wjK7nrtli+BK9ojb5ck4PJ63qBwAwMO+LxwLlpLDmfT5Egv4KOjOus9VKjJcE2Vir1wemOnFAUDgTqUsBdOAFA1UV/fsQC4txxrI032Sejzs7UjdezBJSLxiMJEBpFmWY9tUyfU+0JAGOZWabshsDb1ebOcHA+hOReceioFPFlerMfVnHXGnZY/k3Fe3wJlg61Zsy4/ouFujOHKt7reu+xBg+GEfXIexUVZJ+jwnHT38My8o1fqhuxZM6gkQrT+Tl9iZlzUCfPygUjhn7WkUp9/vJak6ZkAdO+kwdSzCNl7F6mK+dld/IfQHJrHk35GQrmp+Gwpz/XljAGVeBcLtnX0JQcMDcqJBtMDMmmb3el094h8jArOdSVrUn5svs0gmamf8RrteHtqxJ3nGbErTcQHgrXTmL74nGWzJlSCfYw4MDRz+3PatlIZFcnUWYnuFa9i+XidIk+3B8tm1IJv2e28AVBsxT7VvcvfkEAgNb7rE+vm3SNBX5eFnZicm4c2k6xYlW20yCoewofTI1jcZMXBldZl7CMYbbZRjAqlZ2O37NmNbAdM1NMQw4cnvrKDu90CQsHjOFP6uQeI8EFmxi40psSHI0 oVZYK1OC ktneVD0P0Yl6ObUVSq/Qkr8CpXUcsT/abQ0wvv9KwY86yvlfSknb8a1uda2GjGbdfZEkryhxUW+sqzjI+dI/ZXoylBSMhFEoR7fIV5JwBhFzsB2xJ4j+tCXWyKM4EJw2ugtzaCOa1ExbVTD01wgn48crsqGXGRZCcS1VrrCDa0QpBGB3eFWB0c055uzJBoZ13co2N2DmKLDTPokfFBUuvEWDSvuIjC74AHWyPWujZ4N9ox4U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The khugepaged test has a useful framework for save/restore/pop/push of all thp settings via the sysfs interface. This will be useful to explicitly control small-sized THP settings in other tests, so let's move it out of khugepaged and into its own thp_settings.[c|h] utility. Signed-off-by: Ryan Roberts Tested-by: Alistair Popple --- tools/testing/selftests/mm/Makefile | 4 +- tools/testing/selftests/mm/khugepaged.c | 346 ++-------------------- tools/testing/selftests/mm/thp_settings.c | 296 ++++++++++++++++++ tools/testing/selftests/mm/thp_settings.h | 71 +++++ 4 files changed, 391 insertions(+), 326 deletions(-) create mode 100644 tools/testing/selftests/mm/thp_settings.c create mode 100644 tools/testing/selftests/mm/thp_settings.h -- 2.25.1 diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 78dfec8bc676..a3eb20c186e7 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -117,8 +117,8 @@ TEST_FILES += va_high_addr_switch.sh include ../lib.mk -$(TEST_GEN_PROGS): vm_util.c -$(TEST_GEN_FILES): vm_util.c +$(TEST_GEN_PROGS): vm_util.c thp_settings.c +$(TEST_GEN_FILES): vm_util.c thp_settings.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index fc47a1c4944c..b15e7fd70176 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -22,13 +22,13 @@ #include "linux/magic.h" #include "vm_util.h" +#include "thp_settings.h" #define BASE_ADDR ((void *)(1UL << 30)) static unsigned long hpage_pmd_size; static unsigned long page_size; static int hpage_pmd_nr; -#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/" #define PID_SMAPS "/proc/self/smaps" #define TEST_FILE "collapse_test_file" @@ -71,78 +71,7 @@ struct file_info { }; static struct file_info finfo; - -enum thp_enabled { - THP_ALWAYS, - THP_MADVISE, - THP_NEVER, -}; - -static const char *thp_enabled_strings[] = { - "always", - "madvise", - "never", - NULL -}; - -enum thp_defrag { - THP_DEFRAG_ALWAYS, - THP_DEFRAG_DEFER, - THP_DEFRAG_DEFER_MADVISE, - THP_DEFRAG_MADVISE, - THP_DEFRAG_NEVER, -}; - -static const char *thp_defrag_strings[] = { - "always", - "defer", - "defer+madvise", - "madvise", - "never", - NULL -}; - -enum shmem_enabled { - SHMEM_ALWAYS, - SHMEM_WITHIN_SIZE, - SHMEM_ADVISE, - SHMEM_NEVER, - SHMEM_DENY, - SHMEM_FORCE, -}; - -static const char *shmem_enabled_strings[] = { - "always", - "within_size", - "advise", - "never", - "deny", - "force", - NULL -}; - -struct khugepaged_settings { - bool defrag; - unsigned int alloc_sleep_millisecs; - unsigned int scan_sleep_millisecs; - unsigned int max_ptes_none; - unsigned int max_ptes_swap; - unsigned int max_ptes_shared; - unsigned long pages_to_scan; -}; - -struct settings { - enum thp_enabled thp_enabled; - enum thp_defrag thp_defrag; - enum shmem_enabled shmem_enabled; - bool use_zero_page; - struct khugepaged_settings khugepaged; - unsigned long read_ahead_kb; -}; - -static struct settings saved_settings; static bool skip_settings_restore; - static int exit_status; static void success(const char *msg) @@ -161,226 +90,13 @@ static void skip(const char *msg) printf(" \e[33m%s\e[0m\n", msg); } -static int read_file(const char *path, char *buf, size_t buflen) -{ - int fd; - ssize_t numread; - - fd = open(path, O_RDONLY); - if (fd == -1) - return 0; - - numread = read(fd, buf, buflen - 1); - if (numread < 1) { - close(fd); - return 0; - } - - buf[numread] = '\0'; - close(fd); - - return (unsigned int) numread; -} - -static int write_file(const char *path, const char *buf, size_t buflen) -{ - int fd; - ssize_t numwritten; - - fd = open(path, O_WRONLY); - if (fd == -1) { - printf("open(%s)\n", path); - exit(EXIT_FAILURE); - return 0; - } - - numwritten = write(fd, buf, buflen - 1); - close(fd); - if (numwritten < 1) { - printf("write(%s)\n", buf); - exit(EXIT_FAILURE); - return 0; - } - - return (unsigned int) numwritten; -} - -static int read_string(const char *name, const char *strings[]) -{ - char path[PATH_MAX]; - char buf[256]; - char *c; - int ret; - - ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); - if (ret >= PATH_MAX) { - printf("%s: Pathname is too long\n", __func__); - exit(EXIT_FAILURE); - } - - if (!read_file(path, buf, sizeof(buf))) { - perror(path); - exit(EXIT_FAILURE); - } - - c = strchr(buf, '['); - if (!c) { - printf("%s: Parse failure\n", __func__); - exit(EXIT_FAILURE); - } - - c++; - memmove(buf, c, sizeof(buf) - (c - buf)); - - c = strchr(buf, ']'); - if (!c) { - printf("%s: Parse failure\n", __func__); - exit(EXIT_FAILURE); - } - *c = '\0'; - - ret = 0; - while (strings[ret]) { - if (!strcmp(strings[ret], buf)) - return ret; - ret++; - } - - printf("Failed to parse %s\n", name); - exit(EXIT_FAILURE); -} - -static void write_string(const char *name, const char *val) -{ - char path[PATH_MAX]; - int ret; - - ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); - if (ret >= PATH_MAX) { - printf("%s: Pathname is too long\n", __func__); - exit(EXIT_FAILURE); - } - - if (!write_file(path, val, strlen(val) + 1)) { - perror(path); - exit(EXIT_FAILURE); - } -} - -static const unsigned long _read_num(const char *path) -{ - char buf[21]; - - if (read_file(path, buf, sizeof(buf)) < 0) { - perror("read_file(read_num)"); - exit(EXIT_FAILURE); - } - - return strtoul(buf, NULL, 10); -} - -static const unsigned long read_num(const char *name) -{ - char path[PATH_MAX]; - int ret; - - ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); - if (ret >= PATH_MAX) { - printf("%s: Pathname is too long\n", __func__); - exit(EXIT_FAILURE); - } - return _read_num(path); -} - -static void _write_num(const char *path, unsigned long num) -{ - char buf[21]; - - sprintf(buf, "%ld", num); - if (!write_file(path, buf, strlen(buf) + 1)) { - perror(path); - exit(EXIT_FAILURE); - } -} - -static void write_num(const char *name, unsigned long num) -{ - char path[PATH_MAX]; - int ret; - - ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); - if (ret >= PATH_MAX) { - printf("%s: Pathname is too long\n", __func__); - exit(EXIT_FAILURE); - } - _write_num(path, num); -} - -static void write_settings(struct settings *settings) -{ - struct khugepaged_settings *khugepaged = &settings->khugepaged; - - write_string("enabled", thp_enabled_strings[settings->thp_enabled]); - write_string("defrag", thp_defrag_strings[settings->thp_defrag]); - write_string("shmem_enabled", - shmem_enabled_strings[settings->shmem_enabled]); - write_num("use_zero_page", settings->use_zero_page); - - write_num("khugepaged/defrag", khugepaged->defrag); - write_num("khugepaged/alloc_sleep_millisecs", - khugepaged->alloc_sleep_millisecs); - write_num("khugepaged/scan_sleep_millisecs", - khugepaged->scan_sleep_millisecs); - write_num("khugepaged/max_ptes_none", khugepaged->max_ptes_none); - write_num("khugepaged/max_ptes_swap", khugepaged->max_ptes_swap); - write_num("khugepaged/max_ptes_shared", khugepaged->max_ptes_shared); - write_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan); - - if (file_ops && finfo.type == VMA_FILE) - _write_num(finfo.dev_queue_read_ahead_path, - settings->read_ahead_kb); -} - -#define MAX_SETTINGS_DEPTH 4 -static struct settings settings_stack[MAX_SETTINGS_DEPTH]; -static int settings_index; - -static struct settings *current_settings(void) -{ - if (!settings_index) { - printf("Fail: No settings set"); - exit(EXIT_FAILURE); - } - return settings_stack + settings_index - 1; -} - -static void push_settings(struct settings *settings) -{ - if (settings_index >= MAX_SETTINGS_DEPTH) { - printf("Fail: Settings stack exceeded"); - exit(EXIT_FAILURE); - } - settings_stack[settings_index++] = *settings; - write_settings(current_settings()); -} - -static void pop_settings(void) -{ - if (settings_index <= 0) { - printf("Fail: Settings stack empty"); - exit(EXIT_FAILURE); - } - --settings_index; - write_settings(current_settings()); -} - static void restore_settings_atexit(void) { if (skip_settings_restore) return; printf("Restore THP and khugepaged settings..."); - write_settings(&saved_settings); + thp_restore_settings(); success("OK"); skip_settings_restore = true; @@ -395,27 +111,9 @@ static void restore_settings(int sig) static void save_settings(void) { printf("Save THP and khugepaged settings..."); - saved_settings = (struct settings) { - .thp_enabled = read_string("enabled", thp_enabled_strings), - .thp_defrag = read_string("defrag", thp_defrag_strings), - .shmem_enabled = - read_string("shmem_enabled", shmem_enabled_strings), - .use_zero_page = read_num("use_zero_page"), - }; - saved_settings.khugepaged = (struct khugepaged_settings) { - .defrag = read_num("khugepaged/defrag"), - .alloc_sleep_millisecs = - read_num("khugepaged/alloc_sleep_millisecs"), - .scan_sleep_millisecs = - read_num("khugepaged/scan_sleep_millisecs"), - .max_ptes_none = read_num("khugepaged/max_ptes_none"), - .max_ptes_swap = read_num("khugepaged/max_ptes_swap"), - .max_ptes_shared = read_num("khugepaged/max_ptes_shared"), - .pages_to_scan = read_num("khugepaged/pages_to_scan"), - }; if (file_ops && finfo.type == VMA_FILE) - saved_settings.read_ahead_kb = - _read_num(finfo.dev_queue_read_ahead_path); + thp_set_read_ahead_path(finfo.dev_queue_read_ahead_path); + thp_save_settings(); success("OK"); @@ -798,7 +496,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages, struct mem_ops *ops, bool expect) { int ret; - struct settings settings = *current_settings(); + struct thp_settings settings = *thp_current_settings(); printf("%s...", msg); @@ -808,7 +506,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages, */ settings.thp_enabled = THP_NEVER; settings.shmem_enabled = SHMEM_NEVER; - push_settings(&settings); + thp_push_settings(&settings); /* Clear VM_NOHUGEPAGE */ madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE); @@ -820,7 +518,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages, else success("OK"); - pop_settings(); + thp_pop_settings(); } static void madvise_collapse(const char *msg, char *p, int nr_hpages, @@ -850,13 +548,13 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages, madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE); /* Wait until the second full_scan completed */ - full_scans = read_num("khugepaged/full_scans") + 2; + full_scans = thp_read_num("khugepaged/full_scans") + 2; printf("%s...", msg); while (timeout--) { if (ops->check_huge(p, nr_hpages)) break; - if (read_num("khugepaged/full_scans") >= full_scans) + if (thp_read_num("khugepaged/full_scans") >= full_scans) break; printf("."); usleep(TICK); @@ -911,11 +609,11 @@ static bool is_tmpfs(struct mem_ops *ops) static void alloc_at_fault(void) { - struct settings settings = *current_settings(); + struct thp_settings settings = *thp_current_settings(); char *p; settings.thp_enabled = THP_ALWAYS; - push_settings(&settings); + thp_push_settings(&settings); p = alloc_mapping(1); *p = 1; @@ -925,7 +623,7 @@ static void alloc_at_fault(void) else fail("Fail"); - pop_settings(); + thp_pop_settings(); madvise(p, page_size, MADV_DONTNEED); printf("Split huge PMD on MADV_DONTNEED..."); @@ -973,11 +671,11 @@ static void collapse_single_pte_entry(struct collapse_context *c, struct mem_ops static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *ops) { int max_ptes_none = hpage_pmd_nr / 2; - struct settings settings = *current_settings(); + struct thp_settings settings = *thp_current_settings(); void *p; settings.khugepaged.max_ptes_none = max_ptes_none; - push_settings(&settings); + thp_push_settings(&settings); p = ops->setup_area(1); @@ -1002,7 +700,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o } skip: ops->cleanup_area(p, hpage_pmd_size); - pop_settings(); + thp_pop_settings(); } static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_ops *ops) @@ -1033,7 +731,7 @@ static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_op static void collapse_max_ptes_swap(struct collapse_context *c, struct mem_ops *ops) { - int max_ptes_swap = read_num("khugepaged/max_ptes_swap"); + int max_ptes_swap = thp_read_num("khugepaged/max_ptes_swap"); void *p; p = ops->setup_area(1); @@ -1250,11 +948,11 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o fail("Fail"); ops->fault(p, 0, page_size); - write_num("khugepaged/max_ptes_shared", hpage_pmd_nr - 1); + thp_write_num("khugepaged/max_ptes_shared", hpage_pmd_nr - 1); c->collapse("Collapse PTE table full of compound pages in child", p, 1, ops, true); - write_num("khugepaged/max_ptes_shared", - current_settings()->khugepaged.max_ptes_shared); + thp_write_num("khugepaged/max_ptes_shared", + thp_current_settings()->khugepaged.max_ptes_shared); validate_memory(p, 0, hpage_pmd_size); ops->cleanup_area(p, hpage_pmd_size); @@ -1275,7 +973,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops *ops) { - int max_ptes_shared = read_num("khugepaged/max_ptes_shared"); + int max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared"); int wstatus; void *p; @@ -1443,7 +1141,7 @@ static void parse_test_type(int argc, const char **argv) int main(int argc, const char **argv) { - struct settings default_settings = { + struct thp_settings default_settings = { .thp_enabled = THP_MADVISE, .thp_defrag = THP_DEFRAG_ALWAYS, .shmem_enabled = SHMEM_ADVISE, @@ -1484,7 +1182,7 @@ int main(int argc, const char **argv) default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8; save_settings(); - push_settings(&default_settings); + thp_push_settings(&default_settings); alloc_at_fault(); diff --git a/tools/testing/selftests/mm/thp_settings.c b/tools/testing/selftests/mm/thp_settings.c new file mode 100644 index 000000000000..5e8ec792cac7 --- /dev/null +++ b/tools/testing/selftests/mm/thp_settings.c @@ -0,0 +1,296 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include + +#include "thp_settings.h" + +#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/" +#define MAX_SETTINGS_DEPTH 4 +static struct thp_settings settings_stack[MAX_SETTINGS_DEPTH]; +static int settings_index; +static struct thp_settings saved_settings; +static char dev_queue_read_ahead_path[PATH_MAX]; + +static const char * const thp_enabled_strings[] = { + "always", + "madvise", + "never", + NULL +}; + +static const char * const thp_defrag_strings[] = { + "always", + "defer", + "defer+madvise", + "madvise", + "never", + NULL +}; + +static const char * const shmem_enabled_strings[] = { + "always", + "within_size", + "advise", + "never", + "deny", + "force", + NULL +}; + +int read_file(const char *path, char *buf, size_t buflen) +{ + int fd; + ssize_t numread; + + fd = open(path, O_RDONLY); + if (fd == -1) + return 0; + + numread = read(fd, buf, buflen - 1); + if (numread < 1) { + close(fd); + return 0; + } + + buf[numread] = '\0'; + close(fd); + + return (unsigned int) numread; +} + +int write_file(const char *path, const char *buf, size_t buflen) +{ + int fd; + ssize_t numwritten; + + fd = open(path, O_WRONLY); + if (fd == -1) { + printf("open(%s)\n", path); + exit(EXIT_FAILURE); + return 0; + } + + numwritten = write(fd, buf, buflen - 1); + close(fd); + if (numwritten < 1) { + printf("write(%s)\n", buf); + exit(EXIT_FAILURE); + return 0; + } + + return (unsigned int) numwritten; +} + +const unsigned long read_num(const char *path) +{ + char buf[21]; + + if (read_file(path, buf, sizeof(buf)) < 0) { + perror("read_file()"); + exit(EXIT_FAILURE); + } + + return strtoul(buf, NULL, 10); +} + +void write_num(const char *path, unsigned long num) +{ + char buf[21]; + + sprintf(buf, "%ld", num); + if (!write_file(path, buf, strlen(buf) + 1)) { + perror(path); + exit(EXIT_FAILURE); + } +} + +int thp_read_string(const char *name, const char * const strings[]) +{ + char path[PATH_MAX]; + char buf[256]; + char *c; + int ret; + + ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); + if (ret >= PATH_MAX) { + printf("%s: Pathname is too long\n", __func__); + exit(EXIT_FAILURE); + } + + if (!read_file(path, buf, sizeof(buf))) { + perror(path); + exit(EXIT_FAILURE); + } + + c = strchr(buf, '['); + if (!c) { + printf("%s: Parse failure\n", __func__); + exit(EXIT_FAILURE); + } + + c++; + memmove(buf, c, sizeof(buf) - (c - buf)); + + c = strchr(buf, ']'); + if (!c) { + printf("%s: Parse failure\n", __func__); + exit(EXIT_FAILURE); + } + *c = '\0'; + + ret = 0; + while (strings[ret]) { + if (!strcmp(strings[ret], buf)) + return ret; + ret++; + } + + printf("Failed to parse %s\n", name); + exit(EXIT_FAILURE); +} + +void thp_write_string(const char *name, const char *val) +{ + char path[PATH_MAX]; + int ret; + + ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); + if (ret >= PATH_MAX) { + printf("%s: Pathname is too long\n", __func__); + exit(EXIT_FAILURE); + } + + if (!write_file(path, val, strlen(val) + 1)) { + perror(path); + exit(EXIT_FAILURE); + } +} + +const unsigned long thp_read_num(const char *name) +{ + char path[PATH_MAX]; + int ret; + + ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); + if (ret >= PATH_MAX) { + printf("%s: Pathname is too long\n", __func__); + exit(EXIT_FAILURE); + } + return read_num(path); +} + +void thp_write_num(const char *name, unsigned long num) +{ + char path[PATH_MAX]; + int ret; + + ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name); + if (ret >= PATH_MAX) { + printf("%s: Pathname is too long\n", __func__); + exit(EXIT_FAILURE); + } + write_num(path, num); +} + +void thp_read_settings(struct thp_settings *settings) +{ + *settings = (struct thp_settings) { + .thp_enabled = thp_read_string("enabled", thp_enabled_strings), + .thp_defrag = thp_read_string("defrag", thp_defrag_strings), + .shmem_enabled = + thp_read_string("shmem_enabled", shmem_enabled_strings), + .use_zero_page = thp_read_num("use_zero_page"), + }; + settings->khugepaged = (struct khugepaged_settings) { + .defrag = thp_read_num("khugepaged/defrag"), + .alloc_sleep_millisecs = + thp_read_num("khugepaged/alloc_sleep_millisecs"), + .scan_sleep_millisecs = + thp_read_num("khugepaged/scan_sleep_millisecs"), + .max_ptes_none = thp_read_num("khugepaged/max_ptes_none"), + .max_ptes_swap = thp_read_num("khugepaged/max_ptes_swap"), + .max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared"), + .pages_to_scan = thp_read_num("khugepaged/pages_to_scan"), + }; + if (dev_queue_read_ahead_path[0]) + settings->read_ahead_kb = read_num(dev_queue_read_ahead_path); +} + +void thp_write_settings(struct thp_settings *settings) +{ + struct khugepaged_settings *khugepaged = &settings->khugepaged; + + thp_write_string("enabled", thp_enabled_strings[settings->thp_enabled]); + thp_write_string("defrag", thp_defrag_strings[settings->thp_defrag]); + thp_write_string("shmem_enabled", + shmem_enabled_strings[settings->shmem_enabled]); + thp_write_num("use_zero_page", settings->use_zero_page); + + thp_write_num("khugepaged/defrag", khugepaged->defrag); + thp_write_num("khugepaged/alloc_sleep_millisecs", + khugepaged->alloc_sleep_millisecs); + thp_write_num("khugepaged/scan_sleep_millisecs", + khugepaged->scan_sleep_millisecs); + thp_write_num("khugepaged/max_ptes_none", khugepaged->max_ptes_none); + thp_write_num("khugepaged/max_ptes_swap", khugepaged->max_ptes_swap); + thp_write_num("khugepaged/max_ptes_shared", khugepaged->max_ptes_shared); + thp_write_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan); + + if (dev_queue_read_ahead_path[0]) + write_num(dev_queue_read_ahead_path, settings->read_ahead_kb); +} + +struct thp_settings *thp_current_settings(void) +{ + if (!settings_index) { + printf("Fail: No settings set"); + exit(EXIT_FAILURE); + } + return settings_stack + settings_index - 1; +} + +void thp_push_settings(struct thp_settings *settings) +{ + if (settings_index >= MAX_SETTINGS_DEPTH) { + printf("Fail: Settings stack exceeded"); + exit(EXIT_FAILURE); + } + settings_stack[settings_index++] = *settings; + thp_write_settings(thp_current_settings()); +} + +void thp_pop_settings(void) +{ + if (settings_index <= 0) { + printf("Fail: Settings stack empty"); + exit(EXIT_FAILURE); + } + --settings_index; + thp_write_settings(thp_current_settings()); +} + +void thp_restore_settings(void) +{ + thp_write_settings(&saved_settings); +} + +void thp_save_settings(void) +{ + thp_read_settings(&saved_settings); +} + +void thp_set_read_ahead_path(char *path) +{ + if (!path) { + dev_queue_read_ahead_path[0] = '\0'; + return; + } + + strncpy(dev_queue_read_ahead_path, path, + sizeof(dev_queue_read_ahead_path)); + dev_queue_read_ahead_path[sizeof(dev_queue_read_ahead_path) - 1] = '\0'; +} diff --git a/tools/testing/selftests/mm/thp_settings.h b/tools/testing/selftests/mm/thp_settings.h new file mode 100644 index 000000000000..ff3d98c30617 --- /dev/null +++ b/tools/testing/selftests/mm/thp_settings.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __THP_SETTINGS_H__ +#define __THP_SETTINGS_H__ + +#include +#include +#include + +enum thp_enabled { + THP_ALWAYS, + THP_MADVISE, + THP_NEVER, +}; + +enum thp_defrag { + THP_DEFRAG_ALWAYS, + THP_DEFRAG_DEFER, + THP_DEFRAG_DEFER_MADVISE, + THP_DEFRAG_MADVISE, + THP_DEFRAG_NEVER, +}; + +enum shmem_enabled { + SHMEM_ALWAYS, + SHMEM_WITHIN_SIZE, + SHMEM_ADVISE, + SHMEM_NEVER, + SHMEM_DENY, + SHMEM_FORCE, +}; + +struct khugepaged_settings { + bool defrag; + unsigned int alloc_sleep_millisecs; + unsigned int scan_sleep_millisecs; + unsigned int max_ptes_none; + unsigned int max_ptes_swap; + unsigned int max_ptes_shared; + unsigned long pages_to_scan; +}; + +struct thp_settings { + enum thp_enabled thp_enabled; + enum thp_defrag thp_defrag; + enum shmem_enabled shmem_enabled; + bool use_zero_page; + struct khugepaged_settings khugepaged; + unsigned long read_ahead_kb; +}; + +int read_file(const char *path, char *buf, size_t buflen); +int write_file(const char *path, const char *buf, size_t buflen); +const unsigned long read_num(const char *path); +void write_num(const char *path, unsigned long num); + +int thp_read_string(const char *name, const char * const strings[]); +void thp_write_string(const char *name, const char *val); +const unsigned long thp_read_num(const char *name); +void thp_write_num(const char *name, unsigned long num); + +void thp_write_settings(struct thp_settings *settings); +void thp_read_settings(struct thp_settings *settings); +struct thp_settings *thp_current_settings(void); +void thp_push_settings(struct thp_settings *settings); +void thp_pop_settings(void); +void thp_restore_settings(void); +void thp_save_settings(void); + +void thp_set_read_ahead_path(char *path); + +#endif /* __THP_SETTINGS_H__ */ From patchwork Wed Nov 22 16:29:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3836AC61D9B for ; Wed, 22 Nov 2023 16:30:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE50F8D0048; Wed, 22 Nov 2023 11:30:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B95838D0008; Wed, 22 Nov 2023 11:30:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A35C28D0048; Wed, 22 Nov 2023 11:30:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8C7A68D0008 for ; Wed, 22 Nov 2023 11:30:28 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5AAFA810BA for ; Wed, 22 Nov 2023 16:30:28 +0000 (UTC) X-FDA: 81486128136.15.1F708A6 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf19.hostedemail.com (Postfix) with ESMTP id 586ED1A000C for ; Wed, 22 Nov 2023 16:30:26 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670626; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c00HuZh8f7EUjjIu9Dd6BGADK5NutdsxCPhz2j3JNic=; b=OlP8VpIaHmM29jNtKXKoyB9nZTI+Wp48frL7OOQ60NBlfeJIhFkIiwikZZKJHoIA2LymC5 8BM/RWqdIR6byzsTpZ85i8u/NYuLceosYEon2MMWTvnp/J86EtZVUdiCWs6Cj2Y60dVLpu 0PS/PdWamff2SsbOEAbG8AKxEdD7Rio= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670626; a=rsa-sha256; cv=none; b=NJKBY6By9vse4+Rf5u3oLaArB1UlzWFjxMegTjZow1cSrfa9jG397CddUc3K+pKVm6hO4n Erlwe4RpJ8RbdV67lEOLf9MwDtuMdca3bpKHLDmrcsVxZFrluAhK9vbauv8YHXu+CNHtWK tBgBDiwUMcucT0CC8orG1yilWP4Vmhw= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8D2FA1595; Wed, 22 Nov 2023 08:31:12 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 20A5B3F73F; Wed, 22 Nov 2023 08:30:23 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 07/10] selftests/mm: Support small-sized THP interface in thp_settings Date: Wed, 22 Nov 2023 16:29:47 +0000 Message-Id: <20231122162950.3854897-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: 4o1oxcmqmj5pafuanmtgs4t6xhtsdebq X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 586ED1A000C X-Rspam-User: X-HE-Tag: 1700670626-760610 X-HE-Meta: U2FsdGVkX19518gA23Bb2nUtwSGE53hyj2caZuOd6iQBLIHco5O454zC1lWiZYRLhmyATFvLe0ss6IjFtuLrb1hGAX/VP619ou6mP8mGl/qLi4Bpe8VnZ25cY/q6BKLEq83gUiuDItQT5F/o/lRP6S08fovfSbe0aPtCKazcBbdAGoUokHCtU+iRZShUx8ufWKcXcaOWHIGtzQ4OgkwWiVP/tDyrgo93krbDMLhOu6xmSl3/FPDMEWAyBbf2mVBrXuEZjG79JPe/yFJz9WC9Bz56m6FrMJAVXmYuAifSEeVlHCj+Vf/6RFaqddNFhiStqMfpHHIai0xv4Qf4Z/qgN1TIqlP5Kq764yHRY4UOSHl8Zp2p+AlAQdvqc8FknxmKSsE4b57sTjWVQr/2C5qj/B0Z03yzyMRGwr8+Fx1ShLDU02ehiuRB7ezMAW8A8XwMFoKOPFlM1jSSPdODvDlQbXwyzmSeF7gstpzz1OUPGiPV0k0CNTZDdfXBCpW1ht1PIx7EZxx1TY7kJweXJUBNKPxPqGY5WKLiG9x0lwBTNq6bfIxBY7g/6Zl0JkIYqQfnsK/fo73Ys4Sz1I9iK6QBgHWqcxqgC/7IHHn5CS4m2st4YHfQULfrOWZLxA/DIc0nqiIPDL7GHr49WxuLQlHcY2f3vKgm8Y4aTCHk0AxiG8uw6lcSRU8tbZT87MDhp1q2mZUWTpiDc8sBjzt3nLUB7Sa1Skgh5cdfE6iAlvJ9pz5GHW6VExoiIcKOw7gGeD6PVwZDSfPePRzyw+t2gs4QQ2EY/Y4xEY50xXNT3rKw5KL14MmpkfacXf0BeR8JlyI1lx/puHkUAI3+4ptUHnNN5UPu0zbt+Aq66+HgIQLY0VxESgZuxKzgJg6Wf9xV1YPplQGolJtLfF0q0+NlTNIUSk31w5flBjw+o3mG90flDiStfDD4+H/6TxwTZVewoXBoQcvQW8ziedeaL4DDvuZ nDXQseLj gaatO1emGNnkyoQ9+Q2LCRHa8pI7E4+lzRqiB5DsKrN6Gl5x3lZRNOUMEpaTqJLEm4aflWemhrHhc/XUu8NsMAxwVVMYqKfU6Zt/6EXz6N+UNOUd7k8TDRbT3MAqLg9MSqy/0ZEd9m17ejaiRighB+Y+UI3gsHtoVhbZzEumN0SBAI03myHqRpeoyDj77QqH7nE/iwu6whROBrLd+mtiasOrOgdONd3SYWYNyxZ5zdZxRrjGuDN/l18Jv8Zve6+oRQ9+29oaEdJ6CvhgIn6/0QLVzZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Save and restore the new per-size hugepage enabled setting, if available on the running kernel. Since the number of per-size directories is not fixed, solve this as simply as possible by catering for a maximum number in the thp_settings struct (20). Each array index is the order. The value of THP_NEVER is changed to 0 so that all of these new settings default to THP_NEVER and the user only needs to fill in the ones they want to enable. Signed-off-by: Ryan Roberts --- tools/testing/selftests/mm/khugepaged.c | 3 ++ tools/testing/selftests/mm/thp_settings.c | 55 ++++++++++++++++++++++- tools/testing/selftests/mm/thp_settings.h | 11 ++++- 3 files changed, 67 insertions(+), 2 deletions(-) -- 2.25.1 diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index b15e7fd70176..473ba095cffd 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -1141,6 +1141,7 @@ static void parse_test_type(int argc, const char **argv) int main(int argc, const char **argv) { + int hpage_pmd_order; struct thp_settings default_settings = { .thp_enabled = THP_MADVISE, .thp_defrag = THP_DEFRAG_ALWAYS, @@ -1175,11 +1176,13 @@ int main(int argc, const char **argv) exit(EXIT_FAILURE); } hpage_pmd_nr = hpage_pmd_size / page_size; + hpage_pmd_order = __builtin_ctz(hpage_pmd_nr); default_settings.khugepaged.max_ptes_none = hpage_pmd_nr - 1; default_settings.khugepaged.max_ptes_swap = hpage_pmd_nr / 8; default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2; default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8; + default_settings.hugepages[hpage_pmd_order].enabled = THP_GLOBAL; save_settings(); thp_push_settings(&default_settings); diff --git a/tools/testing/selftests/mm/thp_settings.c b/tools/testing/selftests/mm/thp_settings.c index 5e8ec792cac7..3296995c8e4b 100644 --- a/tools/testing/selftests/mm/thp_settings.c +++ b/tools/testing/selftests/mm/thp_settings.c @@ -16,9 +16,10 @@ static struct thp_settings saved_settings; static char dev_queue_read_ahead_path[PATH_MAX]; static const char * const thp_enabled_strings[] = { + "never", "always", + "global", "madvise", - "never", NULL }; @@ -198,6 +199,10 @@ void thp_write_num(const char *name, unsigned long num) void thp_read_settings(struct thp_settings *settings) { + unsigned long orders = thp_supported_orders(); + char path[PATH_MAX]; + int i; + *settings = (struct thp_settings) { .thp_enabled = thp_read_string("enabled", thp_enabled_strings), .thp_defrag = thp_read_string("defrag", thp_defrag_strings), @@ -218,11 +223,26 @@ void thp_read_settings(struct thp_settings *settings) }; if (dev_queue_read_ahead_path[0]) settings->read_ahead_kb = read_num(dev_queue_read_ahead_path); + + for (i = 0; i < NR_ORDERS; i++) { + if (!((1 << i) & orders)) { + settings->hugepages[i].enabled = THP_NEVER; + continue; + } + snprintf(path, PATH_MAX, "hugepages-%ukB/enabled", + (getpagesize() >> 10) << i); + settings->hugepages[i].enabled = + thp_read_string(path, thp_enabled_strings); + } } void thp_write_settings(struct thp_settings *settings) { struct khugepaged_settings *khugepaged = &settings->khugepaged; + unsigned long orders = thp_supported_orders(); + char path[PATH_MAX]; + int enabled; + int i; thp_write_string("enabled", thp_enabled_strings[settings->thp_enabled]); thp_write_string("defrag", thp_defrag_strings[settings->thp_defrag]); @@ -242,6 +262,15 @@ void thp_write_settings(struct thp_settings *settings) if (dev_queue_read_ahead_path[0]) write_num(dev_queue_read_ahead_path, settings->read_ahead_kb); + + for (i = 0; i < NR_ORDERS; i++) { + if (!((1 << i) & orders)) + continue; + snprintf(path, PATH_MAX, "hugepages-%ukB/enabled", + (getpagesize() >> 10) << i); + enabled = settings->hugepages[i].enabled; + thp_write_string(path, thp_enabled_strings[enabled]); + } } struct thp_settings *thp_current_settings(void) @@ -294,3 +323,27 @@ void thp_set_read_ahead_path(char *path) sizeof(dev_queue_read_ahead_path)); dev_queue_read_ahead_path[sizeof(dev_queue_read_ahead_path) - 1] = '\0'; } + +unsigned long thp_supported_orders(void) +{ + unsigned long orders = 0; + char path[PATH_MAX]; + char buf[256]; + int ret; + int i; + + for (i = 0; i < NR_ORDERS; i++) { + ret = snprintf(path, PATH_MAX, THP_SYSFS "hugepages-%ukB/enabled", + (getpagesize() >> 10) << i); + if (ret >= PATH_MAX) { + printf("%s: Pathname is too long\n", __func__); + exit(EXIT_FAILURE); + } + + ret = read_file(path, buf, sizeof(buf)); + if (ret) + orders |= 1UL << i; + } + + return orders; +} diff --git a/tools/testing/selftests/mm/thp_settings.h b/tools/testing/selftests/mm/thp_settings.h index ff3d98c30617..8e71e323546c 100644 --- a/tools/testing/selftests/mm/thp_settings.h +++ b/tools/testing/selftests/mm/thp_settings.h @@ -7,9 +7,10 @@ #include enum thp_enabled { + THP_NEVER, THP_ALWAYS, + THP_GLOBAL, THP_MADVISE, - THP_NEVER, }; enum thp_defrag { @@ -29,6 +30,12 @@ enum shmem_enabled { SHMEM_FORCE, }; +#define NR_ORDERS 20 + +struct hugepages_settings { + enum thp_enabled enabled; +}; + struct khugepaged_settings { bool defrag; unsigned int alloc_sleep_millisecs; @@ -46,6 +53,7 @@ struct thp_settings { bool use_zero_page; struct khugepaged_settings khugepaged; unsigned long read_ahead_kb; + struct hugepages_settings hugepages[NR_ORDERS]; }; int read_file(const char *path, char *buf, size_t buflen); @@ -67,5 +75,6 @@ void thp_restore_settings(void); void thp_save_settings(void); void thp_set_read_ahead_path(char *path); +unsigned long thp_supported_orders(void); #endif /* __THP_SETTINGS_H__ */ From patchwork Wed Nov 22 16:29:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D24FC61D9B for ; Wed, 22 Nov 2023 16:30:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF6768D0049; Wed, 22 Nov 2023 11:30:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CCD608D0008; Wed, 22 Nov 2023 11:30:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B70458D0049; Wed, 22 Nov 2023 11:30:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A5E4D8D0008 for ; Wed, 22 Nov 2023 11:30:31 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 61D1C120DDC for ; Wed, 22 Nov 2023 16:30:31 +0000 (UTC) X-FDA: 81486128262.27.D1EB73C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf07.hostedemail.com (Postfix) with ESMTP id 5072340004 for ; Wed, 22 Nov 2023 16:30:29 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670629; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zWwgPOnMyq5HoZd3FqMCvj8FMHXJmiw0mbIe3GqzCJ4=; b=2VGC7c1ft+lgk5wsEVLPeYQiDE3/AUylWLHZb/XuNQJnAsBEzwkEM1uqanrB6glzxobG7T Lv6MGVuVxsqseA7c+/qCod/thWx+awx6n+S3EEq3g0XEpaSngkBOFcvBnDkXicaSb3Fu4z Qgw5tXHOat0OlnFkqhyv7vJaIlPjQQo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670629; a=rsa-sha256; cv=none; b=5/CnXXrcdZcNfkxFqvbHWysMcSPSKKvW7YO0UNvDOWNyvBoV61cXIsgmta28ooiSwFYeFF pjWGYaSFJJJbOE3WHgj773anDb0/xZiaT2gVa57TXCzuuNsv+0pCxRKRnznWs505X8mFr8 F+/V5MjYabFSB7lwkuGYWlBDoXupxsA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5F4E2C15; Wed, 22 Nov 2023 08:31:15 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E66B83F73F; Wed, 22 Nov 2023 08:30:25 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 08/10] selftests/mm/khugepaged: Enlighten for small-sized THP Date: Wed, 22 Nov 2023 16:29:48 +0000 Message-Id: <20231122162950.3854897-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5072340004 X-Stat-Signature: c1nx5o8qx831busjauzapz5f1kzo1icc X-Rspam-User: X-HE-Tag: 1700670629-543684 X-HE-Meta: U2FsdGVkX19AFGWdIgNKXAw29f1qOuwYANV4djM41v6WTK6kIHEttTA8J68bqxgEcE6TV/XM0ph5OOc7aGoq6CaHJ11NQtGOmiMeTBhWw+VLKcH3f9NvFShRv7c+NRyb0yWGdiFMj/u4rZ23aueKjsNoqd/Pf0SqkVDZLSjvTTgIhboPU3soc3kNoJSbTdQop4w3WUTewatPCdRjZq05uw5Q2NnATHRG01ursFxZjH99K8RY5HVIv0UXhnx/RZfjSG/0RTkAMenoVu0ps/l3MXr2iVhu/sDlz2bOBH+5wpQIbZR10TIQcNZLIGXYW21eLovpL0yzcFkwIJHIj4J644I/m6ZJlYHwo7IlOyWuixe4umEV8Ie9JS6TZ5Xh9y+rbcUXVakLG01b6FaD3hPV9vMn5LpM4O69NpqUe+Fz5QR8YWiVpjRqv6A7X7hsIT4QcO8orOb4ZfG6h7TgId/dLqRMm9JgVBkBAqyLV5QgE8lZp9DvNvDELX9CXg4nBmAR0vIfKVOoiDvzijE9e+gfsIPbQ36NN4ley6pBJDxERzNcGUXBQn9VDgjwIL24kd4gEhWO2ivcDS+de6QS6b4eSI02qU4YdnrGIAPgJEgp4Jw5DrAj3V1DglfEw1/FwVGRwwUgJKB3LFLjILU5LdoOmGbV3VorzYzZoRhXyair4sfO0Yt01YolW29ZhgqwK9eFhVUFz5FjkFgbASRsNJLMsOQ8HDRi6nWD9a/tNFNI0WpIaHCLdS4p2xFV/WQ2K4X96GFslIMPsYsPVYX33Vrd8UyiUztgYbs5dDVLDqHNsZaCstceQkpMuUz4amyTBCP1btH/0ktfdJzRz8/9oyAEJrP7DD/zL4guUveeH9moTywQW34RH5gy+NMiwUsJyCQYnwOG5Cz7RB7KX3+8Nd4lo3GCt7rlQY0Y9LicCfKtEJLQ4g5VMjPBLHH6WP2U0t4icyAMmz0fRb/CnT7XhQe 3Z2xm/mp rekvfmSQG/f5uU+GBof9cGga1Q4vTu1tyCBNeEz2YFUa9I6a2F9p/TFfJaVTtvpb/Wsv3rchlS+O1ixuysutyK60s/1c99vTrBqqvt794jJv1/MX96DWYIFRA0Z1Xt9LZurCyvrVfuw4TK9mcpcY9TxEcsGwyLPmJSUWzk756drGD5QYlyJkQBfcz94heuUwD4RHTZMcSilEWxbxuCeyV5BNHSc1TuikZZQQHLuIxa1vO3EM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The `collapse_max_ptes_none` test was previously failing when small-sized THP had enabled="always". The root cause is because the test faults in 1 page less than the threshold it set for collapsing. But when using small-sized THP we "over allocate" and therefore the threshold is passed, and collapse unexpectedly succeeds. Solve this by enlightening khugepaged selftest. Add a command line option to pass in the desired small-sized THP that should be used for all anonymous allocations. The harness will then explicitly configure small-sized THP as requested and modify the `collapse_max_ptes_none` test so that it faults in the threshold minus the number of pages in the configured small-sized THP. If no command line option is provided, default to order 0, as per previous behaviour. I chose to use an order in the command line interface, since this makes the interface agnostic of base page size, making it easier to invoke from run_vmtests.sh. Signed-off-by: Ryan Roberts --- tools/testing/selftests/mm/khugepaged.c | 48 +++++++++++++++++------ tools/testing/selftests/mm/run_vmtests.sh | 2 + 2 files changed, 39 insertions(+), 11 deletions(-) -- 2.25.1 diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 473ba095cffd..4d24f2eb158e 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -28,6 +28,7 @@ static unsigned long hpage_pmd_size; static unsigned long page_size; static int hpage_pmd_nr; +static int anon_order; #define PID_SMAPS "/proc/self/smaps" #define TEST_FILE "collapse_test_file" @@ -607,6 +608,11 @@ static bool is_tmpfs(struct mem_ops *ops) return ops == &__file_ops && finfo.type == VMA_SHMEM; } +static bool is_anon(struct mem_ops *ops) +{ + return ops == &__anon_ops; +} + static void alloc_at_fault(void) { struct thp_settings settings = *thp_current_settings(); @@ -673,6 +679,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o int max_ptes_none = hpage_pmd_nr / 2; struct thp_settings settings = *thp_current_settings(); void *p; + int fault_nr_pages = is_anon(ops) ? 1 << anon_order : 1; settings.khugepaged.max_ptes_none = max_ptes_none; thp_push_settings(&settings); @@ -686,10 +693,10 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o goto skip; } - ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none - 1) * page_size); + ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size); c->collapse("Maybe collapse with max_ptes_none exceeded", p, 1, ops, !c->enforce_pte_scan_limits); - validate_memory(p, 0, (hpage_pmd_nr - max_ptes_none - 1) * page_size); + validate_memory(p, 0, (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size); if (c->enforce_pte_scan_limits) { ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none) * page_size); @@ -1076,7 +1083,7 @@ static void madvise_retracted_page_tables(struct collapse_context *c, static void usage(void) { - fprintf(stderr, "\nUsage: ./khugepaged [dir]\n\n"); + fprintf(stderr, "\nUsage: ./khugepaged [OPTIONS] [dir]\n\n"); fprintf(stderr, "\t\t: :\n"); fprintf(stderr, "\t\t: [all|khugepaged|madvise]\n"); fprintf(stderr, "\t\t: [all|anon|file|shmem]\n"); @@ -1085,15 +1092,34 @@ static void usage(void) fprintf(stderr, "\tCONFIG_READ_ONLY_THP_FOR_FS=y\n"); fprintf(stderr, "\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n"); fprintf(stderr, "\tmounted with huge=madvise option for khugepaged tests to work\n"); + fprintf(stderr, "\n\tSupported Options:\n"); + fprintf(stderr, "\t\t-h: This help message.\n"); + fprintf(stderr, "\t\t-s: Small-sized THP size, expressed as page order.\n"); + fprintf(stderr, "\t\t Defaults to 0. Use this size for anon allocations.\n"); exit(1); } -static void parse_test_type(int argc, const char **argv) +static void parse_test_type(int argc, char **argv) { + int opt; char *buf; const char *token; - if (argc == 1) { + while ((opt = getopt(argc, argv, "s:h")) != -1) { + switch (opt) { + case 's': + anon_order = atoi(optarg); + break; + case 'h': + default: + usage(); + } + } + + argv += optind; + argc -= optind; + + if (argc == 0) { /* Backwards compatibility */ khugepaged_context = &__khugepaged_context; madvise_context = &__madvise_context; @@ -1101,7 +1127,7 @@ static void parse_test_type(int argc, const char **argv) return; } - buf = strdup(argv[1]); + buf = strdup(argv[0]); token = strsep(&buf, ":"); if (!strcmp(token, "all")) { @@ -1135,11 +1161,13 @@ static void parse_test_type(int argc, const char **argv) if (!file_ops) return; - if (argc != 3) + if (argc != 2) usage(); + + get_finfo(argv[1]); } -int main(int argc, const char **argv) +int main(int argc, char **argv) { int hpage_pmd_order; struct thp_settings default_settings = { @@ -1164,9 +1192,6 @@ int main(int argc, const char **argv) parse_test_type(argc, argv); - if (file_ops) - get_finfo(argv[2]); - setbuf(stdout, NULL); page_size = getpagesize(); @@ -1183,6 +1208,7 @@ int main(int argc, const char **argv) default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2; default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8; default_settings.hugepages[hpage_pmd_order].enabled = THP_GLOBAL; + default_settings.hugepages[anon_order].enabled = THP_ALWAYS; save_settings(); thp_push_settings(&default_settings); diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 00757445278e..f3fa2238daef 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -359,6 +359,8 @@ CATEGORY="cow" run_test ./cow CATEGORY="thp" run_test ./khugepaged +CATEGORY="thp" run_test ./khugepaged -s 2 + CATEGORY="thp" run_test ./transhuge-stress -d 20 CATEGORY="thp" run_test ./split_huge_page_test From patchwork Wed Nov 22 16:29:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 241AAC61D9B for ; Wed, 22 Nov 2023 16:30:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AACEC8D004A; Wed, 22 Nov 2023 11:30:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A5D2F8D0008; Wed, 22 Nov 2023 11:30:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D6C48D004A; Wed, 22 Nov 2023 11:30:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7A2DB8D0008 for ; Wed, 22 Nov 2023 11:30:34 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 22DBC140F0D for ; Wed, 22 Nov 2023 16:30:34 +0000 (UTC) X-FDA: 81486128388.09.4BA32EE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf19.hostedemail.com (Postfix) with ESMTP id 26EEA1A002B for ; Wed, 22 Nov 2023 16:30:31 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf19.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rAtan5yxh3gshsLYEs1fIKCCaelJr+Yedz3NEnvQJMU=; b=hIll1kETEfHrLWAH8skhJQoHJJvrd4p9gSfWtUkSUhQ10/tlt92kLs4W85Bmbzquvz0ADo CY7ecGEQ/KfjqPv3Boe541siPx3I7Fn4xOMP3kL/Djt/GAuAFP3W1eJBT5MS863115uDcG JeQFxPtEE3DKbgTD35fyriX5b2+ljeY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf19.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670632; a=rsa-sha256; cv=none; b=PWbzofNlPbmAsYmPxCnSFM4GLHZD//tWCHeGC6G3zL+zXp0ToHAHZzHbDwQ9rTeQ+DbYXf j6UDYRP+ixO6MgGuGwKdKXOAqcVnxkvpxXFHMk9P7RU1PKYY4cW0uuCF2ZKwT9AYvuCQsX S4C/eOK49p26f9BrX92BJOCEtpmNeII= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 317C81595; Wed, 22 Nov 2023 08:31:18 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B8B973F73F; Wed, 22 Nov 2023 08:30:28 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper Date: Wed, 22 Nov 2023 16:29:49 +0000 Message-Id: <20231122162950.3854897-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 26EEA1A002B X-Stat-Signature: okr68c5rxi863tineatak4rnq4n954ye X-HE-Tag: 1700670631-907583 X-HE-Meta: U2FsdGVkX1/XFaVndNaLB+Xvc7qKKg/jZziJZuUeL6+ocSslHBdm0tR9SNALI2SRnoyxl1/bTuDDnFjcvi+J+nbblrQmu52o7LS6s74Ou482+/6x/YscxhzNW9N1RfuqZB7B1R5d+bNXQd0qH2ch2PS8ymK8KMndk+W280UKa2s4IEiBDKOv3yrclLVXPjRTLr9Uuf2iIRv1wLjujNZd2CyxfJbyF7aZkAVzSlQIDwFQsYSr9tj/4u8p5DUol0qjsgVoqp7z/WkZwyu5owimXAMHdZSSkOCmTjLZm4GCRWrajOE945aAKYw1gwPeWBUz9wPqK2yiHp2cgevKG1WdifNZv0I08C/i5ciPCWlArg8KfQwrsttg0w6Maw0EloKqBNmZeYoWX7a5DvakWPoZcCFT9QKNLcExdofmFaxh4BNxKr8FkE92t58Ja49EDZMvztVH3/5kHD5j4nwbdxc4ZFv0F/AV3vE1NccjoG+hqR2l9at/foq30D1KLecEJxa0Z/tpOLJdAOl6ooK/ejIL1cv5zAH7lzMYGTfrZq3zUT1f2tdSZH2Al+OldCiidi7MRv3aaG2rp/uGe+ekqB/2CqHA3qti7odw1f7m3nE2d5wgzYrSokjSHkmR6y3p5CoZHvqE5JNzRn/q5Ep88O8HTNsytJA5dWrywOHq+Wgb8g4knBCKsqBQEzZOBI9fFor7PPJKXzAQ4nm/Z3Pb8eiovFQcf5Mkk8mZ8EcXGb/JucJapiS/JgiiUFierYTMJLSUt1SkE60w4veQpqJLLMY/N6l1TDS9UjVfUUfUqnfvtG/2WPjFbW9CP4Yof66qZ8KJgENLC7m4lTsLtRRsJEuNJsYBtoLxqUDTbTRyd5P4xOGKHegOOXfpH5ibNraWIxwWYM3a67O+jyA9eBqTk3mFoV9HGOb7uPe1vSVvwGkZjw4PpJ3aYQjGh3tmPsXO57RL1ab4qLSV2NTkGlmbUp0 e+Kh9kDr 70obOt3qwP30Evf89TiSszJK86V36b0/w6d5VCvpLL/RdCEosXXgf/aAmmBHe9ZG/i8g6df9upe/G71h/5jZFSQK8KEAkfIEfgXTarrQuavtleWgzdd23NZjxx8poHFXw1kOnAnnSgGUbOSCEKZiINDpDA/c7Xd46cU7vPW9wzvV4DWmX0yUR/oiEWy3qlfh63v2ov7wlDrVBotrAj43NepJ4b0E7g6QP8Vbujx9Mk+T6OWoO5Q4OkOjthDxH9CX3ZgztW4H6WUccEoIZw62xzFMi6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of small-sized THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a size parameter which tells the function what size THP it should operate on. A separate commit will utilize this change to add new tests for small-sized THP, where available. Signed-off-by: Ryan Roberts --- tools/testing/selftests/mm/cow.c | 146 +++++++++++++++++-------------- 1 file changed, 79 insertions(+), 67 deletions(-) -- 2.25.1 diff --git a/tools/testing/selftests/mm/cow.c b/tools/testing/selftests/mm/cow.c index 7324ce5363c0..d03c453cfd5c 100644 --- a/tools/testing/selftests/mm/cow.c +++ b/tools/testing/selftests/mm/cow.c @@ -32,7 +32,7 @@ static size_t pagesize; static int pagemap_fd; -static size_t thpsize; +static size_t pmdsize; static int nr_hugetlbsizes; static size_t hugetlbsizes[10]; static int gup_fd; @@ -734,14 +734,14 @@ enum thp_run { THP_RUN_PARTIAL_SHARED, }; -static void do_run_with_thp(test_fn fn, enum thp_run thp_run) +static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t size) { char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED; - size_t size, mmap_size, mremap_size; + size_t mmap_size, mremap_size; int ret; - /* For alignment purposes, we need twice the thp size. */ - mmap_size = 2 * thpsize; + /* For alignment purposes, we need twice the requested size. */ + mmap_size = 2 * size; mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (mmap_mem == MAP_FAILED) { @@ -749,36 +749,40 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) return; } - /* We need a THP-aligned memory area. */ - mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1)); + /* We need to naturally align the memory area. */ + mem = (char *)(((uintptr_t)mmap_mem + size) & ~(size - 1)); - ret = madvise(mem, thpsize, MADV_HUGEPAGE); + ret = madvise(mem, size, MADV_HUGEPAGE); if (ret) { ksft_test_result_fail("MADV_HUGEPAGE failed\n"); goto munmap; } /* - * Try to populate a THP. Touch the first sub-page and test if we get - * another sub-page populated automatically. + * Try to populate a THP. Touch the first sub-page and test if + * we get the last sub-page populated automatically. */ mem[0] = 0; - if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) { + if (!pagemap_is_populated(pagemap_fd, mem + size - pagesize)) { ksft_test_result_skip("Did not get a THP populated\n"); goto munmap; } - memset(mem, 0, thpsize); + memset(mem, 0, size); - size = thpsize; switch (thp_run) { case THP_RUN_PMD: case THP_RUN_PMD_SWAPOUT: + if (size != pmdsize) { + ksft_test_result_fail("test bug: can't PMD-map size\n"); + goto munmap; + } break; case THP_RUN_PTE: case THP_RUN_PTE_SWAPOUT: /* * Trigger PTE-mapping the THP by temporarily mapping a single - * subpage R/O. + * subpage R/O. This is a noop if the THP is not pmdsize (and + * therefore already PTE-mapped). */ ret = mprotect(mem + pagesize, pagesize, PROT_READ); if (ret) { @@ -797,7 +801,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) * Discard all but a single subpage of that PTE-mapped THP. What * remains is a single PTE mapping a single subpage. */ - ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DONTNEED); + ret = madvise(mem + pagesize, size - pagesize, MADV_DONTNEED); if (ret) { ksft_test_result_fail("MADV_DONTNEED failed\n"); goto munmap; @@ -809,7 +813,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) * Remap half of the THP. We need some new memory location * for that. */ - mremap_size = thpsize / 2; + mremap_size = size / 2; mremap_mem = mmap(NULL, mremap_size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (mem == MAP_FAILED) { @@ -830,7 +834,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) * child. This will result in some parts of the THP never * have been shared. */ - ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DONTFORK); + ret = madvise(mem + pagesize, size - pagesize, MADV_DONTFORK); if (ret) { ksft_test_result_fail("MADV_DONTFORK failed\n"); goto munmap; @@ -844,7 +848,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) } wait(&ret); /* Allow for sharing all pages again. */ - ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DOFORK); + ret = madvise(mem + pagesize, size - pagesize, MADV_DOFORK); if (ret) { ksft_test_result_fail("MADV_DOFORK failed\n"); goto munmap; @@ -875,52 +879,60 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) munmap(mremap_mem, mremap_size); } -static void run_with_thp(test_fn fn, const char *desc) +static void run_with_thp(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with THP\n", desc); - do_run_with_thp(fn, THP_RUN_PMD); + ksft_print_msg("[RUN] %s ... with THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_PMD, size); } -static void run_with_thp_swap(test_fn fn, const char *desc) +static void run_with_thp_swap(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc); - do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT); + ksft_print_msg("[RUN] %s ... with swapped-out THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT, size); } -static void run_with_pte_mapped_thp(test_fn fn, const char *desc) +static void run_with_pte_mapped_thp(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc); - do_run_with_thp(fn, THP_RUN_PTE); + ksft_print_msg("[RUN] %s ... with PTE-mapped THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_PTE, size); } -static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc) +static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc); - do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT); + ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT, size); } -static void run_with_single_pte_of_thp(test_fn fn, const char *desc) +static void run_with_single_pte_of_thp(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc); - do_run_with_thp(fn, THP_RUN_SINGLE_PTE); + ksft_print_msg("[RUN] %s ... with single PTE of THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_SINGLE_PTE, size); } -static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc) +static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc); - do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT); + ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT, size); } -static void run_with_partial_mremap_thp(test_fn fn, const char *desc) +static void run_with_partial_mremap_thp(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc); - do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP); + ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP, size); } -static void run_with_partial_shared_thp(test_fn fn, const char *desc) +static void run_with_partial_shared_thp(test_fn fn, const char *desc, size_t size) { - ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc); - do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED); + ksft_print_msg("[RUN] %s ... with partially shared THP (%zu kB)\n", + desc, size / 1024); + do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED, size); } static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize) @@ -1091,15 +1103,15 @@ static void run_anon_test_case(struct test_case const *test_case) run_with_base_page(test_case->fn, test_case->desc); run_with_base_page_swap(test_case->fn, test_case->desc); - if (thpsize) { - run_with_thp(test_case->fn, test_case->desc); - run_with_thp_swap(test_case->fn, test_case->desc); - run_with_pte_mapped_thp(test_case->fn, test_case->desc); - run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc); - run_with_single_pte_of_thp(test_case->fn, test_case->desc); - run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc); - run_with_partial_mremap_thp(test_case->fn, test_case->desc); - run_with_partial_shared_thp(test_case->fn, test_case->desc); + if (pmdsize) { + run_with_thp(test_case->fn, test_case->desc, pmdsize); + run_with_thp_swap(test_case->fn, test_case->desc, pmdsize); + run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize); + run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize); + run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize); + run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize); + run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize); + run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize); } for (i = 0; i < nr_hugetlbsizes; i++) run_with_hugetlb(test_case->fn, test_case->desc, @@ -1120,7 +1132,7 @@ static int tests_per_anon_test_case(void) { int tests = 2 + nr_hugetlbsizes; - if (thpsize) + if (pmdsize) tests += 8; return tests; } @@ -1329,7 +1341,7 @@ static void run_anon_thp_test_cases(void) { int i; - if (!thpsize) + if (!pmdsize) return; ksft_print_msg("[INFO] Anonymous THP tests\n"); @@ -1338,13 +1350,13 @@ static void run_anon_thp_test_cases(void) struct test_case const *test_case = &anon_thp_test_cases[i]; ksft_print_msg("[RUN] %s\n", test_case->desc); - do_run_with_thp(test_case->fn, THP_RUN_PMD); + do_run_with_thp(test_case->fn, THP_RUN_PMD, pmdsize); } } static int tests_per_anon_thp_test_case(void) { - return thpsize ? 1 : 0; + return pmdsize ? 1 : 0; } typedef void (*non_anon_test_fn)(char *mem, const char *smem, size_t size); @@ -1419,7 +1431,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc) } /* For alignment purposes, we need twice the thp size. */ - mmap_size = 2 * thpsize; + mmap_size = 2 * pmdsize; mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (mmap_mem == MAP_FAILED) { @@ -1434,11 +1446,11 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc) } /* We need a THP-aligned memory area. */ - mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1)); - smem = (char *)(((uintptr_t)mmap_smem + thpsize) & ~(thpsize - 1)); + mem = (char *)(((uintptr_t)mmap_mem + pmdsize) & ~(pmdsize - 1)); + smem = (char *)(((uintptr_t)mmap_smem + pmdsize) & ~(pmdsize - 1)); - ret = madvise(mem, thpsize, MADV_HUGEPAGE); - ret |= madvise(smem, thpsize, MADV_HUGEPAGE); + ret = madvise(mem, pmdsize, MADV_HUGEPAGE); + ret |= madvise(smem, pmdsize, MADV_HUGEPAGE); if (ret) { ksft_test_result_fail("MADV_HUGEPAGE failed\n"); goto munmap; @@ -1457,7 +1469,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc) goto munmap; } - fn(mem, smem, thpsize); + fn(mem, smem, pmdsize); munmap: munmap(mmap_mem, mmap_size); if (mmap_smem != MAP_FAILED) @@ -1650,7 +1662,7 @@ static void run_non_anon_test_case(struct non_anon_test_case const *test_case) run_with_zeropage(test_case->fn, test_case->desc); run_with_memfd(test_case->fn, test_case->desc); run_with_tmpfile(test_case->fn, test_case->desc); - if (thpsize) + if (pmdsize) run_with_huge_zeropage(test_case->fn, test_case->desc); for (i = 0; i < nr_hugetlbsizes; i++) run_with_memfd_hugetlb(test_case->fn, test_case->desc, @@ -1671,7 +1683,7 @@ static int tests_per_non_anon_test_case(void) { int tests = 3 + nr_hugetlbsizes; - if (thpsize) + if (pmdsize) tests += 1; return tests; } @@ -1681,10 +1693,10 @@ int main(int argc, char **argv) int err; pagesize = getpagesize(); - thpsize = read_pmd_pagesize(); - if (thpsize) - ksft_print_msg("[INFO] detected THP size: %zu KiB\n", - thpsize / 1024); + pmdsize = read_pmd_pagesize(); + if (pmdsize) + ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n", + pmdsize / 1024); nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes, ARRAY_SIZE(hugetlbsizes)); detect_huge_zeropage(); From patchwork Wed Nov 22 16:29:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13465129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DDEFC61D97 for ; Wed, 22 Nov 2023 16:30:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF1C58D004B; Wed, 22 Nov 2023 11:30:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA1A58D0008; Wed, 22 Nov 2023 11:30:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D42A98D004B; Wed, 22 Nov 2023 11:30:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C476B8D0008 for ; Wed, 22 Nov 2023 11:30:36 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 943C881056 for ; Wed, 22 Nov 2023 16:30:36 +0000 (UTC) X-FDA: 81486128472.23.F8C9F2F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id DA4741C001A for ; Wed, 22 Nov 2023 16:30:34 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700670635; a=rsa-sha256; cv=none; b=PXzQBawuCLP1hitYHqUv7b6LV8zayCyXFL91KT5A+TVQqybq9X74zL87ngE3FDwEogBocq nVEvXgCntB1DCVCrt5kfhfyTXbitTIH2vFuCg93ytIKm7y+C1PC3OaWy3DSvhxIpx9S34+ 2xpP142jO6yN/9xnvglnGQJQPUSdQAY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700670635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lVXulFLM/BbOzECqCrIvkGa5SvSb3hxRV73P3vTC2Os=; b=ZniqqemjxFxJsmh9ZkolQS/48QSAu/omKPvrAd2nTg44QHMDBEmROwEzvJegoGkofzF2xv U/GSCmWGLN2WLExBGlqSX52+oYfHttElUGdtDuhifIadEttcUsb7vya1DBOXMQwkDp8v9N O8AYxHzMtnxjkS6m6Yuri7z21z4Qht4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 036CA1650; Wed, 22 Nov 2023 08:31:21 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8A75A3F73F; Wed, 22 Nov 2023 08:30:31 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v7 10/10] selftests/mm/cow: Add tests for anonymous small-sized THP Date: Wed, 22 Nov 2023 16:29:50 +0000 Message-Id: <20231122162950.3854897-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231122162950.3854897-1-ryan.roberts@arm.com> References: <20231122162950.3854897-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DA4741C001A X-Stat-Signature: 98k1ueeqts5suyn5tqdmqy3iyaa9ajgj X-HE-Tag: 1700670634-876889 X-HE-Meta: U2FsdGVkX19TYCllEfD1wCFUOXOo2lk8xSajsxqghRx5v2W+cFltIBNCTr8UL5+gruLGDzRnQ6iJdUq7Hodwy5/dW3y2xq98BST1Gdr4S7AXaFJTX+Z8vs3laHKKklhVuxjXK1x7EwH++2ERM89OMGaj7+1a/PEVj3pfOtI5X6c1/Bd+c4nSNKk0UpLnn1E/88RDXSKLYlZysYP9HxsQUz3/dQnbJjSm9KGIHs6CyA7NcmW7pE1fh730Bgq10/ku/Xe6ENdA3kA2YOAd6KbLFOwwO7V9c93HHcl8YBTLRXUYv4Nhw8MzObBhxmZaqty5qoRKr83X6mkntLkE8Pjon3KY7AamsmtvK7k1QmL5pgZqzcJ//AskIrRGoTRVq8BBw3TNoPUiBm2YfkqVYAQXPPqCpQsQ0yVv3NsSvfluynStMqI+UmkN2lPyh6aTIEZGru0v/c1CaTP1+bcws8kAdDty0g6gr0mNilZlEfBWegW8fm+WEgOyHjjg9SFYkWpnQc9j705IS8OMAUXhqrIR6y3T26QZr32PyFg6FT7VXgZCn9rpJXcAA2khqCla/7DAgCV9usScoSvnsb96v1Rdux00sh57149zEu9LXoS8hJf0xm4LJWH7HglkdDdGoPTQxf9PaY3FvpXZ8NNgEDLh2q7y4JdSUcDUUSYqCduGyunNbSRXHPmnudB2mdWW5E9x+jBMVXKM9tI/GI5QR/4Yr3rPRo5ZGTYLGzA6Cjfa9SiYQQmJ5guP2K0Ts1Mp0CR/pcqBegi4DeIovhxp2M8E6Ff8SfFff2vpTkf1cmEqv12t41AoRi35W2Bp8YDQcQx63W1ZJsWk9+VNrYMQD5KkcFWFUi5KFbVyhKA+/BCYt88txhYaylH3G4zZJU5yla8N+SCZMfYv/LNNrWQY8jho8x3JXEzLtCKWQhS/XiAFHdBDpjmH3te9c/8HBsqfHDGScfe+3LkFR5JBgJwRgx7 ziWtN6mW nA5SxPaL27gKneW/jrj8EDmdAobcu2YyjurGmjyHMl/MQOC8W//pnflB0zy8RQ2JlinekIdCkpr5zl3wVxO1dUJP+dkyJORbZwBHinb3qUuMyNEiXW/kBmt8SD6qdCBFgyAL1wJT22/RdQoj2sbLUnfD6GhSZEi6lHFhLMxE+tvaFt5gfHdcSSOMu/IsZkCBBIKZLsvzluYQKfD0O5ww1JDVs68v2aQczhstB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add tests similar to the existing PMD-sized THP tests, but which operate on memory backed by (PTE-mapped) small-sized THP. This reuses all the existing infrastructure. If the test suite detects that small-sized THP is not supported by the kernel, the new tests are skipped. Signed-off-by: Ryan Roberts --- tools/testing/selftests/mm/cow.c | 71 +++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-) -- 2.25.1 diff --git a/tools/testing/selftests/mm/cow.c b/tools/testing/selftests/mm/cow.c index d03c453cfd5c..3efc395c7077 100644 --- a/tools/testing/selftests/mm/cow.c +++ b/tools/testing/selftests/mm/cow.c @@ -29,15 +29,49 @@ #include "../../../../mm/gup_test.h" #include "../kselftest.h" #include "vm_util.h" +#include "thp_settings.h" static size_t pagesize; static int pagemap_fd; static size_t pmdsize; +static int nr_thpsmallsizes; +static size_t thpsmallsizes[20]; static int nr_hugetlbsizes; static size_t hugetlbsizes[10]; static int gup_fd; static bool has_huge_zeropage; +static int sz2ord(size_t size) +{ + return __builtin_ctzll(size / pagesize); +} + +static int detect_smallthp_sizes(size_t sizes[], int max) +{ + int count = 0; + unsigned long orders; + size_t kb; + int i; + + /* thp not supported at all. */ + if (!pmdsize) + return 0; + + orders = thp_supported_orders(); + + /* Only interested in small-sized THP (less than PMD-size). */ + for (i = 0; i < sz2ord(pmdsize); i++) { + if (!(orders & (1UL << i))) + continue; + kb = (pagesize >> 10) << i; + sizes[count++] = kb * 1024; + ksft_print_msg("[INFO] detected small-sized THP size: %zu KiB\n", + kb); + } + + return count; +} + static void detect_huge_zeropage(void) { int fd = open("/sys/kernel/mm/transparent_hugepage/use_zero_page", @@ -1113,6 +1147,23 @@ static void run_anon_test_case(struct test_case const *test_case) run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize); run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize); } + for (i = 0; i < nr_thpsmallsizes; i++) { + size_t size = thpsmallsizes[i]; + struct thp_settings settings = *thp_current_settings(); + + settings.hugepages[sz2ord(pmdsize)].enabled = THP_NEVER; + settings.hugepages[sz2ord(size)].enabled = THP_ALWAYS; + thp_push_settings(&settings); + + run_with_pte_mapped_thp(test_case->fn, test_case->desc, size); + run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, size); + run_with_single_pte_of_thp(test_case->fn, test_case->desc, size); + run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, size); + run_with_partial_mremap_thp(test_case->fn, test_case->desc, size); + run_with_partial_shared_thp(test_case->fn, test_case->desc, size); + + thp_pop_settings(); + } for (i = 0; i < nr_hugetlbsizes; i++) run_with_hugetlb(test_case->fn, test_case->desc, hugetlbsizes[i]); @@ -1134,6 +1185,7 @@ static int tests_per_anon_test_case(void) if (pmdsize) tests += 8; + tests += 6 * nr_thpsmallsizes; return tests; } @@ -1691,12 +1743,24 @@ static int tests_per_non_anon_test_case(void) int main(int argc, char **argv) { int err; + struct thp_settings default_settings; pagesize = getpagesize(); pmdsize = read_pmd_pagesize(); - if (pmdsize) + if (pmdsize) { + /* Only if THP is supported. */ + thp_read_settings(&default_settings); + default_settings.hugepages[sz2ord(pmdsize)].enabled = THP_GLOBAL; + thp_save_settings(); + thp_push_settings(&default_settings); + ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n", pmdsize / 1024); + + nr_thpsmallsizes = detect_smallthp_sizes(thpsmallsizes, + ARRAY_SIZE(thpsmallsizes)); + } + nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes, ARRAY_SIZE(hugetlbsizes)); detect_huge_zeropage(); @@ -1715,6 +1779,11 @@ int main(int argc, char **argv) run_anon_thp_test_cases(); run_non_anon_test_cases(); + if (pmdsize) { + /* Only if THP is supported. */ + thp_restore_settings(); + } + err = ksft_get_fail_cnt(); if (err) ksft_exit_fail_msg("%d out of %d tests failed\n",