From patchwork Wed Jul 26 09:51:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13327780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B529C001DE for ; Wed, 26 Jul 2023 09:52:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BAE26B0071; Wed, 26 Jul 2023 05:52:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 46AC16B0074; Wed, 26 Jul 2023 05:52:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35A3B6B0075; Wed, 26 Jul 2023 05:52:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 27ABA6B0071 for ; Wed, 26 Jul 2023 05:52:03 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EE2ACB2627 for ; Wed, 26 Jul 2023 09:52:02 +0000 (UTC) X-FDA: 81053296884.30.830AE4A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf07.hostedemail.com (Postfix) with ESMTP id 0235540005 for ; Wed, 26 Jul 2023 09:52:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690365121; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=Ss/QHfvNMmYn0ZeWMf7UfXztgsSQoqDeFHWHrQZgS8w=; b=g2S54j7FJwSHQLpbTDPWXZreJM9r9ZJFKbV5u6k+/7aDMYbixpXYAv5VECmNfZkovMskRd pfYrtUp8LYGE6tld/ypU4P29g4MeR18/8hfjBQOD+TfVwnzcBj7cDQ52bnlXM1xtVxZo60 r7JMdv6C9Lbwxz1hClNYpOWnjX05H5A= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690365121; a=rsa-sha256; cv=none; b=MSvjYESQz37yT+5K1u2ccmvhL9hZ7pva9CDt6fgzlVL9VpLdZQQ2Tk0DyL3REbMgTWzrLV hmntwa94ZNyy2qzyknmEE0vqztoT7/Phnzi7/h1iSDZydMh7NCLtmC7oUserUF7LOgDtQ1 cSGozHbrMK7Cjj30t5M7el7bGZdnZxw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9EDDC1692; Wed, 26 Jul 2023 02:52:42 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 61A1B3F67D; Wed, 26 Jul 2023 02:51:57 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama Cc: Ryan Roberts , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH v4 0/5] variable-order, large folios for anonymous memory Date: Wed, 26 Jul 2023 10:51:41 +0100 Message-Id: <20230726095146.2826796-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 0235540005 X-Rspam-User: X-Stat-Signature: 4oar6zjpkfyt5bxeznkprceu9fow6145 X-Rspamd-Server: rspam01 X-HE-Tag: 1690365120-499112 X-HE-Meta: U2FsdGVkX18HGcNYlh2S7N+2vmUmEzKCB7MK9fHuBxedNHLJqaE4iiqUNz1rnKrzAKY2H12nZ4p6Jzsytg0/AmH2gYgiZH8G+hhGRx5R9oqauI7wqUNjv7bZN2DdHkTWrB5k70oV6K50lNxtuMBcOXaQhI24uNQrtm1TQQZT/Jjtmu6Qw6q4sb5MgdKyf4UZNYICn+OMkaK1dABHDUmEJMC2mf2lUiFjsHdC8AsPlnOBamdjYQVdRqw+bKc0CTizpBbtQ25TbFmvOKdSuUIKj64DMAXmqsMQ6vWggNZQ+bCmnBtvj5tZ2C7Rbv/D3vboxQ3DeOCfheTCFn2GGtMhscX+Txuhlc1e2s26hC9EXJyDE5Nih+GuddGji91WJ0TYvy+c/skPEVmuNBdJo5jDF0PtX8osmdvbYmxKPUyvTVMaKn2y4KEpzbGRQ3OlJ4gTrpoy8mGjdH9sLKmJQ71nJ1RlZizVKWZiv6qerZowNYyPOZZX9aAZ49zKhZg2ZHx0UDZ0HJHyL0LtGfhgnV7gsAgbZwot25MWWAxjF/wmALVo/qWUoVQLugckV4+ZE0nWLLySjzUPScrkth1I988mNJNO/B2Zy92oycaaBWcYjkDqPzCvBBT84tMKkvZPS/MBJgpnNl5eagj0XMH8JgstKtLbAhLiNyliaGI8DzRO4QWqD2T/e82iBPH4TqN3neWt5RQAeZZ9Je0KqhC1jH5kV2YmdE0qzowDyxefdbSQ84KaEijAHvtjaSxv+D+jOg90abLnR0lTo8NOrMYdUstxycB8pQaLKLgy3ozAX+ZpGTOhPVH7PoeiUd4Xxee8WTYGsVafjwfFedHtKm/ZfoCHd4ypfAFrUu+4skyNGUqWBe1sGvAqDWO5UiChj2zWSPCCMF0zf0favG6moCOOraR5l/ZNeYGjseOzXiYhBFd+crhua+gpTgZsohFZKt9++YsJ976zGv0kCgjpfyivdx3 j+L51DcW 26gV9Ez72vqEVUh++FXB5vEz4n3TP3SyCg2bBRVbFWMciGvxrJtCfXPXOo5sViLIiMr03QfmpR3cCLw4Jfjv6iUEpkwjPEeJ+F68ABASFlk/PmSUek8upPWcJ9a9+JLEvlMyurLJFdVLnF4MJQsxHgIqnkEOG3i6Dri97qDUvyEYvHZZ2USXGTLkyH6ldMPIOTTG0OUodIvLNcUE3zMILFC0+9jfuorNPUxCbK0FXnR3g3EKd+fEM5HJIKT82ZRkPqYCGTESA/GD9bKxA2HK+/ohUn0B5THN+axz+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi All, This is v4 of a series to implement variable order, large folios for anonymous memory. (currently called "LARGE_ANON_FOLIO", previously called "FLEXIBLE_THP"). The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults: 1) Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel overhead. This should benefit all architectures. 2) Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch). This patch set deals with the SW side of things (1). (2) is being tackled in a separate series. The new behaviour is hidden behind a new Kconfig switch, LARGE_ANON_FOLIO, which is disabled by default. Although the eventual aim is to enable it by default. The "arm64: mm: Override arch_wants_pte_order()" patch is for arm64 to explicitly override the default arch_wants_pte_order() and is intended as an example. If this series is accepted I suggest ignoring this patch which I will take through the arm64 tree separately. Neither has any build dependency on the other. My hope is that we are pretty much there with the changes at this point. Although they should not be merged until the prerequisites are complete. These are in progress and tracked at [4]. The patches are based on top of v6.5-rc3. I have a branch at [5]. Testing ------- This version adds patches to mm selftests so that the cow tests explicitly test large anon folios, in the same way that thp is tested. When enabled you should see something similar at the start of the test suite: # [INFO] detected large anon folio size: 64 KiB Then the following results are expected. The fails and skips are due to existing issues: # Totals: pass:263 fail:16 xfail:0 xpass:0 skip:29 error:0 Existing mm selftests reveal 1 regression in khugepaged tests when LARGE_ANON_FOLIO is enabled: Run test: collapse_max_ptes_none (khugepaged:anon) Maybe collapse with max_ptes_none exceeded.... Fail Unexpected huge page I believe this is because khugepaged currently skips non-order-0 pages when looking for collapse opportunities and should get fixed with the help of DavidH's work to create a mechanism to precisely determine shared vs exclusive pages. Changes since v3 [3] -------------------- - Renamed feature from FLEXIBLE_THP to LARGE_ANON_FOLIO. - Removed `flexthp_unhinted_max` boot parameter. Discussion concluded that a sysctl is preferable but we will wait until real workload needs it. - Fixed uninitialized `addr` on read fault path in do_anonymous_page(). - Added mm selftests for large anon folios in cow test suite. Changes since v2 [2] -------------------- - Dropped commit "Allow deferred splitting of arbitrary large anon folios" - Huang, Ying suggested the "batch zap" work (which I dropped from this series after v1) is a prerequisite for merging FLXEIBLE_THP, so I've moved the deferred split patch to a separate series along with the batch zap changes. I plan to submit this series early next week. - Changed folio order fallback policy - We no longer iterate from preferred to 0 looking for acceptable policy - Instead we iterate through preferred, PAGE_ALLOC_COSTLY_ORDER and 0 only - Removed vma parameter from arch_wants_pte_order() - Added command line parameter `flexthp_unhinted_max` - clamps preferred order when vma hasn't explicitly opted-in to THP - Never allocate large folio for MADV_NOHUGEPAGE vma (or when THP is disabled for process or system). - Simplified implementation and integration with do_anonymous_page() - Removed dependency on set_ptes() Changes since v1 [1] -------------------- - removed changes to arch-dependent vma_alloc_zeroed_movable_folio() - replaced with arch-independent alloc_anon_folio() - follows THP allocation approach - no longer retry with intermediate orders if allocation fails - fallback directly to order-0 - remove folio_add_new_anon_rmap_range() patch - instead add its new functionality to folio_add_new_anon_rmap() - remove batch-zap pte mappings optimization patch - remove enabler folio_remove_rmap_range() patch too - These offer real perf improvement so will submit separately - simplify Kconfig - single FLEXIBLE_THP option, which is independent of arch - depends on TRANSPARENT_HUGEPAGE - when enabled default to max anon folio size of 64K unless arch explicitly overrides - simplify changes to do_anonymous_page(): - no more retry loop Performance ----------- Kernel compilation with 8 jobs: | kernel | real-time | kern-time | user-time | |:---------------------------|------------:|------------:|------------:| | baseline-v6.5-rc3 | 0.0% | 0.0% | 0.0% | | anonfolio-v4 | -4.9% | -38.2% | -0.7% | Kernel compilation with 80 jobs: | kernel | real-time | kern-time | user-time | |:---------------------------|------------:|------------:|------------:| | baseline-v6.5-rc3 | 0.0% | 0.0% | 0.0% | | anonfolio-v4 | -5.1% | -42.3% | -0.4% | Speedometer: | kernel | runs_per_min | |:---------------------------|---------------:| | baseline-v6.5-rc3 | 0.0% | | anonfolio-v4 | 0.6% | [1] https://lore.kernel.org/linux-mm/20230626171430.3167004-1-ryan.roberts@arm.com/ [2] https://lore.kernel.org/linux-mm/20230703135330.1865927-1-ryan.roberts@arm.com/ [3] https://lore.kernel.org/linux-mm/20230714160407.4142030-1-ryan.roberts@arm.com/ [4] https://lore.kernel.org/linux-mm/f8d47176-03a8-99bf-a813-b5942830fd73@arm.com/ [5] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/anonfolio-for_lkml_v4 Thanks, Ryan Ryan Roberts (5): mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() mm: LARGE_ANON_FOLIO for improved performance arm64: mm: Override arch_wants_pte_order() selftests/mm/cow: Generalize do_run_with_thp() helper selftests/mm/cow: Add large anon folio tests arch/arm64/include/asm/pgtable.h | 6 + include/linux/pgtable.h | 13 ++ mm/Kconfig | 10 ++ mm/memory.c | 166 +++++++++++++++++++--- mm/rmap.c | 27 +++- tools/testing/selftests/mm/cow.c | 229 ++++++++++++++++++++++--------- 6 files changed, 365 insertions(+), 86 deletions(-) --- 2.25.1