From patchwork Fri Mar 17 10:57:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94199C6FD1D for ; Fri, 17 Mar 2023 10:58:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09BCF6B0078; Fri, 17 Mar 2023 06:58:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F17516B007B; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDF056B007E; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D02A56B0078 for ; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A05FD8183D for ; Fri, 17 Mar 2023 10:58:24 +0000 (UTC) X-FDA: 80578091328.19.8CAA74D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf08.hostedemail.com (Postfix) with ESMTP id 008EC16000A for ; Fri, 17 Mar 2023 10:58:22 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf08.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rXVqB0ZsfllNiDoZhurNV3aR20+gMc/UXYF4nAJ7SiE=; b=PEKqeIr9B+eVTZWY5xZcsVH7N4YmHD5yoWrJ6yCa97m7FMtqYIP312lGGQEIhiatmYCGRe BZCZdGWxzfsoZ+qNvg7x8vNIY8tZm6R0EsC4GFOp7iz8ixrR5q23808E7Ea7cGbalscy6B jlyEC/3GnTojShh5iwpmSolLLllG4yg= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf08.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050703; a=rsa-sha256; cv=none; b=dgJzY2wgHLldtg8v0VQsv87H2hh5XUOFqTvFfutIcDjQqaH1UvUG3RWFIyjshxcVZyAj4b 2O458EkWGqSTbbH0UB1b9oG0DLLFfHsJlM+grUHQBY33ElsFLzX4vZI8FHyyt+DJZk3gGH yeXksJa6NrhsYZob6B6PEuiKlxBXoNs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D83851763; Fri, 17 Mar 2023 03:59:05 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1DD5F3F885; Fri, 17 Mar 2023 03:58:21 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 1/6] mm: Expose clear_huge_page() unconditionally Date: Fri, 17 Mar 2023 10:57:57 +0000 Message-Id: <20230317105802.2634004-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 008EC16000A X-Stat-Signature: bn7rgcf14i5qo5u9egpqiw96hebqwcds X-HE-Tag: 1679050702-861763 X-HE-Meta: U2FsdGVkX1/L6N8mcBm21EhvaFzSrLSX9qvX3ycJBOtoZlbZSy1fYAU3Qme4eEZfEKrFt7jsQFU4lkWNMRM8Lqzmpa8MeMevfcywwuCqC6SCfYLkwnovJoPGZ6TfH+EwAKBuA2xRuz3cAvWhui9hlCoRyldUcaPN+mgqTCv/ZIhFjw7qmhW8mG/jSWaYiYT+Zq+mmKOoZwZERYKvI44K0Bhf0XiePmAjNlk+Rt9/j0G9CZzFkxV06jyp5CIOOvuVwGAJN/GGVJ3Arkb3ZCEoGcwr8SSEh/3pWqztoUMdWosgsvNeUj1rJ8lOQOQYuBWoYk+IotJAurylQnEQaeTJ63LG7T0o5yaf4Avac1YpxKnHWq11dd8FjNf53VxrXKpchtYApwdm1FZYTrZPl/HioMaA3ewuPTPbayfOnKnyD8yL5Jhm/TeI45ibrjyfVqa/kSCgCf7XkPnBz+xj7AvBwYKnN/w69sMRbgHVCZuVp/i8ir+UtWzPioS1shoNYiKksdr6VNPexu4PqeKvbYHxkNRcPhK1Gf8z1b18no9y3QHVqLErVe34lt29nvMc2yN/TwnzZlJBqxN87jVL5wR6T2DotmXlf86IKJ4IMH6dsGnqa5bsfWZ+07m4ZIZO/HJvx4AQbSdsPlcsBGvrHLsjIqzzwtLLDO9iSrKW/sLqliPw6vdVU9u9r3p11CbfCTQ8AKnyHQkV6pCLESMs7EXQAkCV3cxYRz56fk56qROeu4184h0ZKhOYijEK8cgGxD0E+M1vQM7Trp8OYrsOlEq9hi+QhcTz7HB/atxHcI6ICDxlIjeHzyuqF5pr6uNO5ntmVIbyfMIKXGNJdGZC/KHOeQQFapGrI6jMuFNmsXLJFnnzBcZoNE71+EhfTJdO/xn6kz94Ha4z8DO7AIHBP3IVH0N6Y51HvCU9U+WF8SUYU7GPUOzbMyQfp2D1zVj+eXUu8XjTBrxY72HSVVOi0N1 6/C07l8A W2Hw2srgwiniNFDnO+yCxFXGycW9MCeQaDvgw7Yrxo4KgGHEd0ceJ1lSbC/4XlAh5R+oQjVYuRNbDDt6sZgSv1xz/SAWcjRq9o5Qfg5ULG5LEjhUknT6RQSR+hMA/D8GafuD4CZoz5MOGBDsM9EjrP6pLVeqVSbCPYCipNnAt2raDmJG6oDe70O5E5NljAuBT+B1s3akIvpPw30blSusse7+Fd/3yUSEP++6I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio. Signed-off-by: Ryan Roberts --- include/linux/mm.h | 3 ++- mm/memory.c | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) -- 2.25.1 diff --git a/include/linux/mm.h b/include/linux/mm.h index 1f79667824eb..cdb8c6031d0f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3538,10 +3538,11 @@ enum mf_action_page_type { */ extern const struct attribute_group memory_failure_attr_group; -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) extern void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page); + +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) extern void copy_user_huge_page(struct page *dst, struct page *src, unsigned long addr_hint, struct vm_area_struct *vma, diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..c08645908ee2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5628,7 +5628,6 @@ void __might_fault(const char *file, int line) EXPORT_SYMBOL(__might_fault); #endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) /* * Process all subpages of the specified huge page with the specified * operation. The target subpage will be processed last to keep its @@ -5716,6 +5715,8 @@ void clear_huge_page(struct page *page, process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); } +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) + static void copy_user_gigantic_page(struct page *dst, struct page *src, unsigned long addr, struct vm_area_struct *vma, From patchwork Fri Mar 17 10:57:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CA50C76195 for ; Fri, 17 Mar 2023 10:58:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FF3B6B007B; Fri, 17 Mar 2023 06:58:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 888186B007E; Fri, 17 Mar 2023 06:58:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 750256B0080; Fri, 17 Mar 2023 06:58:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 633506B007B for ; Fri, 17 Mar 2023 06:58:26 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 24A261617C3 for ; Fri, 17 Mar 2023 10:58:26 +0000 (UTC) X-FDA: 80578091412.08.DD1136E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 6C12B40005 for ; Fri, 17 Mar 2023 10:58:24 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050704; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i8vpyiuakDg4IaGPZg8fhta9U0S6yowD12SXWlo6jeg=; b=2TPPpKEdTT6gSJJw8cLsCm8z0TOTsAkhYtdEXRUplU7LsF1SOr9p/plD4/X1a2t/17QJMv jCHi9WUgiyRDG5kNzDH60PwERivLU4g3YEaBRlprcmMH+dJzoEy/MS9tUAYVDosND8Kpo2 IDoRwYY3aVycnGtJ/RMRF/8XIfh0hDw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050704; a=rsa-sha256; cv=none; b=VesGWAWZwwbQny6aHTySzg0UZcHmoKYFrTsSxLuQ5AFCJuVYktsq2rNQfoXRBgCnmTFmE8 ZaSeMzEk0xN4Oek4LwWka/UGHj9aL73UOwNMEBybWVL9EKMj77Fj4GQ16NW8m/RXBzHe/A xgbmdpB6wZrrjCvleGSv96h+PvlUIhQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 30B22176C; Fri, 17 Mar 2023 03:59:07 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 503C93F885; Fri, 17 Mar 2023 03:58:22 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 2/6] mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio() Date: Fri, 17 Mar 2023 10:57:58 +0000 Message-Id: <20230317105802.2634004-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6C12B40005 X-Stat-Signature: 5twpo1onqgggetjqcdg34rtwx8pkok8h X-Rspam-User: X-HE-Tag: 1679050704-708860 X-HE-Meta: U2FsdGVkX1+J7PJO0ShPJ2LAQhXJ6HASmWWN6cj5Sa9fvXC9M2baW3AxxL3+RY+/aDQeUuF6a/YMGezsZyO8fKnVNydq+ehOjF6sIkKsxiUms9MUWwP2/8d2XkvGoG5b5NH+4JIcZT23w64v8V/iqcwUlkmVg31wB+PegaUkObKBD/ZgaQPfOb0zKcT5DmeODcHXbuADO2z5HksB8SDsMYcRMFfa7dPLhDYLFXdRcPKRWQY7/LaUOUCxFPL26GZqyUlstyffKfy/eY7B0EgZsEs+Qxj61ksJTZVXUrgOOJKd2YIPB2nwhXHO8u+PbA3p48rhUHkWcxV43UWMuSXRU//sEKtL9SfcN79QwLGk4khYzyTxcimA/ttnidnhYD73DzQ2ZAk3Cvu3KmxU2uhONJ9/DsOBz3t9VNdwh5HHJ2OLq2YumYgtWx+KvqVEOUYvQCpezr99VnT1X4wlIx4sBgsr336jQlKnS/9XhzM2BpRSeu3RD76mTZVsI+Rxn3C7AF2uttPDoXNTtXGLwFefCI5bBn4+PU5fbTrbptNlKsgMnC4PJVGC8nh7sZeQ2gsijKfF35NTmwyexmEQBiKlT6oVVxVmPnNMk37nbwsXwheGkIkISGl+18PFT87RmombGhlL6u5SPE4w4EQmEI51tkFupWZ6McFifA8dE4ZS56F11ZqsoxIflUCIJsaJNZW1hFapZPvveK8xyyUPI2TRy4Gek9G+fal56f86lIJehb3Gl00ZBNkxaCUJH/T7co5t+NCXEfc6txvblDVTj7skS2ZQbLMocHcV0bD6Fk053nWo+A2iMSJc9pYBiLRLdYnAyWGSv1fU8dYe05ot+dLd7MSAikxnuVPA/RGwCSrIJzRj3NQAcUfkVZDDG+7Ye8gCsn396hpJ/KwqneGkVcsEuXsvCKVj9Ku3WOwZIkx/z9/GMzFP5WBpOgtP1lO9Hng6SrPWp1ly3d/bEX7pLzE Hqstko7n WibBqLx2KcfD4unRRWDzAfhmAQQqHyA5o+n5IiOYN2+mTSHdHzc27IkLGqBkRRJySTDbgVVlTdi/VOZPZdgP9BtGxnIIKz1UcdpeP/S7t3XkI68ShXfiqa2AqPaHkqnyyiO/rX6Nuu0IWJx+ZaqpC/xYsrVegovDrROJung813y6Z63kmiRrS2O7mdiQ4HmPNT3YkKbLhmXDNzRzSBIrPanN7cjLajHOHpxkl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow allocation of large folios with vma_alloc_zeroed_movable_folio(). This prepares the ground for large anonymous folios. The generic implementation of vma_alloc_zeroed_movable_folio() now uses clear_huge_page() to zero the allocated folio since it may now be a non-0 order. Currently the function is always called with order 0 and no extra gfp flags, so no functional change intended. Signed-off-by: Ryan Roberts --- arch/alpha/include/asm/page.h | 5 +++-- arch/arm64/include/asm/page.h | 3 ++- arch/arm64/mm/fault.c | 7 ++++--- arch/ia64/include/asm/page.h | 5 +++-- arch/m68k/include/asm/page_no.h | 7 ++++--- arch/s390/include/asm/page.h | 5 +++-- arch/x86/include/asm/page.h | 5 +++-- include/linux/highmem.h | 23 +++++++++++++---------- mm/memory.c | 5 +++-- 9 files changed, 38 insertions(+), 27 deletions(-) -- 2.25.1 diff --git a/arch/alpha/include/asm/page.h b/arch/alpha/include/asm/page.h index 4db1ebc0ed99..6fc7fe91b6cb 100644 --- a/arch/alpha/include/asm/page.h +++ b/arch/alpha/include/asm/page.h @@ -17,8 +17,9 @@ extern void clear_page(void *page); #define clear_user_page(page, vaddr, pg) clear_page(page) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) extern void copy_page(void * _to, void * _from); #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 2312e6ee595f..47710852f872 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -30,7 +30,8 @@ void copy_highpage(struct page *to, struct page *from); #define __HAVE_ARCH_COPY_HIGHPAGE struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr); + unsigned long vaddr, + gfp_t gfp, int order); #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio void tag_clear_highpage(struct page *to); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..3b4cc04f7a23 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -926,9 +926,10 @@ NOKPROBE_SYMBOL(do_debug_exception); * Used during anonymous page fault handling. */ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) + unsigned long vaddr, + gfp_t gfp, int order) { - gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO; + gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO | gfp; /* * If the page is mapped with PROT_MTE, initialise the tags at the @@ -938,7 +939,7 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, if (vma->vm_flags & VM_MTE) flags |= __GFP_ZEROTAGS; - return vma_alloc_folio(flags, 0, vma, vaddr, false); + return vma_alloc_folio(flags, order, vma, vaddr, false); } void tag_clear_highpage(struct page *page) diff --git a/arch/ia64/include/asm/page.h b/arch/ia64/include/asm/page.h index 310b09c3342d..ebdf04274023 100644 --- a/arch/ia64/include/asm/page.h +++ b/arch/ia64/include/asm/page.h @@ -82,10 +82,11 @@ do { \ } while (0) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ ({ \ struct folio *folio = vma_alloc_folio( \ - GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false); \ + GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false); \ if (folio) \ flush_dcache_folio(folio); \ folio; \ diff --git a/arch/m68k/include/asm/page_no.h b/arch/m68k/include/asm/page_no.h index 060e4c0e7605..4a2fe57fef5e 100644 --- a/arch/m68k/include/asm/page_no.h +++ b/arch/m68k/include/asm/page_no.h @@ -3,7 +3,7 @@ #define _M68K_PAGE_NO_H #ifndef __ASSEMBLY__ - + extern unsigned long memory_start; extern unsigned long memory_end; @@ -13,8 +13,9 @@ extern unsigned long memory_end; #define clear_user_page(page, vaddr, pg) clear_page(page) #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) #define __pa(vaddr) ((unsigned long)(vaddr)) #define __va(paddr) ((void *)((unsigned long)(paddr))) diff --git a/arch/s390/include/asm/page.h b/arch/s390/include/asm/page.h index 8a2a3b5d1e29..b749564140f1 100644 --- a/arch/s390/include/asm/page.h +++ b/arch/s390/include/asm/page.h @@ -73,8 +73,9 @@ static inline void copy_page(void *to, void *from) #define clear_user_page(page, vaddr, pg) clear_page(page) #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) /* * These are used to make use of C type-checking.. diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index d18e5c332cb9..34deab1a8dae 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -34,8 +34,9 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, copy_page(to, from); } -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) #ifndef __pa #define __pa(x) __phys_addr((unsigned long)(x)) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index b06254e76d99..e2127af4997b 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -209,26 +209,29 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr) #ifndef vma_alloc_zeroed_movable_folio /** - * vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA. - * @vma: The VMA the page is to be allocated for. - * @vaddr: The virtual address the page will be inserted into. - * - * This function will allocate a page suitable for inserting into this - * VMA at this virtual address. It may be allocated from highmem or + * vma_alloc_zeroed_movable_folio - Allocate a zeroed folio for a VMA. + * @vma: The start VMA the folio is to be allocated for. + * @vaddr: The virtual address the folio will be inserted into. + * @gfp: Additional gfp falgs to mix in or 0. + * @order: The order of the folio (2^order pages). + * + * This function will allocate a folio suitable for inserting into this + * VMA starting at this virtual address. It may be allocated from highmem or * the movable zone. An architecture may provide its own implementation. * - * Return: A folio containing one allocated and zeroed page or NULL if + * Return: A folio containing 2^order allocated and zeroed pages or NULL if * we are out of memory. */ static inline struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) + unsigned long vaddr, gfp_t gfp, int order) { struct folio *folio; - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vaddr, false); + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, + order, vma, vaddr, false); if (folio) - clear_user_highpage(&folio->page, vaddr); + clear_huge_page(&folio->page, vaddr, 1U << order); return folio; } diff --git a/mm/memory.c b/mm/memory.c index c08645908ee2..8798da968686 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3061,7 +3061,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, + 0, 0); if (!new_folio) goto oom; } else { @@ -4049,7 +4050,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* Allocate our own private page. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0); if (!folio) goto oom; From patchwork Fri Mar 17 10:57:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AC1BC74A5B for ; Fri, 17 Mar 2023 10:58:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A1736B007E; Fri, 17 Mar 2023 06:58:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 72A016B0080; Fri, 17 Mar 2023 06:58:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F0D96B0081; Fri, 17 Mar 2023 06:58:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4BEDF6B007E for ; Fri, 17 Mar 2023 06:58:27 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 22017A02B6 for ; Fri, 17 Mar 2023 10:58:27 +0000 (UTC) X-FDA: 80578091454.04.C7B3A4D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 5A8E9180004 for ; Fri, 17 Mar 2023 10:58:25 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050705; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+3NujFEuDznznAnsPUuZmo48W1iW1AcX1wXuqBIeai8=; b=FueDZPz5OtRJ3eCFxNSGR83Tv1jrP5TUseygrB9vxxV0gqJnjHecOS3/k+r2qmnfHjIkIQ 4MhF21Ne6sXjnycOD2XpoNlwWTdi8VyAdIUtTUaY5UFmJfzcyYsUfRLVV3ovmF9XATTYYC 4xXct8lo/UTZ7Avb7MEsS0Q1vXAaIjI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050705; a=rsa-sha256; cv=none; b=BmQs4Fr0wgijDjWhNaLH4jsE8BFoduKT2DYfHe19ZCI8IMNjtQpABuO8ZW0jXQubMFWqkj gNw3NcehSvJYolO9KIuvVmEuRq50fx57JrZPdcLcL48a86eK5NmXpZsFzUEP7239Tp3HgZ 6nPECOecgH4eEaCmkVZK2IBC+C5A+AU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6294A19F0; Fri, 17 Mar 2023 03:59:08 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9C80C3F885; Fri, 17 Mar 2023 03:58:23 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 3/6] mm: Introduce try_vma_alloc_zeroed_movable_folio() Date: Fri, 17 Mar 2023 10:57:59 +0000 Message-Id: <20230317105802.2634004-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5A8E9180004 X-Stat-Signature: rzph1rzk9kazxua19gt3hzm96474s76d X-Rspam-User: X-HE-Tag: 1679050705-680288 X-HE-Meta: U2FsdGVkX19n8ocMPvhQvhI0ex88HYmrQUFqRfGc5eDwALpyLMt7zaWJNbNqXm7npOWiNiZle6hR8SuajbYrYyODmVBYUh8tIeK6K9rFu2RYIMp/ONJFpI77P+YYeIpGIqpmj2OmWuevPXmsJEM6iP1/e7ho3Fv+s67WGlxWWwmA6tgpn/o5yAaHzst5MciM03EbHkZDF/tar30rQ4YE6y4A6J53QJIRY2tOlEtQvzfcH3OzBaifylyoECLu+kBf31KIyWi4mcf+sPyFZQ0SaBF0XjPlzRK5wZU6SDy6DSKdDcxhDfD8mH/vLmgQ7obw2Hw7LRcHUCAX1TJCnon5cBPqFFYgSRcNKYZCYimxWKg0fmr/0iXIZvSfA9WNDr8d+tXs/cmYxUqBC7S1HVn/CULeTO4pL8W8xTlz/GufAEN/1stO9QfrW9OIwM7lj11ZVUgbvwS0h+mTPX9shJZ1noK/96htD5DcCGhwMm4pA3YAw1j1K/56QZb/02tDaYW88snqws6jSVm6CemP4k7Hm2NGjVbZ94pEaxdV6P5ypEmaUgtXAx9rprD/VkYAM3dKqBUDpvvvEsBjgmcvVzABm7CkjBMmPuIGJBq+qs5sMnOPRiyJZrste0cVVRHAhOC4YTp3mcneyFIdGI2246/+79r7iFGVsuqsuzY3GPCFvWLbpBHycR9ou/N0A+FTaoWP6cBZwxijlX8GFpJZPSr7gKw4qPftk02bk2MjfqPowdVkxUhT1mI6bOnHy86nzdOJgI9c+sZ13gUgeKQlBTdYF9fgxhogkxIH7Zz2mAK1aeyJg7iguUFSgfxUwaaSpp/uMKSJkqFulfgzrXYOf84P0r0IPibdl66vULcKELUC+ujKFFQdy/xGIjEN8XegpH4qLYN07PdXJVQQcgIthg30JSrBKiZY4T4DmhHIaY+s8777OHtBDYZyDT/Fj5dnSXCQSTBFEqL3xE8uujlRNzv WupomARH /Alcw2mRwFAZIunlQ7+ip32pciiNN+UaMpv6y/3pbfsTEkvzkG9TGani+AKbehWpDi5RXXnuGrf69Z7f9iPX1Wu4BJhY2VLY5npdKF5KIrZounUsSGu3QNlbZ1lPjq/dtRGWxJ2MW9/9Pcfnf1egJlgQyZtiZJv544KYlnlzEmvspcx8ZUOJ5fIe0mVBk4ggMQ7WT/VGDKA4q/nIjwJcDibSbnvqOow4cdOym X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like vma_alloc_zeroed_movable_folio(), except it will opportunistically attempt to allocate high-order folios, retrying with lower orders all the way to order-0, until success. The user must check what they got with folio_order(). This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under pressure. For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios. Signed-off-by: Ryan Roberts --- mm/memory.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) -- 2.25.1 diff --git a/mm/memory.c b/mm/memory.c index 8798da968686..c9e09415ee18 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3024,6 +3024,27 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +/* + * Opportunistically attempt to allocate high-order folios, retrying with lower + * orders all the way to order-0, until success. The user must check what they + * got with folio_order(). + */ +static struct folio *try_vma_alloc_zeroed_movable_folio( + struct vm_area_struct *vma, + unsigned long vaddr, int order) +{ + struct folio *folio; + gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN; + + for (; order > 0; order--) { + folio = vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order); + if (folio) + return folio; + } + + return vma_alloc_zeroed_movable_folio(vma, vaddr, 0, 0); +} + /* * Handle the case of a page which we actually need to copy to a new page, * either due to COW or unsharing. @@ -3061,8 +3082,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, - 0, 0); + new_folio = try_vma_alloc_zeroed_movable_folio(vma, + vmf->address, 0); if (!new_folio) goto oom; } else { @@ -4050,7 +4071,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* Allocate our own private page. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0); + folio = try_vma_alloc_zeroed_movable_folio(vma, vmf->address, 0); if (!folio) goto oom; From patchwork Fri Mar 17 10:58:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79CC6C6FD1D for ; Fri, 17 Mar 2023 10:58:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B97296B0082; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B476A6B0081; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0F026B0082; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 92CF96B0080 for ; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 56CCD1216D0 for ; Fri, 17 Mar 2023 10:58:28 +0000 (UTC) X-FDA: 80578091496.25.35D3F31 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id B5C53C0004 for ; Fri, 17 Mar 2023 10:58:26 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kELGhbvkyhE1A6aVwXMfE/mNgpiFCIycE4agf1r/kQg=; b=FZQ3er2IrsLJOxbHfowOacoaWGjdvXTLTi70ID3B72ePTo1qdMAAzXWt14fpXEIXEjWYAm kPFCLKMK9/l99qbzvmBrfTzzt9ZnQ3CyNuc6XAolmb4vnydB40PQw7hxJkcFnhYcA144yj MCuKVIVqSFDZBxoq5FO1oAx5db0Mtlk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050706; a=rsa-sha256; cv=none; b=i2KWNFxglNrvYyteXw7XwYZn+tUxJIUcL7CdVetnjt41OYc65YNj2FC3X0N4ZybXKr7mx9 LmmG0Z+XzvnqX8UfnWJfzJCVXQNeJWhcV4SO/jF5AnCcn7uOEAarrzENoIu8hUGhrzHgwG CYF29VpZKa7QtqI2lxLJkHis/JfG/No= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 948AD1A00; Fri, 17 Mar 2023 03:59:09 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CE8093F885; Fri, 17 Mar 2023 03:58:24 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 4/6] mm: Implement folio_add_new_anon_rmap_range() Date: Fri, 17 Mar 2023 10:58:00 +0000 Message-Id: <20230317105802.2634004-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B5C53C0004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: tahwrbmok53chbreu9u5ns691twag5x5 X-HE-Tag: 1679050706-793486 X-HE-Meta: U2FsdGVkX1+HOmH/mg3GQ0DFaqm4ExGTKBsfgzoROMMTxM3coHlIQ+0viowTFe2ppii/XTTBdFIJtRwn/aRPRqFd99Hu/SZ2XJdGSv8fuZSsjeDtHu90XEknJ7eWG+OqGcKR7GAhQO/7rbeVRYVjFR3zpw7mn2jX8h6DqrgvbzJJhIAORtIRSiUDOQi0u4qr0RJl4kkb3bppCm49uEy5GYiQPC5m7V+46vEO0GLMbta5T5xchxc2XceC8wm8ujPAjl9GQsPyfWls7nUOKTppEa70RD1GqlktHb2mT61xKhqjYbWp1y5ck19nmk2PUjVaoO/QqhUVpC2IaW9yj0Jpg94VzAr7fDMuU5y6V0qRF4MqVAkHjlG0wKAPPILbTUrycZU3ZFRQfCwOhCRtx0IdIod5IeZK8tk3T8IjilheLk+WtNtdMO+MMGs3i3akqnzF0QobUaQ2xZc9LeDA5mN8GflKY4PBFFolKhmdfu3BHjXpYoYejOleWl6cs1H1POdKZ9hmasXzySYY8gD/1F48nH83w7NKRBvWZeyoAbzZifVZpwZuVeFPxe/LuXwnEMkKc52dyw/hkJzUV8RWm0YSFKx9Q6G+mKsuD7iCULA2/5VvwVj8PwNx6GerVTmSRo9WIlGM2ZvYczzMp3G3ng4EK1AEcx8RI0/anlA5k99nK6ozzAKa/4SgsuBnFT5xseU2JN9o/odgRIMsECDmuOg1NBRzIHLGbngUA2x2qE1NZq/foJZTLHYUBlC0Uya3Bfm8w75SxWRs+JP2Br8XeF1CGxvL4UMpAny/PN9wFt2mdGe59HHF7hK52DFLTx2Oet3kcEW0Q/ZxaRi5aYFWM/Ii1WgxltYlenujk1TPgHpcrM0Q7cNm03jrJL/s0kQhZ7hFhT+V8DRvFiNH4Q2pFr2Yk720QLoS4Dy0GjvILveEDUcDlW6VT18NocwFVKFPJHDgwwYIykbzGeIylB4uFBP NQ6NJQzi DmUvjVVIB92Glgb9mm1USWe+V5BmTocDkjRxJiLoSkD2h+YqIBgPuooY+8TrfOvOxOuXsAbG27W55GubH4rX+cflz9nwYSB/irtFfLpxf7Igv4y2KaSL/g7UKg64UcZ748/fesBTHSZZAjUmYrUBCQOY7Yjpd1KiV3lTYCUQTEiuy0FcMPyz7IEzFscnKk+fXK9KP2ZnZpHgE5r9kzgLKhxCECVMcQL7iW45C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like folio_add_new_anon_rmap() but batch-rmaps all the pages belonging to a folio, for effciency savings. Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) -- 2.25.1 diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..d1d731650ce8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +void folio_add_new_anon_rmap_range(struct folio *folio, + struct vm_area_struct *vma, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 8632e02661ac..05a0c0a700e7 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1302,6 +1302,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } +/** + * folio_add_new_anon_rmap_range - Add mapping to a new anonymous potentially + * large but definitely non-THP folio. + * @folio: The folio to add the mapping to. + * @vma: the vm area in which the mapping is added + * @address: the user virtual address of the first page in the folio + * + * Like folio_add_new_anon_rmap() but must only be called for new *non-THP* + * folios. Like folio_add_new_anon_rmap(), the inc-and-test is bypassed and the + * folio does not have to be locked. All pages in the folio are individually + * accounted. + * + * As the folio is new, it's assumed to be mapped exclusively by a single + * process. + */ +void folio_add_new_anon_rmap_range(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) +{ + int i; + int nr = folio_nr_pages(folio); + struct page *page = &folio->page; + + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); + __folio_set_swapbacked(folio); + + if (folio_test_large(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + } + + for (i = 0; i < nr; i++) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + page++; + address += PAGE_SIZE; + } + + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); + +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to From patchwork Fri Mar 17 10:58:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E014DC7618B for ; Fri, 17 Mar 2023 10:58:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 926446B0083; Fri, 17 Mar 2023 06:58:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B23B6B0081; Fri, 17 Mar 2023 06:58:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72CC16B0083; Fri, 17 Mar 2023 06:58:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 544926B0080 for ; Fri, 17 Mar 2023 06:58:29 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2E235A02B6 for ; Fri, 17 Mar 2023 10:58:29 +0000 (UTC) X-FDA: 80578091538.17.13DBE61 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 89A05180013 for ; Fri, 17 Mar 2023 10:58:27 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050707; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rBzd0QwMo2nCREZq1qWZEFUrzGffeQOFlyzkW39kPcw=; b=hRBgn6daVVtxaU1G5fZ/x43sudrT04H4GeozrNIkQa47CkNdVUhMKspoPN4WLkA9ByLMBD PGKIDmsKpeUyS1fDTVe60g2qw4Rx4m6/oGoiflyJkD/ODn0sTPbaqIs2NgEI3rzUUAOTlK 9aJj4/ZomrK4hQT0p1eR8jOkdc2EtfM= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050707; a=rsa-sha256; cv=none; b=toRNiuzHg5rRsaw5X1JszTaZzj6Vd73V3WKp9Vy91iWE4tKZ9zmTVK1aXHuKnN9+xwFS6O H3Oo4mKvJLHWXBD6mWqncIrak+XZx+GtRCn/D6Gwr6TS1AVWPCVCBbBKFY2ESPFlwqdacF 8WrSlp3QAePpDOKCmlW3MGnROzCdEdA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C67C91A25; Fri, 17 Mar 2023 03:59:10 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0C6533F885; Fri, 17 Mar 2023 03:58:25 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 5/6] mm: Allocate large folios for anonymous memory Date: Fri, 17 Mar 2023 10:58:01 +0000 Message-Id: <20230317105802.2634004-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 89A05180013 X-Stat-Signature: dqb45o356xafj6kcnodg78thw1y8bk49 X-Rspam-User: X-HE-Tag: 1679050707-492453 X-HE-Meta: U2FsdGVkX18yJWOOe9c4AMU00kr2THeig2hjrjGTmvR+FwGWLrcU6xwUc4zZjg+xscTJU6IKKjeigrKtY602/WL/nUN9DyhiMYlZOGU8H1WGQkrBieYsNY+s+DAerf1Xv2h6Ky3qzNAxtqOB3jaS5twH08Hit4HCbIvKoSE/o7dFJhmFCJeDCHfqcRNW/BtxRwXc+ml+GDrtRhna9QTN2cNY67IBim2J5fSpinL/7UkQOeUza35Lb5A4vUgA6InQD0vyjczt+Dnu3nbeDwiF7uVUOKWnrhqjCN+b9e9FV8bTYuXdvUCHjIIyFFjE4V7Hg3TkCPtJJdK8u71xo9gFDpVCb2vGtyrURrJKNvdOFqWvVH/EYTpKJEZ7NQb6qECFbyVQX7hbNmjkQH/R/r3yJiIvfaCVyoy26gJY4Izpqt9Jlk2UVZD7awKLUbg3H8steRTUjbAEUjOzZIA0P5vuBDpuERH2Jt/3rM0LpxGGrku/dc1VhS5PighUU2OC3AjzJaKeMjxq5n+5RaFGCfH+R/r2LnGkK0UNVZdHfJ1LbFtFGQdSnj7GUpZKuV94EjY4N8Vtif//JJPZkt0mnOW2rVWeIcbj2Wb+kI6cZqau6d2D3cOivBsDaVyDDfgJhH29tNyoDIQ/kpUh5Sw8/5zewd7hCmnqEM/QJ6s0riqnRuT+NKC+Fmuf+XTK61K/WSZ35bBox0EQ4B/fHu+CW40flJMGVuEY6BZQV5e0nmxZll/ILl1u6Vzr8m+taosefGn+LAMq59Oud7s2eRunXBcfgIHqeWKIY31px+fQLwiZsgfDS6PnAMpOeEWYys6Kmog6b6Xwr3Pm0QM3xakxQgT93R9HwNYarU0fJfOPqgfQskVjQr+1Hhyf96Mkrj3dMif8T4CZO8rBKxr9aRQj0xqjobS0Xu9EUncQPQEak7kpQSIIcWSVRhDJBHTfJhGdnSvqG4fQnQ0ygC5AfQVqCAB 6gKgoRlE eRMGrGS5xBi6U6eB6b4N6FOBxBvP82R+T8tEIq7ohdSntTItoD30UwjpMLjQ6Um5o9Dh0nl14DwNN86l0g4ysqDTBhvaTY9YudxlZYBAJ81DXzuYlHTENgp9GTPQhi/Ln45IGTdAK+/QLlQgdBGwnvxYqNn6364XtGrP9gDvIiekby5E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add the machinery to determine what order of folio to allocate within do_anonymous_page() and deal with racing faults to the same region. TODO: For now, the maximum order is set to 4. This should probably be set per-vma based on factors, and adjusted dynamically. Signed-off-by: Ryan Roberts --- mm/memory.c | 140 ++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 16 deletions(-) -- 2.25.1 diff --git a/mm/memory.c b/mm/memory.c index c9e09415ee18..3d01eab46d9c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4013,6 +4013,77 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) return ret; } +/* + * Returns index of first pte that is not none, or nr if all are none. + */ +static int check_all_ptes_none(pte_t *pte, int nr) +{ + int i; + + for (i = 0; i < nr; i++) { + if (!pte_none(*pte++)) + return i; + } + + return nr; +} + +static void calc_anonymous_folio_order(struct vm_fault *vmf, + int *order_out, + unsigned long *addr_out) +{ + /* + * The aim here is to determine what size of folio we should allocate + * for this fault. Factors include: + * - Folio must be naturally aligned within VA space + * - Folio must not breach boundaries of vma + * - Folio must be fully contained inside one pmd entry + * - Folio must not overlap any non-none ptes + * - Order must not be higher than *order_out upon entry + * + * Note that the caller may or may not choose to lock the pte. If + * unlocked, the calculation should be considered an estimate that will + * need to be validated under the lock. + */ + + struct vm_area_struct *vma = vmf->vma; + int nr; + int order = min(*order_out, PMD_SHIFT - PAGE_SHIFT); + unsigned long addr; + pte_t *pte; + pte_t *first_set = NULL; + int ret; + + for (; order > 0; order--) { + nr = 1 << order; + addr = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE); + pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT); + + /* Check vma bounds. */ + if (addr < vma->vm_start || + addr + nr * PAGE_SIZE > vma->vm_end) + continue; + + /* All ptes covered by order already known to be none. */ + if (pte + nr <= first_set) + break; + + /* Already found set pte in range covered by order. */ + if (pte <= first_set) + continue; + + /* Need to check if all the ptes are none. */ + ret = check_all_ptes_none(pte, nr); + if (ret == nr) + break; + + first_set = pte + ret; + } + + *order_out = order; + *addr_out = order > 0 ? addr : vmf->address; +} + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4024,6 +4095,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) struct folio *folio; vm_fault_t ret = 0; pte_t entry; + unsigned long addr; + int order = 4; // TODO: Policy for maximum folio order. + int pgcount; /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) @@ -4065,24 +4139,41 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_MISSING); } - goto setpte; + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, vmf->address, vmf->pte); + goto unlock; } - /* Allocate our own private page. */ +retry: + /* + * Estimate the folio order to allocate. We are not under the ptl here + * so this estiamte needs to be re-checked later once we have the lock. + */ + vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + calc_anonymous_folio_order(vmf, &order, &addr); + pte_unmap(vmf->pte); + + /* Allocate our own private folio. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = try_vma_alloc_zeroed_movable_folio(vma, vmf->address, 0); + folio = try_vma_alloc_zeroed_movable_folio(vma, addr, order); if (!folio) goto oom; + /* We may have been granted less than we asked for. */ + order = folio_order(folio); + pgcount = folio_nr_pages(folio); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; - cgroup_throttle_swaprate(&folio->page, GFP_KERNEL); + folio_throttle_swaprate(folio, GFP_KERNEL); /* * The memory barrier inside __folio_mark_uptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. + * preceding stores to the folio contents become visible before + * the set_ptes() write. */ __folio_mark_uptodate(folio); @@ -4091,11 +4182,26 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); - if (!pte_none(*vmf->pte)) { - update_mmu_tlb(vma, vmf->address, vmf->pte); - goto release; + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); + + /* + * Ensure our estimate above is still correct; we could have raced with + * another thread to service a fault in the region. + */ + if (check_all_ptes_none(vmf->pte, pgcount) != pgcount) { + pte_t *pte = vmf->pte + ((vmf->address - addr) >> PAGE_SHIFT); + + /* If faulting pte was allocated by another, exit early. */ + if (!pte_none(*pte)) { + update_mmu_tlb(vma, vmf->address, pte); + goto release; + } + + /* Else try again, with a lower order. */ + pte_unmap_unlock(vmf->pte, vmf->ptl); + folio_put(folio); + order--; + goto retry; } ret = check_stable_address_space(vma->vm_mm); @@ -4109,14 +4215,16 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, pgcount - 1); + + add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount); + folio_add_new_anon_rmap_range(folio, vma, addr); folio_add_lru_vma(folio, vma); -setpte: - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + + set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vma, addr, vmf->pte, pgcount); unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; From patchwork Fri Mar 17 10:58:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57AE6C74A5B for ; Fri, 17 Mar 2023 10:58:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAD136B0080; Fri, 17 Mar 2023 06:58:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A5F186B0081; Fri, 17 Mar 2023 06:58:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AF836B0085; Fri, 17 Mar 2023 06:58:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 76F036B0080 for ; Fri, 17 Mar 2023 06:58:30 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 508FE1617CF for ; Fri, 17 Mar 2023 10:58:30 +0000 (UTC) X-FDA: 80578091580.08.0F9CB0E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id AB954C0009 for ; Fri, 17 Mar 2023 10:58:28 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S/jJrLnAXbq8rUkwlmRMy37z3XO9fYbM79Iuu4QhghE=; b=mCQsb4tgfz7ImqRdHe/UHnjGdgYL1YAizVn2mTO2rPEWdtCkON9jLBb43tbAolhq6zkdiJ CZ3zF8M6b+i0QU5YQWjABlMeHQQ0xi2gHzribKscCW6ZSxGIqQvzMfSUZ4VKoMJYflbP2W Tv044+uyu4o88huV6v/atxpbI7f7WkQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050708; a=rsa-sha256; cv=none; b=BFsGTyudxwhYzMpw3u/MVdYRmEl3WpQhJefgyoIaWr/s4Ors3gJ0r1HpKE46SKUk2QsWZL Srf2RpjpH573zOvS5lm6Jysd5qPKW7J+HxcMbMhIVPxXwkhBfiTzau1JseBrQVUYaFMWEZ BWKHXBWsNAroMrACjPcMwEacer5Z0Zs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 043051A2D; Fri, 17 Mar 2023 03:59:12 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3E3FA3F885; Fri, 17 Mar 2023 03:58:27 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 6/6] WORKAROUND: Don't split large folios on madvise Date: Fri, 17 Mar 2023 10:58:02 +0000 Message-Id: <20230317105802.2634004-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AB954C0009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: zwxaf6zr85w6mb3w168t1rc9r8qj6jxc X-HE-Tag: 1679050708-714459 X-HE-Meta: U2FsdGVkX1804EisYJPcNVlLOM6qZ0Qkk1PxOxXZb1vQdspjnCIlfb9bcxMws3z17oeUyPwd5CiQv6ggromtR0qvI0Jp7G9GB5Ip1cbIyd6DmExQI9JBazxUMut88BSjOHNiFZao4msX8OSXBUSzb8rcb3siN531SCTX1jvJkoEWjAN4uHpTy4rppExPc7DLtt524dem5vp/zCyXJFiIZ3sEt3Ay8t5YBi9zVhFqKikMaf/Kz+16WejgkQT4aetCAcTduyStMyNr5z4l78k2/lJEzHZDgeajo7Gwnzmhb5IsRoRfX8O0744XfYRYsjNiXinI4TmTrx87mkjBX50+GC2niCQSkb3pPMQ2LpxN0w1xTckNV30wBGPq/Bkl3H3wreipIC3bB9SIci2z/US97qm5XSO8C3B9JwYWuKNsZb9/KCaIQGCxAGDZPJZ0WUi5H3pi9vs1ZCnpe0zZQfAjxsDnUMIACCVWyXLNwPl4eLhFI4O41L9ad1hdBSO42ImYb7wG1U6oIgWlIuQHX6hy55MSuJGyr2Hqay1W7lTSnACgj9n63Lg3k4B77JSzJjKtvfKoFNRLmOimOGaaAbmqRliF2+h8G0/6Vi1H6+3Bhmzl0HWCUaVch9Fw3j1hYrsdwG66cY54yFb+HrnM5W6Ktqi85s34r+OXINUO626JJRFJHSWxkT+ygDKnTmH/3cBcDE6h0onhAEw4K2AKbTRLmB7ANrE6gpO83BfX/dVpmft87sBRWIdKN2fe+5nES4FsckYCgTGYt8G1YBjw4LEsONTsSXtFAANhUfjhX/7yjTOIGgRSNbDhqfjo2WiZE9pOzCqChkWRsXn6uOIxU5dOERtG5Jtm7Yyrc38V9Sga/BCCuAyA6P2XdcQc7yu9TWq1baw+Ut2fcOEVmHF9aHWuvMYHu1jPXicVU6wGhJoc8AwpAyGor9I4SykirPBuwU96Yz+sqdvK8GZ29kRiRN6 8f5Aqoba 4O4yN4A3g4a30lq0tVZm4gHoLRWY/rqn2gK15JtGH44b3LaaqGdHnljeO/bpr/ZQl39UIg1cDJzrb1Mh4Sg1ggu0cSX0pWdhuH9UzXIQg2zRiRht6tcdoeXQqpv+2y1lFEVt385Jt4U5T1xPJQ6gwUt2EZwEe58LveKqKQPA9W7HxBZw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Ryan Roberts --- mm/madvise.c | 8 ++++++++ 1 file changed, 8 insertions(+) -- 2.25.1 diff --git a/mm/madvise.c b/mm/madvise.c index 340125d08c03..8fb84da744e1 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -447,6 +447,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * are sure it's worth. Split it if we are only owner. */ if (folio_test_large(folio)) { +#if 0 if (folio_mapcount(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) @@ -469,6 +470,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, pte--; addr -= PAGE_SIZE; continue; +#else + break; +#endif } /* @@ -664,6 +668,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, * deactivate all pages. */ if (folio_test_large(folio)) { +#if 0 if (folio_mapcount(folio) != 1) goto out; folio_get(folio); @@ -684,6 +689,9 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, pte--; addr -= PAGE_SIZE; continue; +#else + goto out; +#endif } if (folio_test_swapcache(folio) || folio_test_dirty(folio)) {