From patchwork Wed Aug 23 13:13:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AE14EE4993 for ; Wed, 23 Aug 2023 13:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3CF8900022; Wed, 23 Aug 2023 09:14:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CEC738E0011; Wed, 23 Aug 2023 09:14:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDB74900022; Wed, 23 Aug 2023 09:14:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AEAC78E0011 for ; Wed, 23 Aug 2023 09:14:18 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7FA51120122 for ; Wed, 23 Aug 2023 13:14:18 +0000 (UTC) X-FDA: 81155412996.11.AC809D4 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id CEF714000C for ; Wed, 23 Aug 2023 13:14:16 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sGHTBJfPfbpWrvSaUZpCtVi32GUYp0AP8xhZnfIvmvE=; b=LPidU3oxAj4D1I4eL3NoPpVLFMOE/1pz+CampWEmXgE0TantYEizjovcueoMX3SsTKC2wy CI7SJYuVlXLdOi6TAsABw8AfgENdUQRjRxfmbpkpRXSLBmx68m3YtfoFu+etEx3l6st2fL G/AFcu9senOdyoFoVztYRdbrABU0S5o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796456; a=rsa-sha256; cv=none; b=ktInXFA3tm5C9xC+ye2YFSiU2RPZtyPEwSXQgXSOKgT0baNZrYlu1/l3lDADUKWht1ZzbI umtKpWrjAik2umT2rOZ4FO/dhEETiOm2kXsQ2WfHp6n+sCJ5iNG+RkL4LkjHm6VZkC1UAg C59QDvUzR/fSfpTavrQhs7e0IskLsPg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6E09E11FB; Wed, 23 Aug 2023 06:14:56 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2BD863F740; Wed, 23 Aug 2023 06:14:10 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 01/37] mm: page_alloc: Rename gfp_to_alloc_flags_cma -> gfp_to_alloc_flags_fast Date: Wed, 23 Aug 2023 14:13:14 +0100 Message-Id: <20230823131350.114942-2-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Stat-Signature: kt7d7j5mo6mwmea957pgw831jgkd535o X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: CEF714000C X-Rspam-User: X-HE-Tag: 1692796456-428294 X-HE-Meta: U2FsdGVkX1+cNXfI8CGwuL8fWKt+25QDpkgbgelLiIXR77EDYOEhbS95rKcEGDvEM9m1mzHoX0JGVXF6mWNC9FBgBZRrwXOSnJ+/VadmS/ErGkAWDsqkGLcom0KLj+8t2HgmmNIPR208TONmLrq5B1PNvhYNOgltpi27C5XUyeonyuWtdO7DjsVDrcFPMJ0pBGw1EWRPVojaPziz50TMKlky7yAV2A6cLhNJKmQJspY7Pw9d/JfSx7VFlI8bc9UMGEgIeAB7ELCUulLzj7qFeCGrj2wIdvCIJTW5Yn/oSHwBk+eKLOIBHcB0KyuEqOInlEXmYA5KT7MdzFACAxhzMh+ZHPLNKYGZNF3gdzf7UIiorK6wYU8yYj3YjEBJzWRQa/pR9Nu1++Q+nGQgvEK9RoYrmA3CHzugSV76qEjnaBGnWGvK5ypyG8C77tnWegKoG8NBFRO1lV44vcMprkhJZsmZO6XuPGOZKUZF6TNDuCqEFTmHbZwgUnV/McpwOZwhx4rdlJG54nNy0OlG6GhSgEEMEUeXe1nUNNcbHw5l/x1CCqUdeYWY3dv1covqBKzqyd3Rf3YMi+cEkHReXuq+VWxHFHqpM+0Ihb2xeRmFuBPnwX7yxoxVosD/aguKpQAhH/462l1tS8zau4/xtgjsS/12vKpxevMEG5bsPb8C18c0pnQ/T1lpWdqirqP+7z7lo50DFkb6CdIHQIa0U6rD07xMoMPuRD0udcHDUQObkdubwwCh+Iy8vZHcViq9wABYbRncuu5YQDcPDoX5Qm8ABD9yQtUkRzsH2rkeRSvpBcseCJIz/t+lLvECD3AyZQG5WpemCQ9blr4UomKhQl3dkiqfV1MFJKAgj4+0Cb/GJtB9+C6Q1oLlzwR6ibdaRMkW07/gjkAyD0lwX0skxZanlBYn/02FJSAU5XBhnpf9ANB6pQyfz4MV4Z8TYMDApAhDxNUHgtw+Sx5sNtrdO9K 6Uo7vzBD k94XFdWzmIEeB2RAmQmvTyPcd5HedL578XxDkfFJ1ycKnGtsaElCToRljSKs/hxqD8W5vh1PtXP4ileAYT85ESYfLX9QVxaTIgAJdO5gV/A88IqdNC4GuUm3gCnbkUF+okL+3+mxzQkS05ilJAXyY3M/8mdZeCFYq5ArdohxYRRYXqbR5OB5YjLwPFyFH2DpTVC1h17VCQFyxFSg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: gfp_to_alloc_flags_cma() is called on the fast path of the page allocator and all it does is set the ALLOC_CMA flag if all the conditions are met for the allocation to be satisfied from the MIGRATE_CMA list. Rename it to be more generic, as it will soon have to handle another another flag. Signed-off-by: Alexandru Elisei --- mm/page_alloc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7d3460c7a480..e6f950c54494 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3081,7 +3081,7 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) } /* Must be called after current_gfp_context() which can change gfp_mask */ -static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, +static inline unsigned int gfp_to_alloc_flags_fast(gfp_t gfp_mask, unsigned int alloc_flags) { #ifdef CONFIG_CMA @@ -3784,7 +3784,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) } else if (unlikely(rt_task(current)) && in_task()) alloc_flags |= ALLOC_MIN_RESERVE; - alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, alloc_flags); + alloc_flags = gfp_to_alloc_flags_fast(gfp_mask, alloc_flags); return alloc_flags; } @@ -4074,7 +4074,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) - alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) | + alloc_flags = gfp_to_alloc_flags_fast(gfp_mask, reserve_flags) | (alloc_flags & ALLOC_KSWAPD); /* @@ -4250,7 +4250,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, if (should_fail_alloc_page(gfp_mask, order)) return false; - *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); + *alloc_flags = gfp_to_alloc_flags_fast(gfp_mask, *alloc_flags); /* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE); From patchwork Wed Aug 23 13:13:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F73EE4993 for ; Wed, 23 Aug 2023 13:14:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEA21900023; Wed, 23 Aug 2023 09:14:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E73608E0011; Wed, 23 Aug 2023 09:14:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1420900023; Wed, 23 Aug 2023 09:14:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C06C08E0011 for ; Wed, 23 Aug 2023 09:14:24 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9A937A00F1 for ; Wed, 23 Aug 2023 13:14:24 +0000 (UTC) X-FDA: 81155413248.10.B4DAC84 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id D1D901C0008 for ; Wed, 23 Aug 2023 13:14:22 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796463; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YD3SDcEXetacx92ImuY7U0oJT7AUdjMUF5wK06ms074=; b=4/gDbr8Wy+aciz+1l/Ik/KyEcebQqJfwP+wLAESbhSwNEGgCPZAhXp8QWL0CzKVKl1jEJl rHPv1ftFKm/bAqXrmZ+0Wj1nSWb8jy9jrlbxo8HQmqeEaMxfaMWZZeNoAAyzslfvllxxzW abRH35gMsiJt5dZa2KfMm8UKKa94UTU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796463; a=rsa-sha256; cv=none; b=QK/QyMzroQS95Jk+zN+g2fGt5+nfjSBGT0je4tT2B8M5yPHLqx/cRiv7TCJYDfWBz8f62/ t5k2+4iwvOtaQwrZOnZIpmDiJZTpng7Gz1v1nnR+sHyZPch+n15rS5sjYTI8ngxoXghtCO YFut6CXBQ9IBHOa9lJ79QN5KkvX0Bh4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BF9EF1515; Wed, 23 Aug 2023 06:15:02 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2BAF13F740; Wed, 23 Aug 2023 06:14:16 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 02/37] arm64: mte: Rework naming for tag manipulation functions Date: Wed, 23 Aug 2023 14:13:15 +0100 Message-Id: <20230823131350.114942-3-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D1D901C0008 X-Stat-Signature: cgfzrzjqmyp86dgqtqute6c7o31xy8w4 X-HE-Tag: 1692796462-251654 X-HE-Meta: U2FsdGVkX1+PDSUTxCoi8Tesv+ngmlyLG6hSUcoCV8SwWfOQseL/AfwD2l15GMF/camVQZ/+S9S0ve/FWxM3WuiKLZNwxwsuANEEj+c6JoGLqAnLV4KQ8faMg++nCcqLKhPqyvtCXHHPCGbRpm9Iii7R8xEBzuz3ok0HJI47eITNVCcjuuAdweI4cEk/U17j9hnACXHo6mnGN26jrm3VsIZGZTZ7a1rT4HUKqh5NWNFgOHfQa43nml64TJ+QNVqZcqsH1F+N+nu8H4Vgpoa5jx6D+jg++EYh+gdeJQA+SZglAHr0a+gDNzwxEsadyYdd3DzuzZ+Zi/BhrVx+NQUwEfbmAZ/I8W2dCRHMVFkGACsNny8qurPIklsf3yWlWAjfC/HGdKIGGEM1L3mNl96X46Adooldq3SOnB3tf3Vzi/j7+uC3seSGdY3u5ZB2iXx1ty4m0M1IXu3eU+ea85dLSsG32DHQe6pCyPBqluHPuSrbIo5+y2113hXavFKgiBM8f7gljan7eNzylmNBeLYN/KHpMuzcPH43G0HXiSeDFJ6jlq17B9AUQM3tw5CUsqWPkhMQlcRbEPke8/LO7Szy1G2WuB+oyglSoe1FG1AnSGQ81sJP2x7pN+AVtyoGtEPRKrZ7bBjt4p1Y9pqRAQGlU0v+T96NlqoiqOMqNs/sU1vwi21bkNSbgUbMvX+MBrOoZxqTyFN2WW9WqJlaSTSzFEIuRRz2zOn+3BQAnBn8w2ctZUAHJMziNiKpESw9eNy3uM8Bpmo4G6G+A6PBoftcPtqQrj01lTFE4tgkgsoIELNnCsIrE+2vDM0D5lZJRLH8GXE1UJv2cdEp3jweHadihCw92UKDvzFvrKgjnULRcl0oJocijeq07eL0e8NkS5O1095aGvhl8KEhr3uqjqqg6Y194E0MaCIix74tCT7YWr5c34vFfPJSjtSosPlYouKfHjArsiCXb7Y2Yqby8b+ XZ+n4SPD J2I1erc7N5Ca54eTRE0k/vrKDgo1M+imrDEB467uJ/mVB/PWgohWmFikwdjqmVP+9ez9HFmuZlvlC51yMBWzdY3UDuEzS27tArKDe09OUKNwB91sBigc0y3HPSJdrFyXDXPCujNSTQcg/qu6qWIpKHns4MoofEu/LCrIT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tag save/restore/copy functions could be more explicit about from where the tags are coming from and where they are being copied to. Renaming the functions to make it easier to understand what they are doing: - Rename the mte_clear_page_tags() 'addr' parameter to 'page_addr', to match the other functions that take a page address as parameter. - Rename mte_save/restore_tags() to mte_save/restore_page_tags_by_swp_entry() to 1. distinguish the functions from mte_save/restore_page_tags() and 2. make it clear how they are indexed (this will become important once other ways to save the tags are added). Same applies to mte_invalidate_tags{,_area}_by_swp_entry(). - Rename mte_save/restore_page_tags() to make it clear where the tags are going to be saved, respectively from where they are restored - in a previously allocator memory buffer, not in an xarray, like with the tags preserved when swapping. - Rename mte_allocate/free_tag_storage() to mte_allocate/free_tags_mem() to make it clear the functions have nothing to do with the memory where the live tags are stored for a page. Change the parameter type for mte_free_tags_mem()) to be void *, to match the return value of mte_allocate_tags_mem(). Also because that memory is opaque and it not meant to be directly deferenced. In the name of consistency rename local variables from tag_storage to tags. Give a similar treatment to the hibernation code that saves and restores the tags for all tagged pages. In the same spirit, rename MTE_PAGE_TAG_STORAGE to MTE_PAGE_TAG_STORAGE_SIZE to make it clear that it relates to the size of the memory needed to save the tags for a page. Oportunistically rename MTE_TAG_SIZE to MTE_TAG_SIZE_BITS to make it clear it is measured in bits, not bytes, like the rest of the size variable from the same header file. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte-def.h | 16 +++++----- arch/arm64/include/asm/mte.h | 24 +++++++++------ arch/arm64/include/asm/pgtable.h | 8 ++--- arch/arm64/kernel/elfcore.c | 14 ++++----- arch/arm64/kernel/hibernate.c | 46 ++++++++++++++--------------- arch/arm64/lib/mte.S | 14 ++++----- arch/arm64/mm/mteswap.c | 50 ++++++++++++++++---------------- 7 files changed, 89 insertions(+), 83 deletions(-) diff --git a/arch/arm64/include/asm/mte-def.h b/arch/arm64/include/asm/mte-def.h index 14ee86b019c2..eb0d76a6bdcf 100644 --- a/arch/arm64/include/asm/mte-def.h +++ b/arch/arm64/include/asm/mte-def.h @@ -5,14 +5,14 @@ #ifndef __ASM_MTE_DEF_H #define __ASM_MTE_DEF_H -#define MTE_GRANULE_SIZE UL(16) -#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) -#define MTE_GRANULES_PER_PAGE (PAGE_SIZE / MTE_GRANULE_SIZE) -#define MTE_TAG_SHIFT 56 -#define MTE_TAG_SIZE 4 -#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT) -#define MTE_PAGE_TAG_STORAGE (MTE_GRANULES_PER_PAGE * MTE_TAG_SIZE / 8) +#define MTE_GRANULE_SIZE UL(16) +#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_GRANULES_PER_PAGE (PAGE_SIZE / MTE_GRANULE_SIZE) +#define MTE_TAG_SHIFT 56 +#define MTE_TAG_SIZE_BITS 4 +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE_BITS - 1)), MTE_TAG_SHIFT) +#define MTE_PAGE_TAG_STORAGE_SIZE (MTE_GRANULES_PER_PAGE * MTE_TAG_SIZE_BITS / 8) -#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" +#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" #endif /* __ASM_MTE_DEF_H */ diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 4cedbaa16f41..246a561652f4 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -18,19 +18,25 @@ #include -void mte_clear_page_tags(void *addr); +void mte_clear_page_tags(void *page_addr); + unsigned long mte_copy_tags_from_user(void *to, const void __user *from, unsigned long n); unsigned long mte_copy_tags_to_user(void __user *to, void *from, unsigned long n); -int mte_save_tags(struct page *page); -void mte_save_page_tags(const void *page_addr, void *tag_storage); -void mte_restore_tags(swp_entry_t entry, struct page *page); -void mte_restore_page_tags(void *page_addr, const void *tag_storage); -void mte_invalidate_tags(int type, pgoff_t offset); -void mte_invalidate_tags_area(int type); -void *mte_allocate_tag_storage(void); -void mte_free_tag_storage(char *storage); + +/* page_private(page) contains the swp_entry.val value. */ +int mte_save_page_tags_by_swp_entry(struct page *page); +void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page); + +void mte_save_page_tags_to_mem(const void *page_addr, void *to); +void mte_restore_page_tags_from_mem(void *page_addr, const void *from); + +void mte_invalidate_tags_by_swp_entry(int type, pgoff_t offset); +void mte_invalidate_tags_area_by_swp_entry(int type); + +void *mte_allocate_tags_mem(void); +void mte_free_tags_mem(void *tags); #ifdef CONFIG_ARM64_MTE diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e8a252e62b12..944860d7090e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1020,7 +1020,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, static inline int arch_prepare_to_swap(struct page *page) { if (system_supports_mte()) - return mte_save_tags(page); + return mte_save_page_tags_by_swp_entry(page); return 0; } @@ -1028,20 +1028,20 @@ static inline int arch_prepare_to_swap(struct page *page) static inline void arch_swap_invalidate_page(int type, pgoff_t offset) { if (system_supports_mte()) - mte_invalidate_tags(type, offset); + mte_invalidate_tags_by_swp_entry(type, offset); } static inline void arch_swap_invalidate_area(int type) { if (system_supports_mte()) - mte_invalidate_tags_area(type); + mte_invalidate_tags_area_by_swp_entry(type); } #define __HAVE_ARCH_SWAP_RESTORE static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) { if (system_supports_mte()) - mte_restore_tags(entry, &folio->page); + mte_restore_page_tags_by_swp_entry(entry, &folio->page); } #endif /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/kernel/elfcore.c b/arch/arm64/kernel/elfcore.c index 2e94d20c4ac7..c062c2c3d10d 100644 --- a/arch/arm64/kernel/elfcore.c +++ b/arch/arm64/kernel/elfcore.c @@ -17,7 +17,7 @@ static unsigned long mte_vma_tag_dump_size(struct core_vma_metadata *m) { - return (m->dump_size >> PAGE_SHIFT) * MTE_PAGE_TAG_STORAGE; + return (m->dump_size >> PAGE_SHIFT) * MTE_PAGE_TAG_STORAGE_SIZE; } /* Derived from dump_user_range(); start/end must be page-aligned */ @@ -38,7 +38,7 @@ static int mte_dump_tag_range(struct coredump_params *cprm, * have been all zeros. */ if (!page) { - dump_skip(cprm, MTE_PAGE_TAG_STORAGE); + dump_skip(cprm, MTE_PAGE_TAG_STORAGE_SIZE); continue; } @@ -48,12 +48,12 @@ static int mte_dump_tag_range(struct coredump_params *cprm, */ if (!page_mte_tagged(page)) { put_page(page); - dump_skip(cprm, MTE_PAGE_TAG_STORAGE); + dump_skip(cprm, MTE_PAGE_TAG_STORAGE_SIZE); continue; } if (!tags) { - tags = mte_allocate_tag_storage(); + tags = mte_allocate_tags_mem(); if (!tags) { put_page(page); ret = 0; @@ -61,16 +61,16 @@ static int mte_dump_tag_range(struct coredump_params *cprm, } } - mte_save_page_tags(page_address(page), tags); + mte_save_page_tags_to_mem(page_address(page), tags); put_page(page); - if (!dump_emit(cprm, tags, MTE_PAGE_TAG_STORAGE)) { + if (!dump_emit(cprm, tags, MTE_PAGE_TAG_STORAGE_SIZE)) { ret = 0; break; } } if (tags) - mte_free_tag_storage(tags); + mte_free_tags_mem(tags); return ret; } diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 02870beb271e..f3cdbd8ba8f9 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -215,41 +215,41 @@ static int create_safe_exec_page(void *src_start, size_t length, #ifdef CONFIG_ARM64_MTE -static DEFINE_XARRAY(mte_pages); +static DEFINE_XARRAY(tags_by_pfn); -static int save_tags(struct page *page, unsigned long pfn) +static int save_page_tags_by_pfn(struct page *page, unsigned long pfn) { - void *tag_storage, *ret; + void *tags, *ret; - tag_storage = mte_allocate_tag_storage(); - if (!tag_storage) + tags = mte_allocate_tags_mem(); + if (!tags) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); + mte_save_page_tags_to_mem(page_address(page), tags); - ret = xa_store(&mte_pages, pfn, tag_storage, GFP_KERNEL); + ret = xa_store(&tags_by_pfn, pfn, tags, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + mte_free_tags_mem(tags); return xa_err(ret); } else if (WARN(ret, "swsusp: %s: Duplicate entry", __func__)) { - mte_free_tag_storage(ret); + mte_free_tags_mem(ret); } return 0; } -static void swsusp_mte_free_storage(void) +static void swsusp_mte_free_tags(void) { - XA_STATE(xa_state, &mte_pages, 0); + XA_STATE(xa_state, &tags_by_pfn, 0); void *tags; - xa_lock(&mte_pages); + xa_lock(&tags_by_pfn); xas_for_each(&xa_state, tags, ULONG_MAX) { - mte_free_tag_storage(tags); + mte_free_tags_mem(tags); } - xa_unlock(&mte_pages); + xa_unlock(&tags_by_pfn); - xa_destroy(&mte_pages); + xa_destroy(&tags_by_pfn); } static int swsusp_mte_save_tags(void) @@ -273,9 +273,9 @@ static int swsusp_mte_save_tags(void) if (!page_mte_tagged(page)) continue; - ret = save_tags(page, pfn); + ret = save_page_tags_by_pfn(page, pfn); if (ret) { - swsusp_mte_free_storage(); + swsusp_mte_free_tags(); goto out; } @@ -290,25 +290,25 @@ static int swsusp_mte_save_tags(void) static void swsusp_mte_restore_tags(void) { - XA_STATE(xa_state, &mte_pages, 0); + XA_STATE(xa_state, &tags_by_pfn, 0); int n = 0; void *tags; - xa_lock(&mte_pages); + xa_lock(&tags_by_pfn); xas_for_each(&xa_state, tags, ULONG_MAX) { unsigned long pfn = xa_state.xa_index; struct page *page = pfn_to_online_page(pfn); - mte_restore_page_tags(page_address(page), tags); + mte_restore_page_tags_from_mem(page_address(page), tags); - mte_free_tag_storage(tags); + mte_free_tags_mem(tags); n++; } - xa_unlock(&mte_pages); + xa_unlock(&tags_by_pfn); pr_info("Restored %d MTE pages\n", n); - xa_destroy(&mte_pages); + xa_destroy(&tags_by_pfn); } #else /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 5018ac03b6bf..d3c4ff70f48b 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -119,7 +119,7 @@ SYM_FUNC_START(mte_copy_tags_to_user) cbz x2, 2f 1: ldg x4, [x1] - ubfx x4, x4, #MTE_TAG_SHIFT, #MTE_TAG_SIZE + ubfx x4, x4, #MTE_TAG_SHIFT, #MTE_TAG_SIZE_BITS USER(2f, sttrb w4, [x0]) add x0, x0, #1 add x1, x1, #MTE_GRANULE_SIZE @@ -134,9 +134,9 @@ SYM_FUNC_END(mte_copy_tags_to_user) /* * Save the tags in a page * x0 - page address - * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes + * x1 - memory buffer, MTE_PAGE_TAG_STORAGE_SIZE bytes */ -SYM_FUNC_START(mte_save_page_tags) +SYM_FUNC_START(mte_save_page_tags_to_mem) multitag_transfer_size x7, x5 1: mov x2, #0 @@ -153,14 +153,14 @@ SYM_FUNC_START(mte_save_page_tags) b.ne 1b ret -SYM_FUNC_END(mte_save_page_tags) +SYM_FUNC_END(mte_save_page_tags_to_mem) /* * Restore the tags in a page * x0 - page address - * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes + * x1 - memory buffer, MTE_PAGE_TAG_STORAGE_SIZE bytes */ -SYM_FUNC_START(mte_restore_page_tags) +SYM_FUNC_START(mte_restore_page_tags_from_mem) multitag_transfer_size x7, x5 1: ldr x2, [x1], #8 @@ -174,4 +174,4 @@ SYM_FUNC_START(mte_restore_page_tags) b.ne 1b ret -SYM_FUNC_END(mte_restore_page_tags) +SYM_FUNC_END(mte_restore_page_tags_from_mem) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index cd508ba80ab1..aaeca57f36cc 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -7,78 +7,78 @@ #include #include -static DEFINE_XARRAY(mte_pages); +static DEFINE_XARRAY(tags_by_swp_entry); -void *mte_allocate_tag_storage(void) +void *mte_allocate_tags_mem(void) { /* tags granule is 16 bytes, 2 tags stored per byte */ - return kmalloc(MTE_PAGE_TAG_STORAGE, GFP_KERNEL); + return kmalloc(MTE_PAGE_TAG_STORAGE_SIZE, GFP_KERNEL); } -void mte_free_tag_storage(char *storage) +void mte_free_tags_mem(void *tags) { - kfree(storage); + kfree(tags); } -int mte_save_tags(struct page *page) +int mte_save_page_tags_by_swp_entry(struct page *page) { - void *tag_storage, *ret; + void *tags, *ret; if (!page_mte_tagged(page)) return 0; - tag_storage = mte_allocate_tag_storage(); - if (!tag_storage) + tags = mte_allocate_tags_mem(); + if (!tags) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); + mte_save_page_tags_to_mem(page_address(page), tags); /* page_private contains the swap entry.val set in do_swap_page */ - ret = xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); + ret = xa_store(&tags_by_swp_entry, page_private(page), tags, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + mte_free_tags_mem(tags); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + mte_free_tags_mem(ret); } return 0; } -void mte_restore_tags(swp_entry_t entry, struct page *page) +void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page) { - void *tags = xa_load(&mte_pages, entry.val); + void *tags = xa_load(&tags_by_swp_entry, entry.val); if (!tags) return; if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); + mte_restore_page_tags_from_mem(page_address(page), tags); set_page_mte_tagged(page); } } -void mte_invalidate_tags(int type, pgoff_t offset) +void mte_invalidate_tags_by_swp_entry(int type, pgoff_t offset) { swp_entry_t entry = swp_entry(type, offset); - void *tags = xa_erase(&mte_pages, entry.val); + void *tags = xa_erase(&tags_by_swp_entry, entry.val); - mte_free_tag_storage(tags); + mte_free_tags_mem(tags); } -void mte_invalidate_tags_area(int type) +void mte_invalidate_tags_area_by_swp_entry(int type) { swp_entry_t entry = swp_entry(type, 0); swp_entry_t last_entry = swp_entry(type + 1, 0); void *tags; - XA_STATE(xa_state, &mte_pages, entry.val); + XA_STATE(xa_state, &tags_by_swp_entry, entry.val); - xa_lock(&mte_pages); + xa_lock(&tags_by_swp_entry); xas_for_each(&xa_state, tags, last_entry.val - 1) { - __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + __xa_erase(&tags_by_swp_entry, xa_state.xa_index); + mte_free_tags_mem(tags); } - xa_unlock(&mte_pages); + xa_unlock(&tags_by_swp_entry); } From patchwork Wed Aug 23 13:13:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8E5DEE4993 for ; Wed, 23 Aug 2023 13:14:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A00A900024; Wed, 23 Aug 2023 09:14:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 34FAC8E0011; Wed, 23 Aug 2023 09:14:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F07E900024; Wed, 23 Aug 2023 09:14:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0FB248E0011 for ; Wed, 23 Aug 2023 09:14:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E558D16010E for ; Wed, 23 Aug 2023 13:14:30 +0000 (UTC) X-FDA: 81155413500.06.D0C763A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 13DBE80004 for ; Wed, 23 Aug 2023 13:14:28 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3PTixGCuQdqP16ZuJENwZtq1IDL1MOp3N0SmIUR/L8c=; b=lwX5SM6rk0a1XHcpkAx1rUWnq3X2kgDqDEIo96FSIp0E2YTuw9qvafY+GzKNYjR6oYVHkg 8XxQoc8EFyAiDhSUCTNqfZ4HfyiU+nQzF+PntsK0yOCPizSgaoO16+G4XCx+NMOahDjiEd GuzGxPG/4987XDvnmzu/Nko8kiTzR3A= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796469; a=rsa-sha256; cv=none; b=TQQ7QQ+eDT+lwiK3HE7nFgNVZ+zdkcpLOJdKTmXNAnVPVyn8VSjb8j1qrQOerAUCGqxMuI R5GKFVMJ/B+dwtjjWAGQ1cXtVTrUmD9SWKQozmLJmS9pZLpCP7ECvYl1hFggURft14sswS CIOJUB84QJxbf5n9JhDC7dr4aYzSjZs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DC3DF1516; Wed, 23 Aug 2023 06:15:08 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7BE823F740; Wed, 23 Aug 2023 06:14:22 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 03/37] arm64: mte: Rename __GFP_ZEROTAGS to __GFP_TAGGED Date: Wed, 23 Aug 2023 14:13:16 +0100 Message-Id: <20230823131350.114942-4-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 13DBE80004 X-Rspam-User: X-Stat-Signature: zrg6kxf5n3b7xrmg18nzqchbg78keoir X-Rspamd-Server: rspam01 X-HE-Tag: 1692796468-529990 X-HE-Meta: U2FsdGVkX1+cEIZUjKxHCa2W6ZK/aRfuzrXLOBheOOfN+7e4zGOqgFgq8paJl9XU9pf96r6JnQldVi0FrF0NOTrNcwakFiVRpaWb10I6BgLW1NkqHbTh1MFw1rBmf/DKApk7+tmDjxnRuR4gSWu2vVdDbVYJ7mXWv8f2pmfvfjSqLM0B07RVoGzxtrszZjy4OZh3KTHJaX89eRpPDLs43V7/yAnw7DWeNauCWUlOujYgQzJVOxAbDMigy4+Uaebn0QxQlyTlci9sjBumMlmLuDGVtYIqLt4u8TeWFSFtDOp09eGWL+3JwGd+sY8FAM1PGnuapEOCqBRcV1eku3NdzcXosqlHFrKtMIjd+gHyxm4CyjyOiirsF8Zd8BUeVuC8XclTl3pLY2pphbPBL2HYaWW9OlzhwfqhEFFmuhm9kVD964MVpuZtv1ILp5g11sFOA30Gb5wdOnvXkMxZamTvOde1OoKeJHkjPJAY/x6NHBKAqKEcKmYLzQzZeL7qMuEnaB72sSP1kekI876JjvhdttWbs3duluqlsFhYOTe73Rpj8xswqKApbac7uCAS9RsA1ALbOLKkNx2F+xI/oTpq0OwtlVQK9Cn4eZBktkjHxZKFNuIyjD9rMG3nOIs2oa+SHJgj3dN43KACNYfpC7O9Mi58TqU31JqOFH7AAoDAIiJjtc6J+XLCdcELTDMzCq6LS7IEdWUq3/vVsJ4PcwpGHa5sij/R2/TGnOr4pjIzrKRyVVu/N8t0EiTd0IMWZkojtHy6V3fNBbzgz6Oc7p7rE/IemiydjDikz73nE9B0GSd48QNGgJq36mtA3rDy76u1gmZTi5/0G+0RD5J4QZGk8sZzLpa8qH/ZjfeZ003Hmuhjd8b3sV3WUx8bjkKD4UcOxxhx+L+vFEhUt8mlHyY/tuMmGAR5OuWPP5VaMI2SvqHuGQx8QDfLit4eV/Y0TLZkxNvZfJ3f3PgMQrfdbPN XEqFUff1 ciXc1YiIWVj/5revYj/9gnt+LTGFiAyrxgG2wBqD2Q6E3fwBOZb2VhaTkw+VVCnECSG7So50S9yj5S5GQhf6incKxWSvxB8WI7Dp63oHDQj218yqR586oj+ChP8/dGJtY4lCT+K9GvGN7LCj5YNkkq0NICGWHAh2f7zwgFyYMITQtszynW1YCX6RiFWr7LQFMR1ueBgiaZSvyxO4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __GFP_ZEROTAGS is used to instruct the page allocator to zero the tags at the same time as the physical frame is zeroed. The name can be slightly misleading, because it doesn't mean that the code will zero the tags unconditionally, but that the tags will be zeroed if and only if the physical frame is also zeroed (either __GFP_ZERO is set or init_on_alloc is 1). Rename it to __GFP_TAGGED, in preparation for it to be used by the page allocator to recognize when an allocation is tagged (has metadata). Signed-off-by: Alexandru Elisei --- arch/arm64/mm/fault.c | 2 +- include/linux/gfp_types.h | 14 +++++++------- include/trace/events/mmflags.h | 2 +- mm/page_alloc.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 3fe516b32577..0ca89ebcdc63 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -949,7 +949,7 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, * separate DC ZVA and STGM. */ if (vma->vm_flags & VM_MTE) - flags |= __GFP_ZEROTAGS; + flags |= __GFP_TAGGED; return vma_alloc_folio(flags, 0, vma, vaddr, false); } diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 6583a58670c5..37b9e265d77e 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -45,7 +45,7 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_HARDWALL 0x100000u #define ___GFP_THISNODE 0x200000u #define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_TAGGED 0x800000u #ifdef CONFIG_KASAN_HW_TAGS #define ___GFP_SKIP_ZERO 0x1000000u #define ___GFP_SKIP_KASAN 0x2000000u @@ -226,11 +226,11 @@ typedef unsigned int __bitwise gfp_t; * * %__GFP_ZERO returns a zeroed page on success. * - * %__GFP_ZEROTAGS zeroes memory tags at allocation time if the memory itself - * is being zeroed (either via __GFP_ZERO or via init_on_alloc, provided that - * __GFP_SKIP_ZERO is not set). This flag is intended for optimization: setting - * memory tags at the same time as zeroing memory has minimal additional - * performace impact. + * %__GFP_TAGGED marks the allocation as having tags, which will be zeroed it + * allocation time if the memory itself is being zeroed (either via __GFP_ZERO + * or via init_on_alloc, provided that __GFP_SKIP_ZERO is not set). This flag is + * intended for optimization: setting memory tags at the same time as zeroing + * memory has minimal additional performace impact. * * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation. * Used for userspace and vmalloc pages; the latter are unpoisoned by @@ -241,7 +241,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) -#define __GFP_ZEROTAGS ((__force gfp_t)___GFP_ZEROTAGS) +#define __GFP_TAGGED ((__force gfp_t)___GFP_TAGGED) #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) #define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 1478b9dd05fa..4ccca8e73c93 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -50,7 +50,7 @@ gfpflag_string(__GFP_RECLAIM), \ gfpflag_string(__GFP_DIRECT_RECLAIM), \ gfpflag_string(__GFP_KSWAPD_RECLAIM), \ - gfpflag_string(__GFP_ZEROTAGS) + gfpflag_string(__GFP_TAGGED) #ifdef CONFIG_KASAN_HW_TAGS #define __def_gfpflag_names_kasan , \ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e6f950c54494..fdc230440a44 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1516,7 +1516,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, { bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && !should_skip_init(gfp_flags); - bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); + bool zero_tags = init && (gfp_flags & __GFP_TAGGED); int i; set_page_private(page, 0); From patchwork Wed Aug 23 13:13:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A299EEE49A0 for ; Wed, 23 Aug 2023 13:14:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35166900025; Wed, 23 Aug 2023 09:14:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 302038E0011; Wed, 23 Aug 2023 09:14:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A2A2900025; Wed, 23 Aug 2023 09:14:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 09B698E0011 for ; Wed, 23 Aug 2023 09:14:37 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D485F14010D for ; Wed, 23 Aug 2023 13:14:36 +0000 (UTC) X-FDA: 81155413752.23.FE32F45 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 1069C4001C for ; Wed, 23 Aug 2023 13:14:34 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf17.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TWPBvK+759MlmZvoutGWKwOmOF56oMp2n9QmBdgkL4c=; b=BGPnN82x5R/gK7jxSUqMO7jOQj/WCt2oGV+gbA1UJZpDNLI0Jlus5IhU3yl963NQO1hlX2 RTpwdvjztGiRDRXe7aHG4hDBFZbG26kV9t8sw6b7c2LtsuErfpEBy3q6rLJBkscF2b8ttP u0lJkobl0yI10AgGLb4M13FZrmmUKZ0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf17.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796475; a=rsa-sha256; cv=none; b=amq7nVUHKW0rDFUuYW7YhgjEO/Tqsm8ZE6ZifThIJSo93nrcG0BZIUnfa3BMxcmzyLmdMa nPdhFool3Pe7y8jsZO85pCjIk9Duaa7/5XjW7Hrxm6he890p3PjyQxPqdwkidGewWvV9EC idAqIlqwZRJKggq2ztOvjUusYVk63KM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C581B152B; Wed, 23 Aug 2023 06:15:14 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9A1373F740; Wed, 23 Aug 2023 06:14:28 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 04/37] mm: Add MIGRATE_METADATA allocation policy Date: Wed, 23 Aug 2023 14:13:17 +0100 Message-Id: <20230823131350.114942-5-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: rb3znecgwkcki7y84t9uoax8use19jme X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1069C4001C X-HE-Tag: 1692796474-307179 X-HE-Meta: U2FsdGVkX1+/UQQrqVhGgKFYcdOHOOj3DXcvFxhJS8mL0Wmxtk4FyxQRDlqykco43HVfZCYjig3kzN6iQLSu+IBB73msEWI2aUJhPOP9UcYTdQ8s4dH3Nqws74wDOmFfWQFOiANNzbyGGjCO9mByV8nVCAA0/27SvpSxhGiqwE+Q7O0jD47e6ToLkptjRKe/0qS/ALpovTz1S8onzdmWJELpyQ2huhtYqm2GbDgfJxRMjV0HMYen9q8dZDK+pK14xg0kU97XDNqPTe2ZFHsPGuYrg1sFjK0tKAAiRTLkBH5cPDb59ihN1TXab8EmPVqvH8zxGAUWZdAdpNywyxkBnb5yfIBdpDePK2UIB6rwDAoZEBFJ6gODt9k9PfqxDaE/6+v05/66C1lq0N//cPlOynTjEOFJAUrk+7vkwLFBARlIHRPQgDYjOcujphplFvC8JCr1lFECwSREVNqVoj2wnw8LzKEGw5oI2fwQr06+O7ofESu3XTudJpCYV1/j5SK6MX7LSUAtm1/PGYerOg0suwhzpzS+/p/rbEdKmNmc1dNXWbppOevzpYe44OjUDD8dlwhDdlrRtNTd74d2iPUSe7lNOMtHjjmss2fxUBPIGe66+QJoaRWQz1uotJmz1raMS58oUy57JnYa/yxnPxhw6u1BENpsjIufL0kluE5/mTsIDITRMGApqxoQI8GiPR3xxeedA5VE3G5J8HfOCrdwa00pJ2ObxBiSjA6HiOtRPPKBVXpefsanym7ACDBLIdoJc/IjOXZAsQdws4lUNLW8n50xMsTaWJbT3iXvdefz2idvb3B5HGBwAijzTxn7JfuIhggu/B5OrfrkUvpJivWmFp44WTdSY8g7xDB1qPb1TqWSquLiKL1eA7raL9EokJVzu9gtYvMAVAF9/9NtBRI4VkWA5YzS1E6qQPD1/V4nsR12RpnzKfrBBeswVU08KbjqWZeT5UicM+86F2UErE+ utT/qbVO zGVMUmBomZsoZ3YoLE1LoHC+lYmF0lMzhefwSNfNW1pofmDHbRNjtxl4oFfhtTk99g6DcHs5TPiuj8k0DzLlU1EAGgAxOb0iLGqwz+ELrqn0QqBriqGKqBMrKLE3gE4IBnmOn543YOiVbnhLMwa+Ejf8f6xl1FDLZrLA5bV4Xh6cn7MDlkxKr/zi1FkQd2oEp9jdWlUeGL1g82aQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures implement hardware memory coloring to catch incorrect usage of memory allocation. One such architecture is arm64, which calls its hardware implementation Memory Tagging Extension. So far, the memory which stores the metadata has been configured by firmware and hidden from Linux. For arm64, it is impossible to to have the entire system RAM allocated with metadata because executable memory cannot be tagged. Furthermore, in practice, only a chunk of all the memory that can have tags is actually used as tagged. which leaves a portion of metadata memory unused. As such, it would be beneficial to use this memory, which so far has been unaccessible to Linux, to service allocation requests. To prepare for exposing this metadata memory a new migratetype is being added to the page allocator, called MIGRATE_METADATA. One important aspect is that for arm64 the memory that stores metadata cannot have metadata associated with it, it can only be used to store metadata for other pages. This means that the page allocator will *not* allocate from this migratetype if at least one of the following is true: - The allocation also needs metadata to be allocated. - The allocation isn't movable. A metadata page storing data must be able to be migrated at any given time so it can be repurposed to store metadata. Both cases are specific to arm64's implementation of memory metadata. For now, metadata storage pages management is disabled, and it will be enabled once the architecture-specific handling is added. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 21 ++++++++++++++++++ arch/arm64/mm/fault.c | 3 +++ include/asm-generic/Kbuild | 1 + include/asm-generic/memory_metadata.h | 18 +++++++++++++++ include/linux/mmzone.h | 11 ++++++++++ mm/Kconfig | 3 +++ mm/internal.h | 5 +++++ mm/page_alloc.c | 28 ++++++++++++++++++++++++ 8 files changed, 90 insertions(+) create mode 100644 arch/arm64/include/asm/memory_metadata.h create mode 100644 include/asm-generic/memory_metadata.h diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h new file mode 100644 index 000000000000..5269be7f455f --- /dev/null +++ b/arch/arm64/include/asm/memory_metadata.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023 ARM Ltd. + */ +#ifndef __ASM_MEMORY_METADATA_H +#define __ASM_MEMORY_METADATA_H + +#include + +#ifdef CONFIG_MEMORY_METADATA +static inline bool metadata_storage_enabled(void) +{ + return false; +} +static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) +{ + return false; +} +#endif /* CONFIG_MEMORY_METADATA */ + +#endif /* __ASM_MEMORY_METADATA_H */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 0ca89ebcdc63..1ca421c11ebc 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -956,6 +957,8 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, void tag_clear_highpage(struct page *page) { + /* Tag storage pages cannot be tagged. */ + WARN_ON_ONCE(is_migrate_metadata_page(page)); /* Newly allocated page, shouldn't have been tagged yet */ WARN_ON_ONCE(!try_page_mte_tagging(page)); mte_zero_clear_page_tags(page_address(page)); diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild index 941be574bbe0..048ecffc430c 100644 --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -36,6 +36,7 @@ mandatory-y += kprobes.h mandatory-y += linkage.h mandatory-y += local.h mandatory-y += local64.h +mandatory-y += memory_metadata.h mandatory-y += mmiowb.h mandatory-y += mmu.h mandatory-y += mmu_context.h diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h new file mode 100644 index 000000000000..dc0c84408a8e --- /dev/null +++ b/include/asm-generic/memory_metadata.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_GENERIC_MEMORY_METADATA_H +#define __ASM_GENERIC_MEMORY_METADATA_H + +#include + +#ifndef CONFIG_MEMORY_METADATA +static inline bool metadata_storage_enabled(void) +{ + return false; +} +static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) +{ + return false; +} +#endif /* !CONFIG_MEMORY_METADATA */ + +#endif /* __ASM_GENERIC_MEMORY_METADATA_H */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5e50b78d58ea..74925806687e 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -61,6 +61,9 @@ enum migratetype { */ MIGRATE_CMA, #endif +#ifdef CONFIG_MEMORY_METADATA + MIGRATE_METADATA, +#endif #ifdef CONFIG_MEMORY_ISOLATION MIGRATE_ISOLATE, /* can't allocate from here */ #endif @@ -78,6 +81,14 @@ extern const char * const migratetype_names[MIGRATE_TYPES]; # define is_migrate_cma_page(_page) false #endif +#ifdef CONFIG_MEMORY_METADATA +# define is_migrate_metadata(migratetype) unlikely((migratetype) == MIGRATE_METADATA) +# define is_migrate_metadata_page(_page) (get_pageblock_migratetype(_page) == MIGRATE_METADATA) +#else +# define is_migrate_metadata(migratetype) false +# define is_migrate_metadata_page(_page) false +#endif + static inline bool is_migrate_movable(int mt) { return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE; diff --git a/mm/Kconfig b/mm/Kconfig index 09130434e30d..838193522e20 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1236,6 +1236,9 @@ config LOCK_MM_AND_FIND_VMA bool depends on !STACK_GROWSUP +config MEMORY_METADATA + bool + source "mm/damon/Kconfig" endmenu diff --git a/mm/internal.h b/mm/internal.h index a7d9e980429a..efd52c9f1578 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -824,6 +824,11 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_NOFRAGMENT 0x0 #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ +#ifdef CONFIG_MEMORY_METADATA +#define ALLOC_FROM_METADATA 0x400 /* allow allocations from MIGRATE_METADATA list */ +#else +#define ALLOC_FROM_METADATA 0x0 +#endif #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fdc230440a44..7baa78abf351 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -53,6 +53,7 @@ #include #include #include +#include #include "internal.h" #include "shuffle.h" #include "page_reporting.h" @@ -1645,6 +1646,17 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, unsigned int order) { return NULL; } #endif +#ifdef CONFIG_MEMORY_METADATA +static __always_inline struct page *__rmqueue_metadata_fallback(struct zone *zone, + unsigned int order) +{ + return __rmqueue_smallest(zone, order, MIGRATE_METADATA); +} +#else +static inline struct page *__rmqueue_metadata_fallback(struct zone *zone, + unsigned int order) { return NULL; } +#endif + /* * Move the free pages in a range to the freelist tail of the requested type. * Note that start_page and end_pages are not aligned on a pageblock @@ -2144,6 +2156,15 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype, if (alloc_flags & ALLOC_CMA) page = __rmqueue_cma_fallback(zone, order); + /* + * Allocate data pages from MIGRATE_METADATA only if the regular + * allocation path fails to increase the chance that the + * metadata page is available when the associated data page + * needs it. + */ + if (!page && (alloc_flags & ALLOC_FROM_METADATA)) + page = __rmqueue_metadata_fallback(zone, order); + if (!page && __rmqueue_fallback(zone, order, migratetype, alloc_flags)) goto retry; @@ -3088,6 +3109,13 @@ static inline unsigned int gfp_to_alloc_flags_fast(gfp_t gfp_mask, if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif +#ifdef CONFIG_MEMORY_METADATA + if (metadata_storage_enabled() && + gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE && + alloc_can_use_metadata_pages(gfp_mask)) + alloc_flags |= ALLOC_FROM_METADATA; +#endif + return alloc_flags; } From patchwork Wed Aug 23 13:13:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B26EBEE4993 for ; Wed, 23 Aug 2023 13:14:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46305900026; Wed, 23 Aug 2023 09:14:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43A3C8E0011; Wed, 23 Aug 2023 09:14:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DAFD900026; Wed, 23 Aug 2023 09:14:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1E19A8E0011 for ; Wed, 23 Aug 2023 09:14:44 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E455FA0110 for ; Wed, 23 Aug 2023 13:14:43 +0000 (UTC) X-FDA: 81155414046.24.9471FE6 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 0835FA0020 for ; Wed, 23 Aug 2023 13:14:41 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z3SIzDwrIxiPRKD/aZg484UuGGxwodYj8Csf5fd6iHk=; b=HEn9ZPYiZ7IVG3w5Nw6gs8gqFTNkBHQWoqvQtLUiFmNwPLEbuyXAaAsZ9ZqSOddBB2R53K K2TDLQIszPsjmoXKavVvIKoLtST9XP+xS8S6q6+2XT/MEHAZ1Y4ShCGrpDN4ajsNEplLJs R/YJ6l3LmssivH4cslbgMzAw8nATBtg= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796482; a=rsa-sha256; cv=none; b=2NZnPB79EJ9cyMW49sLd3j5ikcuL7zYpm/OizcpouVcspsVRZXqWgOqjZwkCYZSLf38woO y0KOj5y2grBWekfWa2HphoE8ckDZUdP7OoHctzR1QxUL47ozPbmbjQ1KCGxbPd6AoWpynP p8WXRugpGTmFQjBrMN4TaNKF2qf/Plc= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 85FE01570; Wed, 23 Aug 2023 06:15:21 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B54533F740; Wed, 23 Aug 2023 06:14:34 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 05/37] mm: Add memory statistics for the MIGRATE_METADATA allocation policy Date: Wed, 23 Aug 2023 14:13:18 +0100 Message-Id: <20230823131350.114942-6-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0835FA0020 X-Rspam-User: X-Stat-Signature: qr4jrzati3p4z5hjn5i45dwn5bn1e8nk X-Rspamd-Server: rspam01 X-HE-Tag: 1692796481-964533 X-HE-Meta: U2FsdGVkX1+dUBk0UZySQliIlM8Wx9VhiQ6U1QvPK8x6YoNCgxIRaH7Hioa8UcJyYfHbVWRPv8VXXyxkmP87TycgWmJEYdwPrpHuRmrrNOWASnABJlmkizEKQ+zcdM2Y/WG+0LN+uRT/6LggMhynrvCSD2k6k1nhTiAL1nrLyC9EBzD0/dQ0ogeArEFEq3mJv353Gaq/boVzVlUfYvzX17QZKNLqnG5Q8FfPvYNi29RZMeFdDkZ5RKgicu8EyNrZeR9U0mPrCyCdR5jaUMv5yI6cm87h4KlkBxIz0xO//kK8ANoThqScp168wow1buxUCsqEhnzco+JreFuHZDit8gQjbOHgp0RUi8qFSQg482VygAfvelu+LIYwbsGna1YZKKEaU6S5yQ8ozla316e9EI99B2oH2URkMr0w1C+JicEMY2vwQ18j+6Azbvod09cFoUJvzIKqAjdJEA88VRQNo9nD+7aBKOKCGwIbL8X1vN3vfSDL+iXQaUJUT0IwWH/6HF1aB9bUUoaJRMUPHhEMtQ0GgfbnWGuX8BlyUuQSqL21EqzCN8JfbHmdWe6mWfDbexhpo3as74NgxJ4vFwlSeftaHxvcqsfB4Kh7ONCWQuHb0UZVS8a6GaM1P2Ht9melYuyHaAn28T+KKTSuwBxZOuTjNHfzyQ19wcrI4pRoWBUyyvRosgtXEKP94ciZz+cUHlIJfmZTGORCCND610YXz01rjVH761BcoV7zaohvSL+gfbiPMP9o1huwBIsONS/Y8LYdyr01iwtJVBcN6fQcEaTmv+mCSHSXalJePHyjlryISxV9QF/m1hpvZpE83rmEut90gqMJZrLbCrwvXDEh1Tui5TnngWL7veW3Pe8PQhMkpJsHBXnJDXa19FYAeutDfcT7+/vK2Ubbmxo/pxK13X7V+6t6XX9H4jRHuY3rND+LtZiDJ4o8ZiBuZSGym0M5P+p3z1q72kZOq114keJ LwAyp8zQ Q2MRGEGuyXo1oHr8wEhY9/CmP9don7YEULgARGuCj7B8SbFYkYtGQ5MKVP6FKqeh79VK89a7fpoXS/wBO9xVJqgPHbhC1DX3RAjIU2SXP0DLO62CZF/l+BJHsltnxzYG2TbQg52wWPi4uV7HKbs9cxnY0nfwxTK8FhlBV3V4nz0JNNuwnSBzZh3DcYrhqIfTwatRiel3OU420fa4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Keep track of the total number of metadata pages available in the system, as well as the per-zone pages. Opportunistically add braces to an "if" block from rmqueue_bulk() where the body contains multiple lines of code. Signed-off-by: Alexandru Elisei --- fs/proc/meminfo.c | 8 ++++++++ include/asm-generic/memory_metadata.h | 2 ++ include/linux/mmzone.h | 13 +++++++++++++ include/linux/vmstat.h | 2 ++ mm/page_alloc.c | 18 +++++++++++++++++- mm/page_owner.c | 3 ++- mm/show_mem.c | 4 ++++ mm/vmstat.c | 8 ++++++-- 8 files changed, 54 insertions(+), 4 deletions(-) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 8dca4d6d96c7..c9970860b5be 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -17,6 +17,9 @@ #ifdef CONFIG_CMA #include #endif +#ifdef CONFIG_MEMORY_METADATA +#include +#endif #include #include "internal.h" @@ -167,6 +170,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) show_val_kb(m, "CmaFree: ", global_zone_page_state(NR_FREE_CMA_PAGES)); #endif +#ifdef CONFIG_MEMORY_METADATA + show_val_kb(m, "MetadataTotal: ", totalmetadata_pages); + show_val_kb(m, "MetadataFree: ", + global_zone_page_state(NR_FREE_METADATA_PAGES)); +#endif #ifdef CONFIG_UNACCEPTED_MEMORY show_val_kb(m, "Unaccepted: ", diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index dc0c84408a8e..63ea661b354d 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -4,6 +4,8 @@ #include +extern unsigned long totalmetadata_pages; + #ifndef CONFIG_MEMORY_METADATA static inline bool metadata_storage_enabled(void) { diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 74925806687e..48c237248d87 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -160,6 +160,7 @@ enum zone_stat_item { #ifdef CONFIG_UNACCEPTED_MEMORY NR_UNACCEPTED, #endif + NR_FREE_METADATA_PAGES, NR_VM_ZONE_STAT_ITEMS }; enum node_stat_item { @@ -914,6 +915,9 @@ struct zone { #ifdef CONFIG_CMA unsigned long cma_pages; #endif +#ifdef CONFIG_MEMORY_METADATA + unsigned long metadata_pages; +#endif const char *name; @@ -1026,6 +1030,15 @@ static inline unsigned long zone_cma_pages(struct zone *zone) #endif } +static inline unsigned long zone_metadata_pages(struct zone *zone) +{ +#ifdef CONFIG_MEMORY_METADATA + return zone->metadata_pages; +#else + return 0; +#endif +} + static inline unsigned long zone_end_pfn(const struct zone *zone) { return zone->zone_start_pfn + zone->spanned_pages; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index fed855bae6d8..15aa069df6b1 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -493,6 +493,8 @@ static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); if (is_migrate_cma(migratetype)) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); + if (is_migrate_metadata(migratetype)) + __mod_zone_page_state(zone, NR_FREE_METADATA_PAGES, nr_pages); } extern const char * const vmstat_text[]; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7baa78abf351..829134a4dfa8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2202,9 +2202,14 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - if (is_migrate_cma(get_pcppage_migratetype(page))) + if (is_migrate_cma(get_pcppage_migratetype(page))) { __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(1 << order)); + } + if (is_migrate_metadata(get_pcppage_migratetype(page))) { + __mod_zone_page_state(zone, NR_FREE_METADATA_PAGES, + -(1 << order)); + } } __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); @@ -2894,6 +2899,10 @@ static inline long __zone_watermark_unusable_free(struct zone *z, #ifdef CONFIG_UNACCEPTED_MEMORY unusable_free += zone_page_state(z, NR_UNACCEPTED); #endif +#ifdef CONFIG_MEMORY_METADATA + if (!(alloc_flags & ALLOC_FROM_METADATA)) + unusable_free += zone_page_state(z, NR_FREE_METADATA_PAGES); +#endif return unusable_free; } @@ -2974,6 +2983,13 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, return true; } #endif + +#ifdef CONFIG_MEMORY_METADATA + if ((alloc_flags & ALLOC_FROM_METADATA) && + !free_area_empty(area, MIGRATE_METADATA)) { + return true; + } +#endif if ((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_OOM)) && !free_area_empty(area, MIGRATE_HIGHATOMIC)) { return true; diff --git a/mm/page_owner.c b/mm/page_owner.c index c93baef0148f..c66e25536068 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -333,7 +333,8 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m, page_owner = get_page_owner(page_ext); page_mt = gfp_migratetype(page_owner->gfp_mask); if (pageblock_mt != page_mt) { - if (is_migrate_cma(pageblock_mt)) + if (is_migrate_cma(pageblock_mt) || + is_migrate_metadata(pageblock_mt)) count[MIGRATE_MOVABLE]++; else count[pageblock_mt]++; diff --git a/mm/show_mem.c b/mm/show_mem.c index 01f8e9905817..3935410c98ac 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -22,6 +22,7 @@ atomic_long_t _totalram_pages __read_mostly; EXPORT_SYMBOL(_totalram_pages); unsigned long totalreserve_pages __read_mostly; unsigned long totalcma_pages __read_mostly; +unsigned long totalmetadata_pages __read_mostly; static inline void show_node(struct zone *zone) { @@ -423,6 +424,9 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_CMA printk("%lu pages cma reserved\n", totalcma_pages); #endif +#ifdef CONFIG_MEMORY_METADATA + printk("%lu pages metadata reserved\n", totalmetadata_pages); +#endif #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif diff --git a/mm/vmstat.c b/mm/vmstat.c index b731d57996c5..07caa284a724 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1184,6 +1184,7 @@ const char * const vmstat_text[] = { #ifdef CONFIG_UNACCEPTED_MEMORY "nr_unaccepted", #endif + "nr_free_metadata", /* enum numa_stat_item counters */ #ifdef CONFIG_NUMA @@ -1695,7 +1696,8 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, "\n spanned %lu" "\n present %lu" "\n managed %lu" - "\n cma %lu", + "\n cma %lu" + "\n metadata %lu", zone_page_state(zone, NR_FREE_PAGES), zone->watermark_boost, min_wmark_pages(zone), @@ -1704,7 +1706,8 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, zone->spanned_pages, zone->present_pages, zone_managed_pages(zone), - zone_cma_pages(zone)); + zone_cma_pages(zone), + zone_metadata_pages(zone)); seq_printf(m, "\n protection: (%ld", @@ -1909,6 +1912,7 @@ int vmstat_refresh(struct ctl_table *table, int write, switch (i) { case NR_ZONE_WRITE_PENDING: case NR_FREE_CMA_PAGES: + case NR_FREE_METADATA_PAGES: continue; } val = atomic_long_read(&vm_zone_stat[i]); From patchwork Wed Aug 23 13:13:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B591EE49B0 for ; Wed, 23 Aug 2023 13:14:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1559D900027; Wed, 23 Aug 2023 09:14:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 106898E0011; Wed, 23 Aug 2023 09:14:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE914900027; Wed, 23 Aug 2023 09:14:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DD4D18E0011 for ; Wed, 23 Aug 2023 09:14:49 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A43691C822C for ; Wed, 23 Aug 2023 13:14:49 +0000 (UTC) X-FDA: 81155414298.22.7801242 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id CE71F100022 for ; Wed, 23 Aug 2023 13:14:47 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NyZXYewavhBbZmGyqV77AChMJ6F3CygpLyxWgKDcDkI=; b=NVL+ZNP4FY6tJ2ZfcQEMHEq9iIkihBMltC6Mr5VUYcifOzpabMkcJBDWF7oVvh7RVMM9Ru GR+NwjugBQzJ/cnPf7g94tGLOkGOt6TjRJm0ofNDhh0z7KrLHg+zISMqZX7td3C+uVT+hK zMOfVml7NFamqd9fhzK71xO908zH2jI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796488; a=rsa-sha256; cv=none; b=Q7SeLG+P9mtBL2k+Zjh7bRR192O3kL+mncgysv+/EnayzjDfK7Z9nGR4G0M8wWursdJkcm 3kJVfrqbcTPIyrvZPtXRYjyLX8Av9yz/0fSyOO86b1oz+gBGz15glOcAU/lpgrEzsKnpvf QKRh048sHEmeYYUt4v2PXvozHHI6OKU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 919B61576; Wed, 23 Aug 2023 06:15:27 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 415F23F740; Wed, 23 Aug 2023 06:14:41 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 06/37] mm: page_alloc: Allocate from movable pcp lists only if ALLOC_FROM_METADATA Date: Wed, 23 Aug 2023 14:13:19 +0100 Message-Id: <20230823131350.114942-7-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CE71F100022 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: gwb17wbhhztdw4x7mjw8zhf4fu6yuskp X-HE-Tag: 1692796487-363157 X-HE-Meta: U2FsdGVkX18IabF+8dHytiO3IIiGpKfY+0SPw4Q5eH/J7s1BiqVCs0iUAhwoTdsdRd/e1LEivRRB/0JLAbkz8jXJmHEe0CzNeqO7bb1ro4K6wSzj6S16lVu/FNlfOVPKqWbtlw0lYzjUo7k+Gfo3hcegCMjOly1Vr4KxMQDDdb/DEQ9X5WhKk05MFLWYZDaB1O5UxxFsdrowg58eXlMJ7lSgHkgb0wqFCMQ1k2EzaTW/Z/k6nqvCvsZs2LbH+NUyggWYIdipKkPVBoL4naltJO6829nWkPi3XiIZkRdFPLCerQAjqHDreo+nmrWrcBuH9f809XEXpu9glpVSvwPxp2rVH3YU2c905Gl5mv2cjC2VjGMisG8sZYniVCtliq9a2Wni8BOaUh9fhKq2mBzHBNtoC3dXBnYVFmkR8sf1+2jz1/46dzmwGEWfrjTiAC6tWyd8vr9WTFtm5QaqKG+OIWweJn9p747XHZI5IRecwSpWjohYfh49iWgHWnSHghB0haN4cXXR6RKBNhgryDYxl45V2rDERJS6w9TIPZUV2zxbRGAB6SzERhWtl7b8aOmcuzsdWVk9MYCqYScTeku4xnFqqiiVEQvk+mE6OfE357NjGo0zEM6zWGuLVaZCZshic89Azs1iGgaLE2SkIyMwadQ0NTDguXATv9y5vH5VsAFxmjVQpJhBzG8sqDiyMCRqIy4epieWwwx9a9bjyYhgrC+MxznzKKVIEtcQTSZXtzqEyM9Ur2Cee8yCM+x3UPGdbWLnEO1GI7fStFaC8jKfyZZD2fxIIks7ZtQAOkQ8qH4vwWscaYa9C2Su8vpMY9X+9KdIzmMdO8kbSqU3j8xlZ36z4tvRMMI1I4uVuUo6WWgx7LD+ZonJPZt0W42KkoCHGta9L/9qhoaddNTlRkIdZ4++04p59cdDvCc1obPukv4OxwkzFcZ/VGlrXJ8vO3hMUFO+1OUSYJ0kwEw9TgP 4lSdaOJR s0UeGTt8b8ZELpE8OZ3ko1FbNe6xOf7gIKVHwuBhzY6UA2ys1IGzekk+dHAMnyR9gYH1wm9R8ytQrJfTVYkDKqnkYGkkFz7MdyOGDe7Uu5eTz5f65bOHAA9aXoo+PyBOEd/3FkMhhvqFA52BEkg6G3jzW0ZAjy7soIaxalLwQsL3RSAz4hhS7SDKw+u7ZtFlLFXA6fm7T6EUQGUE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pcp lists keep MIGRATE_METADATA pages on the MIGRATE_MOVABLE list. Make sure pages from the movable list are allocated only when the ALLOC_FROM_METADATA alloc flag is set, as otherwise the page allocator could end up allocating a metadata page when that page cannot be used. __alloc_pages_bulk() sidesteps rmqueue() and calls __rmqueue_pcplist() directly. Add a check for the flag before calling __rmqueue_pcplist(), and fallback to __alloc_pages() if the check is false. Note that CMA isn't a problem for __alloc_pages_bulk(): an allocation can always use CMA pages if the requested migratetype is MIGRATE_MOVABLE, which is not the case with MIGRATE_METADATA pages. Signed-off-by: Alexandru Elisei --- mm/page_alloc.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 829134a4dfa8..a693e23c4733 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2845,11 +2845,16 @@ struct page *rmqueue(struct zone *preferred_zone, if (likely(pcp_allowed_order(order))) { /* - * MIGRATE_MOVABLE pcplist could have the pages on CMA area and - * we need to skip it when CMA area isn't allowed. + * PCP lists keep MIGRATE_CMA/MIGRATE_METADATA pages on the same + * movable list. Make sure it's allowed to allocate both type of + * pages before allocating from the movable list. */ - if (!IS_ENABLED(CONFIG_CMA) || alloc_flags & ALLOC_CMA || - migratetype != MIGRATE_MOVABLE) { + bool movable_allowed = (!IS_ENABLED(CONFIG_CMA) || + (alloc_flags & ALLOC_CMA)) && + (!IS_ENABLED(CONFIG_MEMORY_METADATA) || + (alloc_flags & ALLOC_FROM_METADATA)); + + if (migratetype != MIGRATE_MOVABLE || movable_allowed) { page = rmqueue_pcplist(preferred_zone, zone, order, migratetype, alloc_flags); if (likely(page)) @@ -4388,6 +4393,14 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; gfp = alloc_gfp; + /* + * pcp lists puts MIGRATE_METADATA on the MIGRATE_MOVABLE list, don't + * use pcp if allocating metadata pages is not allowed. + */ + if (metadata_storage_enabled() && ac.migratetype == MIGRATE_MOVABLE && + !(alloc_flags & ALLOC_FROM_METADATA)) + goto failed; + /* Find an allowed local zone that meets the low watermark. */ for_each_zone_zonelist_nodemask(zone, z, ac.zonelist, ac.highest_zoneidx, ac.nodemask) { unsigned long mark; From patchwork Wed Aug 23 13:13:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 807EEEE49A0 for ; Wed, 23 Aug 2023 13:14:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A3E3900028; Wed, 23 Aug 2023 09:14:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 155178E0011; Wed, 23 Aug 2023 09:14:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01C31900028; Wed, 23 Aug 2023 09:14:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E77D58E0011 for ; Wed, 23 Aug 2023 09:14:55 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BD15A120118 for ; Wed, 23 Aug 2023 13:14:55 +0000 (UTC) X-FDA: 81155414550.08.1BF7EEF Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 1A825A001F for ; Wed, 23 Aug 2023 13:14:53 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796494; a=rsa-sha256; cv=none; b=uobyihivSOPoV4Q3zuc/b0C6LEHSSZFvwPd4VDvQG9JTz9TMcEBpoEI0rHk9ONQbcsYmtg D58RACisp1AHJoAXUqQSu8AHzILB2629vdvJKrWqh+cav+H8CGfuNXP2Dc7jYy+gVL7JLw Jc2p76aK/2g+HdKNooptVXvab9rJJZY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A51Ox/2QEasSjpYzQHkVp8hfIbl3ucpbfQjyxPrD1/A=; b=0FsTLFop7hCC4Sra3cDbNXKNctBsZuOr7L/UVewty4ziN2nBTDtKVD2XHHwazx8q8b1Xk0 GXXtX9sDhmapFrhT26tDg3ksJ6MvANiaHMf9VdOj5xrqY8dsiDe0s+HYE31K6Fs9+cvcFy 7qwZ6SrJMebklb+ySB5L6gh5UsurUJM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B6AC91595; Wed, 23 Aug 2023 06:15:33 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 51AF33F740; Wed, 23 Aug 2023 06:14:47 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 07/37] mm: page_alloc: Bypass pcp when freeing MIGRATE_METADATA pages Date: Wed, 23 Aug 2023 14:13:20 +0100 Message-Id: <20230823131350.114942-8-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 1A825A001F X-Stat-Signature: um7fmys5fda4wr5oxebu9y5gc3hm4k8j X-Rspam-User: X-HE-Tag: 1692796493-414253 X-HE-Meta: U2FsdGVkX1/pQjLd6HRwtqE2i9Wj9hShhuU1E6nBb5bhOS+NimfpaKZXnvRxBMRyX6aOyntcuDcLzJx/UW8+rIwxF5ciWSNDqxyfTCIteJ/J02dzcWHHUgCvjyZZdSd4o/1T1AYDQqAczIVwqn04bDiFaFDaI1YgIXcUnhk6iRRQlySbk4/GSshkFp/Wlws1QLY0xLkiNlGDylyjaRStMYJJZn3pfWe25qTKJDjrA4W0X4+Jt4gdGP3GUcwbzorNPfxaSWN/Qi5nl+eToKbcDweOadKheSLdVh1Iue/G4iuzaiytVoDQr2GE3s44RdN2sTpaNaBfT0vrYM9vk8TT7y//mpWPXwzuuD6T1ZYbC5sbUvtz7iflmVQti33+D3J0Ny0AWpbRTmztTse8pR6Nfplt0D+7zVXaA4+93riieRFtHSypH+lYpFMs/xmg2fsco/kVXoJEsSdaQvnXJ5WyFOPMP9HqJOBODglysvjN2KDgj+kWAB+vooOd4/15YDa56zh0MahwL4e20SABqzFw/xtBV27OQ7rCgisYfp1xhuj1fY7nO4csbtGFI512IZWWFEsQVsWj6URRzkV+0/qwJlnnD2xRem0nc/ggD3KOHavCDMhrhUw/pvZnRpkhRdNQwiGF1USI5A/ktZ5HLMLWZRyRjOAv76hNB7LFXWYySCeJi7AUYTq2M0jB4KizXQ9ZTIVeril1T8VVfVCTidhYEVseRUb6wCTJwV0C+Dh94ZhXZw1CT5owtjMzXzeyAykuX5U25wLEFvBx5jUVuJJwdBN/aKP+gkm+g6iF6bDnx/IHCxAhTvQzHtvcnLGzxD3mJSHpFAaPMvAoOk/5WdpB8rhbWO2ymQVuRDeKRu0baXADT8Wghtogtnm9nvKRnX7VijVzrLJL6Mq7E9pbe3rMP8T5nV0/MCUOU0i6WpLlxJEbYY1qcYzIPvh1xngPGVmi0xjdzr04AP3HiZG4tPp 9Q57tPzO i2t/EACZOQNuMJk5jAvbVwSowJX44N2BIlUp9+7Xq/yiUT45WJJCm11D1SVYLGmoOI/C8ll9V7HpSQoMDfxoRIMiE0UJVaTav4cPdRcXUAcRwOy4mj8wXNwYmckUPA4CDzkKVUU0n9XDNEuU5N8ra6jjIjuaKqYLRXR61/8lnBJRIBbUQvHzv946Z+QtlvaQcp7bW+RPvVR85/ds= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a metadata page is returned to the page allocator because all the associated pages with metadata were freed, the page will be returned to the pcp list, which makes it very likely that it will be used to satisfy an allocation request. This is not optimal, because metadata pages should be used as a last resort, to increase the chances they are not in use when they are needed, to avoid costly page migration. Bypass the pcp lists when freeing metadata pages. Note that metadata pages can still end up on the pcp lists when a list is refilled, but this should only happen when memory is running low, which is as intended Signed-off-by: Alexandru Elisei --- mm/page_alloc.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a693e23c4733..bbb49b489230 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2478,7 +2478,8 @@ void free_unref_page(struct page *page, unsigned int order) */ migratetype = get_pcppage_migratetype(page); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { - if (unlikely(is_migrate_isolate(migratetype))) { + if (unlikely(is_migrate_isolate(migratetype) || + is_migrate_metadata(migratetype))) { free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); return; } @@ -2522,7 +2523,8 @@ void free_unref_page_list(struct list_head *list) * comment in free_unref_page. */ migratetype = get_pcppage_migratetype(page); - if (unlikely(is_migrate_isolate(migratetype))) { + if (unlikely(is_migrate_isolate(migratetype) || + is_migrate_metadata(migratetype))) { list_del(&page->lru); free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE); continue; From patchwork Wed Aug 23 13:13:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79AE2EE4993 for ; Wed, 23 Aug 2023 13:15:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E07D900029; Wed, 23 Aug 2023 09:15:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 091248E0011; Wed, 23 Aug 2023 09:15:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E73BD900029; Wed, 23 Aug 2023 09:15:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D8D1D8E0011 for ; Wed, 23 Aug 2023 09:15:01 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A1612A00F1 for ; Wed, 23 Aug 2023 13:15:01 +0000 (UTC) X-FDA: 81155414802.27.0056AA3 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf23.hostedemail.com (Postfix) with ESMTP id DC07E14001B for ; Wed, 23 Aug 2023 13:14:59 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796500; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jgOGMY3Qi84t0cJR/fEb5u08vn2CWmT1JaB2CQA9BPk=; b=vOQEm6YkaUsh0DGlgH3r+DPNNU1OcL7vPa7ivGUybdeoMy/ydgYvNdxjOoOtUVTSudew02 liiqh64TidCcipptzlchNTr/IXDky8jTdsGUF8UMT2ImgqQeKRgu4UtVvzJrugrGfDtB6r sTIzNbzfXoivCfynVeODrwT5qMwlltc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796500; a=rsa-sha256; cv=none; b=3CMnP1x5IiAGYvui1GO6KurL8GnoC6dCLKJw8JTnAulBo10DtkvY01tyxiiKYdAAf4LzX6 QyU5LibYLzySnLX6PVtOVq0FGcY03mBhmqsvX0xtGF2R3Z/RdbyMrbeCE/q1XoufYhxJ2i Gviv4YbeOBwUR+5CkslEl6z5ECCR/Pg= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AB0511596; Wed, 23 Aug 2023 06:15:39 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 756C53F740; Wed, 23 Aug 2023 06:14:53 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 08/37] mm: compaction: Account for free metadata pages in __compact_finished() Date: Wed, 23 Aug 2023 14:13:21 +0100 Message-Id: <20230823131350.114942-9-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: ttjr83a6p1pc4de5snhk8bmejsug6pqs X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DC07E14001B X-HE-Tag: 1692796499-68681 X-HE-Meta: U2FsdGVkX1/hgK7C0WbObYqiBHI8Jbwl1H0kq56fl2Kiqu2XkIbm0q4xUokjbejoIboNf20mo2xLGRn9CJC2gSYU/pu8tU7ED2eNrpxkA3v3W+6hNPKkBhKAO1Xy+OlkKj2k7LEyOJKAT2ZdDDq7MmaxCb+pvizqLR7LqAO7f3lxjKM94q3RH3Ax6LIJKHm0pJ/rUTAKqfPF14BM+XAZQZGVXJJnB0vwF3/cMNY3oQ7Wm/r4xRzA4Audx+vwDaW/2xLhYJdydXgJePpUQ0NLjr4xx20WjdDMYSg2JZwyn5pjtA6mJS2NDPBFOuby/biGpFLqDuGHdc5BW3I0tu4B79UEpMqfVbr/lXpH9dbvTRnZdRkLHN7xZsY88+x4aALvf9a/xZ9zX93aP57XRuiCQ0yUJpsi++foaQNEx4U3FujZxLLTkihtWdffMZAN9m3tGXWRT1dUwPQOY6Bxrq3REhjHWklIbfCk+UwGgzfqWOLFO9kgmJp+5Eyn0KFl0CwqJljDtHPC9MYtSUHiyYE3Nj7hFYS5d0Zi7J9H89DSitAr/sElfVPDipi/A6v7fzkWNovXb7Zl8YkGzr5l6a6RbiKE96lvGKncQGIHPFWlXKqg7jmkfgyXY7SdgYw5jRh8yT0EuexkH17AoRcuvLA8Faz5+gs/pRmEkONBfW+gBI5fWITDWLWzFw9u1Ou5CspKbOe2wGUb3DAcVFR0P9d8pwY4Ymx/Oot4MPMUU9NcIWk7nBXgjAuJgsWLAW23j6Pkmga/SnodTPro/J/wn9VAWEkNGhOeUxKgkcQqXcJcMMM+N/+fvOsYct3ix6cPX+D7s/caFICVMgJ9CdY/3+Y/51GhD906uCHrCPpjgNtjRU7oNrjU7ZueExKqr0SsgZYxDBDxlQdTwgGebccjfBW0KaWYFzKFqNaPc7fui3piqy8+l16guOEwB5ZPUem1GAUrusyXeKSECJcceVIQhD+ r33SfEKB NX5uAtkk5RH1N+JFQK1f42NncSPwx5vfYcbbTYjMW1iLXcUTEgTnuTj0VrT7GMy/YTgBJUl18mfkArk4xT7Z8mlA94zaj+5mrYo1DKtRYfotH4eR3+fvCzFA8wga2IQSBIG6R+ngkmGlr4BtCGEN3tg5GIa6uRg+nXIA6I+dqDZnk1Fo+cwqtaQkdDliEaqlJw9hxRFeuzRaHqDc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __compact_finished() signals the end of compaction if a page of an order greater than or equal to the requested order if found on a free_area. When allocation of MIGRATE_METADATA pages is allowed, count the number of free metadata storage pages towards the request order. Signed-off-by: Alexandru Elisei --- mm/compaction.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index dbc9f86b1934..f132c02b0655 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2208,6 +2208,13 @@ static enum compact_result __compact_finished(struct compact_control *cc) if (migratetype == MIGRATE_MOVABLE && !free_area_empty(area, MIGRATE_CMA)) return COMPACT_SUCCESS; +#endif +#ifdef CONFIG_MEMORY_METADATA + if (metadata_storage_enabled() && + migratetype == MIGRATE_MOVABLE && + (cc->alloc_flags & ALLOC_FROM_METADATA) && + !free_area_empty(area, MIGRATE_METADATA)) + return COMPACT_SUCCESS; #endif /* * Job done if allocation would steal freepages from From patchwork Wed Aug 23 13:13:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47B4EEE4993 for ; Wed, 23 Aug 2023 13:15:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D15DD90002A; Wed, 23 Aug 2023 09:15:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC5BA8E0011; Wed, 23 Aug 2023 09:15:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8DF190002A; Wed, 23 Aug 2023 09:15:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AA3028E0011 for ; Wed, 23 Aug 2023 09:15:07 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 86E398010A for ; Wed, 23 Aug 2023 13:15:07 +0000 (UTC) X-FDA: 81155415054.20.7AF1478 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf19.hostedemail.com (Postfix) with ESMTP id D2AF61A0031 for ; Wed, 23 Aug 2023 13:15:05 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf19.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796505; a=rsa-sha256; cv=none; b=ArmtBWETXxJ3KLz0mNwMz4/anxO9JnspGByFwD27tup9D4fSR/DD1wXcaw6R3VqOAhPfE8 tnVWgWa6QlmrH+kMhxuF8sS61eOzCIYE6Ge33Bva+rZpPbZSFPetfLgcdGN6hy9SELI3/m gcSfzmGzZ2lv6JViNloFoUhiF8TKCdA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf19.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oKha7bxlj9+z1RppzJbqpI8dnCzCeCHgDBljczHqGYk=; b=Fu61jYwgohSXZqGhh5sKbA3aXEEjEYDC/LDpYECGtEnvBbYbT0bnbPDBxlmw7ZjeQkSJC/ TVmd4MfHukrOiUT+lnf352f2/sy1/S2VN78a/B6Sg2BUDEG+4FopISEyv57z38ompB4r0Q tCxAV5Q1e4SJjPhfVScN97J8aEZF2+c= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A7E7415A1; Wed, 23 Aug 2023 06:15:45 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 647253F740; Wed, 23 Aug 2023 06:14:59 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 09/37] mm: compaction: Handle metadata pages as source for direct compaction Date: Wed, 23 Aug 2023 14:13:22 +0100 Message-Id: <20230823131350.114942-10-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D2AF61A0031 X-Stat-Signature: f577mznsnyexn979ghyptafxzj6qpifh X-Rspam-User: X-HE-Tag: 1692796505-702072 X-HE-Meta: U2FsdGVkX1+ex82C7fQ6k2Sg9jia7KD528HkIzf2NZ112xU/Aly8cldq96GV17R5kLya06z3rCR8MjZPk6HXTE9dZR1qjrq+h8FEjNduueV0Y9Le51MfRNrLFbUZ8v29p2jlJGMmUBz6FmQgoM6R6nx1sQ90YU3qJSvy0i7wnzbTYXC4DFzgEa6/of7/hX0UR3zcndITvH9MomqwXMozmDh1sZosF5nG8v1bd/n+XOwNY7nZ3eNwWHNMYkFElfrx0YEDKNTG4DNaMQwrwQjmMoKJYwCN9CoDEWdcGIBHh9btqt/auO7pSSrlHMqKRfSvOf4v7tmxaxG4IBVZlxQ92VwegAVpaZP5zf/R7nBGuS3E9KIa5E/87yjYBcKGChlbuywiJm2n10l2qCcnf/HWmmvz/t68Zak40RMynqSZo7Q8Inek3BoSPYI+Cu1AD4py8Q7H/PaIaQXYl0gX7xTNMmawOl9qeqx0G6boPUSAQZEi6j/sloFblmhfazUakjL7mafOJvCoGAn/iJUe96xeQErz8YKRm+vp9DBYxC8P3kWXqEx3kdCq0iSBBZfAZQoHMVdxS4FNMNBxXqUolNxBPvc0lzPwspGosZWiWBsYYYJlYF5JWwCokNX0w1pqLoQ9adzc2SeX7vMdyqMj5x1rYkbyz7VQ4avXdRUEVhg8VXq9e6h/KgAY5Z915+FLzzhvmrhbxAsz5fywvHnDPpNOI0bZtOVwtIQUHBhTN0eq3MbigTnABCyO7/u8GEJRCq1h6qqntTInbRnU9IItyOIeFbFIBtkxnD9MMCvduHOg7T4O287xuQpi5Lhzx1XiH2kupOEQj5t3qu5mEgvtnA5YkAacvqDFbZtcL9Rw6QFZl8GHBEeFLBlvuqrkxUju1wQJOa83emBVu8o23U23easrMMYXnrhrnHXgne/ogii5kqf7sIiCcgyy6JbOxD3DISpIBHFdPxl2J8Tdd4p1aoh 1Zv0aBzv dXRtbdizD6n/UhAK4KzC3wkqKp3VL/9bJ0AEmmmcMg3pSKGXSamWIBvsDhxJEr3MNBnYMFprWOTZn8RglmDvkgYjeUkFtywsGfV9g+8n70SpoZjiJNchId9eSoNU1Q6NAU0j1skMW0GF7m1qykqr2nBzycgIlrCj50yUDfrmCtkYAb85sJFQPW1zq+x3HVfqQbcQx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Metadata pages can have special requirements and can only be allocated if an architecture allows it. In the direct compaction case, the source pages that will be migrated will then be used to satisfy the allocation request that triggered the compaction. Make sure that the allocation allows the use of metadata pages when considering them for migration. When a page is freed during direct compaction, the page allocator will try to use that page to satisfy the allocation request. Don't capture a metadata page in this case, even if the allocation request would allow it, to increase the chances that the page is free when it needs to be taken from the allocator to store metadata. Signed-off-by: Alexandru Elisei --- mm/compaction.c | 10 ++++++++-- mm/page_alloc.c | 1 + 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index f132c02b0655..a29db409c5cc 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "internal.h" #ifdef CONFIG_COMPACTION @@ -1307,11 +1308,16 @@ static bool suitable_migration_source(struct compact_control *cc, if (pageblock_skip_persistent(page)) return false; + block_mt = get_pageblock_migratetype(page); + + if (metadata_storage_enabled() && cc->direct_compaction && + is_migrate_metadata(block_mt) && + !(cc->alloc_flags & ALLOC_FROM_METADATA)) + return false; + if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction) return true; - block_mt = get_pageblock_migratetype(page); - if (cc->migratetype == MIGRATE_MOVABLE) return is_migrate_movable(block_mt); else diff --git a/mm/page_alloc.c b/mm/page_alloc.c index bbb49b489230..011645d07ce9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -654,6 +654,7 @@ compaction_capture(struct capture_control *capc, struct page *page, /* Do not accidentally pollute CMA or isolated regions*/ if (is_migrate_cma(migratetype) || + is_migrate_metadata(migratetype) || is_migrate_isolate(migratetype)) return false; From patchwork Wed Aug 23 13:13:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C605EE4993 for ; Wed, 23 Aug 2023 13:15:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A086E90002B; Wed, 23 Aug 2023 09:15:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B8DB8E0011; Wed, 23 Aug 2023 09:15:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8809590002B; Wed, 23 Aug 2023 09:15:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 775D98E0011 for ; Wed, 23 Aug 2023 09:15:19 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4138E80114 for ; Wed, 23 Aug 2023 13:15:19 +0000 (UTC) X-FDA: 81155415558.14.87F1108 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 81BEC1C002B for ; Wed, 23 Aug 2023 13:15:17 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FxMdaHr1rl+oIUwSSlfEqiIEz4TCyf52oqI+OCU1h+k=; b=cHuQMu5yxtLDPJEK4DOkptQLxnHbzeRT4LzmyIH5WdibEHyM+jUQBeC6yd6EOPI611HItE pahLNagDgpXdp76DlnIAAnx7nQ726cDBh8Xou1WjfAFF/smSn3isXIMrTcYJaycsffGgeu 6hgw5tn90dXkMZX3CtvjzVrrVDGtJm8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796517; a=rsa-sha256; cv=none; b=KkYGi3eUcMSb82YKSy9nyGhmWgVyjv63t6xaxWo2iXu5xYLsa8ynRk6ATnR+gxFV2yv6LQ ECkycW1Ruh4KWXcxt0ZoZabjgAtH46gzDeEP5TRWF+EPIyNvNxbtOCSmouHDkyMW0bMv1m LuSBmtb34beZCreU+uqF/5984fUsPzs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 72F7D15BF; Wed, 23 Aug 2023 06:15:52 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 69A753F740; Wed, 23 Aug 2023 06:15:05 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 10/37] mm: compaction: Do not use MIGRATE_METADATA to replace pages with metadata Date: Wed, 23 Aug 2023 14:13:23 +0100 Message-Id: <20230823131350.114942-11-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 81BEC1C002B X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: r9dcco87h4ydcs3o5sfesj8zm66suo3e X-HE-Tag: 1692796517-321475 X-HE-Meta: U2FsdGVkX1/STRGfXs4T29bbu82wKd9urizQLoYQqUb489LZEElXoAF6aKWncPIXF2gFpGvreqUvxMej9WTFCFpnRS/qmovL8ttKvTu4wnn6t21mdr2MZag0b43e18XUjhSF0hGUZxAnxdunkdDw2twJ7YaPoMZeRih5nGbC5hWTR/AMqxJNumnDQAv+D6EGHA7GG31O8AVRquaVoAyVSbvwJS4yY5Zs62VDcfzEfTZoOXzvG2K9oadEy3+qAg6i9XjyhXxX57OpGcvVStFxz8Pc8kDLv2Qrjqu0CxxehWYSwLr1FsawiyId3HmGW+Yz2nViMJqkLzw9WIl5P9ctxd2bbt8lC2888/MkYYWXlhQeCOmjwBqZxAwaoGXedclfIgHFbmF+FwBIc0XBeiWYXnC5dqfx7zzIIreBOWkhLBO43Cu2sJPE0DHlu3GBSe1Sxb53UDI1zL8o2YatFwkZbZLZMrkhKjeUrPpa+9nCc1uhtrTEXdS7140n/vXKQf+huE7WBPtJlPUaVqd6JhHbfTIZrM5RPwqNKRSodRg4vfeka4xL04dhVLMOFBreR66LX/tTG7s/MB1IX7M81ZC7Vmee8OdMRoXxBQ+ZiEAugylfgOf1AEKBaQkU1PlQ53y1Fi5rrMPPvFQibiZjpqA2IxB5ZgEn8Vf545rm7UVok4Z/wysFDKuN0xv6dnCcfgwOZGCfrtXk7HeTBnkRnvh5YTZOqxDoN8SyPRIJhC/95U3hqztUnhofnDxNEr1qsZEhHAlMN2Ef7cvx0G4OschQ0sT0zhs3XAz+fzXYAu+llDUBhl5oHRdwH0JpP8Wf9Fc6ctoaKEtPtZlxq6EP6Z6n5ZWRmzYR5Hh0xIfuBexQ+OmDWd5h666g0qE1S1BOxwuiNvnmtXPRcPqVOo6VPnuArajfLI0JPPSo6MlxMb7PuiyDEf0wRGe6mEgmrz19VAteQG+doDJPF41l/l1SwpL c1PoprGP 8NCEZ3pVdCaqlgPjkfdBjdtOsZVPyOUDjBsD/FvrMNqnWr3eNeYNTGrdyJ+1xsqIc7P1FVZAAK/bWPFRHrnMCBwaB/AyONdCY7Tj/Tbn51R49wWdc0SVA0VZA4jDdDKDsvk7Or2XqeoE5lNk97CUPwZTVHudImxXmwZxVpObpdCwpq4YRDcB4SczmtGzhCuB96GqBjhxhbJuCAgIgTK1ML1y21c0jbj+PxRqGVPkqFXLoeJALVuhP3vsxYw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: MIGRATE_METADATA pages are special because for the one architecture (arm64) that use them, it is not possible to have metadata associated with a page used to store metadata. To avoid a situation where a page with metadata is being migrated to a page which cannot have metadata, keep track of whether such pages have been isolated as the source for migration. When allocating a destination page for migration, deny allocations from MIGRATE_METADATA if that's the case. fast_isolate_freepages() takes pages only from the MIGRATE_MOVABLE list, which means it is not necessary to have a similar check, as MIGRATE_METADATA pages will never be considered. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 5 +++++ include/asm-generic/memory_metadata.h | 5 +++++ include/linux/mmzone.h | 2 +- mm/compaction.c | 19 +++++++++++++++++-- mm/internal.h | 1 + 5 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index 5269be7f455f..c57c435c8ba3 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -7,6 +7,8 @@ #include +#include + #ifdef CONFIG_MEMORY_METADATA static inline bool metadata_storage_enabled(void) { @@ -16,6 +18,9 @@ static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) { return false; } + +#define page_has_metadata(page) page_mte_tagged(page) + #endif /* CONFIG_MEMORY_METADATA */ #endif /* __ASM_MEMORY_METADATA_H */ diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index 63ea661b354d..02b279823920 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -3,6 +3,7 @@ #define __ASM_GENERIC_MEMORY_METADATA_H #include +#include extern unsigned long totalmetadata_pages; @@ -15,6 +16,10 @@ static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) { return false; } +static inline bool page_has_metadata(struct page *page) +{ + return false; +} #endif /* !CONFIG_MEMORY_METADATA */ #endif /* __ASM_GENERIC_MEMORY_METADATA_H */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 48c237248d87..12d5072668ab 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -91,7 +91,7 @@ extern const char * const migratetype_names[MIGRATE_TYPES]; static inline bool is_migrate_movable(int mt) { - return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE; + return is_migrate_cma(mt) || is_migrate_metadata(mt) || mt == MIGRATE_MOVABLE; } /* diff --git a/mm/compaction.c b/mm/compaction.c index a29db409c5cc..cc0139fa0cb0 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1153,6 +1153,9 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, nr_isolated += folio_nr_pages(folio); nr_scanned += folio_nr_pages(folio) - 1; + if (page_has_metadata(&folio->page)) + cc->source_has_metadata = true; + /* * Avoid isolating too much unless this block is being * fully scanned (e.g. dirty/writeback pages, parallel allocation) @@ -1328,6 +1331,15 @@ static bool suitable_migration_source(struct compact_control *cc, static bool suitable_migration_target(struct compact_control *cc, struct page *page) { + int block_mt; + + block_mt = get_pageblock_migratetype(page); + + /* Pages from MIGRATE_METADATA cannot have metadata. */ + if (is_migrate_metadata(block_mt) && cc->source_has_metadata) + return false; + + /* If the page is a large free page, then disallow migration */ if (PageBuddy(page)) { /* @@ -1342,8 +1354,11 @@ static bool suitable_migration_target(struct compact_control *cc, if (cc->ignore_block_suitable) return true; - /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */ - if (is_migrate_movable(get_pageblock_migratetype(page))) + /* + * If the block is MIGRATE_MOVABLE, MIGRATE_CMA or MIGRATE_METADATA, + * allow migration. + */ + if (is_migrate_movable(block_mt)) return true; /* Otherwise skip the block */ diff --git a/mm/internal.h b/mm/internal.h index efd52c9f1578..d28ac0085f61 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -491,6 +491,7 @@ struct compact_control { * ensure forward progress. */ bool alloc_contig; /* alloc_contig_range allocation */ + bool source_has_metadata; /* source pages have associated metadata */ }; /* From patchwork Wed Aug 23 13:13:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C186EE49A0 for ; Wed, 23 Aug 2023 13:15:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 135838E0011; Wed, 23 Aug 2023 09:15:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 043D190002C; Wed, 23 Aug 2023 09:15:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8A9F8E0015; Wed, 23 Aug 2023 09:15:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C76E28E0011 for ; Wed, 23 Aug 2023 09:15:20 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A85DFB11B2 for ; Wed, 23 Aug 2023 13:15:20 +0000 (UTC) X-FDA: 81155415600.24.4DD3B4F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id E2BEA2002A for ; Wed, 23 Aug 2023 13:15:18 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ys9H5PnldVXevxD/YSg/rx5YKtIBi2TNqs30lPiZcP4=; b=oVF85SCdLoWc+Ec6tiRmmzyjBjiwqqZ4VXpn4rn9Q9ocBbb6A+ujjykV3++mSM7gSD3u/y xjsJ/EdznKAFjDqmFb4r6SqDjdeAL/+yWij6Ux1j7Va3jrQUUUxkSJ2Quq9F5DIifC7KGN 8yDp/is453MiTQrNQ+c0ozQvpc2HZm4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796519; a=rsa-sha256; cv=none; b=zd+ZnxAGPy+vW1GAC28331YOvekTQdgZt8yA0IauOHnW4m2TtZC3/8G9FOMwELkt20/CDa 8MZ6JflRJG+J2b0UD+nadnZpf6KFZGD88Q6Swu16vfq/GfsHlXVQDQr7cG/GLfYFENJgjW rW+h85ehjS2WkH7ueEHX54xgG9Qr0TM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9081C15DB; Wed, 23 Aug 2023 06:15:58 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2C7E63F740; Wed, 23 Aug 2023 06:15:12 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 11/37] mm: migrate/mempolicy: Allocate metadata-enabled destination page Date: Wed, 23 Aug 2023 14:13:24 +0100 Message-Id: <20230823131350.114942-12-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E2BEA2002A X-Stat-Signature: 76r8i9fhjky6z9rjbbfiodbz1rj4dkgu X-Rspam-User: X-HE-Tag: 1692796518-65387 X-HE-Meta: U2FsdGVkX187iPphQE/w3ESbP2g/lEzqvOu7LKDu5ytT7ZCwkhsAFnBAw92j08QDaGNmthZNSjBJIoTaZDhF3SGMhxpp4D1mGsQoEilmW/yvD3+QNpzs2TqwAb+Z6pzxMwW4D+KtdDRsTU46iXz4KsPSLDy+PrWC8rNEw3LXLKZQw3OwknVZDlxp3t0UBtwvj4P9QxDEg+tP0hIqh1ihXZJRlGDPyMwQ2zXIqdlsQndulqvgjAiClKVNCPKB1HQ7sQYcXknPYt+jl1w6+cuLYb08/MmljNHm0YmlEroUkJXmThxQD/ho9NwrohqRX4vSv9o4oKhrUZgxv48qb+cS5d4VELW0EhMDqlsG9PwY96MYPLunNDINaOvFcs+mKjWuqSVzv5V5lUcEJGMutPXATiT6nXoAHs6GSyznc+rQssOC3mEkN4/P75JfUsPp7PR9fmnHUFYSqPndH3OpYR4iBogo3TH/eJttEvCEWRnLte84jmxqNSZmMcLYz2cEg57TSWoFEVZ8qDACw33Xl93vGeH/ilrvezET8AOxnGO1XCR42L/reWOTJFH+v/epPOPZgQSMY1g8mDsN8uJfsFDVtG6FyBcuRdzdw4kLz15i3aJfoLw2yfktMdKUCYXjCm9I2RGAytpx8+bsQJzT+oqp0ZIAVuuMGmzyNsr8xGKq/bDF9O7OlwVFeBVByLDJnMj4NNx9ZlENukYM5MbaqjLReMQnQ2yMnmOMhStYNHopwzfn6g5L7QlS2cfOE1LL6fDvvGgpQZpc+3hvgBtT03VRD0tf6MtWur3F0RAMITYfxC0LBJyG3HPJRCCzWPDAVGNhx/m19FCBjv3TWx/Vls8pUNNQ3Q4l/VRQBeCKDHVNA0h385jfD3OD1zQHXRMuEPH6pkEvdBHmCA6yuPa5bNkch2FW46Gv4aEWGND9Bx93Ee9oIZ/PUahpDhi8p1+NzUU3Z8kh3CJ1XTE4YUIr1nS AAZjA3VK 5P0eCRWWqvbC9Rif005wKHnVv1PyU3sjxoMogVSae7hXPeBWRUfrRCeYlcGjgki/yY8uYz9fr6gLAu455dHEgY6HhMOy/G97d9K3TvGiTDZkmy9zpT3o0hWJ9EM1PCksp6/wa21464e2LAAhJdz26RSBYBhY4vUs8lzfJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With explicit metadata page management support, it's important to know if the source page for migration has metadata associated with it for two reasons: - The page allocator knows to skip metadata pages (which cannot have metadata) when allocating the destination page. - The associated metadata page is correctly reserved when fulfilling the allocation for the destination page. When choosing the destination during migration, keep track if the source page has metadata. The mbind() system call changes the NUMA allocation policy for the specified memory range and nodemask. If the MPOL_MF_MOVE or MPOL_MF_MOVE_ALL flags are set, then any existing allocations that fall within the range which don't conform to the specified policy will be migrated. The function that allocates the destination page for migration is new_page(), teach it too about source pages with metadata. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 4 ++++ include/asm-generic/memory_metadata.h | 4 ++++ mm/mempolicy.c | 4 ++++ mm/migrate.c | 6 ++++++ 4 files changed, 18 insertions(+) diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index c57c435c8ba3..132707fce9ab 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -21,6 +21,10 @@ static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) #define page_has_metadata(page) page_mte_tagged(page) +static inline bool folio_has_metadata(struct folio *folio) +{ + return page_has_metadata(&folio->page); +} #endif /* CONFIG_MEMORY_METADATA */ #endif /* __ASM_MEMORY_METADATA_H */ diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index 02b279823920..8f4e2fba222f 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -20,6 +20,10 @@ static inline bool page_has_metadata(struct page *page) { return false; } +static inline bool folio_has_metadata(struct folio *folio) +{ + return false; +} #endif /* !CONFIG_MEMORY_METADATA */ #endif /* __ASM_GENERIC_MEMORY_METADATA_H */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index edc25195f5bd..d164b5c50243 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -103,6 +103,7 @@ #include #include +#include #include #include #include @@ -1219,6 +1220,9 @@ static struct folio *new_folio(struct folio *src, unsigned long start) if (folio_test_large(src)) gfp = GFP_TRANSHUGE; + if (folio_has_metadata(src)) + gfp |= __GFP_TAGGED; + /* * if !vma, vma_alloc_folio() will use task or system default policy */ diff --git a/mm/migrate.c b/mm/migrate.c index 24baad2571e3..c6826562220a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -51,6 +51,7 @@ #include #include +#include #include #include @@ -1990,6 +1991,9 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private) if (nid == NUMA_NO_NODE) nid = folio_nid(src); + if (folio_has_metadata(src)) + gfp_mask |= __GFP_TAGGED; + if (folio_test_hugetlb(src)) { struct hstate *h = folio_hstate(src); @@ -2476,6 +2480,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src, __GFP_NOWARN; gfp &= ~__GFP_RECLAIM; } + if (folio_has_metadata(src)) + gfp |= __GFP_TAGGED; return __folio_alloc_node(gfp, order, nid); } From patchwork Wed Aug 23 13:13:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D2BBEE4993 for ; Wed, 23 Aug 2023 13:15:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B414490002E; Wed, 23 Aug 2023 09:15:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF13590002C; Wed, 23 Aug 2023 09:15:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91D1090002E; Wed, 23 Aug 2023 09:15:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8348890002C for ; Wed, 23 Aug 2023 09:15:26 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6453016010B for ; Wed, 23 Aug 2023 13:15:26 +0000 (UTC) X-FDA: 81155415852.19.F1D77CD Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf10.hostedemail.com (Postfix) with ESMTP id BDE9BC0010 for ; Wed, 23 Aug 2023 13:15:24 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796524; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mQXtYP7R5BBbKPu5paKceFa7P/n6vcGbguF9R7S7xqM=; b=PQqtGkN3AVjeLPT3nR991artA/4dPu6swRa1bRP68fnnBOWeQLGoSfFs1JJZic5gy5nc9t rAAL4z6Rg9wjApCLKN0yEpmDJGCw9EGDpdOoxxwiqgt5hYTnuBJhaYZhUheiRvmA5m3x+y NuciRIf44n1zJjHfztT0wXeZ9yFptRk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796524; a=rsa-sha256; cv=none; b=neL7O2PQpNvSdKMqi8c/PLOBVL3gpDaqyzpE7IkcqpVVr3cw+TbA4oJ7ps3d/RY20va65K d1wImHe0bSs0hTjErKVIhDLRKo9GKdFpgwwDTLzumDtpREov/wgE78IzyQjPLENL/EXVXC Hg8H043I12QcFg/OmIJp5GqFTIrxxnQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 91C481650; Wed, 23 Aug 2023 06:16:04 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4AFC03F740; Wed, 23 Aug 2023 06:15:18 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 12/37] mm: gup: Don't allow longterm pinning of MIGRATE_METADATA pages Date: Wed, 23 Aug 2023 14:13:25 +0100 Message-Id: <20230823131350.114942-13-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BDE9BC0010 X-Rspam-User: X-Stat-Signature: ma3fp66wfw8f8ozdnocsgpwmr3ydhn43 X-Rspamd-Server: rspam01 X-HE-Tag: 1692796524-870582 X-HE-Meta: U2FsdGVkX19/7KJNLRFj33c8iN3k4w/uMMJsIma9aaGuKXOlV+2MhFkqkpfBdoNPehGUwrBpuFhZE597+cKbjAsJNmcQ3ciTO4gPzSUBQnIYf33IG/OuvScS2FtahiMWYpqRHmnR7m51xKtU6GDJXBLzfadop7hZ0aaakbcs5KhSRjYxRBXm7g61fmjjgBuEEpFBmzDwWjdZ81DY7BXkXdziTWParrJrIIZMXGtwCfArursgE043EChlGv10V0W3w4P0Xq5nkBlEDEOtgeTrcvtsf3Zd5gUM00y/0EZwxXPM73j4tT46OwLDKMms0YAqxoy8OgrvkD+3ab5Y31OVeZO3m/CPqNBSIYwaElp12dqZxTZTqeilZoFgv2/f+JYhzDvYLRwOzyUuBq8RJUcTjdFwk2quPYTdXSMAoHacTNoBfz0wA4pVC0A8bG9P6XdMmSR4AalkCUA5qLpwbaE6SN05ncl/wW/FroGHbzU/H+1vssSKZ+a/qI+dsUtYcwoRUosLcU8AWdLwHCxVOUNMrrx2JwRKZeZWAEnm/IitFvmOo7tm/AsjNBgvn8u2DSp6V/Y09MHb3js9mV1Ov0pOzsIJj2chFE8/myIoewRPAVuOnYRI6WmRlCTmpR6Bdnk3pCCJpCgtQIoRMM0hTMHZx4Si3+FSn4Ik8SgfEcINGtORY5CrSkxBw6PGEhPjViYjWCyLIywxUllZkQJptKx73KvPiUccja2m46OXMYbI1UUMBaowa7B1gFdgncymVKbsLu5k46ar7rL6ULVqcqK9gmtaps5guIwUDYKyPeKh8X6Ge3bdzmcy9w8xmiqAZ0XzTqeQq8OPDenEe/fxu3o2VLArpCV9Udarwn5k6GJkLE3sesns2ObnOrOTuZdEiyB0rFVJ8J8JsgAFhD0t61MyaZvcTsHDbXvJOSfMfM/asIzPYLwNnJQg3d9yT6R+KQ/GRmdeKWDPRcX2oHljX0N g7HjfR25 LNfO905LbydX77RPPdSnMRkj5pOd5w3zizLrJHzULOofITwqZqt0T2SFC/hztlkpfpVcrbdHOh/B1ud2EXcwdk7LgGaFrN4lWq9uY+5BHiym7yHxGhnEtNsF01tquqyjyrX/RuaJ37ZtmKOA58wihhxCCt4TdVY5ipluE+FjEh28Q+zK09Yi/hzBX28tXKH3QSaGxdVfLqiopnWg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Treat MIGRATE_METADATA pages just like movable or CMA pages and don't allow them to be pinned longterm. No special handling needed for migrate_longterm_unpinnable_pages() because the gfp mask for allocating the destination pages is GFP_USER. GFP_USER doesn't include __GFP_MOVABLE, which makes it impossible to accidently allocate metadata pages for migrating the pinned pages. Signed-off-by: Alexandru Elisei --- include/linux/mm.h | 10 +++++++--- mm/Kconfig | 2 ++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2dd73e4f3d8e..ce87d55ecf87 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1925,16 +1925,20 @@ static inline bool is_zero_folio(const struct folio *folio) return is_zero_page(&folio->page); } -/* MIGRATE_CMA and ZONE_MOVABLE do not allow pin folios */ +/* MIGRATE_CMA, MIGRATE_METADATA and ZONE_MOVABLE do not allow pin folios */ #ifdef CONFIG_MIGRATION static inline bool folio_is_longterm_pinnable(struct folio *folio) { -#ifdef CONFIG_CMA +#if defined(CONFIG_CMA) || defined(CONFIG_MEMORY_METADATA) int mt = folio_migratetype(folio); - if (mt == MIGRATE_CMA || mt == MIGRATE_ISOLATE) + if (mt == MIGRATE_ISOLATE) + return false; + + if (is_migrate_cma(mt) || is_migrate_metadata(mt)) return false; #endif + /* The zero page can be "pinned" but gets special handling. */ if (is_zero_folio(folio)) return true; diff --git a/mm/Kconfig b/mm/Kconfig index 838193522e20..847e1669dba0 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1238,6 +1238,8 @@ config LOCK_MM_AND_FIND_VMA config MEMORY_METADATA bool + select MEMORY_ISOLATION + select MIGRATION source "mm/damon/Kconfig" From patchwork Wed Aug 23 13:13:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07409EE49A0 for ; Wed, 23 Aug 2023 13:15:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D04E900030; Wed, 23 Aug 2023 09:15:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 880A990002C; Wed, 23 Aug 2023 09:15:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D3D1900030; Wed, 23 Aug 2023 09:15:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5979B90002C for ; Wed, 23 Aug 2023 09:15:38 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 23D9840002 for ; Wed, 23 Aug 2023 13:15:38 +0000 (UTC) X-FDA: 81155416356.23.5DE7FD2 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf10.hostedemail.com (Postfix) with ESMTP id 40974C0017 for ; Wed, 23 Aug 2023 13:15:36 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796536; a=rsa-sha256; cv=none; b=0OMNdQXHQEWzwWac1c4L5WRRCxh5QE8bY+BbX5mJpr3GhG8RILIteQb8VJuCiZP1PvE6FU tyE7Qjb5sIgIY8KNMbjOyEplLpTKdHy4w3EgviEBNUwgpX/lT+LxSYJvaRUa4Ve7qlenbF fJiFhbaYAGeBWz/1CHS5nD5p4MhUQQI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796536; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qj3NQx0RNfQ+L4wpgGM4+UupLlv4VRmKSNrZ91Gty4E=; b=nmtBc67qkMhnt6QLaaJmLhi/5Clb3fTe+sOf+t/DSAwEMEVphhUABj4llj01FD8WPKeWT7 QVieCrl5DheniK7NzzjlyLxKrjJWVWyIo89wpQ7pwR9RDt0g99TtxHs9iPjq/j6+LyQX0m ynGKezsQiZIKumHSnKb51UX8Hq/BRbk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1E9101655; Wed, 23 Aug 2023 06:16:11 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 54BD83F740; Wed, 23 Aug 2023 06:15:24 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 13/37] arm64: mte: Reserve tag storage memory Date: Wed, 23 Aug 2023 14:13:26 +0100 Message-Id: <20230823131350.114942-14-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 40974C0017 X-Stat-Signature: 8g8h9pijogk5qe5xhd66uw9pda9ceq4c X-HE-Tag: 1692796536-858372 X-HE-Meta: U2FsdGVkX18BB/niZq0FnRLIEwMMNFXkDYDM393zD8eyM+ynh9G0kZUSP38T8BBypfTGEekhjLs87/FUwAvKj8nfE8cWmtdBzKhRXy5+k+mqAjnGY7sJw1wIAbKcrPUKx6LMH7m2964GOwjtzTsVSGNKtCpp3ksj+avk6xlN/nCqxz1kDfroVARROA2srEOPvgHl9BqmCCnA8anZ3pEaPJ8AHcU5/2Jjg7XiaMAvzoPIpktR6EcPobqB8ZyWzAnCgVCgNR0xYphn/kHcBV/IxvVz6xBKrTX2+//I4DWMltBQryE5REh2VaYKWe+AEdp5JWaniDgDk2bNFQc7a93rzEvbUrTMa5raRUHlqMF376ypNr6h57r80dCIE2T9P6Ajdwu73Xr67ZnlJ8d8otUXGbdIvxfyV9YQl+n5nM+7FM07VQcE8y+RmarPYfmrP4g5r5mFe1fgATwBWqqhPiT8uNvFNXwhb2vJWvNUt0XIc+B86eW95QPW6RRTaAtv+0qUnTDK0jDYWZYyjcR7LZoqPJ0CVgQYeWywv2MWtKgfaOIjpZKnwUlvPaMYZUWCcJbHM/9XNAWTUxqwTmyqJkB5/fRIEPkSK98ku4T+EugCVP69E3r17iSvlewNsdlZEfKaseqTwBy9j/X95u6hjIOnHiFp7dL3hywxonyD4hAl+jTtioyroHSeWP9v61yr4lC+CRZTWYmlU0JeOYJO5U2N3ABSb8Wrl0Q+4dWOAHc/sVlElPNYvEhgzn4wB/o6HF0Jxl/rDo+gR9KQNbI7EkjxDOCMtOdwbRf+/KoUd9L9fTGryx90HC+LzWc/DrnARqlgJrOLijuRSxKtR0wPAMpdZCV14AL/uXiwDVVy3REsioZv8wu/LD3ik0UtpsRvCe5ojAZWbqr1cPwYCjObuZPNOtaxKwiz0qQGRSLaCV1vLqlRLurHcKUZneU5xYRe7kNYrI9aH6lLWyquPc9kbQ0 z0op0JG+ o8HzeEJuzEWLirPgqEVrt0dBG0aDByVMHTr2oVRVxhXj5mGWeDKRpFZObykmno2de9NWRyUA/ajshKnpwR6g5eT4RmVa4KpJ7vBarE/Uk+R/ZtXEbXOOJhqYjO4/hPM6TOWmfmMPaa1LpuGypGoxLyphRn3jKd1QEXSO6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow the kernel to get the size and location of the MTE tag storage memory from the DTB. This memory is marked as reserved for now, with later patches adding support for making use of it. The DTB node for the tag storage region is defined as: metadata1: metadata@8c0000000 { compatible = "arm,mte-tag-storage"; reg = <0x08 0xc0000000 0x00 0x1000000>; block-size = <0x1000>; // 4k memory = <&memory1>; // Associated tagged memory }; The tag storage region represents the largest contiguous memory region that holds all the tags for the associated contiguous memory region which can be tagged. For example, for a 32GB contiguous tagged memory the corresponding tag storage region is 1GB of contiguous memory, not two adjacent 512M memory regions. "block-size" represents the minimum multiple of 4K of tag storage where all the tags stored in the block correspond to a contiguous memory region. This in needed for platforms where the memory controller interleaves tag writes to memory. For example, if the memory controller interleaves tag writes for 256KB of contiguous memory across 8K of tag storage (2-way interleave), then the correct value for "block-size" is 0x2000. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 12 ++ arch/arm64/include/asm/memory_metadata.h | 3 +- arch/arm64/include/asm/mte_tag_storage.h | 15 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/mte_tag_storage.c | 262 +++++++++++++++++++++++ arch/arm64/kernel/setup.c | 7 + 6 files changed, 299 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/mte_tag_storage.h create mode 100644 arch/arm64/kernel/mte_tag_storage.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2511b30d0f6..ed27bb87babd 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2077,6 +2077,18 @@ config ARM64_MTE Documentation/arch/arm64/memory-tagging-extension.rst. +if ARM64_MTE +config ARM64_MTE_TAG_STORAGE + bool "Dynamic MTE tag storage management" + select MEMORY_METADATA + help + Adds support for dynamic management of the memory used by the hardware + for storing MTE tags. This memory can be used as regular data memory + when it's not used for storing tags. + + If unsure, say N +endif # ARM64_MTE + endmenu # "ARMv8.5 architectural features" menu "ARMv8.7 architectural features" diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index 132707fce9ab..3287b2776af1 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -14,9 +14,10 @@ static inline bool metadata_storage_enabled(void) { return false; } + static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) { - return false; + return !(gfp_mask & __GFP_TAGGED); } #define page_has_metadata(page) page_mte_tagged(page) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h new file mode 100644 index 000000000000..8f86c4f9a7c3 --- /dev/null +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023 ARM Ltd. + */ +#ifndef __ASM_MTE_TAG_STORAGE_H +#define __ASM_MTE_TAG_STORAGE_H + +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +void mte_tag_storage_init(void); +#else +static inline void mte_tag_storage_init(void) +{ +} +#endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ +#endif /* __ASM_MTE_TAG_STORAGE_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index d95b3d6b471a..5f031bf9f8f1 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -70,6 +70,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o +obj-$(CONFIG_ARM64_MTE_TAG_STORAGE) += mte_tag_storage.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c new file mode 100644 index 000000000000..5014dda9bf35 --- /dev/null +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Support for dynamic tag storage. + * + * Copyright (C) 2023 ARM Ltd. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +struct tag_region { + struct range mem_range; /* Memory associated with the tag storage, in PFNs. */ + struct range tag_range; /* Tag storage memory, in PFNs. */ + u32 block_size; /* Tag block size, in pages. */ +}; + +#define MAX_TAG_REGIONS 32 + +static struct tag_region tag_regions[MAX_TAG_REGIONS]; +static int num_tag_regions; + +static int __init tag_storage_of_flat_get_range(unsigned long node, const __be32 *reg, + int reg_len, struct range *range) +{ + int addr_cells = dt_root_addr_cells; + int size_cells = dt_root_size_cells; + u64 size; + + if (reg_len / 4 > addr_cells + size_cells) + return -EINVAL; + + range->start = PHYS_PFN(of_read_number(reg, addr_cells)); + size = PHYS_PFN(of_read_number(reg + addr_cells, size_cells)); + if (size == 0) { + pr_err("Invalid node"); + return -EINVAL; + } + range->end = range->start + size - 1; + + return 0; +} + +static int __init tag_storage_of_flat_get_tag_range(unsigned long node, + struct range *tag_range) +{ + const __be32 *reg; + int reg_len; + + reg = of_get_flat_dt_prop(node, "reg", ®_len); + if (reg == NULL) { + pr_err("Invalid metadata node"); + return -EINVAL; + } + + return tag_storage_of_flat_get_range(node, reg, reg_len, tag_range); +} + +static int __init tag_storage_of_flat_get_memory_range(unsigned long node, struct range *mem) +{ + const __be32 *reg; + int reg_len; + + reg = of_get_flat_dt_prop(node, "linux,usable-memory", ®_len); + if (reg == NULL) + reg = of_get_flat_dt_prop(node, "reg", ®_len); + + if (reg == NULL) { + pr_err("Invalid memory node"); + return -EINVAL; + } + + return tag_storage_of_flat_get_range(node, reg, reg_len, mem); +} + +struct find_memory_node_arg { + unsigned long node; + u32 phandle; +}; + +static int __init fdt_find_memory_node(unsigned long node, const char *uname, + int depth, void *data) +{ + const char *type = of_get_flat_dt_prop(node, "device_type", NULL); + struct find_memory_node_arg *arg = data; + + if (depth != 1 || !type || strcmp(type, "memory") != 0) + return 0; + + if (of_get_flat_dt_phandle(node) == arg->phandle) { + arg->node = node; + return 1; + } + + return 0; +} + +static int __init tag_storage_get_memory_node(unsigned long tag_node, unsigned long *mem_node) +{ + struct find_memory_node_arg arg = { 0 }; + const __be32 *memory_prop; + u32 mem_phandle; + int ret, reg_len; + + memory_prop = of_get_flat_dt_prop(tag_node, "memory", ®_len); + if (!memory_prop) { + pr_err("Missing 'memory' property in the tag storage node"); + return -EINVAL; + } + + mem_phandle = be32_to_cpup(memory_prop); + arg.phandle = mem_phandle; + + ret = of_scan_flat_dt(fdt_find_memory_node, &arg); + if (ret != 1) { + pr_err("Associated memory node not found"); + return -EINVAL; + } + + *mem_node = arg.node; + + return 0; +} + +static int __init tag_storage_of_flat_read_u32(unsigned long node, const char *propname, + u32 *retval) +{ + const __be32 *reg; + + reg = of_get_flat_dt_prop(node, propname, NULL); + if (!reg) + return -EINVAL; + + *retval = be32_to_cpup(reg); + return 0; +} + +static u32 __init get_block_size_pages(u32 block_size_bytes) +{ + u32 a = PAGE_SIZE; + u32 b = block_size_bytes; + u32 r; + + /* Find greatest common divisor using the Euclidian algorithm. */ + do { + r = a % b; + a = b; + b = r; + } while (b != 0); + + return PHYS_PFN(PAGE_SIZE * block_size_bytes / a); +} + +static int __init fdt_init_tag_storage(unsigned long node, const char *uname, + int depth, void *data) +{ + struct tag_region *region; + unsigned long mem_node; + struct range *mem_range; + struct range *tag_range; + u32 block_size_bytes; + u32 nid; + int ret; + + if (depth != 1 || !strstr(uname, "metadata")) + return 0; + + if (!of_flat_dt_is_compatible(node, "arm,mte-tag-storage")) + return 0; + + if (num_tag_regions == MAX_TAG_REGIONS) { + pr_err("Maximum number of tag storage regions exceeded"); + return -EINVAL; + } + + region = &tag_regions[num_tag_regions]; + mem_range = ®ion->mem_range; + tag_range = ®ion->tag_range; + + ret = tag_storage_of_flat_get_tag_range(node, tag_range); + if (ret) { + pr_err("Invalid tag storage node"); + return ret; + } + + ret = tag_storage_get_memory_node(node, &mem_node); + if (ret) + return ret; + + ret = tag_storage_of_flat_get_memory_range(mem_node, mem_range); + if (ret) { + pr_err("Invalid address for associated data memory node"); + return ret; + } + + /* The tag region must exactly match the corresponding memory. */ + if (range_len(tag_range) * 32 != range_len(mem_range)) { + pr_err("Tag region doesn't cover exactly the corresponding memory region"); + return -EINVAL; + } + + ret = tag_storage_of_flat_read_u32(node, "block-size", &block_size_bytes); + if (ret || block_size_bytes == 0) { + pr_err("Invalid or missing 'block-size' property"); + return -EINVAL; + } + region->block_size = get_block_size_pages(block_size_bytes); + if (range_len(tag_range) % region->block_size != 0) { + pr_err("Tag storage region size is not a multiple of allocation block size"); + return -EINVAL; + } + + ret = tag_storage_of_flat_read_u32(mem_node, "numa-node-id", &nid); + if (ret) + nid = numa_node_id(); + + ret = memblock_add_node(PFN_PHYS(tag_range->start), PFN_PHYS(range_len(tag_range)), + nid, MEMBLOCK_NONE); + if (ret) { + pr_err("Error adding tag memblock (%d)", ret); + return ret; + } + memblock_reserve(PFN_PHYS(tag_range->start), PFN_PHYS(range_len(tag_range))); + + pr_info("Found MTE tag storage region 0x%llx@0x%llx, block size %u pages", + PFN_PHYS(range_len(tag_range)), PFN_PHYS(tag_range->start), region->block_size); + + num_tag_regions++; + + return 0; +} + +void __init mte_tag_storage_init(void) +{ + struct range *tag_range; + int i, ret; + + ret = of_scan_flat_dt(fdt_init_tag_storage, NULL); + if (ret) { + pr_err("MTE tag storage management disabled"); + goto out_err; + } + + if (num_tag_regions == 0) + pr_info("No MTE tag storage regions detected"); + + return; + +out_err: + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + memblock_remove(PFN_PHYS(tag_range->start), PFN_PHYS(range_len(tag_range))); + } + num_tag_regions = 0; +} diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 417a8a86b2db..1b77138c1aa5 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -342,6 +343,12 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) FW_BUG "Booted with MMU enabled!"); } + /* + * Must be called before memory limits are enforced by + * arm64_memblock_init(). + */ + mte_tag_storage_init(); + arm64_memblock_init(); paging_init(); From patchwork Wed Aug 23 13:13:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DE1EEE49B0 for ; Wed, 23 Aug 2023 13:15:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC5FD900031; Wed, 23 Aug 2023 09:15:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E749490002C; Wed, 23 Aug 2023 09:15:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC715900031; Wed, 23 Aug 2023 09:15:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B6F4190002C for ; Wed, 23 Aug 2023 09:15:40 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8EBEB12012B for ; Wed, 23 Aug 2023 13:15:40 +0000 (UTC) X-FDA: 81155416440.24.18E00AE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id C1B934001D for ; Wed, 23 Aug 2023 13:15:38 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8EJ+2QqpCbHyjqA+CxCqOVdJDVMnVbpZW0doDrqRPVQ=; b=v7I6ef9J4bHliOq9CPEqyzvuQYdbUs3p3Z1lD6rqC3O7Ajifzm52zpEXP/7aM5emsMqKZW 7QfvvymYWCfArFn05I3lubHO7nX26QhEurRtA5aoMW74OdU89T0EAusPuzqymazK9Dqy+S 3L2EwMa/hUDEE0/MYC9HWA3rF15vIDs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796538; a=rsa-sha256; cv=none; b=Lw5NRG/+JWRiVDlXevHrF9aPsQ7Pgh+eiun+B9ao4xIWrP5BinWfGeXbVlx22ncqItxkd0 v5ad5JAW/wVBsZIZurn0UTrDB3J3GPZ8NtjgtnfrjPQmZB3ZG+GBG5EMu3CfzVqMzeXXKF KyAaaXhlx4PQCrpeZ0DOxdTSnaA7vZY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8850B165C; Wed, 23 Aug 2023 06:16:18 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CEDA63F740; Wed, 23 Aug 2023 06:15:30 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 14/37] arm64: mte: Expose tag storage pages to the MIGRATE_METADATA freelist Date: Wed, 23 Aug 2023 14:13:27 +0100 Message-Id: <20230823131350.114942-15-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C1B934001D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 747enxub1jkc5xa6cnisryne6yqs5itq X-HE-Tag: 1692796538-399019 X-HE-Meta: U2FsdGVkX1+0Ac0LtDJYtWxH7UsEw3rJatcbId0+Est33OlYfTaJNSePxsAz6bq4mPwUizD46WH8GSKFi/T7AQgRdEGt01E+WVp9DlvtnA8jzmA1tko9Vk4b9q9e5jYAFCPy+tb9EtnpNeA6iGiGO2nIY/kTCpBlQnNGA4G3VdQ4EtTBEZxAsDsslAP+gr2VZAcNN8d5PhpjaIUYlnvMHc5aFTDnVdnFt9c1rxog6vx1S+JjYhkHVoNvoLZfjUDXAOXwk6sjiOycvpPDuXpfXuRJBSiL8NFM0KStMDJfHTLHiHh38pI5TfTXoTG1UC5/goYXgnjmpy9Dir30csBz5kPoDU7m7kYa7r7wykj06htqD068VJDqJXDPyF13DLKInKpdbnAktAmui55ASzJ1FDQiMKNLKqsLWqUvfD0JqrBzbhPhWZVFqusjUSlrauBjInJPR39XvIYAnHtbjPTSusffm1j265S7CLJ6uw/Xz4Fo4ldygh5LudkUAPFZvjbN7nxQYQ4FmEJ2EsQ4qQClYGqeiQd3RVflHP9rs1hgzyPnsn0TZG8N7VS2ii4h71OfYOgi+58wmLrSdGVEBuiq88+b5+C88s17KpIRlhckkQyhLBS41blZLBkaCclffXL4jsFKdM1cuAjgOA6yvno7NMKgA66FZCyjsNCloT1P3cDnwzCEFtFde290/1reUZyayc/U+thYG3tyFFiKTHZ+FH4gzKupBJ483Pj6+zhl8rPnYAGLoSD7TjtHNBP9etl3VynZguNZrM29IrP61SK5Xg+Un5xyNckm/ZjCqqV1A5ahEcipzTeW/f192JDcXWz78mQKzirWF/n9h2GMHohqYWKWJqEoi5JxsR5sfoWuSLwJxcCqmvGl9nSOSkMAeAaR8m+VLq1GMD2bLqNEFMSph0+6mr0Z+0VJJds6QQGo04OlDpXEpzB2R2Che8jclxB1A5t8awZZzhla41q1MXZ tSiSI5dm 0iqCOfuoKUg3X4iGwg88xpyVGEqWKhNizZEjX0Td3mLlp3rW8FQSJOvyHI163/LOx8Ocqy9/mF7UkQTqnMKFK6Pm59qjPAD8E0rIVW9MH1Zt9tQX6p41krmCwwaWqRcoDYmKQ4wtmNtZYX0PZR0eVFp94WdN6ElfWRgnvpYLAyw0PcSMQG1xj6pUXEgPoiVpMAi3IuipbYfQrbXhKmxTpIfabYN5KtPWsV9XRnQMHOtnBmPiT54bLZElBxgS2hfq/7zP3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add the MTE tag storage pages to the MIGRATE_METADATA freelist, which allows the page allocator to manage them like (almost) regular pages. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 47 +++++++++++++++++++++++++++++ include/linux/gfp.h | 8 +++++ mm/mm_init.c | 24 +++++++++++++-- 3 files changed, 76 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 5014dda9bf35..87160f53960f 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -5,10 +5,12 @@ * Copyright (C) 2023 ARM Ltd. */ +#include #include #include #include #include +#include #include #include #include @@ -190,6 +192,12 @@ static int __init fdt_init_tag_storage(unsigned long node, const char *uname, return ret; } + /* Pages are managed in pageblock_nr_pages chunks */ + if (!IS_ALIGNED(tag_range->start | range_len(tag_range), pageblock_nr_pages)) { + pr_err("Tag storage region not aligned to 0x%lx", pageblock_nr_pages); + return -EINVAL; + } + ret = tag_storage_get_memory_node(node, &mem_node); if (ret) return ret; @@ -260,3 +268,42 @@ void __init mte_tag_storage_init(void) } num_tag_regions = 0; } + +static int __init mte_tag_storage_activate_regions(void) +{ + phys_addr_t dram_start, dram_end; + struct range *tag_range; + unsigned long pfn; + int i; + + if (num_tag_regions == 0) + return 0; + + dram_start = memblock_start_of_DRAM(); + dram_end = memblock_end_of_DRAM(); + + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + /* + * Tag storage region was clipped by arm64_bootmem_init() + * enforcing addressing limits. + */ + if (PFN_PHYS(tag_range->start) < dram_start || + PFN_PHYS(tag_range->end) >= dram_end) { + pr_err("Tag storage region 0x%llx-0x%llx outside addressable memory", + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end + 1)); + return -EINVAL; + } + } + + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) { + init_metadata_reserved_pageblock(pfn_to_page(pfn)); + totalmetadata_pages += pageblock_nr_pages; + } + } + + return 0; +} +core_initcall(mte_tag_storage_activate_regions) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 665f06675c83..fb344baa9a9b 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -354,4 +354,12 @@ extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); +#ifdef CONFIG_MEMORY_METADATA +extern void init_metadata_reserved_pageblock(struct page *page); +#else +static inline void init_metadata_reserved_pageblock(struct page *page) +{ +} +#endif + #endif /* __LINUX_GFP_H */ diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..467c80e9dacc 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2329,8 +2329,9 @@ bool __init deferred_grow_zone(struct zone *zone, unsigned int order) #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ -#ifdef CONFIG_CMA -void __init init_cma_reserved_pageblock(struct page *page) +#if defined(CONFIG_CMA) || defined(CONFIG_MEMORY_METADATA) +static void __init init_reserved_pageblock(struct page *page, + enum migratetype migratetype) { unsigned i = pageblock_nr_pages; struct page *p = page; @@ -2340,15 +2341,32 @@ void __init init_cma_reserved_pageblock(struct page *page) set_page_count(p, 0); } while (++p, --i); - set_pageblock_migratetype(page, MIGRATE_CMA); + set_pageblock_migratetype(page, migratetype); set_page_refcounted(page); __free_pages(page, pageblock_order); adjust_managed_page_count(page, pageblock_nr_pages); +} + +#ifdef CONFIG_CMA +/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ +void __init init_cma_reserved_pageblock(struct page *page) +{ + init_reserved_pageblock(page, MIGRATE_CMA); page_zone(page)->cma_pages += pageblock_nr_pages; } #endif +#ifdef CONFIG_MEMORY_METADATA +/* Free whole pageblock and set its migration type to MIGRATE_METADATA. */ +void __init init_metadata_reserved_pageblock(struct page *page) +{ + init_reserved_pageblock(page, MIGRATE_METADATA); + page_zone(page)->metadata_pages += pageblock_nr_pages; +} +#endif +#endif /* CONFIG_CMA || CONFIG_MEMORY_METADATA */ + void set_zone_contiguous(struct zone *zone) { unsigned long block_start_pfn = zone->zone_start_pfn; From patchwork Wed Aug 23 13:13:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F0E1EE4993 for ; Wed, 23 Aug 2023 13:15:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 267D2900032; Wed, 23 Aug 2023 09:15:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23F1090002C; Wed, 23 Aug 2023 09:15:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 107C2900032; Wed, 23 Aug 2023 09:15:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 003C590002C for ; Wed, 23 Aug 2023 09:15:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C6A67A0110 for ; Wed, 23 Aug 2023 13:15:46 +0000 (UTC) X-FDA: 81155416692.06.D9D2FB5 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id E7653120016 for ; Wed, 23 Aug 2023 13:15:44 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796545; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LKitmhj8fv3HGM2ZEl15TmS0y71aN9S6+KBiyCMKdQE=; b=Rjq9WkpTRUo/7WTfGD/jaruNBKICUEPHfvnpeXrJuK2k8efZ8ZfX/dav423YpeZYMd+ZAw 4bxOUoie3GfMXn8Ft57K6UTlLueNoS6nfiB4BaJsJGUN2Kn69OTFYB3gfoT+w4H9Ppl4PD h1dpPYedX0k0QkJjXuHW6z2UxbvX27E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796545; a=rsa-sha256; cv=none; b=VdWczJSQCkl9vOMSOypK92MtREA1eYT4SJflqWh5mHSQ4Q8NZdCSCUzSqEjvP+eW7KK1Lh a/lsI6UCvbcCF5KHFs209f50sIRLXt/kcNyTTdL0vZJaPcfTSptY3Q/fpOv9IW8GlUH7mq kh0Nu1LMdIpx3K9UYl4wO38jZEjBDA4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D1B7C1682; Wed, 23 Aug 2023 06:16:24 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4F9513F740; Wed, 23 Aug 2023 06:15:38 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 15/37] arm64: mte: Make tag storage depend on ARCH_KEEP_MEMBLOCK Date: Wed, 23 Aug 2023 14:13:28 +0100 Message-Id: <20230823131350.114942-16-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E7653120016 X-Rspam-User: X-Stat-Signature: gh3kd48ace3xq1qicpuezepm38qd1pyz X-Rspamd-Server: rspam03 X-HE-Tag: 1692796544-173148 X-HE-Meta: U2FsdGVkX197tVnga4trlnGrREKJGcEbmtpvPeVGE1tJ+fgmab74K1MnwulXjkP5M2929Qgi7pEzLJMFtZWI7dipaI8cDEsrhtpumLOVc1/L38zxw4jg9QBmtndXhNb5uNgGDyX+xH7NWWFuY29DnWZpckozaMHcsWT6mcM6wLdgOm9/Br3uujpS70aL1xrZvihmwkJrbnInqoJkfZiwudhtu2JHsUdllncNmi6+TKIbuxvkRAS/zUlkHrjHYPa6K/4YTKuCb8P7Ge4saCogt/CzZitDsEiYsp+chnE6gXaTqGiIDIBC29xI+7PnX2oCalllB7zPqHu5sKNzu2ZuJ9a9k/i0U4oX0CEmoZ+9t8Z9PPRyTvL3cddvSiO5BOeiH2YJn3n6kvkuI7zwcpSUtuQJmws4R5emwghlyc9Cf0BhlM90OftwaR+XPbGF9hmU7YRDSOQjg/LKrEoql3KLWSG2QZupn/QV/SAVNjRjJ7cc6cxeFOeUcjWuHdogKTPmscX2cOzIVKI1d4atml22ZwgMnWjykr+twbHyVop3OtegiEbHYvsirL57ClbBbybmCOFUGXsHhoI2SncxuJDP05eJ8kl73zk6Vnu2Eb4Uml4do8H4l7xfdtxGcdakkAHPudKQEGNNoRTDESGf5LcZpdUHUH3QoScKDssCcJMYaHc6H9GYYGHDkUlpxJcMfG/sngJixRQYoa1IE6ZERW8Ne08MYIdFOE/plvEsKcF1Md2tTZsJQ/M9b5On3RwwHqI4Wzyf62wDvOajtGFn51bGwwrTNiQyF5/xA74SeQyn8tunLuDjhKn1JzB804UbZ0ChF1+Xq0cl2YyAj+fGijNCnJvyR8ArU3YTbXPe3jhEzdBhjcvtQ27bxkSNeCq71hD2KYPQmkOSjbsUTwH8riQZE+EqcGZ69WpCqsIfVQKL9F3Vsd/7IS9q1L9IX67QvRREC/zKZKlxjjFAAMUC8ty EBOmd9oO D2F8v5VIzPAZJZMUpoQbzH+uSHST2ZpZL7negRsOD00mhySiCFLyVWCXXs2NMlNVF2OUlt8uGQhty6YUw6HsNNy7+DD9qqTytmj0JjIPiQGNEb8mT9YPEzShoqwiNLfY2HLtn2KhDLpkweMiLvXlizQ+FNYotPlCnxkhK7PVgh7uCVAUMHwHDKsZCLDt4GfVNFeE052noI4eMF50= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Tag storage memory requires that the tag storage pages used for data can be migrated when they need to be repurposed to store tags. If ARCH_KEEP_MEMBLOCK is enabled, kexec will scan all non-reserved memblocks to find a suitable location for copying the kernel image. The kernel image, once loaded, cannot be moved to another location in physical memory. The initialization code for the tag storage reserves the memblocks for the tag storage pages, which means kexec will not use them, and the tag storage pages can be migrated at any time, which is the desired behaviour. However, if ARCH_KEEP_MEMBLOCK is not selected, kexec will not skip a region unless the memory resource has the IORESOURCE_SYSRAM_DRIVER_MANAGED flag, which isn't currently set by the initialization code. Make ARM64_MTE_TAG_STORAGE depend on ARCH_KEEP_MEMBLOCK to make it explicit that that is required for it to work correctly. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index ed27bb87babd..1e3d23ee22ab 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2081,6 +2081,7 @@ if ARM64_MTE config ARM64_MTE_TAG_STORAGE bool "Dynamic MTE tag storage management" select MEMORY_METADATA + depends on ARCH_KEEP_MEMBLOCK help Adds support for dynamic management of the memory used by the hardware for storing MTE tags. This memory can be used as regular data memory From patchwork Wed Aug 23 13:13:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83FE9EE4993 for ; Wed, 23 Aug 2023 13:15:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F2E9900033; Wed, 23 Aug 2023 09:15:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1CB2E90002C; Wed, 23 Aug 2023 09:15:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 092AE900033; Wed, 23 Aug 2023 09:15:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EAE7190002C for ; Wed, 23 Aug 2023 09:15:52 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C5DA61C83EC for ; Wed, 23 Aug 2023 13:15:52 +0000 (UTC) X-FDA: 81155416944.16.A87DB29 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id EC1FB20007 for ; Wed, 23 Aug 2023 13:15:50 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796551; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y+6K9RhNe7jOhIChMm8dU4Uk3x6aZgKT0cYlxD+b1zA=; b=gor8k9vR/tEztGW6ozIrPOgYTgMiGvM7SbuMlSVmwLiScUacfpOqmkm7M95dWcyBc3HZEW UXsAeYiT6sc2uHwdCGe8863GYKaRKynjDnO7EyDExKFuw6yaZGE/8na+z96+m/n223MdxM bctjXHdSdXOXZIiFcbbMvGwpXnbOlGc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796551; a=rsa-sha256; cv=none; b=c3nlwuD6XdsjfVR3Q1PMgHTZ4BsIrG99QVWzrMHWigxyjLPK1Z06upa9NBjsQ7RN4KtSXl e+qm6lv9a1KF662c2duDSHsKogog2dscHhm0BJBQCZBQpUlt2qrl/kJqlsJzQvMaFa6Otz LHEAcFl3He1F38lUcJbOkWVkgORIRNg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CC9FC1684; Wed, 23 Aug 2023 06:16:30 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8E4913F740; Wed, 23 Aug 2023 06:15:44 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 16/37] arm64: mte: Move tag storage to MIGRATE_MOVABLE when MTE is disabled Date: Wed, 23 Aug 2023 14:13:29 +0100 Message-Id: <20230823131350.114942-17-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EC1FB20007 X-Rspam-User: X-Stat-Signature: k7bf4n68ntt6gqz457nbu1d4q88qu5cw X-Rspamd-Server: rspam03 X-HE-Tag: 1692796550-15317 X-HE-Meta: U2FsdGVkX1/0FY2nBsR+Fg1YkiFcfC6e9QHWy8hV6E1YdOBYKOj51YQhu/USbFlriCEslimKuotoo7mJE0WLxBDMAQNGMhnmvwpVf8oq88V2cUrtP8DPI/22WQ64yqmcEKf7oYQuFWHwnvssyHqP0RM9LSpILywtQOv8b5ywKajmyb85jSMxoQ+HMd9Hq1hgFQPZqUeaJ5azoSZ/eO2/bgkkd+KUmuFJ85DcBxjhpGCcKpvcZ9ATqNgJyhM8ZhWfX2L5pWWWyFn6j+FJriU2cJTpZTBT9Aqa2oUvEviCrjwb1Sgcr9vHdWi1hxdIPxoT2fG/fxEd/7j9atengeN48HzLSknoHZjwTQ3CCly3jZuJsqN+t2N2/SZXlBhWQ4+1rQop1HOltjl+2hcb0suDJW1O7Mf/wriLrxQe5pFhgN3/UJh39qv0gZd8WFVhd0xIhkSNlaWFzSIIBpR1nHDEqa4kn5iVsYFVjdahByazUNUJj0xTzDU1TnvrMNcQ2UWXdn8Xybd/1p7NtWXPSVWkI0MWN2ux1r0if5hIEHbrv++cHIEmZl0yRCt8DtH/bG9Ax4ZXo5RmwoMZ7QSFmEvBoJgjBPp0UWsF77JQhxUioS6ptQsf2x8JmzkFJrnlgoWty7R3QT2pBi19kAGJqO3mUc4YfJN8hNWkYdXM2+JNo4tXEqqgomr/16kiJORi8vKZXzUN0FFKayVNrFgbXfKGcqVzF7ph6ckySyGaudYFzwbpgOmqP3QZNIYUAwcQ9W0P4EDapswJHhlZP+czeiCpsBoLA79NFtuiJMFGWcR1e/RjWfDUDe14kg0JF8h56J8kdAfCIkhKjAd0Z4HUbrp6JR0N+Q/A6KJF5VDVA89WAWhQ5pmXm6olOWAVZOFcXD2jnlT6Q75YefRgxdzNhjDFT+949XrwvGUf6Y2V7VnAmJKN9bPPvf7FINAmvY1M0gMSXlfDCn70mw5GaXelJ6h ThodZdQS 09YiPmW/JBd7z5CrooBK3npCCp790Upxfpo0BJRYHUZ2MALZrH7xC6RiAWMyPFFGH3Ijt5QLNDOYuDYq3anW1aXfcaA+KIJHpKR/mFMdHvkBu31WQR/AormkTyWdRX0MFvRy5hwHypTOC5WBf5i+QYbSfcWW2pKSjvkWiFOh6lGIGWi9zAcQmEyyy0OvRxpavRBpS67LaXj4HNY4rT98J2R08XIEGcbRo8aL4BINCXyUvar+P5uc/G7iWUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If MTE is disabled (for example, from the kernel command line with the arm64.nomte option), the tag storage pages behave just like normal pages, because they will never be used to store tags. If that's the case, expose them to the page allocator as MIGRATE_MOVABLE pages. MIGRATE_MOVABLE has been chosen because the bulk of memory allocations comes from userspace, and the migratetype for those allocations is MIGRATE_MOVABLE. MIGRATE_RECLAIMABLE and MIGRATE_UNMOVABLE requests can still use the pages as a fallback. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 18 ++++++++++++++++++ include/linux/gfp.h | 2 ++ mm/mm_init.c | 3 +-- 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 87160f53960f..4a6bfdf88458 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -296,6 +296,24 @@ static int __init mte_tag_storage_activate_regions(void) } } + /* + * MTE disabled, tag storage pages can be used like any other pages. The + * only restriction is that the pages cannot be used by kexec because + * the memory is marked as reserved in the memblock allocator. + */ + if (!system_supports_mte()) { + for (i = 0; i< num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + for (pfn = tag_range->start; + pfn <= tag_range->end; + pfn += pageblock_nr_pages) { + init_reserved_pageblock(pfn_to_page(pfn), MIGRATE_MOVABLE); + } + } + + return 0; + } + for (i = 0; i < num_tag_regions; i++) { tag_range = &tag_regions[i].tag_range; for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) { diff --git a/include/linux/gfp.h b/include/linux/gfp.h index fb344baa9a9b..622bb9406cae 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -354,6 +354,8 @@ extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); +extern void init_reserved_pageblock(struct page *page, enum migratetype migratetype); + #ifdef CONFIG_MEMORY_METADATA extern void init_metadata_reserved_pageblock(struct page *page); #else diff --git a/mm/mm_init.c b/mm/mm_init.c index 467c80e9dacc..eedaacdf153d 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2330,8 +2330,7 @@ bool __init deferred_grow_zone(struct zone *zone, unsigned int order) #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ #if defined(CONFIG_CMA) || defined(CONFIG_MEMORY_METADATA) -static void __init init_reserved_pageblock(struct page *page, - enum migratetype migratetype) +void __init init_reserved_pageblock(struct page *page, enum migratetype migratetype) { unsigned i = pageblock_nr_pages; struct page *p = page; From patchwork Wed Aug 23 13:13:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BC31EE49B0 for ; Wed, 23 Aug 2023 13:15:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30454900034; Wed, 23 Aug 2023 09:15:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B43290002C; Wed, 23 Aug 2023 09:15:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17C3C900034; Wed, 23 Aug 2023 09:15:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0968290002C for ; Wed, 23 Aug 2023 09:15:59 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B62171C81F7 for ; Wed, 23 Aug 2023 13:15:58 +0000 (UTC) X-FDA: 81155417196.28.BD30DE2 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id F1D45C0023 for ; Wed, 23 Aug 2023 13:15:56 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796557; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ibseDnUSNnUHw0ddS9Rgi0iYzaH2wNqC2tvgyGRHr8=; b=sBlj0K1NSCyRnWG1bUTJgnoFIMcKlVPZSzpeZrM+0kXa1TLvcXabU6pTTCY5nBjw8MwmSD SXuAGvsJLC2lNbJkzNT4K3XaSXlvRlN3h3HDTF78CDyqocFy/b+9CSJvLh1L6xY6l/EzYO KWhhk0by+ROsvZO1HoQhnAUC+8M41Gk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796557; a=rsa-sha256; cv=none; b=WKuGxK3ZvVESxZMH69h+CqxuGNqexYgi7lKp0b1Blxgni4Mk6lkvskuUWzoFtvMcF3jnYn VccGcyBUQGqGU9tkpWcHSe1A9JaqYt62EqvkuAnzIPOUZiLILUvFTgjyasXh4HXKBeHY2K Q7phMcCawfLf5+tGFatVW2w3Cdlshzk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ECD8715BF; Wed, 23 Aug 2023 06:16:36 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8BF703F740; Wed, 23 Aug 2023 06:15:50 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 17/37] arm64: mte: Disable dynamic tag storage management if HW KASAN is enabled Date: Wed, 23 Aug 2023 14:13:30 +0100 Message-Id: <20230823131350.114942-18-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F1D45C0023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: dpf64kcb6zasrd3zn54b573xexsebgzh X-HE-Tag: 1692796556-241486 X-HE-Meta: U2FsdGVkX1+AEfqjnrmxTTkQjimEYyEI1KX/+Ns45kk6vNsXzk4DBOv+PFBjJSxIp/ULM2sdKelScx9W30qlorIjwRYM0EzKicAsJsDdBQR7EidAh8F1LDuQdgPFYzk20o6peq+0CwBG9rrQQdUhIfueNoRHb+yLqwmbKUy8+Nmap1d8qAicgy6ATINY6aMbJcZup9tFxf/LHz/OkqcLWoQS3WfXw/8gmH9tPvUiq33l9nYMi9SOXrXWzQ2zbAv45yym/JD/Y4niHN/cy0pJA18jUd8AC++R6GYviOU9Uw/NPybYBREYUGwYgx0ficABb3jj9xuCv3vYE0bKAQiN0Wnfe2z2Lm+EIJY2ORdafhCeMZpNij6YK2PIA5XR+a+Ew67BNeSE3iBqnVh78GaYMrq4JsERbFSVelukiH390yrrd8j2J873/jnd64ViWB32i9vZk8ufDus2zhrvVV/eee3shdAIO23Z2yVSIMtQJaFAuWzcCPnck/Zw9FB/nGgWhoZOgqTg43AQNWC3v8xv4GnX8OSlR4V61UmzEatj7WqsOShNAUPFjd/NY3O46ph3tr5EyCgI5U+P6spBI5c002i54AbN5CJFB7aNdHZK6WKKeRDLfPFp/nR2RTIudQ9QZoW9cvyMI1T1E7guReZd4ggWmaC21J8QWYKKcWyybdyO5HMQm0JXSxg2iTeSWyj5Z2jFNoM2CMK5G2gSnDObwbiWfTxb7cnaPUZ32eDuwxnTKRgnoI7gh0oNTmib1EY9/uqhUkNQC+GqTkS136i75A2fbBFNCRYhqFnXwzmiCa1g7DrQfMGmHKTQOv5C+b0VL/VzN0MfjcrEuukVYcmQbkdUPIMtrVWxBnpZs9wcMimJaKVTJi/0vhMVtzgUTWLQb4Q8bc2ZhOds0Z3bO6HN+VXICYPswKwpI75hx1/Vh76hn1UKzCftNWLGw7Y/KvizfmrDuZ0sD0YvCeG58IX Kx9sTgX2 gGl+DYEHThNu5n3HdB51hzq+/vZf2EucyWidMeNq+uoRTR3sr5hIwK+rRBizr914oWW1kgJpKMpgQ7bI6614caRJFA0t7esH64Q77lu8Cz3EiT6gXdJ5YKeineltFCnICp0GEIb+gzCCAVVaTUFYVHL6Tj9pD/CuuZALGLsqe99Gcwyn0rAZbu4ZZetP1n36bjSwc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reserving the tag storage associated with a tagged page requires the ability to migrate existing data if the tag storage is in use for data. The kernel allocates pages, which are now tagged because of HW KASAN, in non-preemptible contexts, which can make reserving the associate tag storage impossible. Don't expose the tag storage pages to the memory allocator if HW KASAN is enabled. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 4a6bfdf88458..f45128d0244e 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -314,6 +314,18 @@ static int __init mte_tag_storage_activate_regions(void) return 0; } + /* + * The kernel allocates memory in non-preemptible contexts, which makes + * migration impossible when reserving the associated tag storage. + * + * The check is safe to make because KASAN HW tags are enabled before + * the rest of the init functions are called, in smp_prepare_boot_cpu(). + */ + if (kasan_hw_tags_enabled()) { + pr_info("KASAN HW tags enabled, disabling tag storage"); + return 0; + } + for (i = 0; i < num_tag_regions; i++) { tag_range = &tag_regions[i].tag_range; for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) { From patchwork Wed Aug 23 13:13:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C155EE49A3 for ; Wed, 23 Aug 2023 13:16:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BAAF280038; Wed, 23 Aug 2023 09:16:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06AC390002C; Wed, 23 Aug 2023 09:16:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9B15280038; Wed, 23 Aug 2023 09:16:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D1CFD90002C for ; Wed, 23 Aug 2023 09:16:05 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B4D6CB0D6A for ; Wed, 23 Aug 2023 13:16:05 +0000 (UTC) X-FDA: 81155417490.19.F0A2B27 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id EE35718002A for ; Wed, 23 Aug 2023 13:16:03 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796564; a=rsa-sha256; cv=none; b=mPMl9X8vAlt82i5ZiKYlPHoc+KJcGnCCQ7+E+s9jbM92PCREqTQlUMpRZ1aV3tKTx+KjH+ KyPsnFWsBI9PWmTy+igFT88HmG3S37eN8BFNkR+NQlyC6yCeH0ODSvpg2fKSeOO3vfErs+ ivTGRfE0ZSZkGyOChpN+zZGTVSQomjw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796564; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GFBau2s/4i9ddIgPVNLkTwGwLRQklrxctctrBRRRHXw=; b=gL97kU1hyq80ncrypiattvKa99eq2jhb8/qtJtQZ/zolKzTzHSUYAuPc0CUHzkIWfvOyao mBh9NHT62GWYPYI2t4jVJyGMTSco0G/hbR7BOS56fLIW5+VJVVNR7BWezKUqZcTRcbIO38 usTNsHZ39h1t809phVx+ZLQAddH6e9o= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DA46F15DB; Wed, 23 Aug 2023 06:16:43 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A7DD83F740; Wed, 23 Aug 2023 06:15:56 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 18/37] arm64: mte: Check that tag storage blocks are in the same zone Date: Wed, 23 Aug 2023 14:13:31 +0100 Message-Id: <20230823131350.114942-19-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EE35718002A X-Stat-Signature: qdhhri9ygbyuxo17ji1ru1nqjji3hkw8 X-HE-Tag: 1692796563-212676 X-HE-Meta: U2FsdGVkX18H4JAqxNrsg/p0aIN+zaN33ersniR1X3wMdp911+Z/uPKiDBjy2E0obNpR8wUoSYE63XMrWAVR1Inm55+Vp6uj+Vs16B6s4xo1MmcOSfPPGdAismuUoGpfethYLlGWzEWX68zDiaKAe0UC6TE9l1b243AcnKptAVQWjZCLM8YqbczrAj0/EBV+dQgP3wqm5RYoWzbgrDlcU3U33zp2bgtDOPSAI3N8I03i5NUp0M+xiWXawlxKE7thP9jPO7Zln2erd9pQSO4Xbd6c2a8PzFj+2VdsA4rJ5VdD3Z+I3nXe2HSzdvojmo+UpUm04xw3PFy7DPyIW62A7yL2UenCxBqf6dW5xvFPlhH4B6wi2B1ecEEsfxjcNY8KflbzCiGFTwKiQhNL//R3/eiHnmi0Gz18Nv+wkX2dJehyeVoZedZGp0NZ8ZdC0OWQNqKa9bd3ZleNX0e9tMaQ3Z7E48F5Lzz/vQ1P+EAVwXgSFkuHoLy/uqO7+MQ6FWyd+S+Vg5l4/XHAZkyuwkw3uuZHXOGmXDp1KBf3n7iNeL92NLSceuZXkUjGojcNc8bAJTxtrn109df5xE93v2XhiMEmzD/RZTgYyuZF6Tz1u+R161xVosiL3Qz7uQ7qhd1Xr8iUL7bXfKYMpR4qEAObN/ooiUAtYeu+LTYPPg1Yi3Qht9WMb/Aqw0F0F2qM4Q6Vq1P+xsbkME8f4Vjy2QIQgoQtG8/iqYn4EEqow68APRyIUdRaGO3MDpGfE/ZhhONb8XBdLhNcDbbUyTwNlZ9cRmFNUhJhj+47FEaPQKOe0/m2z10ehsZwYeGr9KO8y850+UFPyxW6enH0JOr4zmBo41x3O4MIqqUydq5YFD2mQVW7HVUuiZwN6FvSbh+nJxm1jtmO0InDKItH6kh+6q06d3ZPJ/FisZPbR/9PR8fQ5mzsTpn+sPLSceOeHQDJg0t9f/LIb1AabKC7QgD67Ao dAliExsI PfDoRj1NIbEQPZzG2Nh7XO9LZ3INffzy6a7TjqnVNMAN3cFnYruVp8RF57Q7gzyux+O8CrvGQBxm5mSNFvesb6B+r0Yk7n2Y+dczgb8o0IG7Z+6fjf6IWu8qV9fi9nqePdJ/nprDcnkcPeUO6MRiB0Wr/qJ9r5392WUrC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range() requires that the requested pages are in the same zone. Check that this is indeed the case before initializing the tag storage blocks. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 35 ++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index f45128d0244e..3e0123aa3fb3 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -269,12 +269,41 @@ void __init mte_tag_storage_init(void) num_tag_regions = 0; } +/* alloc_contig_range() requires all pages to be in the same zone. */ +static int __init mte_tag_storage_check_zone(void) +{ + struct range *tag_range; + struct zone *zone; + unsigned long pfn; + u32 block_size; + int i, j; + + for (i = 0; i < num_tag_regions; i++) { + block_size = tag_regions[i].block_size; + if (block_size == 1) + continue; + + tag_range = &tag_regions[i].tag_range; + for (pfn = tag_range->start; pfn <= tag_range->end; pfn += block_size) { + zone = page_zone(pfn_to_page(pfn)); + for (j = 1; j < block_size; j++) { + if (page_zone(pfn_to_page(pfn + j)) != zone) { + pr_err("Tag block pages in different zones"); + return -EINVAL; + } + } + } + } + + return 0; +} + static int __init mte_tag_storage_activate_regions(void) { phys_addr_t dram_start, dram_end; struct range *tag_range; unsigned long pfn; - int i; + int i, ret; if (num_tag_regions == 0) return 0; @@ -326,6 +355,10 @@ static int __init mte_tag_storage_activate_regions(void) return 0; } + ret = mte_tag_storage_check_zone(); + if (ret) + return ret; + for (i = 0; i < num_tag_regions; i++) { tag_range = &tag_regions[i].tag_range; for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) { From patchwork Wed Aug 23 13:13:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 668C4EE49B0 for ; Wed, 23 Aug 2023 13:16:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E28A728003C; Wed, 23 Aug 2023 09:16:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB0A3900035; Wed, 23 Aug 2023 09:16:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C795428003C; Wed, 23 Aug 2023 09:16:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B9394900035 for ; Wed, 23 Aug 2023 09:16:12 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 696288010F for ; Wed, 23 Aug 2023 13:16:12 +0000 (UTC) X-FDA: 81155417784.09.6FA84CC Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 735C5180043 for ; Wed, 23 Aug 2023 13:16:10 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796570; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vclm0xvSy/zaNzjMwsfX1Mqpruo7FlHCQh6iPYLZLAs=; b=tGe+USmdG8kcYfluTfCLPJebQWw5mbWiFFeCwgchMnAlRgrIwTuO9ASnhhpvtvE0b8gmzV kFtpoWEgXG08WvTpn6OJI23rkHCR7+xVTlPdFz+66E6G8Pk6r+Ujfyne7CC3/UdWAx34Tu qAjJXAHTQKZSMVyKSaLeoGBzwl2QLGM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796570; a=rsa-sha256; cv=none; b=g0vUj9QjD2CNcgzdHd8Jo/wRABd+6ZwhMsM+THro0kMSQrJcj21Ii2KtZCX5naBFTFmkbb LOXnT2HeevS25DVyYo6uO8QTPODep8VEw/Q2QKfJwEGUeFjC7yztOd8Dk/Gn4Dav5ElFKt +z1TcstJ8eQdhrK870wIm6hqeCTgk/E= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 472231688; Wed, 23 Aug 2023 06:16:50 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 99D093F740; Wed, 23 Aug 2023 06:16:03 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 19/37] mm: page_alloc: Manage metadata storage on page allocation Date: Wed, 23 Aug 2023 14:13:32 +0100 Message-Id: <20230823131350.114942-20-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 735C5180043 X-Stat-Signature: rr8tyazuw54ktkfinj9dmnzmnixykt5j X-Rspam-User: X-HE-Tag: 1692796570-103654 X-HE-Meta: U2FsdGVkX19T/ol4eBCsTCWcvE66XSN/vT6AorGfZRFU08YQqxhoSt7jxoHnFjPhVicqW07GUJ5jamMBKCrA3NvsFKfeDaHSwVCP9t8nBdqQC6SSH6IfRAddK3Z9vZsnYjx1CYEqmDbZRL210I9In7XdU7RlCujmQ+FiylSrMi54kLB57FJ2iX5MpjMOmT7v5rbu+YnNrP08+hrBJLP/T3EP3I+a5LmJASP8inPsgJz2ZK4T4npBr2upF5cckMolr7n1FQAs+tt54CF0qgvs2dWtvbJC/xb1zuXIaiCyrEk913stJnvylyxW7aGhJPvvAZ0lvrcoxKVwgTx4FbGVNaLN3LhraS0x+bKAa9pjJ99U3P9rB5bqGULLP0FybM9bYapyCCbwDmBa4xhBMfY1d/HDxwfVf6YTgmHBdpC8InP0HR+2nNU8u3uhlVHMwwoBEsBJ1T5bxROhmEnyM/RHS4FLxAvvFiWYk9SNsZvwp8Am4g78ExFqA7DpycJPnz0LqQY5LpbUG+mafv9bU7Kv5RHaOX+y3oxBqQp05biLnB0lcbE/s/RlwhQE17BVN5bKM7opVptsgqAqpstqp//qIR3hqZ+3aS6jn/u7oTgTyFAcb1IdCD5ot9LrJIf7iZ80jGjhzSylScNanfT9I4+UykIQS397sQQHcxeMwLmPO+k/7IpCcXR3a51Rbw4AmA7L+IVsw8opL5JIqL7Wd5kR0iND3U2zB736CzZU3tyKaUzL7+uDV5DGYRypQCNYAw3BD75BaWEm3rAFjt79aDfyuMJkv4+BAHxAGKtlyGqmAhT7L2+iGcCrukJ3H5sA/QlfvCzg5GKNZvvsvv/0Yk6TKrNocMf+TTGCcXWlTkWiIurBBL1OLH0ZyPpfKcX6qYKonyPbMFk64YTSMuAxaNhsSMbIOoUq0cBVZFfDzB9rPpSI7n3GQln4WCyjDLmNHRSyIydKbf7UeSlmIBaOGmf R3MkNTte 1FyV6Mej5kc3xTwuQM4AFcKwB7XmcTKda8oH0NLX8tYuMDsywIW3PH75i/IkBKuRnlxGGE67YVUziNVKADTkEKAKSOo2onhEsCqZtXH/fe2TnXE2yjBY9LCsNEh8RFWHuWcaesZV/Ug9rkSYq8JBReGK6TrrF3j23c6enY1rvVPWxlAo/BjjBQaYyRjmmE5cnjZig8pnFhnSa028= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a page is allocated with metadata, the associated metadata storage cannot be used for data because the page metadata will overwrite the metadata storage contents. Reserve metadata storage when the associated page is allocated with metadata enabled. If metadata storage cannot be reserved, because, for example, of a short term pin, then the page with metadata enabled which triggered the reservation will be put back at the tail of the free list and the page allocator will repeat the process for a new page. If the page allocator exhausts all allocation paths, then it must mean that the system is out of memory and this is treated like any other OOM situation. When a metadata-enabled page is freed, then also free the associated metadata storage, so it can be used to data allocations. For the direct reclaim slowpath, no special handling for metadata pages has been added - metadata pages are still considered for reclaim even if they cannot be used to satisfy the allocation request. This behaviour has been preserved to increase the chance that the metadata storage is free when the associated page is allocated with metadata enabled. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 14 ++++++++ include/asm-generic/memory_metadata.h | 11 ++++++ include/linux/vm_event_item.h | 5 +++ mm/page_alloc.c | 43 ++++++++++++++++++++++++ mm/vmstat.c | 5 +++ 5 files changed, 78 insertions(+) diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index 3287b2776af1..1b18e3217dd0 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -20,12 +20,26 @@ static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) return !(gfp_mask & __GFP_TAGGED); } +static inline bool alloc_requires_metadata(gfp_t gfp_mask) +{ + return gfp_mask & __GFP_TAGGED; +} + #define page_has_metadata(page) page_mte_tagged(page) static inline bool folio_has_metadata(struct folio *folio) { return page_has_metadata(&folio->page); } + +static inline int reserve_metadata_storage(struct page *page, int order, gfp_t gfp_mask) +{ + return 0; +} + +static inline void free_metadata_storage(struct page *page, int order) +{ +} #endif /* CONFIG_MEMORY_METADATA */ #endif /* __ASM_MEMORY_METADATA_H */ diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index 8f4e2fba222f..111d6edc0997 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -16,6 +16,17 @@ static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) { return false; } +static inline bool alloc_requires_metadata(gfp_t gfp_mask) +{ + return false; +} +static inline int reserve_metadata_storage(struct page *page, int order, gfp_t gfp_mask) +{ + return 0; +} +static inline void free_metadata_storage(struct page *page, int order) +{ +} static inline bool page_has_metadata(struct page *page) { return false; diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 8abfa1240040..3163b85d2bc6 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -86,6 +86,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_CMA CMA_ALLOC_SUCCESS, CMA_ALLOC_FAIL, +#endif +#ifdef CONFIG_MEMORY_METADATA + METADATA_RESERVE_SUCCESS, + METADATA_RESERVE_FAIL, + METADATA_RESERVE_FREE, #endif UNEVICTABLE_PGCULLED, /* culled to noreclaim list */ UNEVICTABLE_PGSCANNED, /* scanned for reclaimability */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 011645d07ce9..911d3c362848 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1111,6 +1111,9 @@ static __always_inline bool free_pages_prepare(struct page *page, trace_mm_page_free(page, order); kmsan_free_page(page, order); + if (metadata_storage_enabled() && page_has_metadata(page)) + free_metadata_storage(page, order); + if (unlikely(PageHWPoison(page)) && !order) { /* * Do not let hwpoison pages hit pcplists/buddy @@ -3143,6 +3146,24 @@ static inline unsigned int gfp_to_alloc_flags_fast(gfp_t gfp_mask, return alloc_flags; } +#ifdef CONFIG_MEMORY_METADATA +static void return_page_to_buddy(struct page *page, int order) +{ + struct zone *zone = page_zone(page); + unsigned long pfn = page_to_pfn(page); + unsigned long flags; + int migratetype = get_pfnblock_migratetype(page, pfn); + + spin_lock_irqsave(&zone->lock, flags); + __free_one_page(page, pfn, zone, order, migratetype, FPI_TO_TAIL); + spin_unlock_irqrestore(&zone->lock, flags); +} +#else +static void return_page_to_buddy(struct page *page, int order) +{ +} +#endif + /* * get_page_from_freelist goes through the zonelist trying to allocate * a page. @@ -3156,6 +3177,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, struct pglist_data *last_pgdat = NULL; bool last_pgdat_dirty_ok = false; bool no_fallback; + int ret; retry: /* @@ -3270,6 +3292,15 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, page = rmqueue(ac->preferred_zoneref->zone, zone, order, gfp_mask, alloc_flags, ac->migratetype); if (page) { + if (metadata_storage_enabled() && alloc_requires_metadata(gfp_mask)) { + ret = reserve_metadata_storage(page, order, gfp_mask); + if (ret != 0) { + return_page_to_buddy(page, order); + page = NULL; + goto no_page; + } + } + prep_new_page(page, order, gfp_mask, alloc_flags); /* @@ -3285,7 +3316,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, if (try_to_accept_memory(zone, order)) goto try_this_zone; } + } +no_page: + if (!page) { #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* Try again if zone has deferred pages */ if (deferred_pages_enabled()) { @@ -3475,6 +3509,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; + int ret; if (!order) return NULL; @@ -3498,6 +3533,14 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, */ count_vm_event(COMPACTSTALL); + if (metadata_storage_enabled() && page && alloc_requires_metadata(gfp_mask)) { + ret = reserve_metadata_storage(page, order, gfp_mask); + if (ret != 0) { + return_page_to_buddy(page, order); + page = NULL; + } + } + /* Prep a captured page if available */ if (page) prep_new_page(page, order, gfp_mask, alloc_flags); diff --git a/mm/vmstat.c b/mm/vmstat.c index 07caa284a724..807b514718d2 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1338,6 +1338,11 @@ const char * const vmstat_text[] = { #ifdef CONFIG_CMA "cma_alloc_success", "cma_alloc_fail", +#endif +#ifdef CONFIG_MEMORY_METADATA + "metadata_reserve_success", + "metadata_reserve_fail", + "metadata_reserve_free", #endif "unevictable_pgs_culled", "unevictable_pgs_scanned", From patchwork Wed Aug 23 13:13:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59CB7EE4993 for ; Wed, 23 Aug 2023 13:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD9A428003D; Wed, 23 Aug 2023 09:16:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D89C9900035; Wed, 23 Aug 2023 09:16:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C78C928003D; Wed, 23 Aug 2023 09:16:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B537E900035 for ; Wed, 23 Aug 2023 09:16:23 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 832BB40100 for ; Wed, 23 Aug 2023 13:16:23 +0000 (UTC) X-FDA: 81155418246.20.5F167FF Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id C65658002C for ; Wed, 23 Aug 2023 13:16:21 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796581; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cxIim5XoZri2irZTfRyizJUXpnkJfHUOphqzC+7dZ2g=; b=aB0CU7xpWWvtHLqW2Tnj9tJr8hBCw9R3IW8ExsYz8HSUA3ieArP1N8CMv3qLUUepM80Hew +uFSmE9SSHwHalqqBI98cghN3X9pk03w/WbblAjk3hDZ4RTB0Gkw4iGRZYg8v/PbyixcN4 E7YtFYRUbsRYxYZG8fyp4WpCoXN+2ns= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796581; a=rsa-sha256; cv=none; b=jwoYWn0Ml4ux/1FFMZcFMKUCHW/l+oxlW3NI7Bsqyaw8wseK2k8JdJGB+2i0wUmxEZazQP xprrmZTZD0D/9QMzSSAdNtztwfTs7pMbKfkj3knjUcOPuAYtx+b7dkpeuTkeeDePLleeHJ CMSloEoomZbjiVhLGbA1e4rSDhIkM58= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 93C321691; Wed, 23 Aug 2023 06:16:56 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 033CD3F740; Wed, 23 Aug 2023 06:16:09 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 20/37] mm: compaction: Reserve metadata storage in compaction_alloc() Date: Wed, 23 Aug 2023 14:13:33 +0100 Message-Id: <20230823131350.114942-21-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C65658002C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: j1mx1ga4sbneyapxkmd485hu6qkxo6r4 X-HE-Tag: 1692796581-630870 X-HE-Meta: U2FsdGVkX18kEcQL3lbtMSwshYD1FNHv5kEnlpYbddB3p4qMaUnQPDRrDPWcrA5ktEwBrv6HX07LZjWkeGrt1muA/q+1QcUowD3Y+yrUhlKiyex/XfLZG7imxN07/QxfKdyrC+xbN5qDBZN5Eq1+ENkaiwvyZKxevWdGqQX/0zVcrAaVeYww9975Q3022HC+vnci3S6+WY/lDhS9qIEnPFdZbHx5nNmV5sKO+q19ao9jMG08UHvLg+R+jTJa7ntSB4fFklRZxOTWwetO4JxIo/AMQWmH2mMTMq0840H6crVoEtqvxdpiwgg2WUlhtO1QPxjvf7rwwvN7bVN+RexxJAawXKVstgfrjMWS4wT0XmV9GbLoO9nltKeUzOVxksutmbSTrHyLg1lTQUXgyh86402NcsNM+mDtMcNOPsFE2GUh+CdPmMiMvpayWIMqXhdBJ+xChnUiN+RoItx5+UgNqctuTtPX6LYDRWbcuyVfRAVBTHMvw2Smp4ljfB9E505gU4AR/hWjWJuAkovLxGjW5okXZR9FLQMj1qnE/kDLfyQiO3jqG6XicKVtDG91S/Ge3TQaOyslidoHEohbOeAfNSqJLIWRrLUklR/hgdbDXRTeg3OBoXy6ec8nHK67iPJj4UNRD4gpPIzDffHo9UX9BcNh8kPUGu7sNADsaZwiqDxQrq7WJDqS1imfjM2L/j+OtuIwFdz5pQnMgudv26cpe8/Zx7M5gbkFd5KtBhL/YmcBt7Q74+9GjDEAqUR9IEIj9iiYD2KGa/uwQDdkLAH4ttGIYLt+kQJFMYwYMmxIaxJ8kGLqeMpX8HUaOvzpIop1P6ynYpE+vIrLehAveOUN8BNM/vUkNB4CB5wRhJBw9n3ka7AMrn0Y9ecE5k+l9SsHcXL5c8zkqpMNHPPWLzkzhoKYSeZr+QFJhD7OcnJ8Dgt0dfDCtVoKYfy6lZsD+/lAwx7MWMrA7N7d5qE8C0Q D36M/lAz gcEAyWgBAhMgZfcEn9ly9kEE+4kzbGIc4GouZpiAevYI67dIU9ONzb/7Ue8W87UHd8vQysl0dB9U34fs53tW4bhKprywy2Hw/2J5H7KV6smGYQbQexHvUfM6OLSkx6Cbg6oBU63vLHXPJSzOV/4uDBkiOOHJODHy4Hhf/7C6T76y+WA0ypuVIdRu9wliNJg7duLiwiZKHNPHvLbA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the source page being migrated has metadata associated with it, make sure to reserve the metadata storage when choosing a suitable destination page from the free list. Signed-off-by: Alexandru Elisei --- mm/compaction.c | 9 +++++++++ mm/internal.h | 1 + 2 files changed, 10 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index cc0139fa0cb0..af2ee3085623 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -570,6 +570,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, bool locked = false; unsigned long blockpfn = *start_pfn; unsigned int order; + int ret; /* Strict mode is for isolation, speed is secondary */ if (strict) @@ -626,6 +627,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, /* Found a free page, will break it into order-0 pages */ order = buddy_order(page); + if (metadata_storage_enabled() && cc->reserve_metadata) { + ret = reserve_metadata_storage(page, order, cc->gfp_mask); + if (ret) + goto isolate_fail; + } isolated = __isolate_free_page(page, order); if (!isolated) break; @@ -1757,6 +1763,9 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) struct compact_control *cc = (struct compact_control *)data; struct folio *dst; + if (metadata_storage_enabled()) + cc->reserve_metadata = folio_has_metadata(src); + if (list_empty(&cc->freepages)) { isolate_freepages(cc); diff --git a/mm/internal.h b/mm/internal.h index d28ac0085f61..046cc264bfbe 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -492,6 +492,7 @@ struct compact_control { */ bool alloc_contig; /* alloc_contig_range allocation */ bool source_has_metadata; /* source pages have associated metadata */ + bool reserve_metadata; }; /* From patchwork Wed Aug 23 13:13:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94FE6EE49A3 for ; Wed, 23 Aug 2023 13:16:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26837280040; Wed, 23 Aug 2023 09:16:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F0CB900035; Wed, 23 Aug 2023 09:16:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B914280040; Wed, 23 Aug 2023 09:16:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EFB8A900035 for ; Wed, 23 Aug 2023 09:16:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B45C280114 for ; Wed, 23 Aug 2023 13:16:26 +0000 (UTC) X-FDA: 81155418372.11.1FB3F7C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id DDC02120024 for ; Wed, 23 Aug 2023 13:16:24 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796585; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Epfbq3/qTmsKFVi06SipgNFOGiN49JPuFxN4Fsx+vdw=; b=bH96fv/xSHKrLiI2p+ZzETOJ0sTuEX5sR8csqH2pY/7L7r5GNzyEbf+7nl9SrD/JJkeicR YjVMej3i6MuzLdtV/rWXuMXsPYNeo7OanUKUn/dZutxysRGSyUes6tcLKlWH7fcE7mxUB7 pgGRkSLJRnKJuBYMrpyQInEtqdYp3tI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796585; a=rsa-sha256; cv=none; b=38ctmSfqm71F25IqsB5Ia5r2uLwUF14zq+UjlH27PXLdNXCf89Hk9sL8Xj0wmEEhh1qrIr UFHKPkGmOxh656b9NPMs+rnXuCmPGUAFWqLl2cBG2e/xtWzTuOM7rO0CMu18F8CaMPi9YT 7Np8rSu+ZBiCQW9CSsVIs6cy7qMVu7Y= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C95B2168F; Wed, 23 Aug 2023 06:17:03 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B4A793F740; Wed, 23 Aug 2023 06:16:16 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 21/37] mm: khugepaged: Handle metadata-enabled VMAs Date: Wed, 23 Aug 2023 14:13:34 +0100 Message-Id: <20230823131350.114942-22-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DDC02120024 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: z35it518r3xs199ffz7bycffs8ykrxcg X-HE-Tag: 1692796584-63618 X-HE-Meta: U2FsdGVkX18RjS6MGjOBbNUw7Qr42PeM0H2GdKG4mzXYXnLRMyBJM7JKa9+En0gzWc8ZG6qLRPgPrC3sbvuAVVITDfTQtvnf3+IccDtLtMkiM9MTGJzjF484fTNFUA5+fHOgZee/+zPB8A7c1DjL92KpUj97OKPUF0z/mW99nKu2NzrebvlU4DsN5BTS5nIxYKzpGxH7qERkeFVX897MxnCuTN2x3HZC3tgtt+3yPFZmMCbp8sovNYlt8F4gk2Q7WR5LfiVZYBNFYybZiLd3GdR9Ww23n4rnCiwJ/Rp7demaZSdTaLJSSoO/kq1qxbG8CezevJMo3igRl5HkXus/1ATdvT3C9LA5qvV4dD9ETudR+grfQqC1kWeoFERwCm+lHwVaeTadqUDTQc6spjyrOsjw6rDZYFw1ridmNxCGUhjxMliHAC5lG52kMSKL1JsX2Gtdudm2MuNleWk/QRcFxs/Pt3XFdUij0Ip5cc9qg1gSC+YBCyxeJyJOs6aL7sxJkewLE83CYNPhkHkpFlFP+UbGJZCBQU3AkB3Dc4MivpIftJmkbiYi/B+e5eyDNbRdUYir6oZxKPCfnH3tuJuAaq8qzaG/NUf1TxB2x2ioncxvd73oXeyLtX8fZz+ofMKcpe8t0elHVECA74Ff/JoW3MKaMEw+dXXdITmcBJcWvDXypsC6sH25+geHMK0+QMFNhCI5kYIEC+swk1/1VmiwX6fMsrRGfQarkk8jRT49kPA81Y1neLndAwGNvD6+SfjPDUt9ulClyEzaEaH3/dza+lzjJjYmpll6So/PZZl4xWDJuAojUCiGMc4wyC4C20OgrefHWVi7Tj870yGikpVRkvj5GwCenyTEsvEhC/oZDeOv126efnQ4xkU8GAdgLz5GnaPXbTdojKKW+ud0JO+gnXuz6rYbHTD/G/YBgFvvjKfrrFPGKbFjRV9vwUwqqO5lzYXJl3fUhQX3ih5iTr2 lUpY+nh0 rnGHK0kN6q0a4T7Zhp1FE2QnhVhoSwXHKg4XpHlAgVUNdezGWtCxtGJowmzD8O92r2J/vMGIQVwphZ01/7k77TxC4PnCygyM8PIedm2HIZ++0w6lwR2weG8PX3QTSA0RPX3BfypL01wtGH7v5Ule46crsxiCnSlNSuee5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both madvise(MADV_COLLAPSE) and khugepaged can collapse a contiguous THP-sized memory region mapped as PTEs into a THP. If metadata is enabled for the VMA where the PTEs are mapped, make sure to allocate metadata storage for the compound page that will be replacing them. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 7 +++++++ include/asm-generic/memory_metadata.h | 4 ++++ mm/khugepaged.c | 7 +++++++ 3 files changed, 18 insertions(+) diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index 1b18e3217dd0..ade37331a5c8 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -5,6 +5,8 @@ #ifndef __ASM_MEMORY_METADATA_H #define __ASM_MEMORY_METADATA_H +#include + #include #include @@ -40,6 +42,11 @@ static inline int reserve_metadata_storage(struct page *page, int order, gfp_t g static inline void free_metadata_storage(struct page *page, int order) { } + +static inline bool vma_has_metadata(struct vm_area_struct *vma) +{ + return vma && (vma->vm_flags & VM_MTE); +} #endif /* CONFIG_MEMORY_METADATA */ #endif /* __ASM_MEMORY_METADATA_H */ diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index 111d6edc0997..35a0d6a8b5fc 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -35,6 +35,10 @@ static inline bool folio_has_metadata(struct folio *folio) { return false; } +static inline bool vma_has_metadata(struct vm_area_struct *vma) +{ + return false; +} #endif /* !CONFIG_MEMORY_METADATA */ #endif /* __ASM_GENERIC_MEMORY_METADATA_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 78c8d5d8b628..174710d941c2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -20,6 +20,7 @@ #include #include +#include #include #include #include "internal.h" @@ -96,6 +97,7 @@ static struct kmem_cache *mm_slot_cache __read_mostly; struct collapse_control { bool is_khugepaged; + bool has_metadata; /* Num pages scanned per node */ u32 node_load[MAX_NUMNODES]; @@ -1069,6 +1071,9 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, int node = hpage_collapse_find_target_node(cc); struct folio *folio; + if (cc->has_metadata) + gfp |= __GFP_TAGGED; + if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) return SCAN_ALLOC_HUGE_PAGE_FAIL; @@ -2497,6 +2502,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, if (khugepaged_scan.address < hstart) khugepaged_scan.address = hstart; VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK); + cc->has_metadata = vma_has_metadata(vma); while (khugepaged_scan.address < hend) { bool mmap_locked = true; @@ -2838,6 +2844,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, if (!cc) return -ENOMEM; cc->is_khugepaged = false; + cc->has_metadata = vma_has_metadata(vma); mmgrab(mm); lru_add_drain_all(); From patchwork Wed Aug 23 13:13:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 745C9EE4993 for ; Wed, 23 Aug 2023 13:16:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 045AB900035; Wed, 23 Aug 2023 09:16:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01CDD28005D; Wed, 23 Aug 2023 09:16:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFF5A900036; Wed, 23 Aug 2023 09:16:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C1E1D900035 for ; Wed, 23 Aug 2023 09:16:31 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9E1BF1A006F for ; Wed, 23 Aug 2023 13:16:31 +0000 (UTC) X-FDA: 81155418582.12.2275F0D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id DA77DA000A for ; Wed, 23 Aug 2023 13:16:29 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796590; a=rsa-sha256; cv=none; b=SH8rmuPKc9BmSAxn6GdBkxJTNpyl66nJpow1/fHb/ci4uYBGZhLLUcbLFY0JjdFhkakMG2 KMrZGTenxK6gaViC95E/TV0bDtUt3eEyD6E/rFP4b92vbwR1lvQiYUYhdoBNtCZHYc3euc 9o17nfzQZUHy6Bhtf1/ilIodRH/wojo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4ONpPYaHebA/SF1xz6SHCYgtyIsQhmlhVB3Qoug5JjE=; b=tPkUy/wSMnt0UulEJ3hboPIh4tNYVvdAxecjVdRpeAqlCrM2pWvbRxzTFGKnVDR/JCviq2 5FX0ZmBLvuFKNxpdjvdjT9wWWkptbBCzep4JUPNJP5fHGaUyC/2q0mAvG1uXDFujC5x4Nz g62gLbD4tFhXUHEW3QKgnH32JscJBhY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CC2FD169C; Wed, 23 Aug 2023 06:17:09 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 866CD3F740; Wed, 23 Aug 2023 06:16:23 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 22/37] mm: shmem: Allocate metadata storage for in-memory filesystems Date: Wed, 23 Aug 2023 14:13:35 +0100 Message-Id: <20230823131350.114942-23-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DA77DA000A X-Stat-Signature: rbsdfqxazrcitt7jmka9owmib5nhf5mf X-Rspam-User: X-HE-Tag: 1692796589-286739 X-HE-Meta: U2FsdGVkX18rc+HaSLifUSkEPbLc5Vl0gecGwYo/GBwBFi8S3y9TVNNFMIFyze00uTjJLmrvXTpQCFLiJI4EBcom0hCiabQPs3/a6HdO3zK8uTMGJAh5gWvsuOTA5hEBZzc3CMvTqAu2YEvdgyiWLcH4xA/CSCKK4CA5IaLTg8AvOfx0Ibg9bISYpwQ72BT3nCdJhWYDJn9o0jScwarl0fFdUTW37hw3TtK+HphI1Yp4/XC+3plc669aU2GVo6hTqb7/UmgFrkfs3Ohf6FdcinQbxnJWyRpPxP1qGgc5HdCs8dxaxQ4Hey38Cf5LIOpMVHDdWLQfhP6+VN5nGEUxvZzUUHB/Pf73BNl+nlxYONBM+w0TFCTso+ZRXlj80Bpl6HTcbsbjjcoT116gcswLrQ0vf74zlbx9dFB0qg1aGWkR2Tl0to3m+G73c/SC9Lja15A0CnKnKAqGFoK4i55LW0pDhe4ySaB2LTQU1Mp91fS5FCAdbHYkzBEAk5gGAlEsgFwnEAKcglgjZEm3IXVa9s+yLyL7e+kQdXAGuRMDdg3H06henagh/aqHIH5T52v4FM5p1zvdGdpLP+fjhMLvQqnK0L45TFGeOYpO3rxxguQLUYOuaRX3l7Oxknr5xLlMMCW34Wtmlo2TEFMrR/xYEvXnb+noRtpMWxKaWVZyY2ACm/UzvDPGi/5hOx7OohbnfmyOLmiBTIvk6LLcrh0PqNalJfZ+mKxMc7s7DAYjLXXEs9Rgtuf9Gjf5DniNYKooynMkB4q7YuR32fEqAcO9HVcZHIaka5L9yv7Q2l75hfXsr+qdxcEq21QuRTmYnMB6nIOMgJWqv9m12K4p9CgX/l06Wh5UQG6TGL1nePseXMTIaTW3+up9t1uynb15y7IG+xeQfnwHCRtW0BmuZsbIXETE8EWc5zd8x8gQgoDL6VzpfUJkouqkqenxKCg9iwujZYsHbVxIlxEOV08nZr/ 1baazusB lMMvsLm8h0GgG6mIOB9dsnfL5c7TlSpbLtvcfy9Vx2Z9AlTKODzoZGZ8eTpHA6I5Ubj8VxIv3zXMXTmUP7H36buCEMllWdi+kLDS2Jzx4hoIdc5w24Oj+rL/HwTmz471Oi8X5Hbs1AAmSX06wTTc07+u5BPfQXU4Tje4G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set __GFP_TAGGED when a new page is faulted in, so the page allocator reserves the corresponding metadata storage. Signed-off-by: Alexandru Elisei --- mm/shmem.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index 2f2e0e618072..0b772ec34caa 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -81,6 +81,8 @@ static struct vfsmount *shm_mnt; #include +#include + #include "internal.h" #define BLOCKS_PER_PAGE (PAGE_SIZE/512) @@ -1530,7 +1532,7 @@ static struct folio *shmem_swapin(swp_entry_t swap, gfp_t gfp, */ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) { - gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM; + gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM | __GFP_TAGGED; gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY; gfp_t zoneflags = limit_gfp & GFP_ZONEMASK; gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK); @@ -1941,6 +1943,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, goto alloc_nohuge; huge_gfp = vma_thp_gfp_mask(vma); + if (vma_has_metadata(vma)) + huge_gfp |= __GFP_TAGGED; huge_gfp = limit_gfp_mask(huge_gfp, gfp); folio = shmem_alloc_and_acct_folio(huge_gfp, inode, index, true); if (IS_ERR(folio)) { @@ -2101,6 +2105,10 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) int err; vm_fault_t ret = VM_FAULT_LOCKED; + /* Fixup gfp flags for metadata enabled VMAs. */ + if (vma_has_metadata(vma)) + gfp |= __GFP_TAGGED; + /* * Trinity finds that probing a hole which tmpfs is punching can * prevent the hole-punch from ever completing: which in turn From patchwork Wed Aug 23 13:13:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63FEDEE4993 for ; Wed, 23 Aug 2023 13:16:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 044C428005E; Wed, 23 Aug 2023 09:16:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F374D28005D; Wed, 23 Aug 2023 09:16:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFFB728005E; Wed, 23 Aug 2023 09:16:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D15CA28005D for ; Wed, 23 Aug 2023 09:16:38 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A5BF1C0111 for ; Wed, 23 Aug 2023 13:16:38 +0000 (UTC) X-FDA: 81155418876.23.D389EE3 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 8CB8320014 for ; Wed, 23 Aug 2023 13:16:36 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796596; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4uPh9wMSKhUVzYjVNVvG0TPCuujYvgNW9n6bQKO5bMc=; b=LEmeNbgmNUCL3+NRiyx0lKS/Z+yAuAd7607cTHTdDuzDMdVDWaFBatL3ZveuLyZAPzLlnw ZZ1m1hAdtcaGW8icaNqbgwmRLBhhR0tmqQYUaJwnKvIWMMErsFwxJhzp2G7hhSxoC5BYnR BKHSTTT0OH/hPIOHHo5kbQxC4x85NeI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796596; a=rsa-sha256; cv=none; b=lbxkyYmBB87KAwumypS3EVKINkUGtcmBIASxBsDciI6QiVSHX2g3k2SiHTn4ySUY/rzRMZ JIdgzC4ua67gcMcxDBFjNbUeD+TtaXb65HS9lG2lMWdG83yN1l7wyY8MjyACBuxuTZGRWj rSJT8jxO2UR9PPe6KCLOiL/uLLhDOuk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6F7C11655; Wed, 23 Aug 2023 06:17:16 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 880023F740; Wed, 23 Aug 2023 06:16:29 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 23/37] mm: Teach vma_alloc_folio() about metadata-enabled VMAs Date: Wed, 23 Aug 2023 14:13:36 +0100 Message-Id: <20230823131350.114942-24-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8CB8320014 X-Stat-Signature: 3xwgo9fyhwmox9aacrtod7ar97imq86a X-HE-Tag: 1692796596-857260 X-HE-Meta: U2FsdGVkX1/IjBmDmw2dbsQ0zS1G6PwYttEiokb0RYKoGBt0pX/359LcK83bfZUUkCHe9SbArrtzQbLM1pTickEcR4IqDQSZaFl4JHC+lZYx/zs1xOwznK5Qyx5X75ZQ/euavU97ax9JAopOyqRbrv6nqtCjy0FajNzqrteqxMJBzSMfIsha8BDSgOjmLBJ9gnZKTCQLe+13Q80erRqIiG4H11EERNokX2tnE/B4r+x53I7JP3lTklNf3+4KXSR2ZyTBknWoyCUH/WFHaWzco/mQcS5zVf0fk/qpNzmcaTPTY1hkY6Haa96RccAkJvAV8eMoIH541d7jgvp6jlinOOhTir1XowtI2p5amT69KIrwKrcERtQ8zCiv65qBGPa2ppyenUVDbBC1RPScgUQ336HdCLuNzqMI5hlofegNsox+r4EW2pPyqvZsJDt1bJwlgkxrkHC888FbY0MNFxiLV/6faR9w+cXl+RgYe6fjFdt6O5bIGqaPzAfXFG3Mz4IWgmodQTFvM2LtvjiDorNx8fzLVKce+lgRnbuV1aqo/w6zqKLTmU6PvrZsB0n1uCpJGEl9tL4lB08EKNWgOdwO+i/6KynumLGGuQhL785/ivTg9fdUkUNN+1sWAdZXa7JplKr/Wrw3oM3vOMOR0uhIz15t9Plcb7jwr4G4NkPoaAF7ezi+F9FxNpvG0RNTPtSuUxgdzKeQXT6oNjttUD0qe3hiRQUFHA6nWb4ygd0PqmlAQlCOOClkuMCUJ3/qonFqkwaMsfJfYrqpbDvKCw1uRGObwf4VaGeyowz490seqGJ4tHmNbVKe2cvv4ffZelEYJu+BztbaOQj9dXvwh7274wwkRMTSFffWsvXRfChl1xTfuwSk8Za8QhFLaPy/u2Asv7Ns+5ShPFMeuMkdBEiax2wUqXGznK3RWs+WoS42irZrqY8huLNyvZONqkJ/qHqQl5aT8vD9YN+LmKlenpB GmPMX5MN rXSpTnnEJ6O30/X+Hrv/AkUFpRYVBAOUrxLHepT7R/uIPscvzPTrXtXR3DWsQ7IHGBcw0puuNoSwbzykrqXL96q88z2wXBME9V1yWikzoXxD9KOgVB21WpoNnmh/yS1X574fFbqdd/piG+TgO/bgl3FevWAlSdQNc8fPMBPWPF9AjSw8klwGFXyzJa9mkmsKkg20RrUgq4T8r49Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When an anonymous page is mapped into the user address space as a result of a write fault, that page is zeroed. On arm64, when the VMA has metadata enabled, the tags are zeroed at the same time as the page contents, with the combination of gfp flags __GFP_ZERO | __GFP_TAGGED (which used be called __GFP_ZEROTAGS for this reason). For this use case, it is enough to set the __GFP_TAGGED flag in vma_alloc_zeroed_movable_folio(). But with dynamic tag storage reuse, it becomes necessary to have the __GFP_TAGGED flag set when allocating a page to be mapped in a VMA with metadata enabled in order reserve the corresponding metadata storage. Change vma_alloc_folio() to take into account VMAs with metadata enabled. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/page.h | 5 ++--- arch/arm64/mm/fault.c | 19 ------------------- mm/mempolicy.c | 3 +++ 3 files changed, 5 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 2312e6ee595f..88bab032a493 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -29,9 +29,8 @@ void copy_user_highpage(struct page *to, struct page *from, void copy_highpage(struct page *to, struct page *from); #define __HAVE_ARCH_COPY_HIGHPAGE -struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr); -#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio +#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) void tag_clear_highpage(struct page *to); #define __HAVE_ARCH_TAG_CLEAR_HIGHPAGE diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 1ca421c11ebc..7e2dcf5e3baf 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -936,25 +936,6 @@ void do_debug_exception(unsigned long addr_if_watchpoint, unsigned long esr, } NOKPROBE_SYMBOL(do_debug_exception); -/* - * Used during anonymous page fault handling. - */ -struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) -{ - gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO; - - /* - * If the page is mapped with PROT_MTE, initialise the tags at the - * point of allocation and page zeroing as this is usually faster than - * separate DC ZVA and STGM. - */ - if (vma->vm_flags & VM_MTE) - flags |= __GFP_TAGGED; - - return vma_alloc_folio(flags, 0, vma, vaddr, false); -} - void tag_clear_highpage(struct page *page) { /* Tag storage pages cannot be tagged. */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d164b5c50243..782e0771cabd 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2170,6 +2170,9 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, int preferred_nid; nodemask_t *nmask; + if (vma->vm_flags & VM_MTE) + gfp |= __GFP_TAGGED; + pol = get_vma_policy(vma, addr); if (pol->mode == MPOL_INTERLEAVE) { From patchwork Wed Aug 23 13:13:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C68EEE49A3 for ; Wed, 23 Aug 2023 13:16:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B9E228005F; Wed, 23 Aug 2023 09:16:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06AE128005D; Wed, 23 Aug 2023 09:16:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E750828005F; Wed, 23 Aug 2023 09:16:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D827128005D for ; Wed, 23 Aug 2023 09:16:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B5ADB160120 for ; Wed, 23 Aug 2023 13:16:44 +0000 (UTC) X-FDA: 81155419128.12.4DF2114 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 037A080003 for ; Wed, 23 Aug 2023 13:16:42 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796603; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SLpmhuiLqoeOoKiHs0XPEs55KygN5SxAbZ2C12O4yl0=; b=7DSQgNE1AoOoI8Mfb1FHri17C7xMaHkqp1rsKw8+gMbW93fCNDs3PmgN313GP6WnO2s7Vj 9hXKhg3BE07XwFKceU70Wp9LhmM7QP+XQFNduXdI96CKSzVJcElybfNEaIKjicFLJl4huu GFLNYzF8sgDvtIN/n2xZ/85/TvKWpVo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796603; a=rsa-sha256; cv=none; b=ljkS6xXLzrIXkngmm4Pt4hzubLNiV7K49EKqZ1UoXJ9DZEg1dSH1UTKNHxCM4sxjWi9mCv AshIuVYWA//KDH5Hk6hZJIJs+wh+7J/Wt4JqaJA1I6aEp1izlhmtw98qbLBYpmLN92HpSa QJ/dJpzo+NmtaCjhVfUQcUuRafg2lwc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EB6C41692; Wed, 23 Aug 2023 06:17:22 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2EA123F740; Wed, 23 Aug 2023 06:16:36 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 24/37] mm: page_alloc: Teach alloc_contig_range() about MIGRATE_METADATA Date: Wed, 23 Aug 2023 14:13:37 +0100 Message-Id: <20230823131350.114942-25-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 037A080003 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 4xr1e9ze3xcqyrzmwqjjemnjmoexu4ft X-HE-Tag: 1692796602-250729 X-HE-Meta: U2FsdGVkX19DQroeERV3AHsjzKzBo5XLyongmtbFb4N1NLRwdfa0AgZkLLrU9is0QkgP5sGB73ubEuncU3DeElCURSIVCTcImnpvaqMVOd1J8dVWjlIvxH0NoQxTs+3ltSaz3R6LhG2zK0nKR7pHGCrjzV43PFyQIuVzjA5Rfmgd9ydh4dJXMXfx4Vk2UuJpfJx9+ottMSsfOnzwkPn3d0GYAyDkt0NR+B4deX/vlQjyk2XOanFX5BR1shp+GD+9SM0HiKoOBFuB5Nwmaj7YU84Azue+aeJWblMQwRM3ikMGxtMF3TSkAwjEu76FnQBJLCur+WKYZJaRdwQTmChopENT3HnfesUOb2cqIuTYUjDTvdjA73okslWFASZdcmOl2XqHXqC1zjmrkvRSrUCSkV08BYxGK4fTdBiTsLfly6XTUYxWhR6/48/T1+zwy1XmhpIX+361aESSEDruET+czOjTV63VSVsmtL/K1Jme2Czf6xmHtbWZaI2M3blHhGWxkYmA60EK7eDaxWvSQAonSgYve+cd8UMJvsINQzd9DM6RxQRzOAulLsHOB5EFkelRDyJ3GQWAkwhb/96T2tebcHzwYxIMbOLX6mUQZmbJ9ZnVDPSjpIK8mhyOtqIOVaTC2QgaOPfimGSBu6o/iWvCJDJ5k0K2rUrFthUbQ413hvlFiRlvUZ1cwE4EKNqVTPf0PRi74faLU3T1kRnwXa+E8Z+hbFlJXM9XVrrZPEJE7/LW9KCFCpwhe5UXsQSZHSXIv/cmyloi4ZwAkz56AHhlzrmXf8LtrQ1auXmGZTYU0VQtSKMmYliN/HntO88eOKwdpLMNwUQsuaRBux7wFMXQZzn1nGns90M0oJeRWbPXW5n2O12yHEoXDZJWBbEUGUQdXkTHqZSVkjqJy4k9rqWBM+P42iPyTJmS1cWwiLAE5GksIh3dNyEgh7gdPu3IqD5ZCq5u8ZAGv7yQePvMTEM mavI9/+L n2BsA0jE/6uBu5NNlJEb9qijfOB0wMhwitIGaG3Ik5zrqmw1PXhYzor50bjq05DZB5irLlzVD8S65ZIMxPZayEXGMdTX7vLH6nONkftNg+TBiVgZq33B+x0CVZCLPjCAQmd86qZM6hj9QWcjHzttR5ymt/qZS3rL0KPFHqSikTvCOVdv06wf+UL/OGRpKmE7489m7f2of5QhOm2c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range() allocates a contiguous range of physical memory. Metadata pages in use for data will have to be migrated and then taken from the free lists when they are repurposed to store tags, and that will be accomplished by calling alloc_contig_range(). The first step in alloc_contig_range() is to isolate the requested pages. If the pages are part of a larger huge page, then the hugepage must be split before the pages can be isolated. Add support for metadata pages in isolate_single_pageblock(). __isolate_free_page() checks the WMARK_MIN watermark before deleting the page from the free list. alloc_contig_range(), when called to allocate MIGRATE_METADATA pages, ends up calling this function from isolate_freepages_range() -> isolate_freepages_block(). As such, take into account the number of free metadata pages when checking the watermark to avoid false negatives. Signed-off-by: Alexandru Elisei --- mm/compaction.c | 4 ++-- mm/page_alloc.c | 9 +++++---- mm/page_isolation.c | 19 +++++++++++++------ 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index af2ee3085623..314793ec8bdb 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -46,7 +46,7 @@ static inline void count_compact_events(enum vm_event_item item, long delta) #define count_compact_events(item, delta) do { } while (0) #endif -#if defined CONFIG_COMPACTION || defined CONFIG_CMA +#if defined CONFIG_COMPACTION || defined CONFIG_CMA || defined CONFIG_MEMORY_METADATA #define CREATE_TRACE_POINTS #include @@ -1306,7 +1306,7 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn, return ret; } -#endif /* CONFIG_COMPACTION || CONFIG_CMA */ +#endif /* CONFIG_COMPACTION || CONFIG_CMA || CONFIG_MEMORY_METADATA */ #ifdef CONFIG_COMPACTION static bool suitable_migration_source(struct compact_control *cc, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 911d3c362848..1adaefa22208 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2624,7 +2624,8 @@ int __isolate_free_page(struct page *page, unsigned int order) * exists. */ watermark = zone->_watermark[WMARK_MIN] + (1UL << order); - if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) + if (!zone_watermark_ok(zone, 0, watermark, 0, + ALLOC_CMA | ALLOC_FROM_METADATA)) return 0; __mod_zone_freepage_state(zone, -(1UL << order), mt); @@ -6246,9 +6247,9 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either - * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks - * in range must have the same migratetype and it must - * be either of the two. + * #MIGRATE_MOVABLE, #MIGRATE_CMA or #MIGRATE_METADATA). + * All pageblocks in range must have the same migratetype + * and it must be either of the three. * @gfp_mask: GFP mask to use during compaction * * The PFN range does not have to be pageblock aligned. The PFN range must diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 6599cc965e21..bb2a72ce201b 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -52,6 +52,13 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e return page; } + if (is_migrate_metadata_page(page)) { + if (is_migrate_metadata(migratetype)) + return NULL; + else + return page; + } + for (pfn = start_pfn; pfn < end_pfn; pfn++) { page = pfn_to_page(pfn); @@ -396,7 +403,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = head_pfn + nr_pages; continue; } -#if defined CONFIG_COMPACTION || defined CONFIG_CMA +#if defined CONFIG_COMPACTION || defined CONFIG_CMA || defined CONFIG_MEMORY_METADATA /* * hugetlb, lru compound (THP), and movable compound pages * can be migrated. Otherwise, fail the isolation. @@ -466,7 +473,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = outer_pfn; continue; } else -#endif +#endif /* CONFIG_COMPACTION || CONFIG_CMA || CONFIG_MEMORY_METADATA */ goto failed; } @@ -495,10 +502,10 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, * @gfp_flags: GFP flags used for migrating pages that sit across the * range boundaries. * - * Making page-allocation-type to be MIGRATE_ISOLATE means free pages in - * the range will never be allocated. Any free pages and pages freed in the - * future will not be allocated again. If specified range includes migrate types - * other than MOVABLE or CMA, this will fail with -EBUSY. For isolating all + * Making page-allocation-type to be MIGRATE_ISOLATE means free pages in the + * range will never be allocated. Any free pages and pages freed in the future + * will not be allocated again. If specified range includes migrate types other + * than MOVABLE, CMA or METADATA, this will fail with -EBUSY. For isolating all * pages in the range finally, the caller have to free all pages in the range. * test_page_isolated() can be used for test it. * From patchwork Wed Aug 23 13:13:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A68D8EE4993 for ; Wed, 23 Aug 2023 13:16:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A092280062; Wed, 23 Aug 2023 09:16:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3509628005D; Wed, 23 Aug 2023 09:16:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21873280062; Wed, 23 Aug 2023 09:16:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 107D028005D for ; Wed, 23 Aug 2023 09:16:53 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CFA9A1A0113 for ; Wed, 23 Aug 2023 13:16:52 +0000 (UTC) X-FDA: 81155419464.08.D4DA10F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id 19AB980026 for ; Wed, 23 Aug 2023 13:16:50 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796611; a=rsa-sha256; cv=none; b=Rn6y4Nd8I1JJN8ltNzZLAIrbDvcxQaHAutgUwXJLvVjhpOkhWnkqmaExYvbk37Uf+B4tQ8 twtTCiWClHf99c5en8bKrMhghQwKc+bpSihr5LbtN9yxMcJ/tsIjQMl7zUCvnuLfv6WxQg fam7SLmtN+QWEPoBYStcq+tVpNxcP9E= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yzlkZdnLlUiULXsdZF7vOjY49ZMfvnYeuD0zJDtuhc4=; b=am5m+OASOcxY0FdIxSWRdvi/JCK9Nv3lA1D940XsMPl7DoudTuVc6AbWPoiTyr2GLJjg0Q SDNIXCVMK5Z2d4ye/KohdCnQExMfphwIM3+TYESYf7IxQLgThdrgozJQBiV3XIT7ru73j8 pa9K96l1xGwYmWKcFEA8uLQi/qkC6ls= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E999E165C; Wed, 23 Aug 2023 06:17:30 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AC7ED3F740; Wed, 23 Aug 2023 06:16:42 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 25/37] arm64: mte: Manage tag storage on page allocation Date: Wed, 23 Aug 2023 14:13:38 +0100 Message-Id: <20230823131350.114942-26-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 19AB980026 X-Stat-Signature: kh96eibntjub4ynjifxpiibgbkm71ymx X-HE-Tag: 1692796610-516303 X-HE-Meta: U2FsdGVkX1/Jo4mktXAJFgefyvIQdW/L6dHcBZ7EUmIayLYaRaJ78CwSmKeGNJQ/AVJcao/ulH7ATD2HltA37pqTBb1xdFah6i4qF2mGfRrwghaob8oxFtFESHEX1Tu1Ir72WfAiovMxRzdPEAgrMGWj03EUR7TafKKWUlTXeRcO/SYs0Sbt9WprBSr0xw6P9u/TZhrEghT5ee2qigD+WyCXOmCnDHhwxRNHf6NGwjKtObWl3oK/MOdgP4BafRPYKB/D/nNvEprzAkdnn5KO7EuIIMNabLH6gOjkAUTuEKlV9tE2nummCvZeIyXuaTXl52KRQ3evOXiam7nRs3dp/ZiQodEhCZMNCnf1JbG1WmeSSGNqhCjEgTl00uY88Jvb+DuSYtRf/QIfCJ7fOFKb57PDi3+MMSxmexBevZbJIAzgFC0xOkWqf0gzkdkQrqUsEzwbyalmsrZ8nlh2M4gOWCd0ec8h+McqYEeEsRC8mVRoIkBfWy63PGN6bKV8oDQ0hmHQjF6/DzrGiStE1gFsHcHMpTHwAv4E5NYt4NH5V0KlOSfxKHgbCQGk+httj5UwI+9sUn/wA9rHG2PTHdoi68NfNwx2jpgIUM6JH2iIrs2DqHflhwFEv+oKnsu4xua32z2CTM7ZJ7W9b17WsdyfaMvFRcBvPpYArJRxDrN1efu0RR2wNjn8k2BBsWohtDT2DmaF700BhfZ9bALGYhHiUHphUIrqnC9wu6r72zY3baGevwIr5a/tc7/TzZd+qt/4rxA4UrkoX5iLSEUrQjbgkO3h6/AAsBK/hdIsejbPAJFuqYPA/TjBUUNR72r7y54uMh9XDu3bPR7ImTQozny5mnQbQEP7UfPMRfmm1ocmfx2T/dQoLMI0/8MKDSdOHguxeazB24mqHsGflF5AIZL+ccnaCoUdyhMt8Ukcn5dql+8+g3tXJw8cqeCeQgu7H57CsFqgOX9VRBQZAxwVX2K IJ/4tYug HPIq+ro6CRLC4CSbm/cVtzzjEraVL2kkHofbKTnauMGiNWf3u5a9wEMnds8SPo8lNs51qACgLHYUqVljl9zgG/jOPq25T6r6Olgk2u4P017NayjiHA0gb4qVlpoThf7m20IDQ6L3yy1IQtg9OkG/I0oAaecw4IUu9Uhj2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reserve tag storage for a tagged allocation by migrating the contents of the tag storage (if in use for data) and removing the pages from page allocator by using alloc_contig_range(). When all the associated tagged pages have been freed, put the tag storage pages back to the page allocator, where they can be used for data allocations. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 16 +- arch/arm64/include/asm/mte.h | 12 ++ arch/arm64/include/asm/mte_tag_storage.h | 8 + arch/arm64/kernel/mte_tag_storage.c | 242 ++++++++++++++++++++++- fs/proc/page.c | 1 + include/linux/kernel-page-flags.h | 1 + include/linux/page-flags.h | 1 + include/trace/events/mmflags.h | 3 +- mm/huge_memory.c | 1 + 9 files changed, 273 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index ade37331a5c8..167b039f06cf 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -12,9 +12,11 @@ #include #ifdef CONFIG_MEMORY_METADATA +DECLARE_STATIC_KEY_FALSE(metadata_storage_enabled_key); + static inline bool metadata_storage_enabled(void) { - return false; + return static_branch_likely(&metadata_storage_enabled_key); } static inline bool alloc_can_use_metadata_pages(gfp_t gfp_mask) @@ -34,19 +36,13 @@ static inline bool folio_has_metadata(struct folio *folio) return page_has_metadata(&folio->page); } -static inline int reserve_metadata_storage(struct page *page, int order, gfp_t gfp_mask) -{ - return 0; -} - -static inline void free_metadata_storage(struct page *page, int order) -{ -} - static inline bool vma_has_metadata(struct vm_area_struct *vma) { return vma && (vma->vm_flags & VM_MTE); } + +int reserve_metadata_storage(struct page *page, int order, gfp_t gfp_mask); +void free_metadata_storage(struct page *page, int order); #endif /* CONFIG_MEMORY_METADATA */ #endif /* __ASM_MEMORY_METADATA_H */ diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 246a561652f4..70cfd09b4a11 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -44,9 +44,21 @@ void mte_free_tags_mem(void *tags); #define PG_mte_tagged PG_arch_2 /* simple lock to avoid multiple threads tagging the same page */ #define PG_mte_lock PG_arch_3 +/* Track if a tagged page has tag storage reserved */ +#define PG_tag_storage_reserved PG_arch_4 + +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +DECLARE_STATIC_KEY_FALSE(metadata_storage_enabled_key); +extern bool page_tag_storage_reserved(struct page *page); +#endif static inline void set_page_mte_tagged(struct page *page) { +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE + /* Open code mte_tag_storage_enabled() */ + WARN_ON_ONCE(static_branch_likely(&metadata_storage_enabled_key) && + !page_tag_storage_reserved(page)); +#endif /* * Ensure that the tags written prior to this function are visible * before the page flags update. diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 8f86c4f9a7c3..7b69a8af13f3 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -5,11 +5,19 @@ #ifndef __ASM_MTE_TAG_STORAGE_H #define __ASM_MTE_TAG_STORAGE_H +#include + #ifdef CONFIG_ARM64_MTE_TAG_STORAGE void mte_tag_storage_init(void); +bool page_tag_storage_reserved(struct page *page); #else static inline void mte_tag_storage_init(void) { } +static inline bool page_tag_storage_reserved(struct page *page) +{ + return true; +} #endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ + #endif /* __ASM_MTE_TAG_STORAGE_H */ diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 3e0123aa3fb3..075231443dec 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -11,13 +11,19 @@ #include #include #include +#include +#include #include +#include #include +#include #include #include #include +__ro_after_init DEFINE_STATIC_KEY_FALSE(metadata_storage_enabled_key); + struct tag_region { struct range mem_range; /* Memory associated with the tag storage, in PFNs. */ struct range tag_range; /* Tag storage memory, in PFNs. */ @@ -29,6 +35,30 @@ struct tag_region { static struct tag_region tag_regions[MAX_TAG_REGIONS]; static int num_tag_regions; +/* + * A note on locking. Reserving tag storage takes the tag_blocks_lock mutex, + * because alloc_contig_range() might sleep. + * + * Freeing tag storage takes the xa_lock spinlock with interrupts disabled + * because pages can be freed from non-preemptible contexts, including from an + * interrupt handler. + * + * Because tag storage freeing can happen from interrupt contexts, the xarray is + * defined with the XA_FLAGS_LOCK_IRQ flag to disable interrupts when calling + * xa_store() to prevent a deadlock. + * + * This means that reserve_metadata_storage() cannot run concurrently with + * itself (no concurrent insertions), but it can run at the same time as + * free_metadata_storage(). The first thing that reserve_metadata_storage() does + * after taking the mutex is increase the refcount on all present tag storage + * blocks with the xa_lock held, to serialize against freeing the blocks. This + * is an optimization to avoid taking and releasing the xa_lock after each + * iteration if the refcount operation was moved inside the loop, where it would + * have had to be executed for each block. + */ +static DEFINE_XARRAY_FLAGS(tag_blocks_reserved, XA_FLAGS_LOCK_IRQ); +static DEFINE_MUTEX(tag_blocks_lock); + static int __init tag_storage_of_flat_get_range(unsigned long node, const __be32 *reg, int reg_len, struct range *range) { @@ -367,6 +397,216 @@ static int __init mte_tag_storage_activate_regions(void) } } + return ret; +} +core_initcall(mte_tag_storage_activate_regions); + +bool page_tag_storage_reserved(struct page *page) +{ + return test_bit(PG_tag_storage_reserved, &page->flags); +} + +static int tag_storage_find_block_in_region(struct page *page, unsigned long *blockp, + struct tag_region *region) +{ + struct range *tag_range = ®ion->tag_range; + struct range *mem_range = ®ion->mem_range; + u64 page_pfn = page_to_pfn(page); + u64 block, block_offset; + + if (!(mem_range->start <= page_pfn && page_pfn <= mem_range->end)) + return -ERANGE; + + block_offset = (page_pfn - mem_range->start) / 32; + block = tag_range->start + rounddown(block_offset, region->block_size); + + if (block + region->block_size - 1 > tag_range->end) { + pr_err("Block 0x%llx-0x%llx is outside tag region 0x%llx-0x%llx\n", + PFN_PHYS(block), PFN_PHYS(block + region->block_size), + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end)); + return -ERANGE; + } + *blockp = block; + + return 0; +} + +static int tag_storage_find_block(struct page *page, unsigned long *block, + struct tag_region **region) +{ + int i, ret; + + for (i = 0; i < num_tag_regions; i++) { + ret = tag_storage_find_block_in_region(page, block, &tag_regions[i]); + if (ret == 0) { + *region = &tag_regions[i]; + return 0; + } + } + + return -EINVAL; +} + +static void block_ref_add(unsigned long block, struct tag_region *region, int order) +{ + int count; + + count = min(1u << order, 32 * region->block_size); + page_ref_add(pfn_to_page(block), count); +} + +static int block_ref_sub_return(unsigned long block, struct tag_region *region, int order) +{ + int count; + + count = min(1u << order, 32 * region->block_size); + return page_ref_sub_return(pfn_to_page(block), count); +} + +static bool tag_storage_block_is_reserved(unsigned long block) +{ + return xa_load(&tag_blocks_reserved, block) != NULL; +} + +static int tag_storage_reserve_block(unsigned long block, struct tag_region *region, int order) +{ + int ret; + + ret = xa_err(xa_store(&tag_blocks_reserved, block, pfn_to_page(block), GFP_KERNEL)); + if (!ret) + block_ref_add(block, region, order); + + return ret; +} + +bool alloc_can_use_tag_storage(gfp_t gfp_mask) +{ + return !(gfp_mask & __GFP_TAGGED); +} + +bool alloc_requires_tag_storage(gfp_t gfp_mask) +{ + return gfp_mask & __GFP_TAGGED; +} + +static int order_to_num_blocks(int order) +{ + return max((1 << order) / 32, 1); +} + +int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) +{ + unsigned long start_block, end_block; + struct tag_region *region; + unsigned long block; + unsigned long flags; + int i, tries; + int ret = 0; + + VM_WARN_ON_ONCE(!preemptible()); + + /* + * __alloc_contig_migrate_range() ignores gfp when allocating the + * destination page for migration. Regardless, massage gfp flags and + * remove __GFP_TAGGED to avoid recursion in case gfp stops being + * ignored. + */ + gfp &= ~__GFP_TAGGED; + if (!(gfp & __GFP_NORETRY)) + gfp |= __GFP_RETRY_MAYFAIL; + + ret = tag_storage_find_block(page, &start_block, ®ion); + if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) + return 0; + + end_block = start_block + order_to_num_blocks(order) * region->block_size; + + mutex_lock(&tag_blocks_lock); + + /* Make sure existing entries are not freed from out under out feet. */ + xa_lock_irqsave(&tag_blocks_reserved, flags); + for (block = start_block; block < end_block; block += region->block_size) { + if (tag_storage_block_is_reserved(block)) + block_ref_add(block, region, order); + } + xa_unlock_irqrestore(&tag_blocks_reserved, flags); + + for (block = start_block; block < end_block; block += region->block_size) { + /* Refcount incremented above. */ + if (tag_storage_block_is_reserved(block)) + continue; + + tries = 5; + while (tries--) { + ret = alloc_contig_range(block, block + region->block_size, MIGRATE_METADATA, gfp); + if (ret == 0 || ret != -EBUSY) + break; + } + + if (ret) + goto out_error; + + ret = tag_storage_reserve_block(block, region, order); + if (ret) { + free_contig_range(block, region->block_size); + goto out_error; + } + + count_vm_events(METADATA_RESERVE_SUCCESS, region->block_size); + } + + for (i = 0; i < (1 << order); i++) + set_bit(PG_tag_storage_reserved, &(page + i)->flags); + + mutex_unlock(&tag_blocks_lock); + return 0; + +out_error: + xa_lock_irqsave(&tag_blocks_reserved, flags); + for (block = start_block; block < end_block; block += region->block_size) { + if (tag_storage_block_is_reserved(block) && + block_ref_sub_return(block, region, order) == 1) { + __xa_erase(&tag_blocks_reserved, block); + free_contig_range(block, region->block_size); + } + } + xa_unlock_irqrestore(&tag_blocks_reserved, flags); + + mutex_unlock(&tag_blocks_lock); + + count_vm_events(METADATA_RESERVE_FAIL, region->block_size); + + return ret; +} + +void free_metadata_storage(struct page *page, int order) +{ + unsigned long block, start_block, end_block; + struct tag_region *region; + unsigned long flags; + int ret; + + if (WARN_ONCE(!page_mte_tagged(page), "pfn 0x%lx is not tagged", page_to_pfn(page))) + return; + + ret = tag_storage_find_block(page, &start_block, ®ion); + if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) + return; + + end_block = start_block + order_to_num_blocks(order) * region->block_size; + + xa_lock_irqsave(&tag_blocks_reserved, flags); + for (block = start_block; block < end_block; block += region->block_size) { + if (WARN_ONCE(!tag_storage_block_is_reserved(block), + "Block 0x%lx is not reserved for pfn 0x%lx", block, page_to_pfn(page))) + continue; + + if (block_ref_sub_return(block, region, order) == 1) { + __xa_erase(&tag_blocks_reserved, block); + free_contig_range(block, region->block_size); + count_vm_events(METADATA_RESERVE_FREE, region->block_size); + } + } + xa_unlock_irqrestore(&tag_blocks_reserved, flags); } -core_initcall(mte_tag_storage_activate_regions) diff --git a/fs/proc/page.c b/fs/proc/page.c index 195b077c0fac..e7eb584a9234 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -221,6 +221,7 @@ u64 stable_page_flags(struct page *page) #ifdef CONFIG_ARCH_USES_PG_ARCH_X u |= kpf_copy_bit(k, KPF_ARCH_2, PG_arch_2); u |= kpf_copy_bit(k, KPF_ARCH_3, PG_arch_3); + u |= kpf_copy_bit(k, KPF_ARCH_4, PG_arch_4); #endif return u; diff --git a/include/linux/kernel-page-flags.h b/include/linux/kernel-page-flags.h index 859f4b0c1b2b..4a0d719ffdd4 100644 --- a/include/linux/kernel-page-flags.h +++ b/include/linux/kernel-page-flags.h @@ -19,5 +19,6 @@ #define KPF_SOFTDIRTY 40 #define KPF_ARCH_2 41 #define KPF_ARCH_3 42 +#define KPF_ARCH_4 43 #endif /* LINUX_KERNEL_PAGE_FLAGS_H */ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 92a2063a0a23..42fb54cb9a54 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -135,6 +135,7 @@ enum pageflags { #ifdef CONFIG_ARCH_USES_PG_ARCH_X PG_arch_2, PG_arch_3, + PG_arch_4, #endif __NR_PAGEFLAGS, diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 4ccca8e73c93..23f1a76d66a7 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -125,7 +125,8 @@ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ IF_HAVE_PG_IDLE(young) \ IF_HAVE_PG_ARCH_X(arch_2) \ -IF_HAVE_PG_ARCH_X(arch_3) +IF_HAVE_PG_ARCH_X(arch_3) \ +IF_HAVE_PG_ARCH_X(arch_4) #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index eb3678360b97..cf5247b012de 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2458,6 +2458,7 @@ static void __split_huge_page_tail(struct page *head, int tail, #ifdef CONFIG_ARCH_USES_PG_ARCH_X (1L << PG_arch_2) | (1L << PG_arch_3) | + (1L << PG_arch_4) | #endif (1L << PG_dirty) | LRU_GEN_MASK | LRU_REFS_MASK)); From patchwork Wed Aug 23 13:13:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91DF1EE49B5 for ; Wed, 23 Aug 2023 13:17:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B67D280069; Wed, 23 Aug 2023 09:17:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 266CE28005D; Wed, 23 Aug 2023 09:17:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10767280069; Wed, 23 Aug 2023 09:17:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0073428005D for ; Wed, 23 Aug 2023 09:17:01 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D7422A0108 for ; Wed, 23 Aug 2023 13:17:01 +0000 (UTC) X-FDA: 81155419842.25.CFFFF9B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 15347A001F for ; Wed, 23 Aug 2023 13:16:58 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796619; a=rsa-sha256; cv=none; b=g+DWyXfMqNzARIulbgdDHBP/tD4fL22d1UeSJNUtpljRgfA3X+JIjxydSjIuwTyW+MJWXv yf1IM6vo4Bnl+DmmAfdEMB3zNfyHCJlGm+gTReMYOEKbKajRz4jK+eAGQQKKNP2BZpqgXI K+kvsvg5n+QsDkLTtk3Hi1NgIUqucvQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796619; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hggf+d5pT7l1VCYsLbalVNgtWCznf34+9T1gNLr534I=; b=prbZxzoTk/EnNZy5w6oj1I7TD4QbYz52A+wkqTGR7SQAbyNzU84wWfK9LK5CgHaRhb9S0K CLG5GDt2kbLVK9fGKeo9lWpEzUghsI00UWTWdd4IebQUariY1bV0CW/G9aU9nxl2tvFTK4 eQSp/KTenDV/y4EO4pfFihJbA5UfEW8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D68951682; Wed, 23 Aug 2023 06:17:38 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BFDA13F740; Wed, 23 Aug 2023 06:16:50 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 26/37] arm64: mte: Perform CMOs for tag blocks on tagged page allocation/free Date: Wed, 23 Aug 2023 14:13:39 +0100 Message-Id: <20230823131350.114942-27-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 15347A001F X-Stat-Signature: bpp8mhu91c1cabnkjge4oqiw7b1uhwtm X-Rspam-User: X-HE-Tag: 1692796618-149935 X-HE-Meta: U2FsdGVkX19bK3720guIRXY/q5K5NXcOGKYcHAs1z+yVHM5W3wjtelcWr62rRTgQfBekCS5ahMJWcyBgwzqn0UObiHa/KnW5/vq9JyRMXTfZcdzlHOkBPNu9zZn1wf/Gd7/gE2dMD/W1cgu5pW/Msa6xcuVieEzQ9TC5v5R1NZdK5dmJhJIplFbyB0ZCUWikg54ymQAHL09t1KWEIvGyogqscqi6/vLXoMUjeQdSgpLxnREZbQlHOyc9CSlyq2fzJ+HArZUIso/iOv67ccROErYvgo7Ay1m2yENeiMwMXVd9UrCxzt3FpejxPGtYw6PkAQy2NX0ilwo653FXGJTZrOfwonl8ptudox3CXrjzvXvPCI9Lz2BRH68KuaPBqdxZeyzXUfiTCaaNqlxjZM9abCyNFKgpfYf037yUQ8fE31BUGbMXIoHXu/XrLTOOo7VpjTpkRHQ8h/wNZYForbRfTfOwE4U5q56/s4H+Qs+Br+o4669MKnmXf4pU06lRIsjT0RxKWk4sRMQ13cgXjijsWiUAZX7/dAarb3nZ09J+EsMejjhBXWrCZnMxOu+cHWlbFYCt+h3Pk17KGW4BoJsq0X1g8LoIDZ0L9ehDVLsLKJZxR1ThVXzfxFPcwlEPQuj/1d1kYaGIFVP3lOZO7E+IaKf1X7RXLsWOPi/b2IbddfBGy87temltAlmEaLgrNniLLwQQlsz3/94A4qqR78Qa31bLH3Gz9oTVs1h6kkfO0XtMMegRl4uMr/pWtfr8LJ5YQ874Td5VkWlpIno+vOqwqo7ESdYy2mOZgMP9R3chSoKBVuOvc2/ZMFy/9p/z9q3e/KqSK8ZawqwQ3t9b7izNiR3zWCwDR4dQRQk6ZoSRazE2YkUcT3aSvF36xAQefGJdHUOcgo7c5hhtEkzUL7E3ewk1Y6ABEfv9eru+MiMLghS8ujp+X7/ETGvo9TzFdwz7iOHImnnBXzrK0JeL30i 5+NKlJRS oZeacjmPqn4eSDjtKMPz9MVK+C75mlL6/xRFAaDmThjtNcnF3pgSa0LjKt5QoEl1iCNlItDyvHCkHhtye6F2BtZQvkMZj9rFyY3a0TJWOfVFsOXZNwHfoKMaLApd/XTsZBJOOyWOzTahOkvFZDseuGMQyaS2BMDzfecd4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make sure that the contents of the tag storage block is not corrupted by performing: 1. A tag dcache inval when the associated tagged pages are freed, to avoid dirty tag cache lines being evicted and corrupting the tag storage block when it's being used to store data. 2. A data cache inval when the tag storage block is being reserved, to ensure that no dirty data cache lines are present, which would trigger a writeback that could corrupt the tags stored in the block. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/assembler.h | 10 ++++++++++ arch/arm64/include/asm/mte_tag_storage.h | 2 ++ arch/arm64/kernel/mte_tag_storage.c | 14 ++++++++++++++ arch/arm64/lib/mte.S | 16 ++++++++++++++++ 4 files changed, 42 insertions(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 376a980f2bad..8d41c8cfdc69 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -310,6 +310,16 @@ alternative_cb_end lsl \reg, \reg, \tmp // actual cache line size .endm +/* + * tcache_line_size - get the safe tag cache line size across all CPUs + */ + .macro tcache_line_size, reg, tmp + read_ctr \tmp + ubfm \tmp, \tmp, #32, #37 // tag cache line size encoding + mov \reg, #4 // bytes per word + lsl \reg, \reg, \tmp // actual tag cache line size + .endm + /* * raw_icache_line_size - get the minimum I-cache line size on this CPU * from the CTR register. diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 7b69a8af13f3..bad865866eeb 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -7,6 +7,8 @@ #include +extern void dcache_inval_tags_poc(unsigned long start, unsigned long end); + #ifdef CONFIG_ARM64_MTE_TAG_STORAGE void mte_tag_storage_init(void); bool page_tag_storage_reserved(struct page *page); diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 075231443dec..7dff93492a7b 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -19,6 +19,7 @@ #include #include +#include #include #include @@ -470,8 +471,13 @@ static bool tag_storage_block_is_reserved(unsigned long block) static int tag_storage_reserve_block(unsigned long block, struct tag_region *region, int order) { + unsigned long block_va; int ret; + block_va = (unsigned long)page_to_virt(pfn_to_page(block)); + /* Avoid writeback of dirty data cache lines corrupting tags. */ + dcache_inval_poc(block_va, block_va + region->block_size * PAGE_SIZE); + ret = xa_err(xa_store(&tag_blocks_reserved, block, pfn_to_page(block), GFP_KERNEL)); if (!ret) block_ref_add(block, region, order); @@ -584,6 +590,7 @@ void free_metadata_storage(struct page *page, int order) { unsigned long block, start_block, end_block; struct tag_region *region; + unsigned long page_va; unsigned long flags; int ret; @@ -594,6 +601,13 @@ void free_metadata_storage(struct page *page, int order) if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) return; + page_va = (unsigned long)page_to_virt(page); + /* + * Remove dirty tag cache lines to avoid corruption of the tag storage + * page contents when it gets freed back to the page allocator. + */ + dcache_inval_tags_poc(page_va, page_va + (PAGE_SIZE << order)); + end_block = start_block + order_to_num_blocks(order) * region->block_size; xa_lock_irqsave(&tag_blocks_reserved, flags); diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index d3c4ff70f48b..97f0bb071284 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -175,3 +175,19 @@ SYM_FUNC_START(mte_restore_page_tags_from_mem) ret SYM_FUNC_END(mte_restore_page_tags_from_mem) + +/* + * dcache_inval_tags_poc(start, end) + * + * Ensure that any tags in the D-cache for the interval [start, end) + * are invalidated to PoC. + * + * - start - virtual start address of region + * - end - virtual end address of region + */ +SYM_FUNC_START(__pi_dcache_inval_tags_poc) + tcache_line_size x2, x3 + dcache_by_myline_op igvac, sy, x0, x1, x2, x3 + ret +SYM_FUNC_END(__pi_dcache_inval_tags_poc) +SYM_FUNC_ALIAS(dcache_inval_tags_poc, __pi_dcache_inval_tags_poc) From patchwork Wed Aug 23 13:13:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCAC3EE4993 for ; Wed, 23 Aug 2023 13:17:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5271C28006A; Wed, 23 Aug 2023 09:17:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D79B28005D; Wed, 23 Aug 2023 09:17:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C69228006A; Wed, 23 Aug 2023 09:17:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2A1AE28005D for ; Wed, 23 Aug 2023 09:17:07 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0C62E160112 for ; Wed, 23 Aug 2023 13:17:07 +0000 (UTC) X-FDA: 81155420094.19.1A35D45 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id 444EFC0017 for ; Wed, 23 Aug 2023 13:17:05 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796625; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wR07bmEfdnhrLlB8kiJZfNPNesPOD1NKIT94k9iUsy8=; b=0cMtNgc7Cxx+T5jqw5tui5y5j6GzAOQ9+2fmC9CcNoaltAXMweUF5aQH0Uq8P6+t9qeVuE PqnM198btViCJE70JUSIYMqFYuvVvLle9MGQp1bVEYZZt38vFasfGsItGVjKufSRTkHFoE 0zazZmPnXhsdzJv2oDSE+LdbZctSfsY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796625; a=rsa-sha256; cv=none; b=cUQdV6ws8FsHI53u70SxFYKuUPOxZ3u6MoQhaAhWMkdgDAk1ZgBAN5Qt2XBuZWCaqwNWkX B2RK78mweW8Oufr0SYH/7NpbXpb4DQzgMn1mRr5jOz0r7HCtgfqVil8JOwq4idQWYKz2Z9 V6kPd5uPk38QHbLs4AsMW53N76N3UWE= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1A4C816F2; Wed, 23 Aug 2023 06:17:45 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9BF043F740; Wed, 23 Aug 2023 06:16:58 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 27/37] arm64: mte: Reserve tag block for the zero page Date: Wed, 23 Aug 2023 14:13:40 +0100 Message-Id: <20230823131350.114942-28-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 444EFC0017 X-Stat-Signature: cj14ahoey6zs1s8iipe5mxmzfxut1kww X-Rspam-User: X-HE-Tag: 1692796625-3007 X-HE-Meta: U2FsdGVkX19uJwceGPXDTBdTbkMIwqNzAJgv+HkQGrJKT4uq/h08ZCizmfPElfaZy9fWLw2iC2akjMEN1uN7gj/a0lry+XF1GrRcGLYVd1ASRYq79JSPZ+sAukDVmjIrzib53ipXfipr2wDAZtw9sRzEVB4I2jIbY1AhEns39dbEQUWUiPe9XDEgZbN8zpRY8OrU/Q0GrWP2VXQEpAwJVKiOmdUDl3xgKEAApHo50+4f4kVqK/FycrysPpInJYQUEPbuYeoTKZ9rzT2yFu6iO7Fn+ygl8MB8I3Wmy5hjGfF7M2IH6cxB10mswqP8ic+Sr2kBr38PP98D5ja0CKdX9xuOeR/38/E6iLVrPc0UBjmilTDFMzvNJBB+NP3oP/qRrlCLRWVDRHfGm2ZICFo2lXwG5kNZUWNsuXOGfFp2IYmfEqGNEp42j6+MfTbLo0Hm6yFihP6i5WmkBmQFd8PFFapK8R8v3l6VjeevxpKHDNFFj2YWH+groO3dlPHST6EXU0k6xoVBcR9tN6eYYekDoNbG73g89prugOFdw30BMsWUbR4uAg9CdbjUbNyVwEfBet16s2+N4RXe6hVVouio0H3THieBfv2djLmEjwIyeyHlP1nH0ZtTDZA3FpUFeRwTnjz3dx/M50nvfaOEqCe7ac4DaN30mf5GM7WF2dVBfaRPFJw+2zhUmQrUmNBmNfxcg2NN6jbFAnU046oNgxtIRXraiuqJyG9LkwxzkB4ebWYqIsbIqp3qJbf614AQ43EaizuFx8PIP6J7oFRTgENYr9WGK/c+20i3Av3fwWCoWjd7QbmLTlbPsmrPhkp5+XZeUi7s1W6anzwXPFQRA5iTQceCwcaroi8E5owYW+J0OegkVV6/9AKYe7dnM7gpm1jaiwG56/2ehKFk14ewTWOPkq740WJxrqqmpopwbaV4+HLJLZVbhNMKyigurTqCJLHK7unN1cHr1IY6wp+wq3l NFqX2QuV MYAqr/OGrf1P2ksrrObnIJSIhT9T6Pl0Ja0MxoFWRHFZesR0J3VktBfgNq1keWcbdk2Nc1gWo5A7pTVX7jZDlBwb6C/lxE/tTUu4GtVEI3B5nSe+e/CrSU3gG21yzgGc4IDXvG1Dbk+doY//ELQ8lTYghKtjoXXnxP3d9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On arm64, the zero page receives special treatment by having the tagged flag set on MTE initialization, not when the page is mapped in a process address space. Reserve the corresponding tag block when tag storage is being activated. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 7dff93492a7b..1ab875be5f9b 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -398,6 +398,8 @@ static int __init mte_tag_storage_activate_regions(void) } } + ret = reserve_metadata_storage(ZERO_PAGE(0), 0, GFP_HIGHUSER_MOVABLE); + return ret; } core_initcall(mte_tag_storage_activate_regions); From patchwork Wed Aug 23 13:13:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E25A0EE49B0 for ; Wed, 23 Aug 2023 13:17:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E62528006B; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BDC128005D; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6858628006B; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 55FEE28005D for ; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 30340B0D00 for ; Wed, 23 Aug 2023 13:17:13 +0000 (UTC) X-FDA: 81155420346.11.C206B02 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 6E68F10001A for ; Wed, 23 Aug 2023 13:17:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tgGYsPuAcTk/paK1LuxSZMiq6gyvIL+TuHYCaCWFD+c=; b=AEGVqOOPQLVHHh0B5sVcsH3qI2N6MymrjWhCyW+SGlrU4daYXo6ymWONNlFAyRPtD4vnts J51GkBnVAFj+1wjZ8CY6fIc3NzeTu543Mkk/TI3lMkHcSrrq+21PtHLStRXMOBsnX+Lx/2 dSFRqE6KGsJNP8ro32pGFc97vTcB27Y= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796631; a=rsa-sha256; cv=none; b=NV8bZ8xXIa7U3aZWig3ravoCp/15pipXXAZaAA+wh7P8E0AO+tZ3JV0W8B0Q6L9Ubq2LyS 6+dgOsi5l2FBj+xtusBIQlWTjc59M/6Fw7DmollStrPfuxI6VKoD/02Fz278gQ4Q0Dysk7 ybJlbFpMPoqMIHuufA/c1fJukRrmYmA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5D19016F3; Wed, 23 Aug 2023 06:17:51 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C9D2F3F740; Wed, 23 Aug 2023 06:17:04 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 28/37] mm: sched: Introduce PF_MEMALLOC_ISOLATE Date: Wed, 23 Aug 2023 14:13:41 +0100 Message-Id: <20230823131350.114942-29-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6E68F10001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3tfi1hznh1ccwqxbhc8q6a9s3wrinyre X-HE-Tag: 1692796631-764338 X-HE-Meta: U2FsdGVkX18ZRE0e3QobrHY2JY5mYmpCIuWnm2wB0N4vfbscbf4pogw9oYz4rvf3w0AnGHzgoHeBqsE0JUU5dU7lPpPz+CkJodCUBqWeP6j8Y5A9hyExvrdxpMbw6jGs0x0qR+wU6eV4ov0s5leAnSR+1UgMxQUrujdk6lDRgMD2vwtVaWUyVo5xTIDk1EwqyNeHNr+s0s3+7YhOaL8yJqB94mFnWVLOq2x5vWElH/sgRps+GuIAV5iehBmx4YAd5fAU4RW58xdo6m67dZ+G83UV3NQDJ4poSWsO9broiQS+Ok3IvZwc/GpxbOL/jO9jm2JnBcuhqklTvJFcd7a/HJAZ/FPqlhAiiQlm69a8BzAhx3VSaz52U6ktGGno5xk7rZZjPd1Dduqx0f+s5uCDAyByej80fgrEErG7+KZvmYIiabRH1vnFdS+kmVQxpsEsRJveHqGjq969sy4eABG7ansg1GOTszyA8qCP4CruHtUHxf47AGTQ4Z+bOWzqN+lJH+xArcaoCE+7tcQKXByDJ0IYgJ+WMx7rlK8V58hcHLp5TfmT3Ny1e6iL+Hw4HMlXK7/f/PmTIWQaa/2F9ShHkRNQsyCqwOcUuJW01Xj+qM7NtGKiTdqFhUfwwM3mlVbjiyFMgC+iYx2/sbNoUZDN3Ayz6DFfLg9ePr6hfy01XSK0FGXzPQOUazLuUeFr5Ixdmzc68xzNLjWS8bH3sckeqHPuIUx6bQBj8UGSnE5n9Y7C040eKwUhsM1lihxRdTsOP3q0UOX8ESZWsJdU1HY2z0f6iHnPHPxdWXnq4PUHtzx3sRTIEqFYfT1FqVjJQmrxkrgyRFd4sP+majCPT7xg3fDNIo3VrJlAeQdOWgj1w+dCtVcva6n0wMkRnKxOEKU9O/rpVKaXmhCdJSi4U4pQr5vWNbe6grWluXwm1a2lqSEaqUv+wVBZ2YjSlUwZsPtehN/UJjWG43Ju8nYlR2G xOqBJSIg FAFaj8VYEU7RbSS0jKY5/jChh67WnbThDm23OHygNIXmMkUzFZBaaysSrAscP1jynfYX9WecUpi0vm/ZjDTxCfXuWZglbQtRkarvNfHFPbikuJ88YuAwASNaXHhoSkgPePE/Csd8vnnvV73kyZORS+X1+KujreYr0iwrcgts1ExZ1FuTdvBYTVbEhlbwQgZ1lx2HSFgMk+5WO3Bk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On arm64, when reserving tag storage for an allocated page, if the tag storage is in use, the tag storage must be migrated before it can be reserved. As part of the migration process, the tag storage block is first isolated. Compaction also isolates the source pages before migrating them. If the target for compaction requires metadata pages to be reserved, those metadata pages might also need to be isolated, which, in rare circumstances, can lead to the threshold in too_many_isolated() being reached, and isolate_migratepages_pageblock() will get stuck in an infinite loop. Add the flag PF_MEMALLOC_ISOLATE for the current thread, which makes too_many_isolated() ignore the threshold to make forward progress in isolate_migratepages_pageblock(). For consistency, the similarly named function too_many_isolated() called during reclaim has received the same treatment. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 5 ++++- include/linux/sched.h | 2 +- include/linux/sched/mm.h | 13 +++++++++++++ mm/compaction.c | 3 +++ mm/vmscan.c | 3 +++ 5 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 1ab875be5f9b..ba316ffb9aef 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -505,9 +505,9 @@ static int order_to_num_blocks(int order) int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) { unsigned long start_block, end_block; + unsigned long flags, cflags; struct tag_region *region; unsigned long block; - unsigned long flags; int i, tries; int ret = 0; @@ -539,6 +539,7 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) } xa_unlock_irqrestore(&tag_blocks_reserved, flags); + cflags = memalloc_isolate_save(); for (block = start_block; block < end_block; block += region->block_size) { /* Refcount incremented above. */ if (tag_storage_block_is_reserved(block)) @@ -566,6 +567,7 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) for (i = 0; i < (1 << order); i++) set_bit(PG_tag_storage_reserved, &(page + i)->flags); + memalloc_isolate_restore(cflags); mutex_unlock(&tag_blocks_lock); return 0; @@ -581,6 +583,7 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) } xa_unlock_irqrestore(&tag_blocks_reserved, flags); + memalloc_isolate_restore(cflags); mutex_unlock(&tag_blocks_lock); count_vm_events(METADATA_RESERVE_FAIL, region->block_size); diff --git a/include/linux/sched.h b/include/linux/sched.h index 609bde814cb0..a2a930cab31a 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1734,7 +1734,7 @@ extern struct pid *cad_pid; #define PF_USED_MATH 0x00002000 /* If unset the fpu must be initialized before use */ #define PF_USER_WORKER 0x00004000 /* Kernel thread cloned from userspace thread */ #define PF_NOFREEZE 0x00008000 /* This thread should not be frozen */ -#define PF__HOLE__00010000 0x00010000 +#define PF_MEMALLOC_ISOLATE 0x00010000 /* Ignore isolation limits */ #define PF_KSWAPD 0x00020000 /* I am kswapd */ #define PF_MEMALLOC_NOFS 0x00040000 /* All allocation requests will inherit GFP_NOFS */ #define PF_MEMALLOC_NOIO 0x00080000 /* All allocation requests will inherit GFP_NOIO */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 8d89c8c4fac1..8db491208746 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -393,6 +393,19 @@ static inline void memalloc_pin_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC_PIN) | flags; } +static inline unsigned int memalloc_isolate_save(void) +{ + unsigned int flags = current->flags & PF_MEMALLOC_ISOLATE; + + current->flags |= PF_MEMALLOC_ISOLATE; + return flags; +} + +static inline void memalloc_isolate_restore(unsigned int flags) +{ + current->flags = (current->flags & ~PF_MEMALLOC_ISOLATE) | flags; +} + #ifdef CONFIG_MEMCG DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); /** diff --git a/mm/compaction.c b/mm/compaction.c index 314793ec8bdb..fdb75316f0cc 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -778,6 +778,9 @@ static bool too_many_isolated(struct compact_control *cc) unsigned long active, inactive, isolated; + if (current->flags & PF_MEMALLOC_ISOLATE) + return false; + inactive = node_page_state(pgdat, NR_INACTIVE_FILE) + node_page_state(pgdat, NR_INACTIVE_ANON); active = node_page_state(pgdat, NR_ACTIVE_FILE) + diff --git a/mm/vmscan.c b/mm/vmscan.c index 1080209a568b..912ebb6003a0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2453,6 +2453,9 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, if (current_is_kswapd()) return 0; + if (current->flags & PF_MEMALLOC_ISOLATE) + return 0; + if (!writeback_throttling_sane(sc)) return 0; From patchwork Wed Aug 23 13:13:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACE81EE49A3 for ; Wed, 23 Aug 2023 13:17:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A19A28006D; Wed, 23 Aug 2023 09:17:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3515128005D; Wed, 23 Aug 2023 09:17:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2424F28006D; Wed, 23 Aug 2023 09:17:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 13DE028005D for ; Wed, 23 Aug 2023 09:17:21 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EE470B10D9 for ; Wed, 23 Aug 2023 13:17:20 +0000 (UTC) X-FDA: 81155420640.18.CBDF3C6 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id AF741140013 for ; Wed, 23 Aug 2023 13:17:17 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796637; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Kvvr3fnXbq2HHvMhlDHS/wJAJ6ZRdrhcIm2Sezd7beA=; b=Rl6aH8+NNrWRt+ZuZMZgZQiov2omDPxGfCiSgolaPZJWw8fdNq1Tpc6m13ZrqmL9h2ZSyP HR8eSVeJWChtWc1lGk7EUaNgGaGz+TCSf2VxHciE4GYenjQ9HAh0nZn/6u8ePJ/5eBAIYt rzCCltK/UC82NRmV4Kad6s4PmmQ3NDM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796637; a=rsa-sha256; cv=none; b=3+f8c4PHQ1akzZTt/5lfzvP38UHnBm5OlAEIlNgfCh647iONtCkEN6qf1c7CB2sCK/hZdo lVzSwj57n633b2u29i4QCsInauCo1a7tpZCP/CM47iu/g/ukHZJXhZYzLXO7pTr9mUscL9 H61ag0NvPM8ARFuDI7nx8WqaOUQTWYA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 740FB16F8; Wed, 23 Aug 2023 06:17:57 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1BF5A3F740; Wed, 23 Aug 2023 06:17:10 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 29/37] mm: arm64: Define the PAGE_METADATA_NONE page protection Date: Wed, 23 Aug 2023 14:13:42 +0100 Message-Id: <20230823131350.114942-30-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AF741140013 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qqhaqu8qredef7dytgfxgnnwd53dd7dd X-HE-Tag: 1692796637-346574 X-HE-Meta: U2FsdGVkX19SHzm8jZksvvo8t54stqYhjgks93pAe3uLz9AnTOZsd7HX4+3SJfrgZdFYo38RcLB3jJvLi26I01CncUuL7YlQMaZBrOJ6TfyC4CKd4UGUasEUZSdjBL++jzJzNzhsoFibEYgDuF+RmZM5zOJEIGoy7Oj1IEBN8XE7OqrBEn7iFjH62UY5mLqQBCNCa/BQmLSUxXN0Zv4s07Sm0tuS+B828htS2JjO0+XqwtzZLtsjG8p8mplorb3G4pgdaKz0+NQrrTJ/h3vvStIYjBxNV1sDVVI/9Jehf8R3iQCalZ0ql32f9YP1k1wEpccwHauFAkcaQ5cUJ3OQtbnD6/jG64pyKQ3lsD5HFeLgp//ZbBrLKv9vP0La/SzgtXR66uxrCAun69kR2T5PzdzgEQwIrmPnGVsgz0NpR9+CVdP8B5tk6jJqmr/kLeIJv1/frbqZ8tcEGmNZaB2Oif/ei/yaYume6QgSIrB01ahoRnBO2j4ZARKwFlDPiZMEGuioFa+z1dtT6jdnkudae9S3jvt7pKvF3cxf/vZohP4GQXfMBpZQuUdk6QIFdyPmsLNwjS+7p0ZAigwkRbs4ruZuVKI/TeX0rf1antw2HkpHPxmEtmblzP1LMMjKy0pdWO+oGUef7TCu2pXc6nEW84vx5ZXLzL0vl2atdRw0V2ERbQGituP88QpB1+rhi6L2J0ZB2Aig1RhAJaEFwnQq83HBe50rMfmK2JVd5p2pmWnSsoHhAXDX3o0BAvJ/CHKKL3BaryXVwFMf2RYahxwSCH8YiSK1v9roozCXn+QW8OO/pad11KHkBrkMZetJMBaPefQG/u5l0043anAyDIWj19MKyJmCqJXzxd3EHuOLs9FfCDEBjn6YNe8pqn21CxCCiHkCKIMwmfEcwtP4psr2e1WZM4GntsZIvg4gEB6nDydyamhSF97rcBI6eNOdMeoY6YFcul/aPhCysGD7vT1 +8ccc8uc AhXzu2SR7TEUjnTCKkM5pT5yeKM62VuSzdfIPXshTn77gAfj/ED43Thux/15+md4ZSgBH4Ej7KAIdOU9UtL+EY3mQVgwzcMnUo2545NhizL7q2nCyRXujxzrV52LioLcLlM35etFDVy8A1EUBRKxwxM29vQIRKELRR5AagMP+N0GRXXwQ/HfSF1tyNKQ49XlH7w11LdAjR+ipBpU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define the PAGE_METADATA_NONE page protection to be used when a page with metadata doesn't have metadata storage reserved. For arm64, this is accomplished by adding a new page table entry software bit PTE_METADATA_NONE. Linux doesn't set any of the PBHA bits in entries from the last level of the translation table and it doesn't use the TCR_ELx.HWUxx bits. This makes it safe to define PTE_METADATA_NONE as bit 59. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/pgtable-prot.h | 2 ++ arch/arm64/include/asm/pgtable.h | 16 ++++++++++++++-- include/linux/pgtable.h | 12 ++++++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index eed814b00a38..ed2a98ec4e95 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -19,6 +19,7 @@ #define PTE_SPECIAL (_AT(pteval_t, 1) << 56) #define PTE_DEVMAP (_AT(pteval_t, 1) << 57) #define PTE_PROT_NONE (_AT(pteval_t, 1) << 58) /* only when !PTE_VALID */ +#define PTE_METADATA_NONE (_AT(pteval_t, 1) << 59) /* only when PTE_PROT_NONE */ /* * This bit indicates that the entry is present i.e. pmd_page() @@ -98,6 +99,7 @@ extern bool arm64_use_ng_mappings; }) #define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) +#define PAGE_METADATA_NONE __pgprot((_PAGE_DEFAULT & ~PTE_VALID) | PTE_PROT_NONE | PTE_METADATA_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) /* shared+writable pages are clean by default, hence PTE_RDONLY|PTE_WRITE */ #define PAGE_SHARED __pgprot(_PAGE_SHARED) #define PAGE_SHARED_EXEC __pgprot(_PAGE_SHARED_EXEC) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 944860d7090e..2e42f7713425 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -451,6 +451,18 @@ static inline int pmd_protnone(pmd_t pmd) } #endif +#ifdef CONFIG_MEMORY_METADATA +static inline bool pte_metadata_none(pte_t pte) +{ + return (((pte_val(pte) & (PTE_VALID | PTE_PROT_NONE)) == PTE_PROT_NONE) + && (pte_val(pte) & PTE_METADATA_NONE)); +} +static inline bool pmd_metadata_none(pmd_t pmd) +{ + return pte_metadata_none(pmd_pte(pmd)); +} +#endif + #define pmd_present_invalid(pmd) (!!(pmd_val(pmd) & PMD_PRESENT_INVALID)) static inline int pmd_present(pmd_t pmd) @@ -809,8 +821,8 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) * in MAIR_EL1. The mask below has to include PTE_ATTRINDX_MASK. */ const pteval_t mask = PTE_USER | PTE_PXN | PTE_UXN | PTE_RDONLY | - PTE_PROT_NONE | PTE_VALID | PTE_WRITE | PTE_GP | - PTE_ATTRINDX_MASK; + PTE_PROT_NONE | PTE_METADATA_NONE | PTE_VALID | + PTE_WRITE | PTE_GP | PTE_ATTRINDX_MASK; /* preserve the hardware dirty information */ if (pte_hw_dirty(pte)) pte = pte_mkdirty(pte); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5063b482e34f..0119ffa2c0ab 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1340,6 +1340,18 @@ static inline int pmd_protnone(pmd_t pmd) } #endif /* CONFIG_NUMA_BALANCING */ +#ifndef CONFIG_MEMORY_METADATA +static inline bool pte_metadata_none(pte_t pte) +{ + return false; +} + +static inline bool pmd_metadata_none(pmd_t pmd) +{ + return false; +} +#endif /* CONFIG_MEMORY_METADATA */ + #endif /* CONFIG_MMU */ #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP From patchwork Wed Aug 23 13:13:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362308 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80FD1EE49B5 for ; Wed, 23 Aug 2023 13:17:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F32A28006E; Wed, 23 Aug 2023 09:17:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A42728005D; Wed, 23 Aug 2023 09:17:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0449528006E; Wed, 23 Aug 2023 09:17:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E907728005D for ; Wed, 23 Aug 2023 09:17:25 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A4567A0121 for ; Wed, 23 Aug 2023 13:17:25 +0000 (UTC) X-FDA: 81155420850.22.728862A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id D024F20003 for ; Wed, 23 Aug 2023 13:17:23 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796643; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=73GXIE0LgOzCBY/ZFf7Lz2IoDRXZ17bZk4L1yzuQFYw=; b=JXBAcqUb2lltFb5VPGj01qS71vRB2SlCNkb0k6L6g5UaFX2SbYx2H/1pbWCUapUnq8adJi vt9vS/oksLju6/Wu+OATDYpbtDe2ll5mxVOnFIRZm2hacMvbGUWVR/C4fbds3RY0OaSxjy fl/XRE6swjhCNbrybypiBc9PbhK6vpw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796644; a=rsa-sha256; cv=none; b=cXH5UPuYb6RZR0SYbLAptwfwkE5yb4ef+mcRBPiwI4qahG37ucQFN33kA5xNa6bcB9Rog0 3zOmy3vSc4GGdt0my1Pe6Zao5rgpgKcgmj4k+hmgbmJwI7qJMrmmNL2tmIr/XCSJVtxDc/ ZRKeq/18Wr7iksZpVTN0M4KsR1mwZBc= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B515415DB; Wed, 23 Aug 2023 06:18:03 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 376473F740; Wed, 23 Aug 2023 06:17:17 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 30/37] mm: mprotect: arm64: Set PAGE_METADATA_NONE for mprotect(PROT_MTE) Date: Wed, 23 Aug 2023 14:13:43 +0100 Message-Id: <20230823131350.114942-31-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: nqobxnpur5gnapcqmf6j5x5g7mdypt7x X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D024F20003 X-HE-Tag: 1692796643-212972 X-HE-Meta: U2FsdGVkX1858/sPL4Y8887oCfvozIyj2mOtEse07KbdhAb9aOSdk/23uKiRpS6geQcXXMWS0D5OeaGPwH9MQGcKmhUc2PW7b5MPlYh2pKqMCkk51vAtzHjJZqiXs9TkQNqxRDjp80pYWhPtjNHkm9bXBILkD1BJr+zdKFDzVwKa9f94UWjU6Ipnr//0wHWg28rSils3x2F4YsHoezfoPie4foc7sZixUUUqDWDtMLKiIkdSXc3kjDzc2WWHXm/Y3NeGl6tpMQlO5bJ2fmomwtFUo0TuuyM9XrnrobCvouzfmU+Zsq6YGQz3RDALFDAFHg29JppqCXIMCncfzljBLQ17ElimaV0tyZAyvsgrF1DMgkwflSO48MaMBq1G4LsTQoj1QGKIW6cRtwRsh6FaihjtvcOvXE6tkM84gMrBw4wcAB+kUyVg4qj827yXtW+MMF1G9gtx7CSV+JuchaAVpeL8Nk91/nVcHnl81oOTPEKW1lbpGrvWfzAWcMhZ+GUKck6jIoW/UgBqad45UjQ+R0KzxEOWiTy6nxBCHZRWUTEMnsBvnxCWiW0j6p+VbssYRleLaoeBeqhFBc3SyPWwCE+lRyPxWijzQpRf5uewxxOgW6X2P3BM3u5++2FwaUdFEY+cB47/mKDoa6hsnM14ykOTHSaGj+y52+Wb4iooOCyN474RqA0vx/Jjc7Svyll/G+nJWvkDAqS0BBs66faJxC5/RexdHQIA5/GcYvmrC7Bjq2Te7a3lijxAd9W3bzabJTj8MR1qSXRcUc/yAKNnjabELRmntwdegTCMKvIlie0a7Y5rAaTPTDK4HzmqqoN5I7m9gd3I1Y8EjSej7ve3Mp0VrnV5yYKIp4mSoSFASvRSM9GnMFn9RApQSDZYcNYxmns148SdzYdMVxsskl3VLY/qHVIoQ5wV4Dn+lDhEAJdt5Isa0+BtcNkbPHBjLYU3iyOnvmCXLnykmd6pDgX VJjYSmiU UQAcypovqM5G+o96dNI45Pa3aJjyCqbMgbyWATiIaIcmC9gcCJb/5jEIhFUygP0dAgEFl23gDhDlp2zT/A9BxhmbzJ62opJlOZptYYdwlGSF1ZwvAsdVYFZQxUIn84IZa+n5DqiDLvnZM5Jl06s/1ePBSUYClYhR05pCVlt94K8BpaBSdciJ234qVOxzh8LsALkzVszE7xWSvzjM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To enable tagging on a memory range, userspace can use mprotect() with the PROT_MTE access flag. Pages already mapped in the VMA obviously don't have the associated tag storage block reserved, so mark the PTEs as PAGE_METADATA_NONE to trigger a fault next time they are accessed, and reserve the tag storage as part of the fault handling. If the tag storage for the page cannot be reserved, then migrate the page, because alloc_migration_target() will do the right thing and allocate a destination page with the tag storage reserved. If the mapped page is a metadata storage page, which cannot have metadata associated with it, the page is unconditionally migrated. This has several benefits over reserving the tag storage as part of the mprotect() call handling: - Tag storage is reserved only for pages that are accessed. - Reduces the latency of the mprotect() call. - Eliminates races with page migration. But all of this is at the expense of an extra page fault until the pages being accessed all have their corresponding tag storage reserved. This is only implemented for PTE mappings; PMD mappings will follow. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 6 ++ include/linux/migrate_mode.h | 1 + include/linux/mm.h | 2 + mm/memory.c | 152 +++++++++++++++++++++++++++- mm/mprotect.c | 46 +++++++++ 5 files changed, 206 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index ba316ffb9aef..27bde1d2609c 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -531,6 +531,10 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) mutex_lock(&tag_blocks_lock); + /* Can happen for concurrent accesses to a METADATA_NONE page. */ + if (page_tag_storage_reserved(page)) + goto out_unlock; + /* Make sure existing entries are not freed from out under out feet. */ xa_lock_irqsave(&tag_blocks_reserved, flags); for (block = start_block; block < end_block; block += region->block_size) { @@ -568,6 +572,8 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) set_bit(PG_tag_storage_reserved, &(page + i)->flags); memalloc_isolate_restore(cflags); + +out_unlock: mutex_unlock(&tag_blocks_lock); return 0; diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index f37cc03f9369..5a9af239e425 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -29,6 +29,7 @@ enum migrate_reason { MR_CONTIG_RANGE, MR_LONGTERM_PIN, MR_DEMOTION, + MR_METADATA_NONE, MR_TYPES }; diff --git a/include/linux/mm.h b/include/linux/mm.h index ce87d55ecf87..6bd7d5810122 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2466,6 +2466,8 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma, #define MM_CP_UFFD_WP_RESOLVE (1UL << 3) /* Resolve wp */ #define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) +/* Whether this protection change is to allocate metadata on next access */ +#define MM_CP_PROT_METADATA_NONE (1UL << 4) bool vma_needs_dirty_tracking(struct vm_area_struct *vma); int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot); diff --git a/mm/memory.c b/mm/memory.c index 01f39e8144ef..6c4a6151c7b2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -51,6 +51,7 @@ #include #include #include +#include #include #include #include @@ -82,6 +83,7 @@ #include #include +#include #include #include #include @@ -4681,6 +4683,151 @@ static vm_fault_t do_fault(struct vm_fault *vmf) return ret; } +/* Returns with the page reference dropped. */ +static void migrate_metadata_none_page(struct page *page, struct vm_area_struct *vma) +{ + struct migration_target_control mtc = { + .nid = NUMA_NO_NODE, + .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_TAGGED, + }; + LIST_HEAD(pagelist); + int ret, tries; + + lru_cache_disable(); + + if (!isolate_lru_page(page)) { + put_page(page); + lru_cache_enable(); + return; + } + /* Isolate just grabbed another reference, drop ours. */ + put_page(page); + + list_add_tail(&page->lru, &pagelist); + + tries = 5; + while (tries--) { + ret = migrate_pages(&pagelist, alloc_migration_target, NULL, + (unsigned long)&mtc, MIGRATE_SYNC, MR_METADATA_NONE, NULL); + if (ret == 0 || ret != -EBUSY) + break; + } + + if (ret != 0) { + list_del(&page->lru); + putback_movable_pages(&pagelist); + } + lru_cache_enable(); +} + +static vm_fault_t do_metadata_none_page(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct page *page = NULL; + bool do_migrate = false; + pte_t new_pte, old_pte; + bool writable = false; + vm_fault_t err; + int ret; + + /* + * The pte at this point cannot be used safely without validation + * through pte_same(). + */ + vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd); + spin_lock(vmf->ptl); + if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; + } + + /* Get the normal PTE */ + old_pte = ptep_get(vmf->pte); + new_pte = pte_modify(old_pte, vma->vm_page_prot); + + /* + * Detect now whether the PTE could be writable; this information + * is only valid while holding the PT lock. + */ + writable = pte_write(new_pte); + if (!writable && vma_wants_manual_pte_write_upgrade(vma) && + can_change_pte_writable(vma, vmf->address, new_pte)) + writable = true; + + page = vm_normal_page(vma, vmf->address, new_pte); + if (!page) + goto out_map; + + /* + * This should never happen, once a VMA has been marked as tagged, that + * cannot be changed. + */ + if (!(vma->vm_flags & VM_MTE)) + goto out_map; + + /* Prevent the page from being unmapped from under us. */ + get_page(page); + vma_set_access_pid_bit(vma); + + pte_unmap_unlock(vmf->pte, vmf->ptl); + + /* + * Probably the page is being isolated for migration, replay the fault + * to give time for the entry to be replaced by a migration pte. + */ + if (unlikely(is_migrate_isolate_page(page))) { + if (!(vmf->flags & FAULT_FLAG_TRIED)) + err = VM_FAULT_RETRY; + else + err = 0; + put_page(page); + return 0; + } else if (is_migrate_metadata_page(page)) { + do_migrate = true; + } else { + ret = reserve_metadata_storage(page, 0, GFP_HIGHUSER_MOVABLE); + if (ret == -EINTR) { + put_page(page); + return VM_FAULT_RETRY; + } else if (ret) { + do_migrate = true; + } + } + if (do_migrate) { + migrate_metadata_none_page(page, vma); + /* + * Either the page was migrated, in which case there's nothing + * we need to do; either migration failed, in which case all we + * can do is try again. So don't change the pte. + */ + return 0; + } + + put_page(page); + + spin_lock(vmf->ptl); + if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; + } + +out_map: + /* + * Make it present again, depending on how arch implements + * non-accessible ptes, some can allow access by kernel mode. + */ + old_pte = ptep_modify_prot_start(vma, vmf->address, vmf->pte); + new_pte = pte_modify(old_pte, vma->vm_page_prot); + new_pte = pte_mkyoung(new_pte); + if (writable) + new_pte = pte_mkwrite(new_pte); + ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, new_pte); + update_mmu_cache(vma, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + + return 0; +} + int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, unsigned long addr, int page_nid, int *flags) { @@ -4941,8 +5088,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); - if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) + if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) { + if (metadata_storage_enabled() && pte_metadata_none(vmf->orig_pte)) + return do_metadata_none_page(vmf); return do_numa_page(vmf); + } spin_lock(vmf->ptl); entry = vmf->orig_pte; diff --git a/mm/mprotect.c b/mm/mprotect.c index 6f658d483704..2c022133aed3 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -89,6 +90,7 @@ static long change_pte_range(struct mmu_gather *tlb, long pages = 0; int target_node = NUMA_NO_NODE; bool prot_numa = cp_flags & MM_CP_PROT_NUMA; + bool prot_metadata_none = cp_flags & MM_CP_PROT_METADATA_NONE; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; @@ -161,6 +163,40 @@ static long change_pte_range(struct mmu_gather *tlb, jiffies_to_msecs(jiffies)); } + if (prot_metadata_none) { + struct page *page; + + /* + * Skip METADATA_NONE pages, but not NUMA pages, + * just so we don't get two faults, one after + * the other. The page fault handling code + * might end up migrating the current page + * anyway, so there really is no need to keep + * the pte marked for NUMA balancing. + */ + if (pte_protnone(oldpte) && pte_metadata_none(oldpte)) + continue; + + page = vm_normal_page(vma, addr, oldpte); + if (!page || is_zone_device_page(page)) + continue; + + /* Page already mapped as tagged in a shared VMA. */ + if (page_has_metadata(page)) + continue; + + /* + * The LRU takes a page reference, which means + * that page_count > 1 is true even if the page + * is not COW. Reserving tag storage for a COW + * page is ok, because one mapping of that page + * won't be migrated; but not reserving tag + * storage for a page is definitely wrong. So + * don't skip pages that might be COW, like + * NUMA does. + */ + } + oldpte = ptep_modify_prot_start(vma, addr, pte); ptent = pte_modify(oldpte, newprot); @@ -531,6 +567,13 @@ long change_protection(struct mmu_gather *tlb, WARN_ON_ONCE(cp_flags & MM_CP_PROT_NUMA); #endif +#ifdef CONFIG_MEMORY_METADATA + if (cp_flags & MM_CP_PROT_METADATA_NONE) + newprot = PAGE_METADATA_NONE; +#else + WARN_ON_ONCE(cp_flags & MM_CP_PROT_METADATA_NONE); +#endif + if (is_vm_hugetlb_page(vma)) pages = hugetlb_change_protection(vma, start, end, newprot, cp_flags); @@ -661,6 +704,9 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; vma_set_page_prot(vma); + if (metadata_storage_enabled() && (newflags & VM_MTE) && !(oldflags & VM_MTE)) + mm_cp_flags |= MM_CP_PROT_METADATA_NONE; + change_protection(tlb, vma, start, end, mm_cp_flags); /* From patchwork Wed Aug 23 13:13:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C119EE4993 for ; Wed, 23 Aug 2023 13:17:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A586B28006F; Wed, 23 Aug 2023 09:17:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A0A4928005D; Wed, 23 Aug 2023 09:17:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 900A128006F; Wed, 23 Aug 2023 09:17:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 80D7E28005D for ; Wed, 23 Aug 2023 09:17:32 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5EA771A0120 for ; Wed, 23 Aug 2023 13:17:32 +0000 (UTC) X-FDA: 81155421144.12.BC58DED Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 6A3E540002 for ; Wed, 23 Aug 2023 13:17:30 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf11.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5fj6tIYo4ds0aZkTInoE9tiuTxGBTaiRl4FDFdunSvM=; b=hGVa60JrbmytdRCkmPOZJ/nWtyJxbbB4fiKsBOmO1U+UrhJAV7im5GaN4aI/y1DyAewLmO qcGI4pN0TshO2T5UvCXy7zGvjVvu3ERJ3Qn0+Sh3uC4vCp9ZXC5LXl4THuc+d54t/hrZJA Rs+aEQnB4NC0NvnC2Kl15ztd1FhcPUw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf11.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796650; a=rsa-sha256; cv=none; b=4hwT2qq4ha0xG+NtIhghkskdBsEv8l/Uj6jkrRt0TCisVWoyhtLLRa3w/idJGAt/3XOjoR dJnO6OptBTIpiJXEpksm+HZSyTV6Jk+qgKqwBeNAhPnBXBpSDCG6BY75f4LDppDeikSsTZ 4n9bIyKAKf1hNyNb46N7xbZSHqg3S4Q= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4D5841756; Wed, 23 Aug 2023 06:18:10 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6F2923F740; Wed, 23 Aug 2023 06:17:23 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 31/37] mm: arm64: Set PAGE_METADATA_NONE in set_pte_at() if missing metadata storage Date: Wed, 23 Aug 2023 14:13:44 +0100 Message-Id: <20230823131350.114942-32-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6A3E540002 X-Stat-Signature: qn47nczixura7dr64zmbf8quowsajimc X-HE-Tag: 1692796650-285581 X-HE-Meta: U2FsdGVkX1+WZ6mMjPfh0XBXcjk997IyHKHHInohM+p5Kk68SKKy7aY9QRYLJJbSQaAxxAQGRpNmk9Te/6jz4n6tZ9CCpfqV58d6ai+SlxYJBjtnIAg3mVfghUQ1+8oOkTS5lToRFrozGpHwZcdyMYnr6gGWFjVJFd6wRogK/jCawzLkrsgeCkTAyfS2u/TCIONzEzmDBuysQ/SXvPinPP+QDr06IX7Iuxq6oECxvffBlgQfROfDVRoL4qbTS8sXLOzJh4lWCbqb24vW/OJ+kw0/rxMs3OXA7CdT2U2tDeaDGFarHiInE/+vJMRY3IHSjmqutN76MoZOa93ud+H2D+fSqMkmaDN2STWaP9nqpNLE5ic9mKpIHMAdCTUHtF16AUULe5FGyh8QmIKpNgLoEZiBqUWtpHDnPW3zYTegU4/t1PqY570WHbsSXT+Pz1VDatIW1jkaI7Z0gW2lpLIBFLrMwx6/AmaR8dKgAnFi8Sokh/btgTk6QpkN+VLh92412ytzrfVOra4UD2q9v5rZwyA3T/x+jjzoDNzXGHtiwrrghAheOzm1TLH/wgK51Ru/6CIjqDa8qgA0KcI1/tQnEcCinei7HTpDO/CP0+/MHaRfFcM243KdHPq9ThpuK82duYP7tVrTwmEpvMFmlZFGT+618rFWIL6pHhZuQF4LKgYSSHjh4N1dHLLSOh4xr/bVtbUUuRYoubi3F/3uuAcODuEJ5zKc2qOBi8z299pHw2UlGB3araMKICnOs85+lofkrOtn3AP/VV6a9G8qqcZ6bZURxDdvOMlW1fidenYsvsffmQg1ekxjcCGFd+LZSzP/SYp09BdfdCdFFKoVRsK1AqypCknvFNbN9zy3gnRd/c1EyNDBCXsWQETyZSpYdfjkwvmjFqXWv+yWncEfAxuFk0nZz4FCuWm4bsgicuCc6L6BcVV6Ern+D6LsOkMTAY5Ab8QQ+RVHHoQH+H6LTBi UztE6UAy VlsjRKwNnAH6f1pZ57ahZGUAr85Pv2n2tOfIeZvU00VLMbULpCilVWIyhy0WLjRRBfcGsGn09mUHX3OkzeI5oprWY+cRzUPO5+btR/NcfBnLkfGxYKvduN0pDnG9vCY/Ch0KAmwHb+yx8ZWZhZ1PF4PoQnD/SnjUMSBKagWqgKs1UOPpWftaUy4rbh5I3TUPvN+qF/r/hVT6r2Yw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a metadata page is mapped in the process address space and then mprotect(PROT_MTE) changes the VMA flags to allow the use of tags, the page is migrated out when it is first accessed. But this creates an interesting corner case. Let's consider the scenario: Initial conditions: metadata page M1 and page P1 are mapped in a VMA without VM_MTE. The metadata storage for page P1 is **metadata page M1**. 1. mprotect(PROT_MTE) changes the VMA, so now all pages must have the associated metadata storage reserved. The to-be-tagged pages are marked as PAGE_METADATA_NONE. 2. Page P1 is accessed and metadata page M1 must be reserved. 3. Because it is mapped, the metadata storage code will migrate metadata page M1. The replacement page for M1, page P2, is allocated without metadata storage (__GFP_TAGGED is not set). This is done intentionally in reserve_metadata_storage() to avoid recursion and deadlock. 4. Migration finishes and page P2 replaces M1 in a VMA with VM_MTE set. The result: P2 is mapped in a VM_MTE VMA, but the associated metadata storage is not reserved. Fix this by teaching set_pte_at() -> mte_sync_tags() to change the PTE protection to PAGE_METADATA_NONE when the associated metadata storage is not reserved. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte.h | 4 ++-- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/kernel/mte.c | 14 +++++++++++--- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 70cfd09b4a11..e89d1fa3f410 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -108,7 +108,7 @@ static inline bool try_page_mte_tagging(struct page *page) } void mte_zero_clear_page_tags(void *addr); -void mte_sync_tags(pte_t pte); +void mte_sync_tags(pte_t *pteval); void mte_copy_page_tags(void *kto, const void *kfrom); void mte_thread_init_user(void); void mte_thread_switch(struct task_struct *next); @@ -140,7 +140,7 @@ static inline bool try_page_mte_tagging(struct page *page) static inline void mte_zero_clear_page_tags(void *addr) { } -static inline void mte_sync_tags(pte_t pte) +static inline void mte_sync_tags(pte_t *pteval) { } static inline void mte_copy_page_tags(void *kto, const void *kfrom) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2e42f7713425..e5e1c23afb14 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -338,7 +338,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, */ if (system_supports_mte() && pte_access_permitted(pte, false) && !pte_special(pte) && pte_tagged(pte)) - mte_sync_tags(pte); + mte_sync_tags(&pte); __check_safe_pte_update(mm, ptep, pte); diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 4edecaac8f91..4556989f0b9e 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -20,7 +20,9 @@ #include #include +#include #include +#include #include #include @@ -35,13 +37,19 @@ DEFINE_STATIC_KEY_FALSE(mte_async_or_asymm_mode); EXPORT_SYMBOL_GPL(mte_async_or_asymm_mode); #endif -void mte_sync_tags(pte_t pte) +void mte_sync_tags(pte_t *pteval) { - struct page *page = pte_page(pte); + struct page *page = pte_page(*pteval); long i, nr_pages = compound_nr(page); - /* if PG_mte_tagged is set, tags have already been initialised */ for (i = 0; i < nr_pages; i++, page++) { + if (metadata_storage_enabled() && + unlikely(!page_tag_storage_reserved(page))) { + *pteval = pte_modify(*pteval, PAGE_METADATA_NONE); + continue; + } + + /* if PG_mte_tagged is set, tags have already been initialised */ if (try_page_mte_tagging(page)) { mte_clear_page_tags(page_address(page)); set_page_mte_tagged(page); From patchwork Wed Aug 23 13:13:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E6A5EE4993 for ; Wed, 23 Aug 2023 13:17:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2D1A280070; Wed, 23 Aug 2023 09:17:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDC4128005D; Wed, 23 Aug 2023 09:17:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACBDD280070; Wed, 23 Aug 2023 09:17:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9D8FE28005D for ; Wed, 23 Aug 2023 09:17:39 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7907812011A for ; Wed, 23 Aug 2023 13:17:39 +0000 (UTC) X-FDA: 81155421438.18.674B16B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 98814C0005 for ; Wed, 23 Aug 2023 13:17:37 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796657; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0SsnpIoQ3y7pPNkB5bLokPAkvyjt/i+TrfRWqY2uaYQ=; b=QqHwW1+hBGijxfWXpUxGIoDqpAyfnXCrx5pZZSTWXeg9INZ9iq4euLcS7p/ZsmLuE0sh1a 1ziekHktbabHthl2s//eBHM0xpBeylRP8Rc9f/PH4xkIZj8Jtzis3MpUJL6GePDPHcG5dd GqPARp0RHdUiRsLwgHx5B5tE9IrGy8o= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796657; a=rsa-sha256; cv=none; b=Ipcd0YW6WDG4T8BrDs1McY0OqZZNUtyhb6Hx2jrUdVwq6unptf4qBEsBO31hYakHylyPpo I609KPkMszhknPaVqc5mR3PRvHmZNk0ClYALdSqzlurx5IadgCjX4UgBXkGynqEakBp6Xd m1Y+eusBSjr2OROSxqJMdTXME8UYUQ4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8531815BF; Wed, 23 Aug 2023 06:18:17 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0FC3D3F740; Wed, 23 Aug 2023 06:17:29 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 32/37] mm: Call arch_swap_prepare_to_restore() before arch_swap_restore() Date: Wed, 23 Aug 2023 14:13:45 +0100 Message-Id: <20230823131350.114942-33-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: baj4uafw11wcnmr9o88a59zxopuq76zy X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 98814C0005 X-HE-Tag: 1692796657-896594 X-HE-Meta: U2FsdGVkX1/cxGePVBsBu8m4hAVLsgp8rSE7/dC3HqFJbivACelelSUIlWrDt5qkU0Bx3q3BT2nmbdwC9SDgG0i5PCgVXK+cNNyWXXXawQnZufilgf7AB8Zw2DFlGYInxcyJns5HslfguchBc7ikuST2mUyn6I5ov3utTBUq8avN5NpzxxWnq7rIgNRpAPVgg28M2lkCV9yyOW4mLldm5baSL0GEhTsl2aOBZFTXyiAXkhImkX6nojhz/WaBceSWs0+6h6XPR7NpCOrfg3wjdt814UHUUnUPfAMu75tA/amFqwVMzvIbKdqLjTq4WY75DpRoRM8Ntt41ORQ/wGiMIRQXzXahoX6aKTm2Rwv402Poc1JB65e/01S1Y7z4ceNHpIsRPQ/gDBUQ+bfkb7KC8GElZuD3UvDxLBNblLMTDtZCKL9LFqhwT5/K6TVlF4N5eeUpeWYn9llgpNO+j+rYvRhwaVz3DI66Dw6q7O1aAt/NXoqtPAlvFb+QKXnVgTxSqBzzUUszL0MQb/wFonA6W0pPgdJ28T0NLSJum2emSjoVPg72+iDpt+aQfR967Jxm+7vzUo+y29H50ImYeA51Ux4EbTxV9vfqXiIr4/1B9NCJtsKq/h04bb6zfbKcJBhF6hDC8/Lz+V4EJ4Fi7IaS5bHtQktRT9IahjisWAAlntfYeVbi2psX/FGcB1xGIL/tXNIRcoGN4JOOtEPXbE5eBj5fRsETYD7OAUlnbat6p9QhIfawXs2DcqaBuKOotU83YGJ5vwypFLuH3RKZNB9zlyDotXuhdW1P0PAc5SYptYIeVY5bN8KMVU30ZN38kL6N6bgnemUG1TYE65JqojGJW6+PvmffRo7aUUH/j6l7lDvsL2N0fZQfBxpmgff/XzzOna/MCsB0pfeBpWAjaebty4bHACMRL+2MXGdH/Hfl/GX15lY/VmjLzJsnIbI3fuCckty7ma6zUrFh9uOsJzu 3oXbcLOc iEBlWMp0erRNu7X7utnrr4ow398RDECtN116WMJRpNHW/aQcuulUsYeQ/b86W+cTzrZZFbK0D+dFiNLFLUH+0rqpV+tYPF7A8T1oCa0xrM4wxOS1FMC7CxhkAuH1uSa3lETZsZNU9uvamPeVEtXgPm2lunRLXZ5PvwVE+9CNFOtJ/UY1bPe6D/cN6vustCaRqozDylZno9DpIWno= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: arch_swap_restore() allows an architecture to restore metadata before the page is swapped in and it's called in atomic context (with the ptl lock held). Introduce arch_swap_prepare_to_restore() to allow such architectures to perform extra work in a blocking context. Signed-off-by: Alexandru Elisei --- include/linux/pgtable.h | 7 +++++++ mm/memory.c | 11 +++++++++++ mm/shmem.c | 4 ++++ mm/swapfile.c | 4 ++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 0119ffa2c0ab..0bce12f9eaab 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -816,6 +816,13 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) } #endif +#ifndef __HAVE_ARCH_SWAP_PREPARE_TO_RESTORE +static inline int arch_swap_prepare_to_restore(swp_entry_t entry, struct folio *folio) +{ + return 0; +} +#endif + #ifndef __HAVE_ARCH_PGD_OFFSET_GATE #define pgd_offset_gate(mm, addr) pgd_offset(mm, addr) #endif diff --git a/mm/memory.c b/mm/memory.c index 6c4a6151c7b2..5f7587109ac2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3724,6 +3724,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swp_entry_t entry; pte_t pte; int locked; + int error; vm_fault_t ret = 0; void *shadow = NULL; @@ -3892,6 +3893,16 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_throttle_swaprate(folio, GFP_KERNEL); + /* + * Some architecture may need to perform certain operations before + * arch_swap_restore() in preemptible context (like memory allocations). + */ + error = arch_swap_prepare_to_restore(entry, folio); + if (error) { + ret = VM_FAULT_ERROR; + goto out_page; + } + /* * Back out if somebody else already faulted in this pte. */ diff --git a/mm/shmem.c b/mm/shmem.c index 0b772ec34caa..4704be6a4e9b 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1796,6 +1796,10 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } folio_wait_writeback(folio); + error = arch_swap_prepare_to_restore(swap, folio); + if (error) + goto unlock; + /* * Some architectures may have to restore extra metadata to the * folio after reading from swap. diff --git a/mm/swapfile.c b/mm/swapfile.c index 6d719ed5c616..387971e2c5f0 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1756,6 +1756,10 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, else if (unlikely(PTR_ERR(page) == -EHWPOISON)) hwposioned = true; + ret = arch_swap_prepare_to_restore(entry, folio); + if (ret) + return ret; + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (unlikely(!pte || !pte_same_as_swp(ptep_get(pte), swp_entry_to_pte(entry)))) { From patchwork Wed Aug 23 13:13:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DF26EE49A3 for ; Wed, 23 Aug 2023 13:17:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B693A280071; Wed, 23 Aug 2023 09:17:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B193E28005D; Wed, 23 Aug 2023 09:17:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BA58280071; Wed, 23 Aug 2023 09:17:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8AE5A28005D for ; Wed, 23 Aug 2023 09:17:46 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 60E021C841D for ; Wed, 23 Aug 2023 13:17:46 +0000 (UTC) X-FDA: 81155421732.05.9216A7A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 6FCE018001D for ; Wed, 23 Aug 2023 13:17:44 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796664; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CPdSU4khSHq8eswwVD97Mwk7RgGfdaE4Otuc9ZM1tZo=; b=mauAglrZYtIzJ/XKIWuWL39ug55rmrjYi5e01Xq4LvkSx2rG/jmu3SIOPpZ8juL2d4nYeh E98Wya0ygtgn5dN9eWmOuv/1vXiYE6YBQ0Gx83/nn7kReXERzBZrbKtdbamF73LhVJJ7xf aI2ONr1Q8c9YyAltfc6MRmYg3bcOLS0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796664; a=rsa-sha256; cv=none; b=r3v16zPyLQqd8tX1qnCpgBc7MyZyOv+jSxazIHd4SGkhGFwbhUEd8N0XaHiYrOhlMOmmjw TaGAZp96kVfs+mX3yBeVyJCWQor421dYAWcpWkqmHx27ZOEDIHkJG9mj4lhbv9xrwrENQU Z8JAGDSZ/JVMGolr8JYBW6QlMXBgOVA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6FA3A1713; Wed, 23 Aug 2023 06:18:24 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5E4203F740; Wed, 23 Aug 2023 06:17:37 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 33/37] arm64: mte: swap/copypage: Handle tag restoring when missing tag storage Date: Wed, 23 Aug 2023 14:13:46 +0100 Message-Id: <20230823131350.114942-34-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6FCE018001D X-Rspam-User: X-Stat-Signature: 6z8iqb51umnsrt1ybyri6jctoh1cd1gy X-Rspamd-Server: rspam01 X-HE-Tag: 1692796664-998831 X-HE-Meta: U2FsdGVkX19g798Ydyga+BT0Z5/nvH6FKXU/kkFoQAyOHBU6Qqkw+XEwbwY5afbEwBViT0lVPWeQOrsuiWyz2REfMpV/qH458VmYWj9rVmjfaOFNlzc1axf4/IZmlZCUd6UM3LEz9pxbvoF1PDYxJr9cruj4YxRNtRT9foPHM4yeotuLN6p7vEochtXthVSyiydAc8fGd9cIUysijV/6Vy/cSQzbmEw85GiLd/vKC8l5FED3GemXUIVyR3JYZKgaiwU09NqhhWRvg+jyJb8s5OXseLC3Hl8c5edfYUIAAID76I7dDRm34C5pkWQGmxN/TRtlOga59oVjFPMGD2TfaPuYOFZjO15zW1hwKetYfqitBJ3fZDy1FfgPHF4a70in5FXKbdFQD08oCVB07oeii58W8jZMPIBU3X8fyrhAKggzB1+yF0Iz5eV970UgF8R2CHHCjAtLRVwRnkyPlwKJMWjxAcAziT+RkBdfPmaceMMvqaSg40u8ppVdyLta4nqZVd/BH6EUiGzxmOk2kf2w18WjMqllXZ50bcxGr5nz11k5J4TMFuWGZlzboqPVpYEdMXCoEevvdiAdwLGkeAuig7I9o91fJBIyPalYzd4bin4WBSRXdcuWKTs5sFlOgO0gWPOoKQpo0XFqM5ra/feCSWW7fBwPqWXP2EBOPO9zoSpxpQ1g6NIW58mBuK3xjrd18buQfMNF8BwdglBa/hGGsqrLkitDTAiBenaQzLMHKue4P5Jc/7Ey2/zR0zYTbsFwlGhv6oH/LRtUnTysGvl9FgEJyfUUy2DbPqHGSmlmkTvobHRfpeTizaC3IYsFaNkM36+mhN3dIYa4KKooO5Ril5VA8eIyISosMNNnFsUEYWiXq9b+lP76cq2gtK0/ZJZ+ohiaW//sUemPnVOB3mCmlQdUguL+XGm8kO9TiF9mAUyAtL+ZeWaxzL0ivkYmCWPod+7E0UqXpPUm0321COK 7FcDQiXW 9zFmp97E19xeQtfqD020Eqim2Z64OIqBHs4DiprzQrsBlMfWJHGTXMtZeJvOC2mwnqrhebRuMDLbQEQzzP9Tvc9pmbOGQ5EGxEA+KcAHChVvu9HiY2w2nA/ilUHSnqfHwkLTNx2qOu+eTVESf75nf01ypLGUNz/+IzSM/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Linux restores tags when a page is swapped in and there are tags saved for the swap entry which the new page will replace. The tags are restored even if the page will not be mapped as tagged. This is done so when a shared page is swapped in as untagged, followed by mprotect(PROT_MTE), the process can still access the correct tags. But this poses a challenge for tag storage: when a page is swapped in for the process where it is untagged, the corresponding tag storage block is not reserved, and restoring the tags can overwrite data in the tag storage block, leading to data corruption. Get around this issue by saving the tags in a new xarray, this time indexed by the page pfn, and then restoring them in set_pte_at(). Something similar can happen when a page is migrated: the migration process starts and the destination page is allocated when the VMA does not have MTE enabled (so tag storage is not reserved as part of the allocation), mprotect(PROT_MTE) is called before migration finishes and the source page is accessed (thus marking it as tagged). When folio_copy() is called, the code will try to copy the tags to the destination page, which doesn't have tag storage reserved. Fix this in a similar way to tag restoring when doing swap in, by saving the tags of the source page in a buffer, then restoring them in set_pte_at(). Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/memory_metadata.h | 1 + arch/arm64/include/asm/mte_tag_storage.h | 11 +++++ arch/arm64/include/asm/pgtable.h | 7 +++ arch/arm64/kernel/mte.c | 17 +++++++ arch/arm64/kernel/mte_tag_storage.c | 9 +++- arch/arm64/mm/copypage.c | 26 ++++++++++ arch/arm64/mm/mteswap.c | 63 ++++++++++++++++++++++++ include/asm-generic/memory_metadata.h | 4 ++ mm/memory.c | 12 +++++ 9 files changed, 149 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/memory_metadata.h b/arch/arm64/include/asm/memory_metadata.h index 167b039f06cf..25b4d790e92b 100644 --- a/arch/arm64/include/asm/memory_metadata.h +++ b/arch/arm64/include/asm/memory_metadata.h @@ -43,6 +43,7 @@ static inline bool vma_has_metadata(struct vm_area_struct *vma) int reserve_metadata_storage(struct page *page, int order, gfp_t gfp_mask); void free_metadata_storage(struct page *page, int order); +bool page_metadata_in_swap(struct page *page); #endif /* CONFIG_MEMORY_METADATA */ #endif /* __ASM_MEMORY_METADATA_H */ diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index bad865866eeb..cafbb618d97a 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -12,6 +12,9 @@ extern void dcache_inval_tags_poc(unsigned long start, unsigned long end); #ifdef CONFIG_ARM64_MTE_TAG_STORAGE void mte_tag_storage_init(void); bool page_tag_storage_reserved(struct page *page); + +void *mte_erase_page_tags_by_pfn(struct page *page); +int mte_save_page_tags_by_pfn(struct page *page, void *tags); #else static inline void mte_tag_storage_init(void) { @@ -20,6 +23,14 @@ static inline bool page_tag_storage_reserved(struct page *page) { return true; } +static inline void *mte_erase_page_tags_by_pfn(struct page *page) +{ + return NULL; +} +static inline int mte_save_page_tags_by_pfn(struct page *page, void *tags) +{ + return 0; +} #endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ #endif /* __ASM_MTE_TAG_STORAGE_H */ diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e5e1c23afb14..a1e93d3228fa 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1056,6 +1056,13 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) mte_restore_page_tags_by_swp_entry(entry, &folio->page); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE + +#define __HAVE_ARCH_SWAP_PREPARE_TO_RESTORE +int arch_swap_prepare_to_restore(swp_entry_t entry, struct folio *folio); + +#endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ + #endif /* CONFIG_ARM64_MTE */ /* diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 4556989f0b9e..5139ce6952ff 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -37,6 +37,20 @@ DEFINE_STATIC_KEY_FALSE(mte_async_or_asymm_mode); EXPORT_SYMBOL_GPL(mte_async_or_asymm_mode); #endif +static bool mte_restore_saved_tags(struct page *page) +{ + void *tags = mte_erase_page_tags_by_pfn(page); + + if (likely(!tags)) + return false; + + mte_restore_page_tags_from_mem(page_address(page), tags); + mte_free_tags_mem(tags); + set_page_mte_tagged(page); + + return true; +} + void mte_sync_tags(pte_t *pteval) { struct page *page = pte_page(*pteval); @@ -51,6 +65,9 @@ void mte_sync_tags(pte_t *pteval) /* if PG_mte_tagged is set, tags have already been initialised */ if (try_page_mte_tagging(page)) { + if (metadata_storage_enabled() && + unlikely(mte_restore_saved_tags(page))) + continue; mte_clear_page_tags(page_address(page)); set_page_mte_tagged(page); } diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 27bde1d2609c..ce378f45f866 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -603,7 +603,8 @@ void free_metadata_storage(struct page *page, int order) struct tag_region *region; unsigned long page_va; unsigned long flags; - int ret; + void *tags; + int i, ret; if (WARN_ONCE(!page_mte_tagged(page), "pfn 0x%lx is not tagged", page_to_pfn(page))) return; @@ -619,6 +620,12 @@ void free_metadata_storage(struct page *page, int order) */ dcache_inval_tags_poc(page_va, page_va + (PAGE_SIZE << order)); + for (i = 0; i < (1 << order); i++) { + tags = mte_erase_page_tags_by_pfn(page + i); + if (unlikely(tags)) + mte_free_tags_mem(tags); + } + end_block = start_block + order_to_num_blocks(order) * region->block_size; xa_lock_irqsave(&tag_blocks_reserved, flags); diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c index a7bb20055ce0..e4ac3806b994 100644 --- a/arch/arm64/mm/copypage.c +++ b/arch/arm64/mm/copypage.c @@ -12,7 +12,29 @@ #include #include #include +#include #include +#include + +static bool copy_page_tags_to_page(struct page *to, struct page *from) +{ + void *kfrom = page_address(from); + void *tags; + + if (likely(page_tag_storage_reserved(to))) + return false; + + tags = mte_allocate_tags_mem(); + if (WARN_ON(!tags)) + goto out; + + mte_save_page_tags_to_mem(kfrom, tags); + + if (WARN_ON(mte_save_page_tags_by_pfn(to, tags))) + mte_free_tags_mem(tags); +out: + return true; +} void copy_highpage(struct page *to, struct page *from) { @@ -25,6 +47,10 @@ void copy_highpage(struct page *to, struct page *from) page_kasan_tag_reset(to); if (system_supports_mte() && page_mte_tagged(from)) { + if (metadata_storage_enabled() && + unlikely(copy_page_tags_to_page(to, from))) + return; + /* It's a new page, shouldn't have been tagged yet */ WARN_ON_ONCE(!try_page_mte_tagging(to)); mte_copy_page_tags(kto, kfrom); diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index aaeca57f36cc..f6a9b6f889e6 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -5,7 +5,9 @@ #include #include #include +#include #include +#include static DEFINE_XARRAY(tags_by_swp_entry); @@ -20,6 +22,62 @@ void mte_free_tags_mem(void *tags) kfree(tags); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +static DEFINE_XARRAY(tags_by_pfn); + +int mte_save_page_tags_by_pfn(struct page *page, void *tags) +{ + void *entry; + + entry = xa_store(&tags_by_pfn, page_to_pfn(page), tags, GFP_KERNEL); + if (xa_is_err(entry)) + return xa_err(entry); + else if (entry) + mte_free_tags_mem(entry); + + return 0; +} + +int arch_swap_prepare_to_restore(swp_entry_t entry, struct folio *folio) +{ + struct page *page = &folio->page; + void *swp_tags, *pfn_tags; + int ret; + + might_sleep(); + + if (!metadata_storage_enabled() || page_mte_tagged(page) || + page_tag_storage_reserved(page)) + return 0; + + swp_tags = xa_load(&tags_by_swp_entry, entry.val); + if (!swp_tags) + return 0; + + pfn_tags = mte_allocate_tags_mem(); + if (!pfn_tags) + return -ENOMEM; + + memcpy(pfn_tags, swp_tags, MTE_PAGE_TAG_STORAGE_SIZE); + + ret = mte_save_page_tags_by_pfn(page, pfn_tags); + if (ret) + mte_free_tags_mem(pfn_tags); + + return ret; +} + +void *mte_erase_page_tags_by_pfn(struct page *page) +{ + return xa_erase(&tags_by_pfn, page_to_pfn(page)); +} + +bool page_metadata_in_swap(struct page *page) +{ + return xa_load(&tags_by_pfn, page_to_pfn(page)) != NULL; +} +#endif + int mte_save_page_tags_by_swp_entry(struct page *page) { void *tags, *ret; @@ -53,6 +111,11 @@ void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page) if (!tags) return; + /* Tags already saved in mte_swap_prepare_to_restore(). */ + if (metadata_storage_enabled() && + unlikely(!page_tag_storage_reserved(page))) + return; + if (try_page_mte_tagging(page)) { mte_restore_page_tags_from_mem(page_address(page), tags); set_page_mte_tagged(page); diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index 35a0d6a8b5fc..4176fd89ef41 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -39,6 +39,10 @@ static inline bool vma_has_metadata(struct vm_area_struct *vma) { return false; } +static inline bool page_metadata_in_swap(struct page *page) +{ + return false; +} #endif /* !CONFIG_MEMORY_METADATA */ #endif /* __ASM_GENERIC_MEMORY_METADATA_H */ diff --git a/mm/memory.c b/mm/memory.c index 5f7587109ac2..ade71f38b2ff 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4801,6 +4801,18 @@ static vm_fault_t do_metadata_none_page(struct vm_fault *vmf) put_page(page); return VM_FAULT_RETRY; } else if (ret) { + // TODO: support migrating swap metadata with the page. + if (unlikely(page_metadata_in_swap(page))) { + vm_fault_t err; + + if (vmf->flags & FAULT_FLAG_TRIED) + err = VM_FAULT_OOM; + else + err = VM_FAULT_RETRY; + + put_page(page); + return err; + } do_migrate = true; } } From patchwork Wed Aug 23 13:13:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E9DFEE49A0 for ; Wed, 23 Aug 2023 13:17:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B41D5280072; Wed, 23 Aug 2023 09:17:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF22228005D; Wed, 23 Aug 2023 09:17:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9935D280072; Wed, 23 Aug 2023 09:17:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8A2A528005D for ; Wed, 23 Aug 2023 09:17:52 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5E619B1071 for ; Wed, 23 Aug 2023 13:17:52 +0000 (UTC) X-FDA: 81155421984.21.1314E79 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 93E0B18002D for ; Wed, 23 Aug 2023 13:17:50 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796670; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uWgWK0tZ+ZryHuzLyftTn9MC7JOaN9vOC7vkU1Btibg=; b=B4FtOSVPXZbBBsgOf/m/e6aT18M5OVnJeEpHYE7uMjibhBdecWZRq0RJ/CAJDIZpkKuWkp eLCDdzVUHMs0HWmj3b4vWJya8ec7iJAm9vmX+sTrgBMRdDj24fu8wnsnszSLaMHrx/2KwH 2fDTupMxsYgvOFS2wHDSZejBdlXH/Ls= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796670; a=rsa-sha256; cv=none; b=SOQEPd0RI7t3Biuw1QsCH2hoPGzXzhBDRdckjWckhBuwvhdmPVQUj3VgzKhDQ+Gle9c36p g6Lhm3/FjT0XlyUFkWNYXvOKJNUmbe27Ve9VoUvT2nU2VkmnsYxvjeobGYUVlYwSSiky/v 0GfCubYLfjJBxYp97t7ILfHJ+1y5DTs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6AF6E16F8; Wed, 23 Aug 2023 06:18:30 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2CE8A3F740; Wed, 23 Aug 2023 06:17:44 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 34/37] arm64: mte: Handle fatal signal in reserve_metadata_storage() Date: Wed, 23 Aug 2023 14:13:47 +0100 Message-Id: <20230823131350.114942-35-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 93E0B18002D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: tops98x7qtrmgpz59dgm318fi84naikx X-HE-Tag: 1692796670-530198 X-HE-Meta: U2FsdGVkX1+ZvYZdMmpQNw5iYl7NceaKHxhb4tZlaG6A6axxydiVYh0USZvIKqY+wlG7eBvCGeIRDF6DgikTsruimULQDa+NGEAezDQmuJl89f5TmMo8vQ6YphoCD6a0ewwpib5szw6Bh7JmUp7S15DVowHCVGRgclq0wVnh++EXQAhCZcBJ3+0FKShjKV5aFRRF5BW+k87rygWG/JRer3Otg7oHALHFA6+W0htWIjcfgJfcOgK+F/q0Gv6W93puC7/IR3mJ27QuIDy4/TrQ4IiQVPYWtvRk2Ytd09THcy9l+ZtfJTlASgtzu78JnH1B9m8NlMgipDP5+mi5cL+BCSLSi+MT0ZmQCk7R5+oyB3W5e/zt56/5AD2YyaoM6QD1ZXEAVkOR612pJdzn0mMnK29WKqKnv70mU5kuzB2exIiorBvCHOgFpve/nxNI3OsEdOTiZWjx5HVvgs3iyEB5dCZXrUg4nyu3q2Y7Bvr4HhrfJN7zwBWxwrvsuBkDlW1I0o5H08Y4iBZoPtFXngdWGGnaOV+fT75nFZ9bZ+qgao2FxCUEF08XGLzXycvCsXp/2pyq6lErmJmBfARVlST8jA9BkMiJhXYo5HaE4Mci6NfJu5j2c5MCAmdU0kwSnNiQ/3jPr0zNQ6ZPD9hSMu8pmQwEmtzR4v6UJAV7u0pk6shD7bzrNQXVALWPxrQsr67nU6rgeHMq7MvL4Mz8F4tmZNJCVFu9cMCPHSkgIb97V4MHADSr+bkoRydayr7BHnI651SFvkQDc3j9hzL1gWBcYBresFByFD8hpugTYbrBQ4XPthUEOCSx9qHn/F62KbK5r043cemhD7JaB5ep8UlynwXo149mjXB9o6RTfAOvkxvxhCRY++7lT31YOG6jQ/Q1tMl4Ziy/4I38aZ26lMLuupJxQjpPWg2Bh9yIIxcG37lQOFJOXzIIRTmaCtD1rnlZOBFZU4jrQDSsDQ690Sh 8f/fST48 kkddI8v1KMvZOmOwtbYnzVV28PkU1770nNFSKqWKM5mbc4LNHB7pixfUhuarbPwV5hopAqBN/PpLikiCVbRIATZVIKC5kzyevyuuJBD7IwRJgA3k2xDH9DfBWT7d6oMOXkXJ4PcvsMlAC9+awzO9K+zOSDjA610e7DDLzFLYpq00BCqxajrO//zza8ub/5gptp11wQIdVOPDjp9/5JhRiI9fKDLEd9IIndXcp5In2dzx1rcnAjZrtUW9BZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As long as a fatal signal is pending, alloc_contig_range() will fail with -EINTR. This makes it impossible for tag storage allocation to succeed, and the page allocator will print an OOM splat. The process is going to be killed, so return 0 (success) from reserve_metadata_storage() to allow the page allocator to make progress. set_pte_at() will map it with PAGE_METADATA_NONE and subsequent accesses from different threads will trap until the signal is delivered. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 17 +++++++++++++++++ arch/arm64/mm/fault.c | 23 +++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index ce378f45f866..1ccbcc144979 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -556,6 +556,23 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) break; } + /* + * alloc_contig_range() returns -EINTR from + * __alloc_contig_migrate_range() if a fatal signal is pending. + * As long as the signal hasn't been handled, it is impossible + * to reserve tag storage for any page. Treat it as an error, + * but return 0 so the page allocator can make forward progress, + * instead of printing an OOM splat. + * + * The tagged page with missing tag storage will be mapped with + * PAGE_METADATA_NONE in set_pte_at(), and accesses until the + * signal is delivered will cause a fault. + */ + if (ret == -EINTR) { + ret = 0; + goto out_error; + } + if (ret) goto out_error; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 7e2dcf5e3baf..64c5d77664c8 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -37,7 +37,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -936,10 +938,31 @@ void do_debug_exception(unsigned long addr_if_watchpoint, unsigned long esr, } NOKPROBE_SYMBOL(do_debug_exception); +static void save_zero_page_tags(struct page *page) +{ + void *tags; + + clear_page(page_address(page)); + + tags = kmalloc(MTE_PAGE_TAG_STORAGE_SIZE, GFP_KERNEL | __GFP_ZERO); + if (WARN_ON(!tags)) + return; + + if (WARN_ON(mte_save_page_tags_by_pfn(page, tags))) + mte_free_tags_mem(tags); +} + void tag_clear_highpage(struct page *page) { /* Tag storage pages cannot be tagged. */ WARN_ON_ONCE(is_migrate_metadata_page(page)); + + if (metadata_storage_enabled() && + unlikely(!page_tag_storage_reserved(page))) { + save_zero_page_tags(page); + return; + } + /* Newly allocated page, shouldn't have been tagged yet */ WARN_ON_ONCE(!try_page_mte_tagging(page)); mte_zero_clear_page_tags(page_address(page)); From patchwork Wed Aug 23 13:13:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0417EE4993 for ; Wed, 23 Aug 2023 13:18:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B210280073; Wed, 23 Aug 2023 09:18:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 462B428005D; Wed, 23 Aug 2023 09:18:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32AEF280073; Wed, 23 Aug 2023 09:18:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2456228005D for ; Wed, 23 Aug 2023 09:18:00 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 07C8AA011F for ; Wed, 23 Aug 2023 13:18:00 +0000 (UTC) X-FDA: 81155422320.09.6C508A7 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id A907F14003C for ; Wed, 23 Aug 2023 13:17:56 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796676; a=rsa-sha256; cv=none; b=AnRSDaIsG5LXWw/1/T8lwHaBO8emw+2Ufsxo9kzS8MkuO/YYYvcvdS/OhQKVnx4rFNGYzO ZAmHmm4K4aES/tTMJJBWFWAKlQ+Jzn+qXjL6g69kTMoZf/P/Nkl3r7bS0U4oFVwg+T4hkU sPWAPboHMUA4zS05MRwF7A40NAE72KY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796676; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lpB9doWJr0d4tiEei76WmxsE+/jXguCBnriVq7+8q5c=; b=EcpJkzdCtaIpHN2fFkXj4mr6VtafGfKxYDv4XUCV0or+3FIL1gkvGfaI8lVcLa41Jukjqo haG/ZW0cq+mgIHl09G56f/72LKwqzx4wWFEXo+zZIaNfLHxRO4nZxwKlLrZSi8fcIxXe/z uu58Zc2+dPlBOU69FkNvpiVOJNN1OvM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9A5CF1758; Wed, 23 Aug 2023 06:18:36 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 29A2D3F740; Wed, 23 Aug 2023 06:17:50 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 35/37] mm: hugepage: Handle PAGE_METADATA_NONE faults for huge pages Date: Wed, 23 Aug 2023 14:13:48 +0100 Message-Id: <20230823131350.114942-36-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A907F14003C X-Stat-Signature: pdp1a4jhd7xwi36c1hw9abzjpeuhgbyu X-HE-Tag: 1692796676-167890 X-HE-Meta: U2FsdGVkX18azUm99HbOZWdF1dz96ptmcmj2jF8OBRGm5wWN+c4yzqPJhL6s4Bdh3w6ipI7RrtLZGZz3nNOlm3d8PYu6dpO4t3IpMpzmOQA9QldOLUISntahx6BUg8CkVZ4pbi9afOsoUo35vD22jg5B+ytynhUFZ5SVSFYCzC43/58WBKFvr9pOM7eYsLmSKbrRBEtyw/rQCVANRVRYRx0fxFdruqspP2ehMLsbyMxG17B0irVL8rSDxeKd2rL4WLPYcjyd+6esX5bw/R/mjtLC2W2Vvb2qWspqYaa+WdU3MyFqM/m61hXXC3Y6fQcEfDKQFuATCmf6nXL/3aU/qWeqd6Nh517w7WGl/VCUVqwbsQH5KNW8M3EfJr3JlxPaSbQL60K3RWcLX2y3hgUf9tgjZCrg2g3OWo9WFLnK5Oc4wNFa+v/4QqSUPX2HmEjSIVeLJS3jTVxD1dWFv1V0cWivTiw4JDYvLdAiY68d8Zg9BM+SiKZjepViHO9KFh2svYepWCPHWo58YrfrcPRTVSpXhnceNg1jUWH35sK6lKQB1WyudwLi+aRjvBWagIXFXzMuMNq6HXdinHiIvg/FRtsf7oxN+DyrntPats1HOxkQIJ27hYMVsp45YDVt7qNaPBLowM1dV+mOovOhX+MAsn2rQstm4GnaEWPPaBL8hvBx6Y5kSubnnFrQOjd//0QoTJNb653qxBZVpLtK9EpvZPwADALhbx7dK6pFUXbEAGKe16Mcvt8KwEyu0hKB5QaaXKisOB29BIa/EERURLXGh78O602qB5qPKfzD5Ru4ukNRiUAM4eujDpm8nZY/1KAQ9gUhmt8EylSr7wUEc4juKcUoKGCrcy27pd9uSKCh2AUMbkLGk6x+hjgnhy6D50pRckSzvtt6lkWZ6lrkJJ7RxJAptLNljkX9f1gtvzQ2DTQt9mE15guw48+yQNOt5p3x1Nn99hTi0h3UiN2eNyq wiRyPFhz EzrJEib6lS0eIhQNegJtHO74WpmAQWewWHZ5XPkJTm3MRrYLStPBtPk4+gq3BxMlMLt5svPvJTUxbENElUqzDJI25uzXODgQ6SIo53YEsrkEa4NPJm/rYNkzrYurm/5JET2B4RitcdFcpRwGs/crz0UuOlo8gedhnL6stCg8oyM4KTG0USw3UhyrM4WA0gNkAqMIbHosHnOrDg3e4MxmmKcAFsQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle accesses to huge pages mapped with PAGE_METADATA_NONE in a similar way to how accesses to PTEs are handled. Signed-off-by: Alexandru Elisei --- include/asm-generic/memory_metadata.h | 2 + include/linux/huge_mm.h | 6 ++ mm/huge_memory.c | 108 ++++++++++++++++++++++++++ mm/memory.c | 7 +- 4 files changed, 121 insertions(+), 2 deletions(-) diff --git a/include/asm-generic/memory_metadata.h b/include/asm-generic/memory_metadata.h index 4176fd89ef41..dfdf2dd82ea6 100644 --- a/include/asm-generic/memory_metadata.h +++ b/include/asm-generic/memory_metadata.h @@ -7,6 +7,8 @@ extern unsigned long totalmetadata_pages; +void migrate_metadata_none_page(struct page *page, struct vm_area_struct *vma); + #ifndef CONFIG_MEMORY_METADATA static inline bool metadata_storage_enabled(void) { diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 20284387b841..6920571b5b6d 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -229,6 +229,7 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); +vm_fault_t do_huge_pmd_metadata_none_page(struct vm_fault *vmf); extern struct page *huge_zero_page; extern unsigned long huge_zero_pfn; @@ -356,6 +357,11 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) return 0; } +static inline vm_fault_t do_huge_pmd_metadata_none_page(struct vm_fault *vmf) +{ + return 0; +} + static inline bool is_huge_zero_page(struct page *page) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cf5247b012de..06038424c3a7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -38,6 +39,7 @@ #include #include +#include #include #include #include "internal.h" @@ -1490,6 +1492,112 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, return page; } +vm_fault_t do_huge_pmd_metadata_none_page(struct vm_fault *vmf) +{ + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + struct vm_area_struct *vma = vmf->vma; + pmd_t old_pmd = vmf->orig_pmd; + struct page *page = NULL; + bool do_migrate = false; + bool writable = false; + vm_fault_t err; + pmd_t new_pmd; + int ret; + + vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd); + spin_lock(vmf->ptl); + if (unlikely(!pmd_same(*vmf->pmd, old_pmd))) { + spin_unlock(vmf->ptl); + return 0; + } + + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); + + /* + * Detect now whether the PMD could be writable; this information + * is only valid while holding the PT lock. + */ + writable = pmd_write(new_pmd); + if (!writable && vma_wants_manual_pte_write_upgrade(vma) && + can_change_pmd_writable(vma, vmf->address, new_pmd)) + writable = true; + + page = vm_normal_page_pmd(vma, vmf->address, new_pmd); + if (!page) + goto out_map; + + /* + * This should never happen, once a VMA has been marked as tagged, that + * cannot be changed. + */ + if (!(vma->vm_flags & VM_MTE)) + goto out_map; + + /* Prevent the page from being unmapped from under us. */ + get_page(page); + vma_set_access_pid_bit(vma); + + spin_unlock(vmf->ptl); + writable = false; + + if (unlikely(is_migrate_isolate_page(page))) { + if (!(vmf->flags & FAULT_FLAG_TRIED)) + err = VM_FAULT_RETRY; + else + err = 0; + put_page(page); + } else if (is_migrate_metadata_page(page)) { + do_migrate = true; + } else { + ret = reserve_metadata_storage(page, HPAGE_PMD_ORDER, GFP_HIGHUSER_MOVABLE); + if (ret == -EINTR) { + put_page(page); + return VM_FAULT_RETRY; + } else if (ret) { + if (unlikely(page_metadata_in_swap(page))) { + if (vmf->flags & FAULT_FLAG_TRIED) + err = VM_FAULT_OOM; + else + err = VM_FAULT_RETRY; + + put_page(page); + return err; + } + do_migrate = true; + } + } + + if (do_migrate) { + migrate_metadata_none_page(page, vma); + /* + * Either the page was migrated, in which case there's nothing + * we need to do; either migration failed, in which case all we + * can do is try again. So don't change the pte. + */ + return 0; + } + + put_page(page); + + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(*vmf->pmd, old_pmd))) { + spin_unlock(vmf->ptl); + return 0; + } + +out_map: + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); + new_pmd = pmd_mkyoung(new_pmd); + if (writable) + new_pmd = pmd_mkwrite(new_pmd); + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, new_pmd); + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); + spin_unlock(vmf->ptl); + + return 0; +} + + /* NUMA hinting page fault entry point for trans huge pmds */ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { diff --git a/mm/memory.c b/mm/memory.c index ade71f38b2ff..6d78d33ef91f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4695,7 +4695,7 @@ static vm_fault_t do_fault(struct vm_fault *vmf) } /* Returns with the page reference dropped. */ -static void migrate_metadata_none_page(struct page *page, struct vm_area_struct *vma) +void migrate_metadata_none_page(struct page *page, struct vm_area_struct *vma) { struct migration_target_control mtc = { .nid = NUMA_NO_NODE, @@ -5234,8 +5234,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, return 0; } if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { - if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) + if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) { + if (metadata_storage_enabled() && pmd_metadata_none(vmf.orig_pmd)) + return do_huge_pmd_metadata_none_page(&vmf); return do_huge_pmd_numa_page(&vmf); + } if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && !pmd_write(vmf.orig_pmd)) { From patchwork Wed Aug 23 13:13:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4474EEE49A3 for ; Wed, 23 Aug 2023 13:18:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF0AF280075; Wed, 23 Aug 2023 09:18:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA02628005D; Wed, 23 Aug 2023 09:18:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8F2A280075; Wed, 23 Aug 2023 09:18:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AA89128005D for ; Wed, 23 Aug 2023 09:18:04 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 82E1412011A for ; Wed, 23 Aug 2023 13:18:04 +0000 (UTC) X-FDA: 81155422488.26.E57CC68 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id DBFB9A002A for ; Wed, 23 Aug 2023 13:18:02 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9AOAnFETym/czjm3ze8Okub0g9qDJ3BTMU9kvePKXwE=; b=8iN9sfIcmj4e4ef1MhiEhe5jPIvlxfj8J+4QJUTB8kC6GDdzzmazq283a6fQVnYtON6x77 97OfCTfiJRa1p9Cqk8/78h0kCcb8boKWR+Ry5wfcPsmb6Si4/RxTIDkgmUtWiF+kIL4AhF ko/y6ia0hmDog10Ja8YQ8py31bs8eMM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796683; a=rsa-sha256; cv=none; b=EThdXEk38RvW6xt+ozrf3OiewlrVDnZRMAEg93hjv446gHL7iRBK16pvxLYC/R6nGrX8xQ YmWNyt/4c7g+R8lxOJJzdBZAL5vMQNCTjlxaOeW8vTitaOBgealb/2iSIM6UlgVMKPKjRg sXLh9ahwarKgLI7Gp0doTd58n9xjeHU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B64F11688; Wed, 23 Aug 2023 06:18:42 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 58BFD3F740; Wed, 23 Aug 2023 06:17:56 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 36/37] KVM: arm64: Disable MTE is tag storage is enabled Date: Wed, 23 Aug 2023 14:13:49 +0100 Message-Id: <20230823131350.114942-37-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DBFB9A002A X-Stat-Signature: atrxiwquot531nynew8oa456g58bzoa8 X-Rspam-User: X-HE-Tag: 1692796682-581480 X-HE-Meta: U2FsdGVkX18Mr5qUn0wRiy9u5ct8SpMHr+TvAPTgBNrBtr2KRQDZ+jYO4MZuOAvVRys8P2Jb05022tLTy5jX2xpng53r+wDOM5/RFa5fl3UxPfhA4xu8dBk5s/6IzgN8MUaUkd4bp4BNyiZM7s2j1XK876bXD6+e2q4SGDc+CiZ0v/fZlMpn5394sc62/pqv4NdU2nUuaJWxIyLbhAU7IX6l3RFh6v2lDPf8c5x9fNqIHNk7PLUdrnKznOzA70j13FBTclAdJ2/iTILP7ej/4Ou+4VsK2FX+HmnYqF1GRtbDT5uBJ8FFHF41pgGwwODZ2bmUFk4DDciR4tWvhgBokoEHDZk+jMOQEknxacWW7HjAv32PT1+kjjoeFOnUuCXu4pg7zN5oZzlOS1A4S/rQAsDNBNvcusKzlFXMM9L8B/DgztJseOH6qlDBIZbJn/5Z/EEEaHGEao1qNOFveIgzNUIO/JgNSO7zXtZVekc9KGZJ96/gYO0tCoY7neXYslzqSxDZVVJuqVqK1ETvsSXI5MkoRhTbKtPAvfjmDsi6wOCMQBqADrcmc44yEywQGNKMpXvWKmk2cy2pUU643x9nFyuGFixnGkfMGg1EJKiWvWU1l8600q4fPS6G4PjIhhaLNKkqoSDDCFtE8aDhmHG32iHdjqGR4EgaoKTkh0roIuXWgD5ZwVLsRpZp4y/Dlm2V7u2my5MOeHSfoADipc+txBPEWmsREuugZhDvW/UGjAa7CBIJwo3Vr9U5tmGyaenPLm8e7TG2/dm0g7WEWASWSIENw9myqz20R8YmvpLOsoVrBiyaZi10TkOM+IBV4O+t0q7um2XFE5kfQdE2ECDrFlk9HRMusQXkbHWvVjej6l/O6iHYLgVJiALd+s/8xQbRggVQsI05EoPXlX3cm+qkfi6A43xxks36fkBV8v3uDViTunZ+YeKV5ib3uVC0qNYzljm7RQiaSc82BWNLegp M1qQVG91 uZeVQyH+3IEkyaHvQAEiVBfTlDfcbS7ah9Ktt82qXbJJJHAvt/cM00WGii4UgZ0H+5w41wJ3GfZsD/FCzqJELqNwwfvUCscdSsSzrxiT2pVsfGJTQi/WvpA9lQDNRGgM2MTDEUB2MWplaS35E/F36PQvIOrIcGosRPh/xutPe9fjI4hnsfCKf2Yuf+/uuAEv/oICl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KVM allows MTE enabled VMs to be created when the backing memory is does not have MTE enabled. Without changes to how KVM allocates memory for a VM, it is impossible to discern when the corresponding tag storage needs to be reserved. For now, disable MTE in KVM if tag storage is enabled. Signed-off-by: Alexandru Elisei --- arch/arm64/kvm/arm.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 72dc53a75d1c..1f39c2d5223d 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -85,7 +86,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, break; case KVM_CAP_ARM_MTE: mutex_lock(&kvm->lock); - if (!system_supports_mte() || kvm->created_vcpus) { + if (!system_supports_mte() || metadata_storage_enabled() || + kvm->created_vcpus) { r = -EINVAL; } else { r = 0; @@ -277,7 +279,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = 1; break; case KVM_CAP_ARM_MTE: - r = system_supports_mte(); + r = system_supports_mte() && !metadata_storage_enabled(); break; case KVM_CAP_STEAL_TIME: r = kvm_arm_pvtime_supported(); From patchwork Wed Aug 23 13:13:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13362386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64CFAEE49A0 for ; Wed, 23 Aug 2023 13:18:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6691280076; Wed, 23 Aug 2023 09:18:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D15D628005D; Wed, 23 Aug 2023 09:18:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDE1E280076; Wed, 23 Aug 2023 09:18:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AD2EB28005D for ; Wed, 23 Aug 2023 09:18:10 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 81C051A006F for ; Wed, 23 Aug 2023 13:18:10 +0000 (UTC) X-FDA: 81155422740.30.9AC8B48 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id DE3B51C0021 for ; Wed, 23 Aug 2023 13:18:08 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796689; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9RNoW0r6TgEs1wa+CKHOXDjieIiL2l/WOGRjaJpNO9w=; b=aPEr+DVr+kDuCoNXc3VPf3Gk1Z1e0PWh6SRSpx24fGYW3/ewAJOstvNUGYQujqYOAcZrFa tlUIbxZk/kOiXd8Ndu+rzIqbplOyHzKIIRYiwXyaiqsC9Pcl6L5FZpwjergSCl2NjNlCQH PCx+XD65NZ/jnItZqiFMDdT9hSU21Bk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796689; a=rsa-sha256; cv=none; b=3x9sR26j5panM8rmsDrbfZfMlMS3tjcQeU+j9g8KEcMDTTVst7sFQjJXxx/O0TrIHdE4gi bEgY/q1GurculQwMq4kwE5h9T47TzU4Zg6POAoxYnQFJkrKg1B5MyA/Yv2b2Iyw/s0Zs2n yGwh8P4K32wh6br92Jm+C48KIwyRRcs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D315F15DB; Wed, 23 Aug 2023 06:18:48 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 799E43F740; Wed, 23 Aug 2023 06:18:02 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 37/37] arm64: mte: Enable tag storage management Date: Wed, 23 Aug 2023 14:13:50 +0100 Message-Id: <20230823131350.114942-38-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DE3B51C0021 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 58teuqpnhrefw387yrwnidbs5zohgwbf X-HE-Tag: 1692796688-319568 X-HE-Meta: U2FsdGVkX1+Coe9mUq3LVBMHilDCQxoy2DJabxRxDgMxJSizHQvZ4u6ZdtZ8EjC1nwRMke0KqkXxk20NkNGtXFHi2GsZrYL79i7V2iuWnGtaS+uGCS3z4f9Ge5Lkp1oKLWVIuh8PkYSBKGERRxKC7uC2rhREcAiOSeRFm03+8R+iqneWFFM7mKDAJk60GPyyhMZ7gcEHgXijI4h+DJ2961A8G6rtT8g5+keZHgD4dd237+L2NRe2oO5oyv8QloixpimEICo9+H294UlSNlkYu/zCT1KeFf6jRM+mXlHxd6XGE5B5Y/s9nklIzGzbtkpB8uM6+0IWXH+BKriGYZvMZuqIrKwHxhj37XaSmLcgRmaIfJ/15wvFL5AxmzZHZV0BNS+tAUEyQs0XMcQwlto8WqVcexCQYWnQ1P2KJ+D1jtPzDpaniD+VwiYTjEIEuOAH/2t7BHt/vuXdjql+D8i8YFPwCg7nqLBKTE9FOX1GT3/L4clbC+WDoHsjt2QWB40dCgdDpw6DGPjVcFZ4ntiT5rE1oU5aUjLEMr6f4ULURO1hwb9pzIqmKQY2NCNB69MYwUIxefqTraB82KsPYLAHj8cw/tohOthC8kFO3oZjS9LnDfYsHOeEqJvt2QjeMu7/RxRgY/+nEMHMn8Ocbpaa3bAuXMjT1tB6e1zo9TJFRSVQMmpNxJbRWVG03p0EXKIx25tPLJ6fOYlCbia5YBdBHMaTGRZgdUqgfX0qDN7tvS7tHGb/fGdb3+ntuEzgu9hhpwB0xVE5EgoUVt6yDnlqGHZMQ15uGFMiTw/+tgjEl5pYKzljxezGRbeHwsKbVeMwgTEtkf/Cu+8G3Ce/g8pfCq+4yEgdkiKv/r3PkWfSafUdouMikVqiDD+L1e664gKmTlrYIvi0a4R9St254X59562XAZtJ5ktlz0XvpDrlP+JKTBE7ZdNae0NkwX4nOAv2A17jyfcYhQA8zf2xkIT bA1Kvldp DUZTNNaGoxa0OehOXozDAaFmopHpx0o9DJBk08JA6AfFaegAtO9BRjwaXRslXaEksnNNXCWbRZBVPWbLezUdvCULotfJ+EcKzKqRj6fhHaW3v+xQPTsvYmExzhQ+eDl2fqN5hdzyCqg3/JTFY17ZKQNxIvHV5KWOTwJpI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Everything is in place, enable tag storage management. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 1ccbcc144979..18264bc8f590 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -399,6 +399,12 @@ static int __init mte_tag_storage_activate_regions(void) } ret = reserve_metadata_storage(ZERO_PAGE(0), 0, GFP_HIGHUSER_MOVABLE); + if (ret) { + pr_info("MTE tag storage disabled"); + } else { + static_branch_enable(&metadata_storage_enabled_key); + pr_info("MTE tag storage enabled\n"); + } return ret; }