From patchwork Thu Mar 13 18:11:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D69DDC35FF3 for ; Thu, 13 Mar 2025 18:11:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58145280001; Thu, 13 Mar 2025 14:11:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55415280010; Thu, 13 Mar 2025 14:11:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 313EA280001; Thu, 13 Mar 2025 14:11:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 011B7280010 for ; Thu, 13 Mar 2025 14:11:43 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7FA1D80460 for ; Thu, 13 Mar 2025 18:11:45 +0000 (UTC) X-FDA: 83217320970.14.09E0D4C Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf28.hostedemail.com (Postfix) with ESMTP id 9262BC0021 for ; Thu, 13 Mar 2025 18:11:43 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RlzmKrE9; spf=pass (imf28.hostedemail.com: domain of 33h_TZwgKCK8YPRZbPcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=33h_TZwgKCK8YPRZbPcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0fZcxWQbpwDGRa6fhxDYyz+3EEc+Y2ZKaTPRMrOqTnk=; b=XJ4ZCZeq/HfXjqCuti0CGM1jn55FAa1qqu7/w/LuWjuF/eZheemzBEh5+nEAsuFJQgT0vH kDzKpc+QJvy8STZDSrWGXs0p0e8+o7vM4dwfxSPfjSIoZJ1A0MrskOo6ZUkENtDH+oj209 bRrJsrgpgoow1LAhNLkuiCos+wXBgL4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RlzmKrE9; spf=pass (imf28.hostedemail.com: domain of 33h_TZwgKCK8YPRZbPcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=33h_TZwgKCK8YPRZbPcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889503; a=rsa-sha256; cv=none; b=XnAMxoPCoMoskalHxw1DV015m216FbHKU34dzvRtqXdVWUO2kUF5Ja7kij/eLQgnkAQMNV POvAmJ4YfGN4oFFtUBben8FUMgi6CnuV8nA0gazWtF05XSEQkimsdcXjzKLs+J/8E/kkxI GiMgyay9OUxQ5XnOW4pAxKD6JwhzVAA= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-438e180821aso6816065e9.1 for ; Thu, 13 Mar 2025 11:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889502; x=1742494302; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0fZcxWQbpwDGRa6fhxDYyz+3EEc+Y2ZKaTPRMrOqTnk=; b=RlzmKrE955CHVMcqVq+lOSdiUD8d61GmE5VyHQJVs3+4LLwVBPvLRoehb0GN+bb5AY 5p8VwTqOzv6qZCyng/rHGB0MFfAVaDP9VQKBFRdNY4jU46ZW+bOQfw3TtJ5izBOdrq+k fF/a9M2X6Cnbtk/t73+kTZR8LVTWo+vaRm+Fqk5Au9YMR28cnzsTF6OxQ+CTnkb3YuGn M0GjAWMu08W9AtIZbe7SpLp30fAIfzpgG4eYryXgZO3wpu6MTDtNk/XzoJZL/Nw4R/G7 /oBZgsoMdFY4VJlJ/pfhJhZ7PJdDYL3rMGOELLnzkZkk6wYQINv4R+PYw7ztp6WFP7al bvlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889502; x=1742494302; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0fZcxWQbpwDGRa6fhxDYyz+3EEc+Y2ZKaTPRMrOqTnk=; b=XoepOILaJboXtgJJ+I7i0OMX2foltMEWQYazF1Fkgeb2qUPLYMfJdYnnhxSgwgGg72 JrVa0MAuisP+5haVWLTqPwP/LzOHQLKIQEKBVFagdfYqR8YoN8inV8jIkqJwHHik6uaP eJUKoCChwLHR8uzurXq07KBbQF/i0/bS0hAaPjGGhDnr0PtwbCk5tsB5iQz6YHjUz3zu /SiQQPtbOiMJpENWkYZyEwImQGrSlaQCX9gbQp4myn/AV4hn3eAIovkRUCU1cnKD6EYp 1diGXN/ixPjLpUXejAOI+bUUu09UFIKX7wcw0mJwasA8Fx4vdHz29+bJvq2eZO77lRHx +zHA== X-Forwarded-Encrypted: i=1; AJvYcCXrpNrYs6A/T1w/6m0FjmYHHOvTsE6VzS6UntbnSPlP1jTLRPfr2lVhcMngkv2fg0gv0gE2FBgAGg==@kvack.org X-Gm-Message-State: AOJu0YxT2wO3b9JDlEvpNnmFmPE4OapYnEXVTbGcjq+p/W8II9E8tO+p obxfTKHr3eoGqFWJ5OhI1Vc76bO66Svl/YP+Z9m3hUz0iK6ij7d+WHgqTgjJ1WOFRCHpXz0mXud SWlH/F8VSHg== X-Google-Smtp-Source: AGHT+IHiDt1oL77dvdJgK/87oo1nyciMshPbOYF45qDzwy03t12Mx9YK7JD3EmFk7CS3zUEi1Qz1ZwU+UaX2dg== X-Received: from wmbbd19.prod.google.com ([2002:a05:600c:1f13:b0:43c:eb09:3784]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4593:b0:43c:fe15:41dd with SMTP id 5b1f17b1804b1-43d1d87fd9bmr6539185e9.6.1741889502176; Thu, 13 Mar 2025 11:11:42 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:20 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-1-04972e046cea@google.com> Subject: [PATCH RFC 01/11] x86/mm: Bare minimum ASI API for page_alloc integration From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Queue-Id: 9262BC0021 X-Rspamd-Server: rspam03 X-Stat-Signature: 7ms6yy7r34mtzr8mnb14qjd7qpr1fz71 X-HE-Tag: 1741889503-37498 X-HE-Meta: U2FsdGVkX19Ruk2o9QIZTup5gUpyrQG29K6RAoSdhAzHqNo4lGxKPHCARj4IBNwZneekrpc/iavRPUwVhYylBymPBK+TYTHCPzkfN7wRtrBKOITkfBZlVgnLGuFhFbiO1+qMkGPui47G4jGPWpFdoYT7k4NPt4R6a81eXwX7d4hDaGz8JhPnw82MhjxKj6q9NW8jAGlr0i9vDV+DxgwzV00UBSK5WNuU+eG/7Tysh08XtaVlb4irsSkRNa25njYCOm7gZvnqjGdu7m+VWOrNQJWjB9XXg7xaCxFjdk7LpcMVYFUc/D/eHilkfkK1QaXauqs85FQ3dKFZfhtggH9NNsX3u/4KJGgOATm5NkFZ/G/tUwKzsByhFk5fhI0p5xrxrSP4Sen/T+wk7tqsq/FYIEQylbbanoGGxEUs3te5GGdBKv60LUJYjVzShRLHp7y89+uTIQ9U179tMJPZJQWb4JGueux6p4oM8j0zeqdddrBRWN28onrIh5bpDnEddQqU8swRJhjxtKw11Uw2JexEHxNCayyLH+DnGBtjyJDGelQtLd/piC5sRuvdkEO7SxvcVuRy9hO9rT4yQktLqBiASllDLMj6Vfn/r2jMpKGHtRzoDFftGlPqJjiSP33KksDRN44JQvqP4vSAeofRU6f2FWVyoKueO4x0IdSGBmXjgU/uhjeKyoH3ulHKYIrNai0GR5YjmYC7sao/KsbFykKXgFV552LFVWYB/jefSAbhBmmqtWeq84DVXdF2xs/Yhg2Xq/yQ/iO3Ad/WvpDFI1C2q1J0Vzl6NMdf9UsT8lJwpxsj78WIjKM7wErchMmaDZ0dlJbw9VkGQ8w0bl1BNoIrywmabn+BR3GNtqO/Qy//yH5H0Ws4410elC+8M91LhOo29aVqnIaRE18ITEy0Cspg7rfPAhqrpxiejlbsxTi3GsSJIS9QUyoNPuDtFtM4WD50VhjS8VdP4puwagZOaHS jNM6O5wt hC9ZQ7LwFfGFy8+EJmZMIs5tnTkH4cLr5cKtb75BRz3gcZK9NStFhp7VHBPdODG7nGgFChbLll6AhKV7NR9vuhd0Qw9WSAtWpKuWb6mxOz0WLF1JMDjfx4gwKaqgrmMuSIvIbLkJzq0PyCFj8FhO4WooSZjoXWRWnKKOoPgGD8CJZQgqEsoz5XBWmmVRTijHYWJn9BAS+LdWKe4F9yPzz1fDN/uCilEq5NZG6xv0r09v2RnW2uPWTdPWjYqDRvj/BK2Nrq/aYooqvNGuqIcJlW+O5hRpkWrZ73eLjiJizRaPhwuovxsscP3Oe51sFEhjugBeTIm7uDLELkduEOAXPMVs/Zn4Yvg8ubtl9TO/2aqcVp0UO9uYxlvZ4aySwrZ7L68z+Pkm35Wp7gFMUf60Eu+yDl8w5tQ9fm8vSEV0wrstlvFx/ibMiy/6lqwaC1Mwh7nqJtPJFTS5K7Ep2DBGRFzL1BthYS5xgHMzb01hD+m0Qsfxuh9Q2BNkQd2gXdePAU/acBTs3rud/d7yhktxeRcNieWcfp518t5xj3G513de4aK2IYHj0WcypgXzOo5Q6CbklOAfncXyo26HHHT2ClSRO//tBuSHwDvA/t2z+fXti7kMSjBDT92OshN8mWll25tQv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit serves to provide a minimal framework to present an ASI integration into the page allocator, without getting distracted by irrelevant details. There's no need to review this actively, just refer back to it as-needed when reading the later patches. In a real [PATCH] series this should be several separate commits. Aside from missing the actual core address-space switching and security logic, this is missing runtime-disablement of ASI. If you enable it in Kconfig, ASI's mm logic gets run unconditionally. That isn't what we want in the real implementation (certainly not in the initial version, anyway). - Add CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION. Attempt to follow the proposal by Mike Rapoport here: https://lore.kernel.org/linux-mm/Z8K2B3WJoICVbDj3@kernel.org/ In this RFC, there's only a small amount of x86-specific logic, perhaps it's possible to implement this logic without any arch/ dependency. But, this is absolutely not true of the full ASI implementation. So that's already reflected in the Kconfig stuff here. - Introduce struct asi, which is an "ASI domain", i.e. an address space. For now this is nothing but a wrapper for a PGD. - Introduce the "global nonsensitive" ASI domain. This contains all the mappings that do not need to be protected from any attacker. Maintaining these mappings is the subject of this RFC. Signed-off-by: Brendan Jackman --- arch/Kconfig | 14 ++++++++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/asi.h | 28 ++++++++++++++++++++++++++++ arch/x86/mm/Makefile | 1 + arch/x86/mm/asi.c | 8 ++++++++ arch/x86/mm/init.c | 3 ++- include/linux/asi.h | 18 ++++++++++++++++++ 7 files changed, 72 insertions(+), 1 deletion(-) diff --git a/arch/Kconfig b/arch/Kconfig index b8a4ff36558228240080a5677f702d37f4f8d547..871ad0987c8740205ceec675a6b7304c644f28e1 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -17,6 +17,20 @@ config CPU_MITIGATIONS def_bool y endif +config ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION + bool + +config MITIGATION_ADDRESS_SPACE_ISOLATION + bool "Allow code to run with a reduced kernel address space" + default n + depends on ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION && !PARAVIRT + help + This feature provides the ability to run some kernel code + with a reduced kernel address space. This can be used to + mitigate some speculative execution attacks. + + !PARAVIRT dependency is a temporary hack while ASI has custom + pagetable manipulation code. # # Selected by architectures that need custom DMA operations for e.g. legacy # IOMMUs not handled by dma-iommu. Drivers must never select this symbol. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 0e27ebd7e36a9e3d69ad3e77c8db5dcf11ae3016..19ceecf5978bbe62e0742072c192c8ee952082dc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -36,6 +36,7 @@ config X86_64 select ARCH_HAS_ELFCORE_COMPAT select ZONE_DMA32 select EXECMEM if DYNAMIC_FTRACE + select ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION config FORCE_DYNAMIC_FTRACE def_bool y diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h new file mode 100644 index 0000000000000000000000000000000000000000..b8f604df6a36508acbc10710f821d5f95e8cdceb --- /dev/null +++ b/arch/x86/include/asm/asi.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ASI_H +#define _ASM_X86_ASI_H + +#include + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +extern struct asi __asi_global_nonsensitive; +#define ASI_GLOBAL_NONSENSITIVE (&__asi_global_nonsensitive) + +/* + * An ASI domain (struct asi) represents a restricted address space. The + * unrestricted address space (and user address space under PTI) are not + * represented as a domain. + */ +struct asi { + pgd_t *pgd; +}; + +static __always_inline pgd_t *asi_pgd(struct asi *asi) +{ + return asi ? asi->pgd : NULL; +} + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +#endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 690fbf48e8538b62a176ce838820e363575b7897..89ade7363798cc20d5e5643526eba7378174baa0 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -61,6 +61,7 @@ obj-$(CONFIG_ACPI_NUMA) += srat.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION) += asi.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c new file mode 100644 index 0000000000000000000000000000000000000000..e5a981a7b3192655cd981633514fbf945b92c9b6 --- /dev/null +++ b/arch/x86/mm/asi.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD]; + +struct asi __asi_global_nonsensitive = { + .pgd = asi_global_nonsensitive_pgd, +}; diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 62aa4d66a032d59191e79d34fc0cdaa4f32f88db..44d3dc574881dd23bb48f9af3f6191be309405ef 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -250,7 +250,8 @@ static void __init probe_page_size_mask(void) /* By the default is everything supported: */ __default_kernel_pte_mask = __supported_pte_mask; /* Except when with PTI where the kernel is mostly non-Global: */ - if (cpu_feature_enabled(X86_FEATURE_PTI)) + if (cpu_feature_enabled(X86_FEATURE_PTI) || + IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION)) __default_kernel_pte_mask &= ~_PAGE_GLOBAL; /* Enable 1 GB linear kernel mappings if available: */ diff --git a/include/linux/asi.h b/include/linux/asi.h new file mode 100644 index 0000000000000000000000000000000000000000..2d3049d5fe423e139dcce8f3d68cdffcc0ec0bfe --- /dev/null +++ b/include/linux/asi.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _INCLUDE_ASI_H +#define _INCLUDE_ASI_H + +#include + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#include +#else + +#define ASI_GLOBAL_NONSENSITIVE NULL + +struct asi {}; + +static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +#endif /* _INCLUDE_ASI_H */ From patchwork Thu Mar 13 18:11:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 822F4C282DE for ; Thu, 13 Mar 2025 18:11:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C49F6280011; Thu, 13 Mar 2025 14:11:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD418280010; Thu, 13 Mar 2025 14:11:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A79FF280011; Thu, 13 Mar 2025 14:11:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7DED2280010 for ; Thu, 13 Mar 2025 14:11:46 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DD1AEAE481 for ; Thu, 13 Mar 2025 18:11:47 +0000 (UTC) X-FDA: 83217321054.23.DF176B0 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf29.hostedemail.com (Postfix) with ESMTP id ADB34120008 for ; Thu, 13 Mar 2025 18:11:45 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pxXdKEvR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 34B_TZwgKCLEaRTbdReSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=34B_TZwgKCLEaRTbdReSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889505; a=rsa-sha256; cv=none; b=gYuQAYhUZEYICtROEUBaC9ks0pESxGCfRa/hsE0I1i/5EKIqV2eDPjZY2gDP1R7masoj8c FX2O5vDQfeemRFueXX9i8TqOW1B+2G4EQuzpYLuh1UWqCWxgC5woTk79ffvOLVGQka6tc3 K++8hNdyX2k/tZuWxmffAB2F2XmiH0E= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pxXdKEvR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 34B_TZwgKCLEaRTbdReSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=34B_TZwgKCLEaRTbdReSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Yd3MTUJa5EaKKcsOqm6EUTHveqQqPDqzenzbNHF0zNE=; b=iXtKRpgDL+4Xrt+FcZW+ORwH/VIHO9biHGxTBhgnSdpaBFM+tCAyJ5wR2cvHG/T5BGpK5h 3rzk5rKglGTo/OCRN0n8qq4RX9ioBRoZGtfEPxHC/+crnHtkoimpV8sYUWQqlXedlWULgr pNtk/iJfXHJ0DpLxVki3Y6nMnQ/aGQw= Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3912d5f6689so730476f8f.1 for ; Thu, 13 Mar 2025 11:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889504; x=1742494304; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Yd3MTUJa5EaKKcsOqm6EUTHveqQqPDqzenzbNHF0zNE=; b=pxXdKEvRAwCx90juMcxzeoLJQM0OUXnruLs2Z9IDPL8nLfQ6/oKgV7QV9G4BRpAZQM P3Mdi2goecYBA8toV3Mmb2Fe8QmqA8m3J2/Su+ud9WfHvjo0KEp0FLnhFAWETWpIQYfP vVugWgfby+RIg/8Mua84N1KB5LFpx4ISyy5pMCUjj0Im1i0pzKHBQQnbnGKHwBzVoVrR h7MOl6zn/JqCM5Ra4yy22tynpkSBmAYyEaurVMQlmBcrz+3juUo7GxG54UeB3IE61PUd 8wsb78KQlZSYL8cx2qE/40L2ykrdp3q2bJjij7yyxXr4zzvP4SlcmNUcTMaiL64MPDjW AsLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889504; x=1742494304; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yd3MTUJa5EaKKcsOqm6EUTHveqQqPDqzenzbNHF0zNE=; b=OnGBUZiD7i7WV2N71q9B85l24B5oQDzWFGVNeksk/YCLoLxh7DjCPKUBeYEOz9NnxU zBWIjNBtXgzhgZkGEBlYArve2Mdn3bm2tRWKZ8yTG5B+tN5hyyr03ztrUakIc7x6htnn H4pqPG1sUYo5wufdCkgwesk73X6+wajw14D6ChiA85WEQf1ELnO45BuvIFJosYgOYbHk 0cTtlV5l5q814EITgUOvjgBlZnqVwU8eICZ+nhxdWjbP8RDv90uH2GTkcHDNRIlYAYfw uk94INOfFTYFJERV9MrYOcbTg0J6zrjAY9b8W+25xftYx/kHbRGAc4Kc++ftlj4WA7zO S45w== X-Forwarded-Encrypted: i=1; AJvYcCXXPtvHT2OZeEk4PsYoubgJgB9MIhT1KMO77X9PS0pe8i5hrCpfQ+VJpIDe0mRdGnDqcwlYxD5Z1Q==@kvack.org X-Gm-Message-State: AOJu0YyHewFg1o6hMaMS1hCBdp/fRfkhfpb75hkyEpFaZA/1+vyY9W/k S2J/zetwvOWKHfonUbeRWlM4FcEzAm2d0aAe0SVjqSw18WplIOMRomvaJydILVM+wNrPeAqxBiU fWGTbOerdSQ== X-Google-Smtp-Source: AGHT+IFsOBeq83+HmwcjVeSsK8GiNED4UzJeglwlfgEFKTnF1muToCLx1a+QuwroI+KWJKQsz/7UlqSE3IjxUQ== X-Received: from wmbfp13.prod.google.com ([2002:a05:600c:698d:b0:43c:fb36:c419]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:adf:8b45:0:b0:391:9b2:f49a with SMTP id ffacd0b85a97d-396c3e16b07mr499843f8f.55.1741889504104; Thu, 13 Mar 2025 11:11:44 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:21 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-2-04972e046cea@google.com> Subject: [PATCH RFC 02/11] x86/mm: Factor out phys_pgd_init() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspamd-Server: rspam07 X-Rspam-User: X-Stat-Signature: yu3os3codmowcc5u4yryrcrink7b8ao3 X-Rspamd-Queue-Id: ADB34120008 X-HE-Tag: 1741889505-723247 X-HE-Meta: U2FsdGVkX19nrOfEgWoKj/EwQuscPCO5S0dLOSH8t4zLUjLnrXTkAOyVN0e0m1n3yLUHEoxAHusJqkvZjfFI82S2zkuRwphJs1wmuAs34+BfHHyU680MlTqGvP5q+28Bw66LXq4+156Rvc88VNwIw4qgZ0T396b9p9SLk225KMS9kMpncrpKJ/yQI/CZmDBobX5vl8gK7mnxTxfi7fty2uvUMkrVs/mvWkdQAaS6zodqVWmwPGPNCmH1UEtgPLzIvsq1blxQWm2PSWoGuoNriSh94EeFfVcMZZEBJLPGyLpgSIISblQe/yRM/RuzLwqlLg/M8YOMj2TsjLF832aA+Tn963OxCkHHd7QowtjT4M4xE79G1a0PWLYV2VuibqIGLafFm13yhpNmBVxkbgTntUxdDoTkeD2PuAJD0keOshnT1vrcYpH63thT2KIgzAsZCL+VO9VhKlD55yita1fDI3KG7IangfQIjiso4vuyJzAoK9PN40wEL//7J0VCPujFRZzfk4TthVh4Mqp74+VxOunm5Pnd7NM9wOH6owtbg+MYXWWRf4JrbRXAjAjF8gSlPTTsNNbK/tw4y8k6V6qcz2YQZvLAOpMg9hZskrnYOqLW6KxMCS0EjBr2KRBiyYU3emSoOk5BnDmLiGoQn5CHhV40HUNQxfr0NvrfhFTg3V/GdQZWTp8eBtp4tINapHHHguwEq1s8C/49bnG2vODngMTrPRm0uqkwIRzLtAN8AKABFEBTNFOVwoIVsZAP2JTDCYejRlgq615ulywuMT+tq+7um+IYQH2nerCuPq3tESIlytL1eICTGGYWh4uVMBglT6Ms5kvZIcOSouTXq/2Nj9st4IjqM5wA6zWTI1oGH8MwU7sl7aRLNGeylpNf9mdflD6Jhd6yXtI2R0rUVYQUhfKRT1zxbqOjHpJdhVODz3GiE5nqjcMcucxc2YY3e+odQXpeLhT+jTYM9LBy5/u gM50LNxO 1jHPp5WdzTmZYw5UE7KbqwYGsdG/C6+3QgvNMzvVem0iFvxtGbnbd2ssa+23cLBuKOZZWt2pAPYd9vn0mtIeT+zGpZWEZ2VaaYrjHiDQQ43sGlwqTqxbWxyWFYUiYbrVjp+zFkzakM7mJJUXCIY7omSkc5nUU1K2T+csSRI1kbsNogXtWLNj2LY+SNXhz4FjXTN+mzOkhNka6XTmUxIpSTQ+dxPvoP3i7oFIOjF3yyLZtfNegv1aj8azXiqBKMRLIvE302HcRI9bw0oHXgRKPI14ts41Y3JCbpUZ9N5qN/YArcULqHBDfPuIO4o4N9J2JwMUmdeWnRnQXTEbWQJxGIznEm7s5ubUJnI+zZTfCatL57HmvlhZ2sc1raCcIFbiLOrp6iileaXhCgf2Jga5DWsd9vwMKzuPQzP8dXETD2WUAd/fapSwOiCG/M3ixqQgGEAYBynkc+HfqxEDR4FC0T1CzM3nte2UqkvGp7A1IbZzPnx5jUYDPMla4wNSPM5Yq14KBFRUrN50lSfmnZsMcdbiXrry9Bnw6L3UFMAzQBWXkn2uL1XHb6I7+nszGcaBlISR7My79giuo29PXJB31E0no1frvWVzFL4bXWH6SiyhMW14= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: __kernel_physical_mapping_init() will soon need to work on multiple PGDs, so factor out something similar to phys_p4d_init() and friends, which takes the base of the PGD as an argument. Signed-off-by: Brendan Jackman --- arch/x86/mm/init_64.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 01ea7c6df3036bd185cdb3f54ddf244b79cbce8c..8f75274fddd96b8285aff48493ebad93e30daebe 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -731,21 +731,20 @@ phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, unsigned long paddr_end, } static unsigned long __meminit -__kernel_physical_mapping_init(unsigned long paddr_start, - unsigned long paddr_end, - unsigned long page_size_mask, - pgprot_t prot, bool init) +phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_start, unsigned long paddr_end, + unsigned long page_size_mask, pgprot_t prot, bool init, bool *pgd_changed) { - bool pgd_changed = false; unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last; + *pgd_changed = false; + paddr_last = paddr_end; vaddr = (unsigned long)__va(paddr_start); vaddr_end = (unsigned long)__va(paddr_end); vaddr_start = vaddr; for (; vaddr < vaddr_end; vaddr = vaddr_next) { - pgd_t *pgd = pgd_offset_k(vaddr); + pgd_t *pgd = pgd_offset_pgd(pgd_page, vaddr); p4d_t *p4d; vaddr_next = (vaddr & PGDIR_MASK) + PGDIR_SIZE; @@ -771,15 +770,29 @@ __kernel_physical_mapping_init(unsigned long paddr_start, (pud_t *) p4d, init); spin_unlock(&init_mm.page_table_lock); - pgd_changed = true; + *pgd_changed = true; } - if (pgd_changed) - sync_global_pgds(vaddr_start, vaddr_end - 1); - return paddr_last; } +static unsigned long __meminit +__kernel_physical_mapping_init(unsigned long paddr_start, + unsigned long paddr_end, + unsigned long page_size_mask, + pgprot_t prot, bool init) +{ + bool pgd_changed; + unsigned long paddr_last; + + paddr_last = phys_pgd_init(init_mm.pgd, paddr_start, paddr_end, page_size_mask, + prot, init, &pgd_changed); + if (pgd_changed) + sync_global_pgds((unsigned long)__va(paddr_start), + (unsigned long)__va(paddr_end) - 1); + + return paddr_last; +} /* * Create page table mapping for the physical memory for specific physical From patchwork Thu Mar 13 18:11:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D17ADC28B28 for ; Thu, 13 Mar 2025 18:11:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84AC9280013; Thu, 13 Mar 2025 14:11:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FE17280010; Thu, 13 Mar 2025 14:11:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67988280013; Thu, 13 Mar 2025 14:11:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4163A280010 for ; Thu, 13 Mar 2025 14:11:53 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7E0B412046A for ; Thu, 13 Mar 2025 18:11:54 +0000 (UTC) X-FDA: 83217321348.10.F52A33D Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf30.hostedemail.com (Postfix) with ESMTP id F3B1380002 for ; Thu, 13 Mar 2025 18:11:47 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E9FpoSVq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 34h_TZwgKCLMcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=34h_TZwgKCLMcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889508; a=rsa-sha256; cv=none; b=2VPdsLp8Ldlp9YNfve7nQo9IwJHcatKZoKNVDX4AmafOQENIJY1bt+sAidQ8wt2fuxKdhz Zxyco3s3u4AtUb4nGIAPGmYyev+DLsJciRm4wwmLe9lgpE1JPblRWu1sRLaGJNUU0ZbiMP xa1610OwI/MRE/v/upZzYPo8ZLnFihc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E9FpoSVq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 34h_TZwgKCLMcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=34h_TZwgKCLMcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SFFEGXNq9QK4O4fqhmQmuGwjZkzkfGXtfnqX8VsiIbA=; b=D+WaL60eorehjSCUr0GTEZG9Bi/BATYF1HHJ99mT79mGZMGgvjoBqA4KfjZfYWx9yz37qf TMXrfaHJQ4qWoYD657gi8n3eQEu6OuHFFVKZpTwOq2o3ZpXXgpA0g+ygn/k2H9RutOK+Gq BisMmoCRUf5fuiw+rLx8y+BxzupnZsg= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-438e4e9a53fso9452745e9.1 for ; Thu, 13 Mar 2025 11:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889506; x=1742494306; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SFFEGXNq9QK4O4fqhmQmuGwjZkzkfGXtfnqX8VsiIbA=; b=E9FpoSVqRGQkJJuYUbc8XqyV7dz2z2/CV9iwwSevu3L2pGTlkHCexonHy6da7T0dBB VroJy93L6/TcRWYQKYhmaN13b49BLh1Of5VMqUxdZReBy6dnBI6ubBGKlEurvIxPaA+R VtCnTyEz63x525GPzdmmr/BbdIbcqv/F1WELRrF1rnMJCyTEiEqWqdqNJXtasksqxxjA cKcmP/B0k/B7hqF3zb0CNJ22mVNMshmrGiy+PF4ddNVt2FyHLx4f6J/WEyFOrlfCJcS+ j3CntY10pg9hrIzqvNKe3YTDSIVuuVOJMr6K1l3fSn5P90FKfDK8XxVr47BxHJs3VWWm 7jRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889506; x=1742494306; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SFFEGXNq9QK4O4fqhmQmuGwjZkzkfGXtfnqX8VsiIbA=; b=Y02aEvKVyG8qPlWwBcXKlC+XDMJmVpIVYkf/uhlgXwkQ25E91QcDxCJB5iyTcLQgrC Gbl7o/ar8EArN1pzqXTypuKWqYEf1ZaThjB9yspzawe/ny60/suf/U8xdsLm6UTYUkCo VL0eYjwNeke4h+uMSOwpPqMTxmrz4/r5v/9JOOMEgLeh/J9Wo0XK7fDu97opul6sr7Nk fFOV6iiwzk+pIV5t/IgX/ef4P/bRNbeHmESsmoJ9eI7JvSmnc7zIF63Lu0Xk6LpYW+yO dZTwYZTdJjYXAITyreZxdmx6MJko2a7YWyUPEsN4jZ657pc7EKsR1hAlkwsDfxQDomRy DCYA== X-Forwarded-Encrypted: i=1; AJvYcCXURzzvU+7ukCLA5Yba9d1MkakxB11uln1iynOEssMaqIHW5yHPMNN+ePXUiWUjezijp+lp3D8UDA==@kvack.org X-Gm-Message-State: AOJu0YxkDRTzrv0Nv3IcxHEv14ZlT3Rp5jS7wSbMc+LNAdlAbgcIxMuQ hItS8LYdeKbQH2DBROKVfSz30LQmvczvZTriQlAZ2X87eZBsm2yvuxr46ccWCWix6tyIgpnyM4f O1x2bONs5/Q== X-Google-Smtp-Source: AGHT+IH1VGtpk3xtUCPvw+qm8+MH2zctSvi1XXF/gGWDnzmEetB35farCcixJ7scWW3mkfOJxNfCCFjLdwFb9g== X-Received: from wmbg20.prod.google.com ([2002:a05:600c:a414:b0:43b:b74b:9350]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:46cb:b0:43c:f470:7605 with SMTP id 5b1f17b1804b1-43d01bdd4a4mr133112715e9.12.1741889506406; Thu, 13 Mar 2025 11:11:46 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:22 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-3-04972e046cea@google.com> Subject: [PATCH RFC 03/11] x86/mm: Add lookup_pgtable_in_pgd() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Stat-Signature: qeagip4abnhb7zcj6munpkeiaa63991o X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F3B1380002 X-Rspam-User: X-HE-Tag: 1741889507-693070 X-HE-Meta: U2FsdGVkX183yy/Ae1PzufOFhBBg4HYER3hEh+VT/9yEWV5/wbHB7fy4v/G03nOkTbE/AwEVnxz0Lm1WdmE0NGSQeJWzG5/VMt5rgTQ7T3SO2GVwA9mac8X+L68/847a7YwBuq/oN+Kwnt2S1+1jgNLMPnsR8ejMlpW/AW7VfZfpaGpfpbcxEAvojXvxaL3zCeWCj3rgIUKXlNuNlzphvhlF2M/9TvjJSlRF3H7UY09/UO6T6mvnt6GCZ825ijVlYl2bRrdbbsgpXuW6fLmqEgoP7tI0Z5XxmmuKbgkmLVqnVqyNQj36yHHOig0D6tYfwUpznmHM795ZRg4VB+9Xm1MfgT3JKBEv/29ZUinNtppVcrJBcLs/b9rb51D+HCxY85aLhZXMaFm2/VD0VsFp3s8UxRU3BiAzWZQvPdrmxUpUhSxdbkCl57Sg2lnZ8288GD8Hx7tt8UEnbKuG4kXlNjkdtYnpWM/R66A+f2ULerrgahCnAF4TBLlwOCEDL1D+wAmvAeb+AzpBQ35n/wprY6O0oDERa2RECjwwsYOKt9xh/Vp+/V9c53Xe2IEZyha3hKKiANnx9RlH5awuBOMUN+3yPkcmTO6prfe13LL1R7h1RN9l1YnaWWttY+ztal4onn/uVtBDM5Z1rHXWzXqK+6fWPjyWfyIsl/UuAdfA2y8gBHjWmUkosXChxHDnUhYq/EzgtF7/Zu1On5EydsX32jykIU2yiz9Izzjbpe+Y6u26isFmiViajlYEMxKsMozyzGByx4Qegdk3nI0cdVcWL5XTSV3ioLkm6to/lCGAasx2JIg4yK7Kqd10SwN9yNHNmA4td7P8esGVxDhSi5YkMSCttKy5pWn2kEOB30xdvjOyzhmakNpUSoYN51vtU2MmJQPno0/HwvHqQILheWg289KAa5SQdPvu45Y8K85K/d5qIOR7XJUBXsBtCSf39uvzNSvC8oAtWLJxHeMWHj7 N5idwxg3 CosaMVQ2QRJh5zyXI/PSheYaaT4rPhfwC54cOusHabL5D3kgFpywpodJcoJScoLySgRP+bn+5Fq5bQQzehC/7X6ShrvHeEllqqZAoqHuz7pZoT0uNP8aBxsnBhFWp9AuyXnNaG+zv0wuPFvF1JuE3Tc4jHHqF5Ve1do+j5nTfxCQCYUGoIsd/7YMlPkyc6XBO/JpJ2/yAfYib/nVDdM/r8xdAMLNgCG9ZcYKntn0vXuY7mfUcspuDE3Q9ro+K7Z5+mltIgp75yOJy9P/kV4jR5W2Tc37dJvbWf5sc1GjXEnjAccSReC98xezZ28s//wU4K4D20IJhjMS8OmfPYNv/rsg8YYZDC4d5Su+0+vrVHSm9fEeByhm8YqrFl+WT+45ZGb/f+V4mH8ofFqdrvwnfb59e5WLelMkkuJLTZ876PiLczvS2X9Us0yEKrj71nQtB/8CHpnA3D8BcbCQDlAfyj+vteutN+gulabM+Isi7NT2Mc97VhiedAB+J9T0+9hDdabHSCnviNYZu4v22WmAdDxsCUX7n/aU17BUxvws646igsElMbj3SAzooDH+jy17LWZEoil8leBRlx0IlT0j7OFUysxcrvuIHIvgu6V9j6fLoZv4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is the same thing as lookup_address_in_pgd(), but it returns the pagetable unconditionally instead of returning NULL when the pagetable is none. This will be used for looking up and modifying pages that are *_none() in order to map memory into the ASI restricted address space. For a [PATCH], if this logic is needed, the surrounding code should probably first be somewhat refactored. It now looks pretty repetitive, and it's confusing that lookup_address_in_pgd() returns NULL when pmd_none() but note when pte_none(). For now here's something that works. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/pgtable_types.h | 2 ++ arch/x86/mm/pat/set_memory.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 4b804531b03c3ce5cc48f0a75cb75d58b985777a..e09b509e525534f31c986d705e07b25dd9c04cb7 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -572,6 +572,8 @@ extern pte_t *lookup_address_in_pgd(pgd_t *pgd, unsigned long address, unsigned int *level); pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, unsigned int *level, bool *nx, bool *rw); +extern pte_t *lookup_pgtable_in_pgd(pgd_t *pgd, unsigned long address, + unsigned int *level); extern pmd_t *lookup_pmd_address(unsigned long address); extern phys_addr_t slow_virt_to_phys(void *__address); extern int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index ef4514d64c0524e5854fa106e3f37ff1e1ba10a2..d066bf2c9e93e126757bd32a7a666db89b2488b6 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -658,6 +658,40 @@ static inline pgprot_t verify_rwx(pgprot_t old, pgprot_t new, unsigned long star return new; } +/* + * Lookup the page table entry for a virtual address in a specific pgd. Return + * the pointer to the entry, without implying that any mapping actually exists + * (the returned value may be zero). + */ +pte_t *lookup_pgtable_in_pgd(pgd_t *pgd, unsigned long address, unsigned int *level) +{ + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + *level = PG_LEVEL_256T; + if (pgd_none(*pgd)) + return (pte_t *)pgd; + + *level = PG_LEVEL_512G; + p4d = p4d_offset(pgd, address); + if (p4d_none(*p4d) || p4d_leaf(*p4d) || !p4d_present(*p4d)) + return (pte_t *)p4d; + + *level = PG_LEVEL_1G; + pud = pud_offset(p4d, address); + if (pud_none(*pud) || pud_leaf(*pud) || !pud_present(*pud)) + return (pte_t *)pud; + + *level = PG_LEVEL_2M; + pmd = pmd_offset(pud, address); + if (pmd_none(*pmd) || pmd_leaf(*pmd) || !pmd_present(*pmd)) + return (pte_t *)pmd; + + *level = PG_LEVEL_4K; + return pte_offset_kernel(pmd, address); +} + /* * Lookup the page table entry for a virtual address in a specific pgd. * Return a pointer to the entry (or NULL if the entry does not exist), From patchwork Thu Mar 13 18:11:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BAB1C282DE for ; Thu, 13 Mar 2025 18:11:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 887D8280012; Thu, 13 Mar 2025 14:11:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 837D0280010; Thu, 13 Mar 2025 14:11:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D832280012; Thu, 13 Mar 2025 14:11:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 48272280010 for ; Thu, 13 Mar 2025 14:11:50 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CC402C03E0 for ; Thu, 13 Mar 2025 18:11:51 +0000 (UTC) X-FDA: 83217321222.20.2BE7B28 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf07.hostedemail.com (Postfix) with ESMTP id E1A334000A for ; Thu, 13 Mar 2025 18:11:49 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rGPIhjPh; spf=pass (imf07.hostedemail.com: domain of 35B_TZwgKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=35B_TZwgKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9PPqJkUj7Pibb+TYCrq8XdoSiIxYoCw22sHO21dNMJ4=; b=ZpHv24yRfsOn9KZAH3gzbMEFJOTSZBQUYVR/LMoosvQGuscqmMfhFc2R4eBbJX7mDYajmn 2xAmGIMuFuqN5OvGfIaevRNWq30lSeWWTTzQMr3CJ48NsXELG3Bewatpvpq/cuc/Jk4G+U i5bN5RiME1VNRE6f3T0yxbRu7UuGs/I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889510; a=rsa-sha256; cv=none; b=SlDOzd16JPaunpCdg4bP4UEBOVswFKfC1qvD8foBC9ghMB/EiSrY2aXo5GuzGXGG5EaNS0 gMnVV6eMZeide1Rlv/Jm6NlHW+zIEOAO1KOKulMSr4ZOBNuvYk5kWsDtR4YoXdBkMhhag6 Sn5BbXb8x867WeAow5+PZLXLiP0glsQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rGPIhjPh; spf=pass (imf07.hostedemail.com: domain of 35B_TZwgKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=35B_TZwgKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43cf44b66f7so7384885e9.1 for ; Thu, 13 Mar 2025 11:11:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889508; x=1742494308; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9PPqJkUj7Pibb+TYCrq8XdoSiIxYoCw22sHO21dNMJ4=; b=rGPIhjPheh9+w7NHoA0LZHxbw++gpH+3T/jjeH5ESXV9CRp5EzBC6L+J2MHDUQaMUZ F/bvFBVUI5q0/BgYtAiRG6UD1xtg/dVCo920ZgjbuujZbjAOgOFXNrcx+zb9BUIiFYs+ R3gEnHw4JBy+N/G1AcIFMUv0Ef2gHsq5o9ipDgVTv5BOku6v5PdWkHfclqkpAOUuRzpq +deckG+K0phvPzPSeD5y8fShEjYbSnCvKwm7UWT469Rzf6gFPWvPXBbTW5y8R3zggOlw iLinfEOjUYImWuzIbjKt9MxKNga93kC8Ebfg1HUa6KJ6WesP3iVfyTwNurjOR7JmTSIP g7XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889508; x=1742494308; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9PPqJkUj7Pibb+TYCrq8XdoSiIxYoCw22sHO21dNMJ4=; b=PVJf1ZkRP8gXoHc6Fa/dHQJPRs3nJzTYbjF9QASGMt6N1kbxDahkyd/dVJa19hP3mz fVnWfjDnPAGi73jaZnaYi4f7AMSX2wCWxtUzA+gDnUhieh3prUkcZs5MKFukK9x4f1Uv uMbIGIXUUBLClHU76l9IrOemqhTl8JrIWuXLnVWqZ0WCVoIvf6OQwU5YI3qoumv8Qxub Ok/wpo88v3DPhWzN4HknmaVv1bI7eSC9xASMmM6kZUDEz+Yck1GKtbL83s4qKfD4O03N wXI5whbyaTMtvlpR8cjTtvIzqnLAz5OW9Iq7x2UFmJrJJ40fFqWVswal2o4YVzVnCUxq jfoA== X-Forwarded-Encrypted: i=1; AJvYcCX/rqF91Uku8TWBa69jx0A6UnWvgwElRpRlc8qCtaBQDnins3o/XFxxWDcke4AAfKffahbm1h5wYg==@kvack.org X-Gm-Message-State: AOJu0YxyL1Xr89Yo3rGl/4WdZ90a0/WRzYw7JunSKnhRjk7XAR/oRbyS dFi6hFsdroapR7yEEEOu+6OxRsuT6tF0cZSKgsmtxJiMkgoQ4yWZz7n4mhO25jM0w7JGFFFjug3 xF+AcHu0VLg== X-Google-Smtp-Source: AGHT+IF8ERnTvg3YNcVI2WgDobZeTnOtswaGwtLGXdYxevgaK+phqTml6l4yAXJ9Dw4d5FjayQ84d9+Hw+R0XA== X-Received: from wmbfm8.prod.google.com ([2002:a05:600c:c08:b0:43b:c9cc:b9b3]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:598c:0:b0:391:3988:1c7c with SMTP id ffacd0b85a97d-396c2105ea6mr522202f8f.24.1741889508450; Thu, 13 Mar 2025 11:11:48 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:23 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-4-04972e046cea@google.com> Subject: [PATCH RFC 04/11] x86/mm/asi: Sync physmap into ASI_GLOBAL_NONSENSITIVE From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E1A334000A X-Stat-Signature: is8dk5i1qqgdd9mieh8qrii95zbthaci X-HE-Tag: 1741889509-829561 X-HE-Meta: U2FsdGVkX1+d92mlFDfGGtTRhu1CKY0xxMX04lYky9FKT7duGUhaoTOdzVYeX7KOQWdpMeTRB6DvXPqwYk50ieOp+EeCe7trh5TTUQuZiVXi/1dpd0bML8gu0d2T6UwHAbe8MFfcQQb+Y3VyutI+tJcGw2FRS6Ar8sZ/qnQbn0S8kCCFcgKcTBhQJvqSyAdceXp3iOdOQ3ItBsQ6fdObgaiNJdzFoehtQghs/zBiXKvQwPw5M9WXCx7i078ieBagHJrvOz64YyPfqKRg1lle8YBJ2smYosxZy4r/DljOH+jxjsrPbHeM2wxJWdrcEq6CtG2RQHkTN1TzY3RvbvrslvtWtt8WkfZ83nphNDDt945GzYD/U6EpZHzHwHzEu855+GdMV/9gO92Ybg5tZLI9q1306mRqKIdBYysK8KS1teCbTv0JLdFwRB+M7q7OSZo3QtQPBZgcGmVwxXJ/lHN6Xa9N26EK4SSm1kMNY7VZimNBIYfccIFjF6wBIhaqEsevjAAzlA/X1kfJVbaeJ9/WAs6vb+GUDxzPz0M0ukRZALTU/pg9I4EgPRpVHz7yaf90Nu80y44H2MjAMVrjHaTgEfT79u6lDRxOb5gNG5xFr14CfjAhsNEbNqFiDTgJNdJDQgElzxEPA5zRpmPHSAkP9mQjA/t/R9Up0M6q4ZqEN7FyP2gY6q3COphtTq7AWt8IweOuvRNlZgxShUkG+VgVkbKJbDCzZFeP4QHfgduyKYFTfXbgefo2kadjvR30vO++FOG/aDoB87ZZTFv8RNw9kKr/qaOFi8WjkIUb6QkR7Nk3CHHmQMi7UCfqZR2PBAXFpNfdTdgrscxjRydN2yzEBVGqf7gpWsO3oTeOSDDUc7YlMWximHp19SOoV+vEI+9hVy9N3fD/y53OWyqvOCkSmuJ0PX9mVyDsAPVl9jIFalPOieL4au83wNeuQFWUVXKGosNORoeaHjM4lKbjqJD d4wWarrK uPNebUWSbdFIFbk/MrVHJ+Kx7NbHZ8/69b4YRcnEm5vn5rtPOMahX/UKSidjK3iCcwGfW3ZxwjThuyDK9JdUHUoikqDsY4Aw7BYJ4tMN70mBtCxNBkFcELW9rQQdZZw25A5WUlPvreldewVf563U3svB+Z3n9wR1JIbBCBYbkF1nXu9p0bjyyfa09H0PA+v542oYa8DwPNlp4yWkDkM9TsyY5vFvQ0VYbWieSd44LQgFdUI6PB52mUTcWZPgeSiho2KuwC7nyT/dTloQ5hR+F+BKxsJD4/J0LNb9iACKZlz07CtGYQ03JHQ/MUHLFm+Zb94QFfYSa6of4CfLbBWF45tK9yeRRLgkqPp91v2zNm/JhE8E+JI+EhrPWoU22hdtpKIAXJ7zRskfq8olYKweQh9XFehsDjGe/cYvg6O0Ehk/lWeJaRmTsFQ4VLW9CnyEUuhbxCxsTlalwZnImBDh38HKtVf4rYNzf3j4Q5Bgw03AMipUOO/UN5ttPsvMxkJ9KObg0/XGEu8XambLn2rBrm6D44e3lphoPpitU2m4RhV3Bl8gh7BpO6p9v5ZbsSM8mopi/nCv57i3pgd9hcJ994pd5TTl42Jtr7GnE6czcOQ1rPmk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Mirror the physmap into the ASI pagetables, but with a maximum granularity that's guaranteed to allow changing pageblock sensitivity without having to allocate pagetables. Signed-off-by: Brendan Jackman --- arch/x86/mm/init_64.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 8f75274fddd96b8285aff48493ebad93e30daebe..4ca6bb419b9643b0e72cb5b6da6d905f2b2be84b 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -7,6 +7,7 @@ * Copyright (C) 2002,2003 Andi Kleen */ +#include #include #include #include @@ -736,7 +737,8 @@ phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_start, unsigned long paddr_en { unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last; - *pgd_changed = false; + if (pgd_changed) + *pgd_changed = false; paddr_last = paddr_end; vaddr = (unsigned long)__va(paddr_start); @@ -770,9 +772,13 @@ phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_start, unsigned long paddr_en (pud_t *) p4d, init); spin_unlock(&init_mm.page_table_lock); - *pgd_changed = true; + if (pgd_changed) + *pgd_changed = true; } + if (pgd_changed) + sync_global_pgds(vaddr_start, vaddr_end - 1); + return paddr_last; } @@ -784,9 +790,29 @@ __kernel_physical_mapping_init(unsigned long paddr_start, { bool pgd_changed; unsigned long paddr_last; + pgd_t *pgd_asi = asi_pgd(ASI_GLOBAL_NONSENSITIVE); paddr_last = phys_pgd_init(init_mm.pgd, paddr_start, paddr_end, page_size_mask, prot, init, &pgd_changed); + + /* + * Set up ASI's global-nonsensitive physmap. This needs to mapped at max + * 2M size so that regions can be mapped and unmapped at pageblock + * granularity without requiring allocations. + */ + if (pgd_asi) { + /* + * Since most memory is expected to end up sensitive, start with + * everything unmapped in this pagetable. The page allocator + * assumes that's the case. + */ + pgprot_t prot_np = __pgprot(pgprot_val(prot) & ~_PAGE_PRESENT); + + VM_BUG_ON((PAGE_SHIFT + pageblock_order) < page_level_shift(PG_LEVEL_2M)); + phys_pgd_init(pgd_asi, paddr_start, paddr_end, 1 << PG_LEVEL_2M, + prot_np, init, NULL); + } + if (pgd_changed) sync_global_pgds((unsigned long)__va(paddr_start), (unsigned long)__va(paddr_end) - 1); From patchwork Thu Mar 13 18:11:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 028B3C282DE for ; Thu, 13 Mar 2025 18:12:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93BC6280016; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8781E280010; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C847280016; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4D5BE280010 for ; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C87B080432 for ; Thu, 13 Mar 2025 18:11:59 +0000 (UTC) X-FDA: 83217321558.01.ABE87AF Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf06.hostedemail.com (Postfix) with ESMTP id 48B91180007 for ; Thu, 13 Mar 2025 18:11:52 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IzJSOY8p; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889513; a=rsa-sha256; cv=none; b=PWtIVyS/+IYYvYd+YP19F6yBt3jVxuZld8jCIEWJfIcszw/Y4Qy0LsDVwZO9OflK/A0Ju6 BUAFp+6bMKiCguOUbb1FLJRX+d2alyKq3+BXP3ZVdIry9gXP9iQMV3VLZEsxVMWu1YKMiV tSx1M+gvVwyZ46oprRSI0z6sZV0KLJw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IzJSOY8p; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889513; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=BQ9qRzrPEN1GcbrK0z/y6HJjUqoMRcgLmlmWfOqlnG0znq3yhd9OwDEWGAxyqP5kPSZTh4 Dyw8f2d8xumpOgs96KIsJSKa0wzbUsbRocUZRXcFE5fkKbtGEsQ2PIWrwArDCJMfITAbWi OmDfTqN7mkTzxjDJARHy4Jmoh9MZ/KY= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-438e4e9a53fso9453255e9.1 for ; Thu, 13 Mar 2025 11:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889511; x=1742494311; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=IzJSOY8pX1aTHn4OOp/egkqnRIKxQEOWzcaS7TaQOCpFBiL4XhRUj+hVo8mrimUC/G G4WFmCL2ugr+Gre7c4bfu8bjWJUSwDx18U2EDGnNgoQpbb/Z/Kvde42h04CTIdhpJqrE YEhY8wyY0H/12yb5IX9rOrukdrI/n7QcloZdi+LUYhGekLEqi/DI+koZ4iyAv5J3lmvC 2pBn4siEQ4H6lDEZnTxJRzFQMGg5YZGzkREqXtBM5Al++V0MX+8RWNqGvPZFdeuM6Q2y s04PTbYZAtR+SQCYh5I+GOfnbsiw7/OQfv6+lStlP+tUt7M+7sBI741baoJzKYUYm4iL zBTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889511; x=1742494311; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=qReEEc666AMqIDQRUym2Wwf4KbAvZHrrIxFKZ4H3hE56dztf07jKJUu/wcFjM7tJSW 4S0ILKEhiq886uAEj/dK6+SODETiUmrHWAgoyO05kgi2fuRs1gIwbx4nzJHIjCqSLEjI bqIyojOE5hTjkjKbAnCjbZBvgWffJCu1wl58g+EF8oVgn6boxFlHtR0bW5cPqciA6fDv vXJtXb1fPBWJoTJtf5z0LjaaT2tOcUg7mfjBdT99livD7hq1F6rCy0Ck2njjTFFo/jz1 B9Jo427U0SIsdHT0nQ2MvZzRtjnV8qxXTP8Df/XhJIt5OwKnuh/vqwAXLCs7pBlYCy/a 5HfA== X-Forwarded-Encrypted: i=1; AJvYcCWgfst0qZCHBdTcikRvqmP457kwYdH09xDy6/86bNQdjSjOzsCkmNNLOrlJnhADD0NGRiM3K0e0/w==@kvack.org X-Gm-Message-State: AOJu0YzsIS1O2FeC4IozuPo8zks7qPwaUd3zPUnjy6qQvqV2cxOQe9w7 l9E4ttp3zUb1qZtSsWhxSdqlCscKdde1mq8XzdrjyiGo2iKCDdv+ruh0VAcLon86qMzLWRawddp w1W4wuFLQqQ== X-Google-Smtp-Source: AGHT+IF292MH6JqAkiIN8jtyln5hVYHbTRuF9S/Wm78jwx44s0FdKYuBZ0MucR5gttxIvSprRDkRwKMLpsnBdw== X-Received: from wmbfj6.prod.google.com ([2002:a05:600c:c86:b0:43c:f256:f4b3]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d12:b0:43c:eec7:eab7 with SMTP id 5b1f17b1804b1-43d01bdd224mr152891025e9.11.1741889510788; Thu, 13 Mar 2025 11:11:50 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:24 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-5-04972e046cea@google.com> Subject: [PATCH RFC HACKS 05/11] Add asi_map() and asi_unmap() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Queue-Id: 48B91180007 X-Rspamd-Server: rspam05 X-Stat-Signature: cwdnmpmz7j4j1pzka83csr7d9g5rmksb X-HE-Tag: 1741889512-775319 X-HE-Meta: U2FsdGVkX18kc3++nFYNev/K0E7QwPXqFE7Y6cN99xsmgU4oZVWGEYGglOF3i65SjB3zbsH8c/fjttyglzLx7kfhS/PBNtsa/AqXo9XXB8lO9MREsK1L7HBHprL1TUpX5Uo8nuLVkNvXfOuBnBfWNQ9j0ThtReJGLKDRwrBlkxUs5gHLDY6tUiqaDvGsMJEcHIzqP3C4XH/zltMj/MZZgfyxJXaJGZih4KPoxFRee2xiNB0CRrcRhHBi4/25jXT5yKK2ygXHDDTuO3vJloVeyNmIBoAfZfwgE4G62IV3YoI0Jt1QWy806N45iw48126pDHd7RpmUSP1qRIHMaVwsaRQ57Dw6Jm5zXUt/25Cv7gnaXCg3yWamgN5Y6817U8UtL7yg2idhVrp6gPEoLfq7FwWUgYn6YTRpatJi3pk9z1HdM7rNQ3IancwhszS2fSesSjybuxGBOWRCt935ACOQjkm3F8+beRHAfEdHW8C/GsfmjEG74CTJ7hD3b4LSfgThGMaJSb/ptUg+9p8bGyIYITsE9wv1xeyuBoP+9J5xg8m7FOFiOGWMyZuZLtwGqGB9qQ8PJh4030JeeNhfx12QCuR873QAAy3bVRwcZDB7SkPcLyyOfF2QYbSsDdfVtPzNfzkbdrxHPalW/S+mFxzCfyYcal34/yrIEhVrvyG4e/NUqF/FI468H9y2ztXThUdi0Q0FMobcP2X62TyEXKEyXaKMpAyQyUJYmllDUIU50VqXzxP6JYX+hnEgMfeUavyQbngI9cE6CZnn+UOiSxuric6v5zgITB+LjzS5By1MhiNJ3Ic1UpaGy0mY0kfNp3yxthFoYovd2Vm7rBUlHsLxI7sAzw2o6G4pVjRAjm/lubFnrp+XFaI222HVFHGTjL0t39pKQCDjVJIYfwoJczNfRklQy++yQfchowENlkbBiGFJagY1KkcRcrTiJSW/n59fZxTO34tberzGrKVNtYH Yd43Kugq h/NMBixFYIe2HS0p9FRSchO2LEW/tPAyIQxITXnEP0Mrk0rmuMqZ2OnZPTiYWkitWmvqsa+vR9JSXvQZqIwmSgsj9O3/duTtFNdhysAeHpr7AlecpNbiMshkJJVPO9rtMs+CYkW6js9RkjEhpHQsIuedDNMXip2d66ERJxP+JRt+O4iRKe9LuLAZ+uIATINrNyaa+R3O80MrzqcjUbjeFHsyR4qOWl5pHKFu8eyJss5p1xhd3EVQ3M7JMTG2ZcBRgerVEzORVf5+VAPR1bSKjyOFL41m4JKsb7vFnP+k0D9aoW+Pw0ZHrOWsjhuyBTK0eNpxKcFFteAxoXOBY6gdhOK54Kw+Qh7ymmLwnwo+NhKC8haXycTu7dag6tBTp2+LqwTW9iFoioLLemYl6S13lFJxEZOKoodIUUPMVPfRBoiOyTbPeieAaVOvXjoUIMm6u1xkjo7AgzpUhUTbREtU4SAAC1O6oyNQ3cdvAtU+COgMu4HwFlBCU9V51ndt9PzSpm6ChEZeKkhEgFVkpbrV/7XtFxbn4xoGBwYPnO4qCwFf9XIl0xIa6oEqsC1eQcsDt2Ym3Kvfd2JNYEuZuRgTmelyaCyJS6v5UgGt5R+oRTCu5oh4vOl9b6cK9XaJEDUL/9Mb5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not worry too much about the implementation of this. I am trying to implement this more neatly using the x86 PAT logic but haven't managed to get it working in time for the RFC. To enable testing & review, I'm very hastily throwing something together that basically works, based on a simplified version of what was used for the latest RFC [0]. [0] https://lore.kernel.org/linux-mm/20250110-asi-rfc-v2-v2-0-8419288bc805@google.com/ Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 3 ++ arch/x86/mm/asi.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++ include/linux/asi.h | 2 ++ include/linux/vmalloc.h | 4 +++ mm/vmalloc.c | 32 +++++++++++-------- 5 files changed, 105 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index b8f604df6a36508acbc10710f821d5f95e8cdceb..cf8be544de8b108190b765e3eb337089866207a2 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -23,6 +23,9 @@ static __always_inline pgd_t *asi_pgd(struct asi *asi) return asi ? asi->pgd : NULL; } +void asi_map(struct page *page, int numpages); +void asi_unmap(struct page *page, int numpages); + #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index e5a981a7b3192655cd981633514fbf945b92c9b6..570233224789631352891f47ac2f0453a7adc06e 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -1,8 +1,85 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include + #include +#include static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD]; struct asi __asi_global_nonsensitive = { .pgd = asi_global_nonsensitive_pgd, }; + +/* + * Map the given pages into the ASI nonsensitive physmap. The source of the + * mapping is the regular unrestricted page tables. Only supports mapping at + * pageblock granularity. Does no synchronization. + */ +void asi_map(struct page *page, int numpages) +{ + unsigned long virt; + unsigned long start = (size_t)(page_to_virt(page)); + unsigned long end = start + PAGE_SIZE * numpages; + unsigned long page_size; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + for (virt = start; virt < end; virt = ALIGN(virt + 1, page_size)) { + pte_t *pte, *pte_asi; + int level, level_asi; + pgd_t *pgd = pgd_offset_pgd(asi_global_nonsensitive_pgd, virt); + + pte_asi = lookup_pgtable_in_pgd(pgd, virt, &level_asi); + page_size = page_level_size(level_asi); + + pte = lookup_address(virt, &level); + if (!pte || pte_none(*pte)) + continue; + + /* + * Physmap should already be setup by PAT code, with no pages + * smaller than 2M. This function should only be called at + * pageblock granularity. Thus it should never be required to + * break up pages here. + */ + if (WARN_ON_ONCE(!pte_asi) || + WARN_ON_ONCE(ALIGN_DOWN(virt, page_size) < virt) || + ALIGN(virt, page_size) > end) + continue; + + /* + * Existing mappings should already match the structure of the + * unrestricted physmap. + */ + if (WARN_ON_ONCE(level != level_asi)) + continue; + + set_pte(pte_asi, *pte); + } +} + +/* + * Unmap pages previously mapped via asi_map(). + * + * Interrupts must be enabled as this does a TLB shootdown. + */ +void asi_unmap(struct page *page, int numpages) +{ + size_t start = (size_t)page_to_virt(page); + size_t end = start + (PAGE_SIZE * numpages); + pgtbl_mod_mask mask = 0; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + vunmap_pgd_range(asi_pgd(ASI_GLOBAL_NONSENSITIVE), start, end, &mask); + + flush_tlb_kernel_range(start, end - 1); +} diff --git a/include/linux/asi.h b/include/linux/asi.h index 2d3049d5fe423e139dcce8f3d68cdffcc0ec0bfe..ee9811f04a417556cf2e930644eaf05f3c9bfee3 100644 --- a/include/linux/asi.h +++ b/include/linux/asi.h @@ -13,6 +13,8 @@ struct asi {}; static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +static inline void asi_map(struct page *page, int numpages) { } +static inline void asi_unmap(struct page *page, int numpages) { } #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _INCLUDE_ASI_H */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 31e9ffd936e39334ddaff910222d4751c18da5e7..c498ba127b4a511b5a6f10afa2aae535509fc153 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -11,6 +11,7 @@ #include /* pgprot_t */ #include #include +#include #include @@ -324,4 +325,7 @@ bool vmalloc_dump_obj(void *object); static inline bool vmalloc_dump_obj(void *object) { return false; } #endif +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long end, + pgtbl_mod_mask *mask); + #endif /* _LINUX_VMALLOC_H */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 61981ee1c9d2f769d4a06ab542fc84334c1b0cbd..ffeb823398809388c0599f51929a7f3506ed035f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -427,6 +427,24 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, } while (p4d++, addr = next, addr != end); } +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long end, + pgtbl_mod_mask *mask) +{ + unsigned long next; + pgd_t *pgd = pgd_offset_pgd(pgd_table, addr); + + BUG_ON(addr >= end); + + do { + next = pgd_addr_end(addr, end); + if (pgd_bad(*pgd)) + *mask |= PGTBL_PGD_MODIFIED; + if (pgd_none_or_clear_bad(pgd)) + continue; + vunmap_p4d_range(pgd, addr, next, mask); + } while (pgd++, addr = next, addr != end); +} + /* * vunmap_range_noflush is similar to vunmap_range, but does not * flush caches or TLBs. @@ -441,21 +459,9 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, */ void __vunmap_range_noflush(unsigned long start, unsigned long end) { - unsigned long next; - pgd_t *pgd; - unsigned long addr = start; pgtbl_mod_mask mask = 0; - BUG_ON(addr >= end); - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - if (pgd_bad(*pgd)) - mask |= PGTBL_PGD_MODIFIED; - if (pgd_none_or_clear_bad(pgd)) - continue; - vunmap_p4d_range(pgd, addr, next, &mask); - } while (pgd++, addr = next, addr != end); + vunmap_pgd_range(init_mm.pgd, start, end, &mask); if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); From patchwork Thu Mar 13 18:11:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79CCDC282DE for ; Thu, 13 Mar 2025 18:11:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94E24280014; Thu, 13 Mar 2025 14:11:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FD71280010; Thu, 13 Mar 2025 14:11:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73008280014; Thu, 13 Mar 2025 14:11:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 07581280010 for ; Thu, 13 Mar 2025 14:11:54 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1C2A380485 for ; Thu, 13 Mar 2025 18:11:56 +0000 (UTC) X-FDA: 83217321432.22.5EC314E Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf17.hostedemail.com (Postfix) with ESMTP id 2310940015 for ; Thu, 13 Mar 2025 18:11:53 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cxqb315p; spf=pass (imf17.hostedemail.com: domain of 36B_TZwgKCLkiZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=36B_TZwgKCLkiZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6roQ0y9HMVWb0fLuDGnNUraXTJD3yygGf4+oRkUuB18=; b=aqWS82/X2llguNtL31cu4hLyjICRFcFaWSFJzV+Yzf7UR7u3M2R4TF8Ku2mo1plcxdvLeW n/ppnJmDRu86zq9mk7KtEbRkerf9s+Tbsn4DVSTULllPrYPjMyLvfp+7IDTgDy4bfiKId1 P7IoigD8U0Rq/XZVG1u3KToqF6cRf2E= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cxqb315p; spf=pass (imf17.hostedemail.com: domain of 36B_TZwgKCLkiZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=36B_TZwgKCLkiZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889514; a=rsa-sha256; cv=none; b=PFPSROU3Rb8gZ4KcCasUn4VSZI3S3Pq+8wViE5up6UblbWNU5nZIqKVJueRxFVRMNR0Fze LKT5QLEfSH6+UOC/oFVF4p16XBmQLFATprlVWZpNNByg73vm62mj7BJKhlX+36D4M0kTgd rLT6Y6/v9NrHRh6rokK5gEXb7mX1rIg= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43cf44b66f7so7386235e9.1 for ; Thu, 13 Mar 2025 11:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889512; x=1742494312; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6roQ0y9HMVWb0fLuDGnNUraXTJD3yygGf4+oRkUuB18=; b=cxqb315pe2cZ/tIIlUZl0Ip6CuuG60O6ZpTs4EytIxXsqSMCyBbK3mSCO1yqFArXMU d1kjLiJ2dy6Z6OiPQmrg8Dhl31gsphvB2V7zf4St7Y+ipk9gfBZAcEzTiq1B4A4VmDxt MWtEozvlI3Gm5Ah3dv6ZsciooaA7PX1V9vWGDZW/gW5QGB2ZyxBZ7lw2sQphi8hQi15V 4nEpX3v9NdJ8KHQUmncQuHtCpxX8RnuZvc9hknUvguD7lU2SHmEb4lkLXlc0nQt1fpiw 6dZJlNeOqoHtJedqyH/8WtSbSyP0cVcjnB13VXjkptXX44JvOBL8/0K2ejnefOUyLW5r UfAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889512; x=1742494312; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6roQ0y9HMVWb0fLuDGnNUraXTJD3yygGf4+oRkUuB18=; b=mdoBeKWbFWqDN3nMZNdIQqCV99b8C5d82yZFthnymIZni9yuQ1XoM+51B9g0eQ36Ua 7tvzI218ha4HIrskuTgLRBuMYv3YyuWKHhi9PqO2na103pMnmHtGXmN6e+FhcZMMXOOw Cv9M/1UXJ7kvARseEQdOZ4VEXxKX/IeY6vsME/dOmr45NEIaowmVyN48d6IPA6vjU2jo 6Fdl7xjwoqjWGbecOaX1xiKjOgSY6oFHkyweLvhUKXVIhEa37atHlDiGmrofi+LLCCeq 2nJFi9cutY0bnxvx2bKvLtEGEBqOiX0T3KSUUth2T2IkCdm1riOWLg7zpIodT48WNIxG Vx0g== X-Forwarded-Encrypted: i=1; AJvYcCUbJnkdePHi+epj9pXrL7NWdw8X5bi6il8JXne8qFfqqlap0OW3ExbcAP21KVWMR894VYVySbz9dg==@kvack.org X-Gm-Message-State: AOJu0Yycn8ig3BzxNh48dGQ78DFLr4RKxrku5T3p4g0gpWCmOg5jC8bR sQwg+C5J0CboyzLki4vTChBOYNDsD4DbyNWqEG7fyAxMVU2u97EuKZMopNwZGlln4SxkQ4XvIb1 GTDqV84xu9w== X-Google-Smtp-Source: AGHT+IEIusjvqc+xKBD+wnnYFxl05hffn8LYeM0xBLqlMTC3EbbSvCojjd4lwP+OgtLOTfROmKsVm430j1MBCA== X-Received: from wmrs15.prod.google.com ([2002:a05:600c:384f:b0:43d:c77:3fd8]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e8b:b0:43b:c5a3:2e1a with SMTP id 5b1f17b1804b1-43d1d88c5ddmr6267935e9.2.1741889512673; Thu, 13 Mar 2025 11:11:52 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:25 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-6-04972e046cea@google.com> Subject: [PATCH RFC 06/11] mm/page_alloc: Add __GFP_SENSITIVE and always set it From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2310940015 X-Stat-Signature: c3auu185i34dj4q4g7om7ng9jwb4c5xn X-HE-Tag: 1741889513-236503 X-HE-Meta: U2FsdGVkX1/sDnhpKEZmsGzKSIpW1K6OkM1r/O0vEHjAG1R2a5WqXqRPL5JGKPkP4PjB0L295WPLwIGUWDhQOyGcNyejIGCRl+LkVU24LaP+ZChB8+OrcBupCpZrc0certkJ6+TpUs4LryyoBVpl68G1tyh7D4VfZy6lVT6DfHw2+cQ6jPqxEp3XEeWiFqu48JuijvXPgzdu5wIHB9360M/KdCUyx3ueKsmWKNAVBeSfULrmGKnUS0Kstvrw2YHbmMNdq3GAdZUowUIUIMhtaNJXxdX+rTXEKWPzchWXzQ/c+ABDVighoS/XcPuZ+Yz7stxdJ886tLrd5Z5Pi0lFCytzB8EZ0u8E9C2I9qQnM2lVGYL9+w8Zp9zI86kxYBf8Vxl2+19CGD1tlQXXDySHqcPPXLkTtzaJ+zQUDl9n0CLrYBoQFQAYvb+kWnFAjVsCD3DFZYHrjyT0FI3srnIreymjCE6cNIbOaixzq5YD3jJ1uN+KGJcZTA54besa412xcWu6tQZTLI1Rt97MjmFJFMccMcTRlzjnM+CSYIVDcWDGN7SDFcSbL28XDzJ5AvVoGnYyBA4/eq7ulNVnsTYRa4URodQxjgmKNCLqNzcSf+SS+WJJjqgFFmYN389dUgUYrgYce87nYH3Rl3jd2lxYGEHdirxNfIKxpKyXXd8i2erVoQh1F+3W1eSCMPJRFv4DVkarDmSsPGivJJwCLuxcPQdnc5JsVxk/te31P2JBdgUwLptun0Voqtvhiy32oVaFlcdA/QEGUJYz62rtQCwSslUTdc5P7l6qAGVR6+vG19YCLQmt86jXFeH1p4p5I9LxgP9X9idE7klTzPY9bugEJwA6o48pt2iG1K2G5ogiCpeBAvcPdJiFnFDFBTo2qI+BO2M/21rPRmGP9F6KtLfNgEM96jF9rleYVe0fBQYGinSjI+guMDA37hFHwbixosQDYTYDgRPzQ3C0CjBd4sV CLhP5eJE zF1PM6uCwqBg0aOgMDQzp9HKXDE3G59fPsX/7uWQpO71vULFUbisnhErBk21J2+SGtahc+Seux9h/DU7i7jXGzAPLhnn4+h4AVCmcv1Yu6iZgi/P1ECS69X6L4w39Oxim95U6IuD7+e5UkRJG8vhWPFhWaQwcoZANVffOamYxIPBKodSa1V+ll3K95pyD1gA690FwAnjh+Fa+vfr6L3nR8gJmKJ63GxtcB5AnLguaA8iforHaK8Y48XBTJzBBpi3Vc8RxmdboaFv1E6MruI8/u1sTbXwJQdBFwVc4UzI0A4BwZ3HY3e9nk4KmOKuGa4gkb1MQznqaYl98+vHsLU4xtQtmxruuVnYYFPH/skoo2ibHkWFvf80Jy8i+oy63cG5nL8ZBW3C2EqCkhVII3BjFOwFRf9ltrkhnNHLb5m1KDXoNUj85K8elaL2w4qxzgeDQwsmSrxL6C7JZ9Ws1Il4p84GwF+wNBWSuTbQ3I9CR7919wbPVHlUYvMZwfPLKDTDEiuaX7r6tXt1xc11NZcESJa9Jse4GtfOxYz4faTtLHjNwhQRVstmVjc0B+gmgUnFbcX5e1kUJs2mhR6/QJJLy8cje0lfT0shgl4Ppg+8a00YqgsA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: __GFP_SENSITIVE represents that a page should not be mapped into the ASI restricted address space. This is added as a GFP flag instead of via some contextual hint, because its presence is not ultimately expected to correspond to any such existing context. If necessary, it should be possible to instead achieve this optionality with something like __alloc_pages_sensitive(), but this would be much more invasive to the overall kernel. On startup, all pages are sensitive. Since there is currently no way to create nonsensitive pages, temporarily set the flag unconditionally at the top of the allocator. __GFP_SENSITIVE is also added to GFP_USER since that's the most important data that ASI needs to protect. Signed-off-by: Brendan Jackman --- include/linux/gfp_types.h | 15 ++++++++++++++- mm/page_alloc.c | 7 +++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 65db9349f9053c701e24bdcf1dfe6afbf1278a2d..5147dbd53eafdccc32cfd506569b04d5c082d1b2 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -58,6 +58,7 @@ enum { #ifdef CONFIG_SLAB_OBJ_EXT ___GFP_NO_OBJ_EXT_BIT, #endif + ___GFP_SENSITIVE_BIT, ___GFP_LAST_BIT }; @@ -103,6 +104,11 @@ enum { #else #define ___GFP_NO_OBJ_EXT 0 #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ___GFP_SENSITIVE BIT(___GFP_SENSITIVE_BIT) +#else +#define ___GFP_SENSITIVE 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -299,6 +305,12 @@ enum { /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) +/* + * Allocate sensitive memory, i.e. do not map it into ASI's restricted address + * space. + */ +#define __GFP_SENSITIVE ((__force gfp_t)___GFP_SENSITIVE) + /* Room for N __GFP_FOO bits */ #define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) @@ -380,7 +392,8 @@ enum { #define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM | __GFP_NOWARN) #define GFP_NOIO (__GFP_RECLAIM) #define GFP_NOFS (__GFP_RECLAIM | __GFP_IO) -#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_HARDWALL) +#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | \ + __GFP_HARDWALL | __GFP_SENSITIVE) #define GFP_DMA __GFP_DMA #define GFP_DMA32 __GFP_DMA32 #define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c9a1936c3070836a3633ebd32d423cd1a27b37a3..aa54ef95233a052b5e79b4994b2879a72ff9acfd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4686,6 +4686,13 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac = { }; + /* + * Temporary hack: Allocation of nonsensitive pages is not possible yet, + * allocate everything sensitive. The restricted address space is never + * actually entered yet so this is fine. + */ + gfp |= __GFP_SENSITIVE; + /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. From patchwork Thu Mar 13 18:11:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E427C282DE for ; Thu, 13 Mar 2025 18:12:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB3BA280015; Thu, 13 Mar 2025 14:11:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D60ED280010; Thu, 13 Mar 2025 14:11:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C088E280015; Thu, 13 Mar 2025 14:11:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9E9F2280010 for ; Thu, 13 Mar 2025 14:11:56 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2B3CDC0447 for ; Thu, 13 Mar 2025 18:11:58 +0000 (UTC) X-FDA: 83217321516.29.7360C74 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf01.hostedemail.com (Postfix) with ESMTP id 1057E4000A for ; Thu, 13 Mar 2025 18:11:55 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sCJvoxrm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 36h_TZwgKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=36h_TZwgKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889516; a=rsa-sha256; cv=none; b=708n1500/Vx7WfGPeFN2jrIikLwIQ/FR9nbMeDO9OzrLlQEnW3IdN6Xx92LB2mMVRGQIoS EEVdQ33zl2brWMHQ+NFvCVux7pIPYrBbtY4KOKFw5L/dYkSc6Hgqpwht9MSAX84QTqOo9L KAsDxw+6OcgjR+/eDddDKjizGOxE6KU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sCJvoxrm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 36h_TZwgKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=36h_TZwgKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889516; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+pdfFaB/k81zEw2QO02syAUWJ0A/YbE7SLXQbmysWgY=; b=n6wHnGy7yckiQtYNS59K+dOhCgsPQ5Mbho++B98CQSVy+LFvTyyiqay7xpGlpaxQk2qhQD 5AUOVv1g2Q/cw9RhesW7eofPSrW7L/d5x1QIRSK00UNflml0HVM9jZb/aLAnWP+A6k3qXM wtlH29CzjrX/FAI94DMHNABQnFLnrMc= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43ced8c2eb7so9317495e9.1 for ; Thu, 13 Mar 2025 11:11:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889514; x=1742494314; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+pdfFaB/k81zEw2QO02syAUWJ0A/YbE7SLXQbmysWgY=; b=sCJvoxrmixSbTXuP1fYH5jFL/b7XhAFGAtz4k0lPpDVAkQtv37g+ME8PNjM9UtcZJU 4g9+HfYORkMYAGa2+X8wbmByGH0Egt49KsMPB6406B+pdCxByHndBY+Tm459lcxjcRZB eGy2ahppeG8C+1nmmzC627UJm3x8XkB/Apq4AtVDVLj/GpmrALYGyDXup7uttYJ1F4UE vBfFr0vENjO0WLt0SePEGWwNufxCpIKGm4DjzcmfOcxaWJeQcLLMBvwWtS5FfqxvT9KT oTJ3Unsqpu9Tvv+Ji24MYUQ1oJX4SS985ig0DgMedoZJemD4wlr68Gawb6fkzKA0bsvm K90Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889514; x=1742494314; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+pdfFaB/k81zEw2QO02syAUWJ0A/YbE7SLXQbmysWgY=; b=W2wOPGOxhNnB+FqSAq8fAf+N3tg/c6+YhcU/mo9qh5jXiM+1Br1E5w23bME9LCyOYO OH80EtomWzPOc1vEZ19Z9zIOFhVTJTg+Zv4/k1AhBD2aevY1V4MS9DebqNWT/NhsH7Sc T7nslVA/RzpDMUptwsBb1adMRB4ANOy36688SIOoJBrpZt3fhCFKbxLCVJx8VejQNkhW ryFJwHTKJXbBQ2KdIF0RpDgugvoCIsm9PEPbELFCz8cv3geb7c6j/MYB+9CCFST4dOqj /JehxhCsbwgIBrRrRBVUUdUB9n3gq5srqRstWd6zm8j854hxGR6MmKobfmyUkdkXie3w Y/4w== X-Forwarded-Encrypted: i=1; AJvYcCVm8PULOjaNeKkff1tZp63uLD2LVHRhOy73ht7BV7s4MKpQcSIbgpCIoexwilvmot/6w9LAuGyAdw==@kvack.org X-Gm-Message-State: AOJu0YydU25lq1UMlkmnITGibXhX0rkQrMy9rWBvMIJBnRaeGqyfcARK iEr1kKGibrcknQIwfiYDnnXwRb925aCvYmRphxO8BJIuzCDuOx6jZvEe8LIOWZxREu/JMB1z+40 soDn68zY9Cw== X-Google-Smtp-Source: AGHT+IHRjd714WqfAw15oFXCnMwvJ6lJg1SUh1wqrNovE6dSYYDgAsuTLSBaeJXNp59r6szjiAIaZ5iRK8TFZQ== X-Received: from wmbez8.prod.google.com ([2002:a05:600c:83c8:b0:43c:fc61:959c]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e8f:b0:43c:fb95:c76f with SMTP id 5b1f17b1804b1-43d1d897497mr5845225e9.9.1741889514676; Thu, 13 Mar 2025 11:11:54 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:26 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-7-04972e046cea@google.com> Subject: [PATCH RFC HACKS 07/11] mm/slub: Set __GFP_SENSITIVE for reclaimable slabs From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Stat-Signature: bdmfw9fb1o1z8r9krz6yfygwmy9nreh9 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1057E4000A X-Rspam-User: X-HE-Tag: 1741889515-917019 X-HE-Meta: U2FsdGVkX185eaTu/NEwq5MEts/vaSEqVE3XxZ9LYNlAqzzTZGuJcMpmPVRwJWRFl2xmC2G3mGaY6lAWTAXUK74i6DGkSQ/d/7rLHeaCbcG0mD61r1YRVQzih5rLX44w+RogZs82usbhSFukzazO9RoFjcA/8QePo6RGL7PwxNFXfx1nF7o5lXgxXWAoYGDmjkCRdaLpmaU47dt1A1Duo2c8Fgd4H0iluc5Vm7gD73huZRDUPmplPL//dtf2jD4qnjO5ilyBrNS31GIpz6sDXTyWqmS6Z6NfqVNDlgX0GIRreyV3zhpJ+EIjuwfsAD2ESMZgfXMzdqgHHG4Nzq4JJmJGtcRO/ew0wcERDAWF8Q5jjhQyQjPpBk4HrXzgxGjR/C0tSOFpV5O829l78yPo84p6t+zHoFJQmOxluECn86Rvdt5D+Ow7fblNE3B6kV87PLnMozuzNz8pFx8OCNfMwDRSuT8sgfN3gpaj4pEu8sdXj+6E7YriaObsMbwRxmcjXN0s3rZI8CBoJDLiX/kUF6k/AqHqeuORwGJ1ygv/L71y2bF5ncNZJheK00Ui3Em9+dXW9olQHd9iWaE3v0Rc/E0SULrjIm8SK8gbUPwD65JvH0A5shebV1P62RJFAeUbpCi43HbtXEuuuNKPQtotlRXnz8sPFh6oDvRAAJmv/P+EHew9oPI4XCKQKsL5Bxujui6QYjteDwtxV5YW9SxjNr5zCW27beJk06jvP5yEwbqT0ta2WSozxM3mTqQo84wYAozMRIJ36SvhEe4c9IdWacUTW0fD/FNqxT4AtfbXWdPFQoHhwGb0GJ6Kw7Xt61q/vQOfHaY90PzDKgPBQpEjps/SA+leOxVNu61PJjHzH5jwA6Sfr7GuQIVkIK9u5SCb0IBFYQxt4MCGMEXDdcQn00HH6PuNjFqYlOGU3exO3FqqGQNSv66/7r853tLGIBUgQbJw3Ej5JLAX5yU+Z5Q C+25+lJZ JPsvupUppdcIUIz9mZZ8vLOo9BV+6+yw8XbMnNgb2F9lhLfzp+i+pFgXF4hkIVcmvF/jcrCWwTbCErDrNbqpsOg/hapLHeZcVJr3v+OwTtWLSW2tvPsQzkZb+R3xVqW/c9Tb/jA3xwRhVjffd6K6lN7EF7hMh9sp2LNY2+6fD/n8fxQIaG4uGrdfu3DeMJtKhkjlGjh3EW3oQSrlngpcAqmeuGhY4mshT28tpu8lmd9OwKI4oTw61Hef3XNjp9fTfvD8NgZF7yt7cBJeTh7WD8nBjyDzt0I/Fu7iM/Suzh8zdBP8DWPrQsmobBJUu8DRAHI2zastkux5WZ1tNxHlzPygMFGNwYBKzg/RxbemSSktk/HLsCO5cweIFRjBpbJxwosEwqKzVxtW+Z2SAJj1SvXh6UXv4z5bh2a4PTGXiqlXnlNeJA+CCjfcoAa1WR7AUdGjqH6d/2NvdsWe0so6ramM3w2fqstDe5Pxm2YhxPRjXx4CfcCe9yVhSQw2339Wf2ziJZ2Jln9UtfYAc1D3SPaPAMpjEVuykfVaqpExJzJBdzqgEQ2y7jC5FrqE/NEoOKWufky81DUpaxLUTFaXdlq7RMUkHQpVcxSKn4caFLjKnsog= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It's not currently possible allocate reclaimable, nonsensitive pages. For the moment, just add __GFP_SENSITIVE. This will need to be fixed before this can be a [PATCH]. Signed-off-by: Brendan Jackman --- mm/slub.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 1f50129dcfb3cd1fc76ac9398fa7718cedb42385..132e894e96df20f2e2d69d0b602b4719cdc072f5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5559,7 +5559,11 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) s->allocflags |= GFP_DMA32; if (s->flags & SLAB_RECLAIM_ACCOUNT) - s->allocflags |= __GFP_RECLAIMABLE; + /* + * TODO: Cannot currently allocate reclaimable, nonsensitive + * pages. For the moment, just add __GFP_SENSITIVE. + */ + s->allocflags |= __GFP_RECLAIMABLE | __GFP_SENSITIVE; /* * Determine the number of objects per slab From patchwork Thu Mar 13 18:11:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CBFAC282DE for ; Thu, 13 Mar 2025 18:12:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98EF5280018; Thu, 13 Mar 2025 14:12:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 940DA280010; Thu, 13 Mar 2025 14:12:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B85D280018; Thu, 13 Mar 2025 14:12:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F0345280010 for ; Thu, 13 Mar 2025 14:12:02 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1B31D1C7DB3 for ; Thu, 13 Mar 2025 18:12:04 +0000 (UTC) X-FDA: 83217321768.13.A52F566 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf21.hostedemail.com (Postfix) with ESMTP id A13BF1C000F for ; Thu, 13 Mar 2025 18:11:58 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CWZRmk2e; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 37B_TZwgKCL0mdfnpdqejrrjoh.frpolqx0-ppnydfn.ruj@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=37B_TZwgKCL0mdfnpdqejrrjoh.frpolqx0-ppnydfn.ruj@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tRsJowh9EEZA56z2uUOhJ3rtTOtbuzhctlmDmyeE3Lo=; b=np8xAGDHIisBJIJddNouVAv/PTrMcPqP9lyiXJYT8L1X5RkTRsCSHGx7oqgGVP0c77dgzQ PzFbgxiyhxwavN4QI/DUtprwpLrykOFqenM1N7uaqVZqj/3yD9JnmirRE8DIJNEjpvE7gn 5Mf65iMCL6PWSHSWjJ8ULvv7m6J5POo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889518; a=rsa-sha256; cv=none; b=hEFP4HE0FKJIl9HFzOehZkaxDGuPOP1tb4A5jptnGtYrAk9wPlgO93/AxpBgCj9ZDcd9Og vmj+ailvJAr8Jp69RphRUrlBoO3qAo8SLRK8S4/aBA8+39gbW0GLZlUX3/dLDdJDxklI1s wmDeGC1JqiW2XpGOOU5+xF32y4339rQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CWZRmk2e; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 37B_TZwgKCL0mdfnpdqejrrjoh.frpolqx0-ppnydfn.ruj@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=37B_TZwgKCL0mdfnpdqejrrjoh.frpolqx0-ppnydfn.ruj@flex--jackmanb.bounces.google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-438e4e9a53fso9453785e9.1 for ; Thu, 13 Mar 2025 11:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889517; x=1742494317; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tRsJowh9EEZA56z2uUOhJ3rtTOtbuzhctlmDmyeE3Lo=; b=CWZRmk2empf/JCNd7guXrW5371p4fsUj6hw/UvVoixFaTiwFv4oQyl3UxdSlCEtAJz kpxmV+6fyCAl8Gn+eL5I6JCSDjqJf5m6LAwSaTiXY83EtdqNmelzATYN4GELAAb3G4J1 TwQBRJfPUZ0W6QHiz/EORyGrZYBc6CsXEZEG2LOxDJi5PLOCcY3eRP0mwIjTOb0UajJe 6bEBN4yuJIlf4kFsnigaDYrL+oIghOW7+qSz4uQjwrL422ERcJuJuDLz500WJ8oaj4DJ 9GZS+uXLwk36wy+S6dgD+uA4dJHDK4cdeDXaRGJhlmmgEW2O2ckgv37pZtJL5mkSxulR Mlug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889517; x=1742494317; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tRsJowh9EEZA56z2uUOhJ3rtTOtbuzhctlmDmyeE3Lo=; b=CzvooYhTPIbVLRDyZfSN5z4pirUoVgGEez417/Km/iiLQdqYdV+mESKeR9Pc/ukLtn /eyr0tYfe35U8UHEUFAwG+cMhgxa8QN/HWk+ht6Io9PFWFd5a9tb55h+5VnR5vQul483 TMp1sA0zXKBzVgAw3NhOrNyBkfXFeMpE1FkQ+JEAReg+DT/LIlKfdvAekBl/6GMFoJ/+ jC+PWdxDUPF27irpaB+P4dntrU38Y9gD1zeDMH4oe23K+5/WN4rLHfZDq2yNm4S2GVJF O1z3Yr3Irj1kYlRVSuCm9idoi3iTfeaiabGrn3EVqeKsd2mbBbzLTv9Z7KXVGGTEvM6b 8FPg== X-Forwarded-Encrypted: i=1; AJvYcCUxr/vFqs8p64Xu+iR4hlY7ox9GWzKId7RUpGt/B19W3kWoTh+lXTtQy61bgIr9qw1AkifanFElZQ==@kvack.org X-Gm-Message-State: AOJu0YyVdZiYmUu32ur5s0WTvQ5QujOfdvFSGjy3xsdxn96BApGSWynD DRslrqd/g7eE+FX36SRnhH7SGpeS7k1idwqBAZPGRYWrRVbJRNXCW19nc7/cIRfvk/Nv08GGlud ClLHVmDGdig== X-Google-Smtp-Source: AGHT+IFfyCsrs7ccNNldH6c3NwbPOVylewQWVrKq/G7/S6WFMNKzAcRUB4lJBTdyYKD3llafwzpnB+NDVR+YSw== X-Received: from wmbeq4.prod.google.com ([2002:a05:600c:8484:b0:43b:c967:2f53]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f14:b0:43d:7a:471f with SMTP id 5b1f17b1804b1-43d01be7e4bmr134665765e9.18.1741889516996; Thu, 13 Mar 2025 11:11:56 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:27 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-8-04972e046cea@google.com> Subject: [PATCH RFC HACKS 08/11] mm/page_alloc: Simplify gfp_migratetype() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A13BF1C000F X-Stat-Signature: 33eitpyxadi1ncbjwnhtdyqeqk65dokp X-HE-Tag: 1741889518-91054 X-HE-Meta: U2FsdGVkX1+yecAQoajJ+i5/rV3o3YrTODKtFdrP7hRgTB4MY5Z0Q4VEzRaxScot5cmnw7PRi6vgBNmBhtCOuA62jUtuL6bmtVcPCfCGwtauJNhhhwZyt9rg9l/YJYaBT97Dd6VY5kMmWdw4mBfeJtGZIJYbMglbT9Pp8kk2skjkteKWx8LsL287LWQb+9DkXVVCha6FSK/Bd2mwBliu4YJXDmaYhxSC5nbNnHps+4XOSWX6wh4V2/g7yjJi7/t8rKXNjqfE2UasCrdXs8XidC4NOtpsRcC4f5PvRfqwq4Z2FgsG51yQh9qTbHgtcwLmA+QgS120L9xp/rTtwVvyPVaOd4e/32/mTLDngx6VuqD/MAsiLnxZBF2tZGiWPCx50W2t4Iueha9lzUeU3Unn5Yb0v8X5TJl72LQbe8B21R2vNhd2ZP5pKKSSWXlS0YT/Wi7Ldq7IlEvO+sFBDih0J8374vzpx70YWY9exOV0V1TE5zxsCn2yqSBL5nLz/v2KXct+shUBl/25FEJR2KvEfNOrhrcC53Hp/hVpNjKHOUAlG0CAndL7b3ARYoVpw0YXMoop24cE//ZHNagDfgr/qL2YBvMDSlvJ1fDUIBguYMFlrCy739s8yk3sn4uSg9HDvXBl7dqnotYyR+CrARCbLi6dpo0K2ecUjeq5PeKGY7OrVKj4BNDc9N8A07PCmzvwtmEsOBe8gArqMpOhStxEefjINHXDdat9qpfMwW8bwpFRVCERsm5KYterrwANhp9dDtzjRtAwDLcbO5Wb1KiqtHkBpLjaRruserGHvOSrBnH8aad+IPwWYXyGSSJvIxoP+V/h4vqBPGJRby3Ph4sYQnCTwGauhWtbbs+nSofHNjvGNnzA3+VWvHC9tcF2W6ITbvCR4Sw3oaAgbQCCpqQBbqfFZQLswG6gOD8qzTagXe+7b0elKHWPwscNMbyVVzyntbe0gphlqxpf+TEXB9e 7PiSBoUu VxZVo2fQTS5aLUN06lgyftvvDddFf/7ERiuyohwMkJU+o1xxQyIXP9IT7YdcugdYkaeYWPw8N5lMji1cdlaDpcJ4cE+S2G+5mafYFlJ4R9G3fQ622Ns+FyObvE3oxU/3QrqLehT3ZVYU9NanvFg7uQjVvbH61upa6T1vndwDU8j5tvIf9Jezwf0GXOrvhVaV9uJkzbVyBWAGmg4PT+CCIvqAt4VFxkchUBklx9PMby/ukY3gpvPGwaYASOUSi4GjDc79Z5gsn4tGIpCl8fvJxOo1zm+CUq55NsKIcGNIZH+S7D98BG1ORb0XBPNMRw04Rx9oJO1MnsGVx7+uceirMhTF5abwJMNY1H6fqlXaUa08aVnv8dDIhQP+3/NukDosn6qu2M35RkK6Rn9JkWiMwEhAknmciVx0B0RRA83aJsdYmM1CApxhvK0akPr3Ud4jzGZcrjXeNLQ0594408VNioglSXz+oPvGvVOh86klgG0+xWQTxAIiPftweeZtuF7eJYTUrYJ0hORAWHgq6uhoD3lLNl28pIMOqmwnLG5+HU7BpVy+uoJ1/2gVwEu3CeyJ6pyf4/51H4n8cMzG0l51RxJy6wtejh9lTp8P40XO2NnkNZPM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This currently uses optimised bit-hacks to avoid conditional branches etc. For the purposes of the RFC, let's not get bogged down in those details - temporarily just drop the bit hacking. Signed-off-by: Brendan Jackman --- include/linux/gfp.h | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 6bb1a5a7a4ae3392c1cd39cb79271e05512adbeb..23289aa54b6c38a71a908e5a6e034828a75a3b66 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -14,25 +14,20 @@ struct mempolicy; /* Convert GFP flags to their corresponding migrate type */ #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) -#define GFP_MOVABLE_SHIFT 3 static inline int gfp_migratetype(const gfp_t gfp_flags) { VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) == GFP_MOVABLE_MASK); - BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) != ___GFP_MOVABLE); - BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_MOVABLE); - BUILD_BUG_ON((___GFP_RECLAIMABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_RECLAIMABLE); - BUILD_BUG_ON(((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> - GFP_MOVABLE_SHIFT) != MIGRATE_HIGHATOMIC); if (unlikely(page_group_by_mobility_disabled)) return MIGRATE_UNMOVABLE; - /* Group based on mobility */ - return (__force unsigned long)(gfp_flags & GFP_MOVABLE_MASK) >> GFP_MOVABLE_SHIFT; + switch (gfp_flags & GFP_MOVABLE_MASK) { + case __GFP_RECLAIMABLE: return MIGRATE_RECLAIMABLE; + case __GFP_MOVABLE: return MIGRATE_MOVABLE; + default: return MIGRATE_UNMOVABLE; + } } -#undef GFP_MOVABLE_MASK -#undef GFP_MOVABLE_SHIFT static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) { From patchwork Thu Mar 13 18:11:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE87AC35FF3 for ; Thu, 13 Mar 2025 18:12:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99A59280017; Thu, 13 Mar 2025 14:12:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AF9F280010; Thu, 13 Mar 2025 14:12:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 705A7280017; Thu, 13 Mar 2025 14:12:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4674A280010 for ; Thu, 13 Mar 2025 14:12:01 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C51611C7D6B for ; Thu, 13 Mar 2025 18:12:02 +0000 (UTC) X-FDA: 83217321684.19.5A577F6 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf10.hostedemail.com (Postfix) with ESMTP id ACC24C000B for ; Thu, 13 Mar 2025 18:12:00 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vec6HYKy; spf=pass (imf10.hostedemail.com: domain of 37h_TZwgKCL8ofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=37h_TZwgKCL8ofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889521; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GhuPyoh4JQENDoXkPgzWqVfgnUM5Erl7lymo551VcRQ=; b=mhpNRqsy6KpV6Dx6a3/j8IjkZxWf5/LyUJs4Z1HFBp+/owl82/Vicr/rDK77Ry7pT3Ah2+ /qpmhhFWhTIn16COVAyIViAWSknxpzWUb1cc1yol0OHxhyfHM8CPKHeFPdxgEmkRuy22Fr zP6eA8uT1GTE5Sjlg8bEMXqmBKA7TM0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vec6HYKy; spf=pass (imf10.hostedemail.com: domain of 37h_TZwgKCL8ofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=37h_TZwgKCL8ofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889521; a=rsa-sha256; cv=none; b=cPnII/KAofV6N5t8AYmdQA4LvdTgj+WtNrr/a9IqMe1ctmmzPjSdMjIcwksreF/dhukra+ VxSz2I8/kAvfdskOziw2xgMQsxTR+tjyXzOWe28QzRjzgZ3FR2MQrxPlC9irq72olFGTX4 7ZBzecQ52J2ouLeLsjWD0C1wLSSyLD4= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43ceeaf1524so6821725e9.1 for ; Thu, 13 Mar 2025 11:12:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889519; x=1742494319; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GhuPyoh4JQENDoXkPgzWqVfgnUM5Erl7lymo551VcRQ=; b=vec6HYKy/N41GOGRFPcZTL24V2Z3aOu6LZLcStcSUBqd5vjvOkx1FXJn4nyGE03Ux0 Dv72d4lhVIzeM9zTAhOUJ5H+zuSEp6e97KQiaB6qKsoBx4fJ5SKje5sIOpVXWNCI5pW5 HM13fGbaGWK0M3g2OUE94rETgkdmbEcf4MXx+VE1FIlWeJT/LaracjUr0Y6QJHUvTwjH 78PQvkIrlSlMcgnGtN8kTtnDzJzroMsr/d8LP73hnS9kBMxndpwo86p+TLGLErrSbFsn iwRbQRkMnKhr4R8kpNl8117jgy6eR6YgwnxGmS2Qtw7kUphqeUA4qAhLFUPrBpV7Ye+a 4VCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889519; x=1742494319; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GhuPyoh4JQENDoXkPgzWqVfgnUM5Erl7lymo551VcRQ=; b=ks/LgijGiEj8c0kLrojKHapy95Ur1UNHNSNy/UpLuJyVV1B6+hShIUDNFRMHJTB9Ty ewTShZo1qTnIHcdl8K+jNsSRfJ9W89DMV1J48WDKmyH+OnZT5UMr0o7YijNSQBvebbwn EazPMuPO0z3ykk9kfyf13D/W5XnGM5NTSSFNKLOlgT7BS6X16vMFN97NVInT3pXOPWzt uabfB82I7BFgEfQPqeIBvz8atBeD6G1kzi0xnRjfTScrofhKFXNImNz8E+w65W737GIy xnXlEAuuvQmL52u8m/etfX9NiPqjXrOoJ1g/WA4pA2qjdnRjtYBBg8ND1n/2EJZk6fKw dDjg== X-Forwarded-Encrypted: i=1; AJvYcCXxiOrCvOrsyFudR7GgkTALGa1H38LWoNe98irSuJh2vguKUFDfPdbgK1kqokrc1KweSXam4jh2qQ==@kvack.org X-Gm-Message-State: AOJu0YxsncY1wi/28YYK5P3l7nA1gHomZh8oz0QdmqQIjh0ah0dIJaKT cpRt43S1kmepCcS43p7xPG14H0Tst1msW5bgWwyEI3C45FzNMs0AKwfIeKr9WBBXfrkbNIY4G/Q +r+5NWsratg== X-Google-Smtp-Source: AGHT+IF61pDIZp4JV1y+yCI+Y7/rxQbZzIOQPlfA/NYp0rFg9DJMnWUvBtDooARG1ee4zqcfgN9USmHD1j/7QA== X-Received: from wmsp2.prod.google.com ([2002:a05:600c:1d82:b0:43c:f517:ac4e]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3143:b0:439:4c1e:d810 with SMTP id 5b1f17b1804b1-43d1808593dmr31744305e9.9.1741889518918; Thu, 13 Mar 2025 11:11:58 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:28 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-9-04972e046cea@google.com> Subject: [PATCH RFC 09/11] mm/page_alloc: Split MIGRATE_UNMOVABLE by sensitivity From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Stat-Signature: zrnh3hmasjhf6wcz7t5sbm1k3zt9rwne X-Rspamd-Queue-Id: ACC24C000B X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1741889520-335109 X-HE-Meta: U2FsdGVkX1/JRYVZnTcGkvo2w6ev89+q41ygn83P/rHSKJ9HiOE+Yd+0OyvNp/IqfLgDxMUnoO4PqwqwBvc5GwPikWhmypM5E88WlNaDkKVg24ngkYgYy/v0cXFGjA7xZI0hu5lwKyJy3LFuOPPqggmc8jrImL5viIxio87Bhlph0zFoBkww1XGo/jtkUIEbu9XN+xIOW9AdgiY/yjXIeSuBoKv6vpOrMKtlt6tyAf33zDnYId3T3mb2CFuTniTsMd3xzfMPgrvDD+T8UsvEMSGAktGqAg9Wv6otnkfPxMOsx70NgdNi9BgHl5UM3dVsjKbKS7XZeAJ82HnRVu1ghdf6Ed2pQ86MJab9z7MBB1J6Nvc3xC422A7gFitOfkb5YwXNViONSD+CDGEG7+gQyJ7CwYUoDjOaJywQBeiXN5sJIt+KHnU2UP/+Gbb4wlDID298dvOUYmoUMkCQiyNPRW3zm+Qsgl7OFvjC7Dqr5PeQ2dWib92RRFS43WA/pSngrJ+PELsnWmPQsv/vxngQn4y4gAHg3i95fehhO6B/kNNEeVZIX2/mgRytk0DMfN2yA1Diiknt8JY8TudpQvUe7m6PvARD84aYKP051GLw4xkz8z7r6l1iMBfDYx+jdhK4xDEgzlzXTKE+XJ3bHAe2oujN8jICkuj8MGyCYsbDy6Bu9zzUASZWKXRgx368hzijoqOB8ayYvofeLrLt7F7SFSgQHqlb0oMDvRwbldP1VUBrpqB2gtT/9YHPkiniFfS2Qgh39rtWpbeYgQBYWIM0rvoZ7MPrRjiQJmK64xkmVCCVqhMppKvbuJnTGH6CfQWf/DF95zN9newGUs3KyQ/KJo6H2YQGqSw+KeKxnNVyD1T3QrUzB1Fg/5Xk/BdBNMc937/wvapKlqB7Pl3uVyRWbSdltkX9tdwEtFPRRjnzj5JRocXnE7I70/pLQpn7fFupzAV98sY4uwhqA0kcQA+ LGfKFYIi 53TNmgeHsZofuuGw0zrcr6PbLHi8non6m03db7ieSzNt1iqebgEsHtF+b8r8z8llvkNjZOV6FIhoFksp1mTNUbskBlY3AhD2NR8eYVNCR3Aq2CqRgztFXz09ZdxOF2DcMdDIeSpjAZ+9ND4uy/+93rWLu4Fw8QNSa9Hsq4s64Ub5ekgPFTL/KeI6kLWtuIhaw/I8600dzvZUTvBoIFz1YCdhAiSVbCTkGjOhFoJh+z4L7/dC/VRirmu8k1exCkvbd18/H6TOlnVITrGJM/E94z/j0ksxhXpcrdP9mNdhyDAsgu+LgVChj389et1NuxFaTcG7rbvgmUZHG9ooP0WZ3kOg4Nmm8ZY8wlI8xxkB1BFjimTwuT2t/1asu9FgzE8FkTQ+zwvHUD13oE4CrJvxmzz74hM5fcGAF+BH4IOKwl7wu1lHl3dH90BSxehW/wDkMZladjkfHA2bUbJo56ThlQuLjW2Ob8i0QPTCdxO92KEsmaUjMStVbx7fHAkgLW4GoehtI2/w4ZWzy/oNoLd6ZqwgaK7l1fYv6xk7E2XmuMFJPAyrshXHQlY+PGPTqHLy2cPdGdmbFPnI96qanaO3VX0TY4TFSqq73jJ6fq/S9bZNnA1c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When ASI is compiled in, create two separate unmovable migratetypes. MIGRATE_UNMOVABLE_NONSENSITIVE represents blocks that are mapped into ASI's restricted address space. MIGRATE_UNMOVABLE becomes MIGRATE_UNMOVABLE_SENSITIVE. All other migratetypes retain their original meaning and gain the additional implication that the pageblock is not ASI-mapped. In future extensions it's likely that more migratetypes will need to support different sensitivities; if and when that happens a more invasive change will be needed but for now this should allow developing the necessary allocator logic for flipping sensitivities by modifying the ASI page tables. For builds with ASI disabled, the two new migratetypes are aliases for one another. Some code needs to be aware of this aliasing (for example, the 'fallbacks' array needs an ifdef for the entries that would otherwise alias) while others doesn't (for example, set_pageblock_migratetype() just works regardless). Since there is now a migratetype below MIGRATE_PCPTYPES with no fallbacks, the 'fallbacks' arrays are no longer all the same size, so make them be terminated by a -1 instead of having a fixed size. On non-ASI builds, the new 'if (fallback_mt < 0)' in find_suitable_fallback() is provably always false and can be eliminated by the compiler. Clang 20 seems to be smart enough to do this regardless, but add a 'const' qualifier to the arrays to try and increase confidence anyway. Signed-off-by: Brendan Jackman --- include/linux/gfp.h | 21 +++++++++++++++++---- include/linux/mmzone.h | 19 ++++++++++++++++++- mm/memory_hotplug.c | 2 +- mm/page_alloc.c | 22 +++++++++++++++------- mm/show_mem.c | 13 +++++++------ 5 files changed, 58 insertions(+), 19 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 23289aa54b6c38a71a908e5a6e034828a75a3b66..0253dcfb66cbaa82414528a266271d719bc09d6a 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -20,13 +20,26 @@ static inline int gfp_migratetype(const gfp_t gfp_flags) VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) == GFP_MOVABLE_MASK); if (unlikely(page_group_by_mobility_disabled)) - return MIGRATE_UNMOVABLE; + goto unmovable; + + /* Only unmovable/unreclaimable pages can be nonsensitive right now. */ + VM_WARN_ONCE((gfp_flags & GFP_MOVABLE_MASK) && !(gfp_flags & __GFP_SENSITIVE), + "%pGg", &gfp_flags); switch (gfp_flags & GFP_MOVABLE_MASK) { - case __GFP_RECLAIMABLE: return MIGRATE_RECLAIMABLE; - case __GFP_MOVABLE: return MIGRATE_MOVABLE; - default: return MIGRATE_UNMOVABLE; + case __GFP_RECLAIMABLE: + return MIGRATE_RECLAIMABLE; + case __GFP_MOVABLE: + return MIGRATE_MOVABLE; + default: + break; } + +unmovable: + if (gfp_flags & __GFP_SENSITIVE) + return MIGRATE_UNMOVABLE_SENSITIVE; + else + return MIGRATE_UNMOVABLE_NONSENSITIVE; } static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9540b41894da6d67751506614e90bed142e127b4..b8215b921b71c0b1cbda2d262664b1ee2a0cb750 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -46,7 +46,19 @@ #define PAGE_ALLOC_COSTLY_ORDER 3 enum migratetype { - MIGRATE_UNMOVABLE, + /* + * All movable pages are sensitive for ASI. Unmovable pages might be + * either; the migratetype reflects whether they are mapped into the + * global-nonsensitive address space. + * + * TODO: what about HIGHATOMIC/RECLAIMABLE? + */ + MIGRATE_UNMOVABLE_SENSITIVE, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + MIGRATE_UNMOVABLE_NONSENSITIVE, +#else + MIGRATE_UNMOVABLE_NONSENSITIVE = MIGRATE_UNMOVABLE_SENSITIVE, +#endif MIGRATE_MOVABLE, MIGRATE_RECLAIMABLE, MIGRATE_PCPTYPES, /* the number of types on the pcp lists */ @@ -89,6 +101,11 @@ static inline bool is_migrate_movable(int mt) return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE; } +static inline bool is_migrate_unmovable(int mt) +{ + return mt == MIGRATE_UNMOVABLE_SENSITIVE || mt == MIGRATE_UNMOVABLE_NONSENSITIVE; +} + /* * Check whether a migratetype can be merged with another migratetype. * diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 16cf9e17077e359b98a69dc4bca48f4575b9a28c..28092c51c7a49546f5b7961aaf1500c72de88337 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1127,7 +1127,7 @@ int mhp_init_memmap_on_memory(unsigned long pfn, unsigned long nr_pages, if (mhp_off_inaccessible) page_init_poison(pfn_to_page(pfn), sizeof(struct page) * nr_pages); - move_pfn_range_to_zone(zone, pfn, nr_pages, NULL, MIGRATE_UNMOVABLE); + move_pfn_range_to_zone(zone, pfn, nr_pages, NULL, MIGRATE_UNMOVABLE_SENSITIVE); for (i = 0; i < nr_pages; i++) { struct page *page = pfn_to_page(pfn + i); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index aa54ef95233a052b5e79b4994b2879a72ff9acfd..ae711025da15823e97cc8b63e0277fc41b5ca0f8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -418,8 +418,9 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, void set_pageblock_migratetype(struct page *page, int migratetype) { if (unlikely(page_group_by_mobility_disabled && - migratetype < MIGRATE_PCPTYPES)) - migratetype = MIGRATE_UNMOVABLE; + migratetype < MIGRATE_PCPTYPES && + migratetype != MIGRATE_UNMOVABLE_NONSENSITIVE)) + migratetype = MIGRATE_UNMOVABLE_SENSITIVE; set_pfnblock_flags_mask(page, (unsigned long)migratetype, page_to_pfn(page), MIGRATETYPE_MASK); @@ -1610,10 +1611,14 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, * * The other migratetypes do not have fallbacks. */ -static int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] = { - [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE }, - [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE }, - [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE }, +static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] = { + [MIGRATE_UNMOVABLE_SENSITIVE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE }, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + /* TODO: Cannot fallback from nonsensitive */ + [MIGRATE_UNMOVABLE_NONSENSITIVE] = { -1 }, +#endif + [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE_SENSITIVE }, + [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_MOVABLE }, }; #ifdef CONFIG_CMA @@ -1893,7 +1898,7 @@ static bool should_try_claim_block(unsigned int order, int start_mt) * allocation size. Later movable allocations can always steal from this * block, which is less problematic. */ - if (start_mt == MIGRATE_RECLAIMABLE || start_mt == MIGRATE_UNMOVABLE) + if (start_mt == MIGRATE_RECLAIMABLE || is_migrate_unmovable(start_mt)) return true; if (page_group_by_mobility_disabled) @@ -1929,6 +1934,9 @@ int find_suitable_fallback(struct free_area *area, unsigned int order, *claim_block = false; for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) { fallback_mt = fallbacks[migratetype][i]; + if (fallback_mt < 0) + return fallback_mt; + if (free_area_empty(area, fallback_mt)) continue; diff --git a/mm/show_mem.c b/mm/show_mem.c index 43afb56abbd3e1f436452ef551dd8149545ccb13..59af6fb286354156cc2aa258fdd78c85797248a6 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -141,15 +141,16 @@ static bool show_mem_node_skip(unsigned int flags, int nid, nodemask_t *nodemask static void show_migration_types(unsigned char type) { static const char types[MIGRATE_TYPES] = { - [MIGRATE_UNMOVABLE] = 'U', - [MIGRATE_MOVABLE] = 'M', - [MIGRATE_RECLAIMABLE] = 'E', - [MIGRATE_HIGHATOMIC] = 'H', + [MIGRATE_UNMOVABLE_SENSITIVE] = 'S', + [MIGRATE_UNMOVABLE_NONSENSITIVE] = 'N', + [MIGRATE_MOVABLE] = 'M', + [MIGRATE_RECLAIMABLE] = 'E', + [MIGRATE_HIGHATOMIC] = 'H', #ifdef CONFIG_CMA - [MIGRATE_CMA] = 'C', + [MIGRATE_CMA] = 'C', #endif #ifdef CONFIG_MEMORY_ISOLATION - [MIGRATE_ISOLATE] = 'I', + [MIGRATE_ISOLATE] = 'I', #endif }; char tmp[MIGRATE_TYPES + 1]; From patchwork Thu Mar 13 18:11:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC5DFC28B28 for ; Thu, 13 Mar 2025 18:12:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12241280019; Thu, 13 Mar 2025 14:12:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00C6B280010; Thu, 13 Mar 2025 14:12:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF080280019; Thu, 13 Mar 2025 14:12:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BCBEE280010 for ; Thu, 13 Mar 2025 14:12:04 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D0D11140430 for ; Thu, 13 Mar 2025 18:12:05 +0000 (UTC) X-FDA: 83217321810.12.85FC9B0 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf29.hostedemail.com (Postfix) with ESMTP id 77DF8120014 for ; Thu, 13 Mar 2025 18:12:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=aLvy1TEZ; spf=pass (imf29.hostedemail.com: domain of 38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889523; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=3iLeUMbB+d/7Ly/Sz4eOrmSImKiCtxD8VosRkpzGJ4UBwcdcnmYaoSCDx828mKk/KFFkmr yHtFowjX06uJhblZt8ZamSv+6Qx9qJ5NzVm9fmBrFmaZHSIRclk0tPHgsGXQGreXWyBkEM 1dggzb2fHEtTTOUoOs1bJ6igwmmFJ6s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889523; a=rsa-sha256; cv=none; b=AgH87D9MlVL1P+FLtR0PUPcjAmNtKQowiZikqANfO8tUREH0W7oMQr/8i8svyO53PkK24M DoTLh0KmxeqDcXb46Q7aqxytE6U9beSt/JGrl1WYn1yUADdqMZXXdLawBdpWy8e7gZ2Xz0 Rf8kir9jAdcmnxHb//zMQs1XsKnxcMo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=aLvy1TEZ; spf=pass (imf29.hostedemail.com: domain of 38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-39143311936so606414f8f.0 for ; Thu, 13 Mar 2025 11:12:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889521; x=1742494321; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=aLvy1TEZh6bdIYx9kXpG1JtkZsCMjqTZbqYpkfhn+a8C6masvcnqBAzC3i4wcGhjLj gpN8xXHRFVF9xIuIVMQAwU67LBmj4ObTwPUl9Qc5nPdSQU/jRv9VcMPjxM5DZ+CdIzIY 8jIpk45sUNiYnYWI4mbL/kIrOwN7FhgVBFSV77F+N3wRZsxmREqnp/TSmrwAH1jaL0Xs IRxHHQ+qxtAO5gKuTidZyyvsNjNYc5g8DRikowmqz0bsZJNEU4vYL3wxTqL/7I1/kxXn JoIrgZHJ9zdekqK0xdfymNNBlBtvl8Pn6KJsEKtqYplnR2+awvBaw38ByFNhIOM3LbTz XAzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889521; x=1742494321; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=nNPodanKVIEpv24U854BzYha63o7Wb8Ib4o2m3R7NoJtttjPIK48lAA4gGuek2Rv/w OgVcVrWN9jWCdFjd9beEAMbQiSnk22Sb8Mmq53FaDZekTyWtMj95L5ZRTNmIy7zLFSvd mYB2VwlQgh9HZjV6wFp4Au5JV0SxKdGXvpfV5Ph0woi+Lt4wOiMo2H7dQGoTkr4rR7hE a05Z7GJH/I424tYxazFiEcEruRtGP/4bz9CHPgMEVXODUB2r5uTt4jW71DH6R4iEgbxV CItewwgtjlK3z1VNee5OL+2HFgVjgvSj/PgWZ1lvbCpruqvcTSFcYWng0Nas30cerDJJ +eVg== X-Forwarded-Encrypted: i=1; AJvYcCWSGLa9aFFG+COj8P8KxhoRm64wqzdA82gRb9zc5zrtsxD6O6BB6+/sx+KZ5KHFsKgxjMIpsXbXPg==@kvack.org X-Gm-Message-State: AOJu0YyXNe+0gbP6iYUT3ce5epRZ2s7X7QKVhTEwsdZKaKrAV7pweNL1 RdeuNdGGBg/5Gouj9atxc19K4xWNNuyEufDt4u/9eFz0TALVsq10k8xGt3X5zn6VccSXe09b84V YvnJSGSxjoQ== X-Google-Smtp-Source: AGHT+IGTDPbDngPkmEoWDX7m7yWVpzg47wDDczIbjQCcv6h8SnX4qWQyopmuiT4sdFB+hEFVGiIYdXMgmMnnIQ== X-Received: from wro9.prod.google.com ([2002:a05:6000:41c9:b0:391:2f9a:9a8b]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1f83:b0:390:f9d0:5e3 with SMTP id ffacd0b85a97d-39263affd7amr11502380f8f.1.1741889521421; Thu, 13 Mar 2025 11:12:01 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:29 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-10-04972e046cea@google.com> Subject: [PATCH RFC 10/11] mm/page_alloc: Add support for nonsensitive allocations From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 77DF8120014 X-Stat-Signature: puobnx8e17oyu77ecjzgxpsin7j735tf X-HE-Tag: 1741889523-576836 X-HE-Meta: U2FsdGVkX1/gmNBpA6cdqXtdmXnLtI969O/PWVYZELev47jE/W3Q/1DG20iye5t8WqcMHlCdA/ogPPMg8QhlI46wyEwMvWIoEA3ywx6AWMVGYjmuUMPBXoQ/FjDYyXLXoJCt8YC5EH0mForgVpqgtYaWf4Jr/WJ8n4jK5Rw/el2nDoaujgRTYBzmBsDGcNIv39y9z1BzvUXhBRgasB0m4kxiw962nmYHXgXtk68vtgHFFVvCLMSeFpyXeaSecPnD1blZdcb6psyZiO8qZlOjRATUlYnC8iy9CA38b+zNADHFy4lOgv5qECc/idVLN4K6h0FsNAiLSs+ve9mnapCSjrF/987WRgdfqKYXQwmG5L5gjJYxfBXLKUwqq9fCIeuQWe2ye/F/NlKB7ucWgQ39PBrjrQtlnnu5FgQRtiWoe4QqacjUq+v7OJzPPxQfyX8w8zUYUf8I4gVL83bRC5DslsTT4rSo2M+MbWZWFgPHt2mBKZqRCM4T5Ao+JpoUp0GoynjGZVGy0mfLda2nv/xf0MpMxeyLE+ze8EbTa988AAmK7frYSPjGn5EF0ofkFsHy1yoddgkcS65O4PQp+bc+ZiJ+c30EdD/EICBQYgsr4syFojTx3mRVbvdB5w2PRhif29BY0O2m4jUgQaQXUX2jUDKFqcl33OndxsyAvgQUFdZyomzM9P+tPMGOVueQW5j4Gof+0sJMX1GwZrxEOL3fAPs15HdBF+nhanoXR7E19y6lS2bCkJsdG534OGqR/SRAoIJPJsC7HJjazQSOmteebr5vhOWvi6sLrNbgNkn9IRQk5XTLzvSu3RCVrmbf5pxhOD5GiaE97KLIefHQCpbsJLEVQODRHr54qXwY7v+pjR5jFmG/ii+525kbTMMZP/nq14AlI8/kwBslrZO80lQ5WW+iPNvqK8y1D2PPWJm1ibrwT2z2tkSCS+8F9mNtA9kIHlobkaQRaSittkcHAda YpVzL8fR z1Isx+Js22tRNxQPsuoHXJMIsIfAb8Skk3CYdlB6u+X0+3Zw7ysWvP8bJmOgCglN1LVfN93Up0al26Zux7dgxnB4tkgTuz+pCBXCCCEBcPRjbq3VkmqZ1mRegSIAjy3vjLRiRilAh37ogukmpQiN9u7K+0jiKEkNYAQuEIUSNfcS0luIQFTp8iCr+1on5PZo9KpUc6238qGqHXFmXbzJo9l69CwIzLdTIS4B8evXwDL0lsed3/YKdzZYq4Ct35gjNFuzOEr4K/Dgs9L3YdsFUqwCvxVHAuxE4qJUAbbJsntrCGPXGtEu9Gv9+9kP0s6SKhfP1XRjycAb0eA2qs2hhwfdlUrRRLx6dTavXfBPQRUtdOcZN7P24MGffiKO2ZbgHW+4J7IubcwVEuFCaOQiFaD1fQ4pIESUdP2IC9zMbEejz1TeFzR5aHzEqtF6soCnwzXFW8B6l5MGnuiNmHmNSPlQhvHdCXRnjxAJg4fbU5YqDEWdP72WLVD+0us2sl1D+kdqDEnEv7YOjJdpXFBxHuPd9PdpGqJeGpgrFczaRThM3DiH2scrnqLpFJNYMKsNYbo77ADWBhdUTdWtk9uK6CGIbBhJhDtiCeyXSpyR0+Z4M570= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Creating nonsensitive pages from sensitive ones is pretty easy: Just call asi_map(). Since this doesn't require any allocation or synchronization (at least, the synchronization is implied by the zone lock) just do it immediately in the place where the pageblock's migratetype is updated. Implement a migratetype fallback mechanism to let this happen, restricted to requiring a whole pageblock. Now that it's possible to create nonsensitive pages, the allocator supports non-__GFP_SENSITIVE allocations, so remove the temporary hack setting this flag unconditionally in the entry point. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 5 ++++ mm/page_alloc.c | 67 +++++++++++++++++++++++++++++++++++----------- 2 files changed, 57 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index cf8be544de8b108190b765e3eb337089866207a2..4208b60852ace25fc9716c276bd86bf920ab3340 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -2,7 +2,12 @@ #ifndef _ASM_X86_ASI_H #define _ASM_X86_ASI_H +#include +#include +#include + #include +#include #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ae711025da15823e97cc8b63e0277fc41b5ca0f8..0d8bbad8675c99282f308c4a4122d5d9c4b14dae 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -15,6 +15,7 @@ * (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ +#include #include #include #include @@ -422,6 +423,18 @@ void set_pageblock_migratetype(struct page *page, int migratetype) migratetype != MIGRATE_UNMOVABLE_NONSENSITIVE)) migratetype = MIGRATE_UNMOVABLE_SENSITIVE; + /* + * TODO: doing this here makes this code pretty confusing, since in the + * asi_unmap() case it's up to the caller. Need to find a way to express + * this with more symmetry. + * + * TODO: Also need to ensure the pages have been zeroed since they were + * last in userspace. Need to figure out how to do that without too much + * cost. + */ + if (migratetype == MIGRATE_UNMOVABLE_NONSENSITIVE) + asi_map(page, pageblock_nr_pages); + set_pfnblock_flags_mask(page, (unsigned long)migratetype, page_to_pfn(page), MIGRATETYPE_MASK); } @@ -1606,19 +1619,35 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, /* - * This array describes the order lists are fallen back to when - * the free lists for the desirable migrate type are depleted + * This array describes the order lists are fallen back to when the free lists + * for the desirable migrate type are depleted. When ASI is enabled, different + * migratetypes have different numbers of fallbacks available, in that case the + * arrays are terminated early by -1. * * The other migratetypes do not have fallbacks. + * + * Note there are no fallbacks from sensitive to nonsensitive migratetypes. */ -static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] = { - [MIGRATE_UNMOVABLE_SENSITIVE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE }, #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION - /* TODO: Cannot fallback from nonsensitive */ - [MIGRATE_UNMOVABLE_NONSENSITIVE] = { -1 }, +#define TERMINATE_FALLBACK -1 +#else +#define TERMINATE_FALLBACK #endif - [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE_SENSITIVE }, - [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_MOVABLE }, +static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] = { + [MIGRATE_UNMOVABLE_SENSITIVE] = { + MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, TERMINATE_FALLBACK + }, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + [MIGRATE_UNMOVABLE_NONSENSITIVE] = { + MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE + }, +#endif + [MIGRATE_MOVABLE] = { + MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE_SENSITIVE, TERMINATE_FALLBACK + }, + [MIGRATE_RECLAIMABLE] = { + MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_MOVABLE, TERMINATE_FALLBACK + }, }; #ifdef CONFIG_CMA @@ -1931,6 +1960,21 @@ int find_suitable_fallback(struct free_area *area, unsigned int order, if (area->nr_free == 0) return -1; + /* + * It's not possible to change the sensitivity of individual pages, + * only an entire pageblock. Thus, we can only fallback to a migratetype + * of a different sensitivity when the entire block is free. + * + * This cross-sensitivity fallback occurs exactly when the start + * migratetype is MIGRATE_UNMOVABLE_NONSENSITIVE. This is because it's + * the only nonsensitive migratetype (so if it's the start type, a + * fallback will always differ in sensitivity) and it doesn't appear in + * the 'fallbacks' arrays (i.e. we never fall back to it, so if it isn't + * the start type, the fallback will never differ in sensitivity). + */ + if (migratetype == MIGRATE_UNMOVABLE_NONSENSITIVE && order < pageblock_order) + return -1; + *claim_block = false; for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) { fallback_mt = fallbacks[migratetype][i]; @@ -4694,13 +4738,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac = { }; - /* - * Temporary hack: Allocation of nonsensitive pages is not possible yet, - * allocate everything sensitive. The restricted address space is never - * actually entered yet so this is fine. - */ - gfp |= __GFP_SENSITIVE; - /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. From patchwork Thu Mar 13 18:11:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98B18C282DE for ; Thu, 13 Mar 2025 18:12:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5344B28001A; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49638280010; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33ACA28001A; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 11951280010 for ; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A5FA552A70 for ; Thu, 13 Mar 2025 18:12:07 +0000 (UTC) X-FDA: 83217321894.19.A05692D Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf06.hostedemail.com (Postfix) with ESMTP id 47A43180002 for ; Thu, 13 Mar 2025 18:12:04 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YDi4XS6Z; spf=pass (imf06.hostedemail.com: domain of 38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889525; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=O5b1QZG6hyFjiQrTtA5rhD/QWKE9dTRBhh6fICvxhPMlDzMuEVlo985c/BN5pPp+aV0yLl 40DgLedk18AZkaiBxKvreSeNh3yko4XnuAj55Byg1o9unWKL5udrc8OFghfVOpzN+fTQ9t T1Hcro7bTVf0G/7ZSGOFiXKpgMD8N7w= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YDi4XS6Z; spf=pass (imf06.hostedemail.com: domain of 38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889525; a=rsa-sha256; cv=none; b=5E40XVyngn5PAkKt/xwKSUG+X0XKZ/HIlseTvO+gUhGpcdO+gA1CVfsMs9xuMcbteKyHUY JIXPEZ47a918Y6Or4LarpYxo8wES7lclhazLteImCcWp9PCmtEIPP5NSG0qOcwAM6APA6N bQNsdZXPqkF7iRIntWSfrW5LEw/0aX4= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43ceeaf1524so6822145e9.1 for ; Thu, 13 Mar 2025 11:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889523; x=1742494323; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=YDi4XS6ZV67BTzUJR8FQcP7aBnCeqZ7YoMpOIiwhl8MwNsh94Oss62eNhJvRcxAeYw OA1Q0wMkmdqGVC8z2WdHYHmY9Strddruxw2+gPIdYypxvVAeOAOG8zSio//dc5KLcJNo sD5W9qLNNs0Le0/PdMZ8rH51XobPInJiJpeNxNUbwHTs4yzNNgoP+Nyma/m1pTii1d4S DhyIdF4nN6rG80k9UofJGzcNLYaHIDS8ReSugmC0Tj1m4qi2cJJx0b7TYPy0NH8Q5EUu FTMVNs5//CrtdLzY9TVrPszl7A9qDQ4RXeDiVf//GbGLbTUhEnwi0Gg/iB3FFYwD7XGP 7l2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889523; x=1742494323; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=KqGn1Iuql4dKtSL6cXwyxr9SiO/x0xzYswwjvLFhhkl7jO9GI+elqGdWli9e6geK2s BPtU/cioJDrNuc+JES431QzGFpyjkap9kim7prPXxhvEzq9wBVREM/FvQ1KQbzh6vOBP +wfbYTgwUt8+qzW6bI6+a6H2sDCImWGuXxEMytUtjMCQyEkGa5HZnfAfV6bucFNBSq1b SQzYVhjI0ig1M5zfWmYjMSc1e+uBzHhntLXrSmJdXU2dzWICY2NwpDvT1MtmCmRapQ5O 6FbGYBY+AN8NtKFYqe+Eja8Q3kTd8OEOiAuNkZzmrXOzTB8JAcfffITjefuExqg7cxl0 YBgQ== X-Forwarded-Encrypted: i=1; AJvYcCWjQ2WjKf/UCTSyVkvqz0HXHAaJP6Trw8WW69CgN0IXPwpmSXH36PIf55xBzfENac1TO4sc7X6Sxg==@kvack.org X-Gm-Message-State: AOJu0YxkaPUJ9Aa2VWPeb00w5AGzbZg5R/ZzWWyyjshm04hWuif3jlXP njk6jw3WnOwMsgqCXKCUDBv8WHPruozJQne2FH82C6bmzT1uPGP60JLBgyVopoBpidpxjyBM5hL hPY9Oe0fj/A== X-Google-Smtp-Source: AGHT+IFbs6k3vnl5QQ+C0J0hTTMoncea9PLFMkDkNtbfor/v1Jhe1WlJqAJFv/1WYWGnmKJfkaAhbBEhyKJgWQ== X-Received: from wmrn39.prod.google.com ([2002:a05:600c:5027:b0:43c:f03d:18aa]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3143:b0:439:4c1e:d810 with SMTP id 5b1f17b1804b1-43d1808593dmr31746155e9.9.1741889523325; Thu, 13 Mar 2025 11:12:03 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:30 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-11-04972e046cea@google.com> Subject: [PATCH RFC 11/11] mm/page_alloc: Add support for ASI-unmapping pages From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Queue-Id: 47A43180002 X-Rspamd-Server: rspam08 X-Stat-Signature: zz195gxmfa1tdso4xdsqbexygeeamwh9 X-HE-Tag: 1741889524-975425 X-HE-Meta: U2FsdGVkX1/BE1Cc2ZVM9QzsAKaClfk6rSHlhvb9hNTps/PSgsA4KWp6Jvyth69d69zrg+gUx/ED0A9Y6f7rCAof71Thx10Uo4idgqj+pIwUsB2P6CXc7gsVWc4FK8HwdNhG4ZK8y0ImNNZmDoyFkWCibYNY6Tv0g6UZitEVB6PvkKlXGP6OsQc/N7crvdvmGQZKNPf6xcsRq5Zv1OKdleEvz5PzKXs9qFU/RvfyGZ04LTV5WucYVZtemEymwEETo3meSBrH0HEutG7DvF0TjwffbpBbxW7oLwMidOx33uzq50n2ottLkR4dpmV9jraqWjNkUiot08EfAjpUFTkpDkV3uldP3oPMP6mDQMMOSGClcjQMPTi9j/115TmlyBGi+UvBGzvHeOUuBz+YgDxc3rQvsMU35xNdPAYc/s0eE4yXhdp6bU66/Ypi82KpOg6CJYLRJcA5mlT568nr2dx48A24XWC4TeYZeyX1sJPLq/maDDYCm8KgNaAzFPOj5gyZCdQuSZtKMSGwu0h3iUBCDiaVinOfBWOUgt2xdWkR4VXQ8pQlA0aaSkR4iZ2PqtV4PHwdNyAypGzCapMqVSA+ywJYeKe3ZLv1XAov3a9wqzw5q5tQU+3Ser9S0Icoo/ar/6wrXd2AgG4I2+T6o67dE+ruwN3uXGxzyoOVMrhdGUWGtrw0ZSAASk9MeJDJWOwAEtl5IehcgIFNtFbgpIwDFnOk/erd9ulNaAn/dlgteErS2JXcH8Dw9FUDB04ptypb+8CyZrOvsp45m/FLgj2Kzwu+R/LlefyFV0Puw0N8FjugsaTydr8y60TSw+mCfOPODJSiGew5LdMJHZkzD+UbFsPKAaIyTpe4+O0gDkwUTPK9kW9q/UZ20t+T68lX+FhrXC229/kXR5czNpUNML3lORcqMlbRcHhxRrIa28DErK3ySEPsrn6nmJ3yAdo7Ap8HvuQHeJecwPiMfCGtZJX WruXYgp/ s8G3Mkkwo6/Q5Uiw6hjncqCcpBzX3j7+sOtrSK/FX0BngSm92rsB1oVyJYyr1ZlW7lGIFVRb86jwNP7KZeCu/I0yn6VXRt3uJZqlEhnUc3mLrYjbBqw19yD4eY4wC5tFjduX9p8Du3iEv1RHC57BY7GmO9EVSQ5QC6+BoOSgaxXUrVtSLAmJ/68wQEZS4aSj8wqgLd2ACRWnMjF0e9QMsHlDmA1+TWg19hFdQou8HG9VYlAa2kwr2Ol7U/dfSSnIj0nNMESBEnKaakRkKVRErMwAU6+OgPH+KGCpiCEdNa8qAa7brw4d+Aw8jPYLedNDp23z/velKV8MKxWR+CHux3G/G56DBEfMBa4f/+wgM58+y8EsPhsx3W7xUi57EIqCNRqyVa7ZE+AlNO6DvoSe0oZ6nsV7gnpnZHyvZnW1FoIsoIRGPrQAfIbhzNp8D7HghOjN8sv9HPVvR1jdgvMGvkaTJVZK3Dydl8X8YRECy7j0ZhN3B6koWxoCWb4qumjO4Yw2rlCxtUSKCQquyfgpQ3rBny2pyW2u/JjX5BPrWF5wBkW/iB6LEuBSaQ21G+pSQ56O+F7PlIGG57M2HHyFxuJ07AmDi4M2nMZfY5+NNuAs6cbM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: While calling asi_map() is pretty easy, to unmap pages we need to ensure a TLB shootdown is complete before we allow them to be allocated. Therefore, treat this as a special case of buddy allocation. Allocate an entire block, release the zone lock and enable interrupts, then do the unmap and TLB shootdown. Once that's complete, return any unwanted pages within the block to the freelists. Signed-off-by: Brendan Jackman --- mm/internal.h | 5 ++++ mm/page_alloc.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 86 insertions(+), 7 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b82ab345fb994b7c4971e550556e24bb68f315f6..7904be86fa2c7fded62c100d84fe572c15407ccf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1189,6 +1189,11 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ALLOC_ASI_UNMAP 0x1000 /* allow asi_unmap(), requiring TLB shootdown. */ +#else +#define ALLOC_ASI_UNMAP 0x0 +#endif /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0d8bbad8675c99282f308c4a4122d5d9c4b14dae..9ac883d7a71387d291bc04bad675e2545dd7ba0f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1627,6 +1627,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, * The other migratetypes do not have fallbacks. * * Note there are no fallbacks from sensitive to nonsensitive migratetypes. + * That's instead handled as a totally special case in __rmqueue_asi_unmap(). */ #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION #define TERMINATE_FALLBACK -1 @@ -2790,7 +2791,77 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, #endif } -static __always_inline +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +/* + * Allocate a page by converting some memory to sensitive, by doing an ASI + * unmap. This can't be done via __rmqueue_fallback because that unmap requires + * a TLB flush which can only be done with interrupts on. + */ +static inline +struct page *__rmqueue_asi_unmap(struct zone *zone, unsigned int request_order, + unsigned int alloc_flags, int migratetype) +{ + struct page *page; + int alloc_order; + int i; + + lockdep_assert_irqs_enabled(); + + if (!(alloc_flags & ALLOC_ASI_UNMAP)) + return NULL; + + VM_WARN_ON_ONCE(migratetype == MIGRATE_UNMOVABLE_NONSENSITIVE); + + /* + * Need to unmap a whole pageblock (otherwise it might require + * allocating pagetables). Can't do that with the zone lock held, but we + * also can't flip the block's migratetype until the flush is complete, + * otherwise any concurrent sensitive allocations could momentarily leak + * data into the restricted address space. As a simple workaround, + * "allocate" at least the whole block, unmap it (with IRQs enabled), + * then free any remainder of the block again. + * + * An alternative to this could be to synchronize an intermediate state + * on the pageblock (since since this code can't be called in an IRQ, + * this shouldn't be too bad - it's likely OK to just busy-wait until + * any conurrent TLB flush completes.). + */ + + alloc_order = max(request_order, pageblock_order); + spin_lock_irq(&zone->lock); + page = __rmqueue_smallest(zone, alloc_order, MIGRATE_UNMOVABLE_NONSENSITIVE); + spin_unlock_irq(&zone->lock); + if (!page) + return NULL; + + asi_unmap(page, 1 << alloc_order); + + change_pageblock_range(page, alloc_order, migratetype); + + if (request_order >= alloc_order) + return page; + + spin_lock_irq(&zone->lock); + for (i = request_order; i < alloc_order; i++) { + struct page *page_to_free = page + (1 << i); + + __free_one_page(page_to_free, page_to_pfn(page_to_free), zone, i, + migratetype, FPI_SKIP_REPORT_NOTIFY); + } + spin_unlock_irq(&zone->lock); + + return page; +} +#else +static inline +struct page *__rmqueue_asi_unmap(struct zone *zone, unsigned int order, + unsigned int alloc_flags, int migratetype) +{ + return NULL; +} +#endif + +static noinline struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned int order, unsigned int alloc_flags, int migratetype) @@ -2814,13 +2885,14 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, */ if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); - - if (!page) { - spin_unlock_irqrestore(&zone->lock, flags); - return NULL; - } } spin_unlock_irqrestore(&zone->lock, flags); + + if (!page) + page = __rmqueue_asi_unmap(zone, order, alloc_flags, migratetype); + + if (!page) + return NULL; } while (check_new_pages(page, order)); __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); @@ -3356,6 +3428,8 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif + if (gfp_mask & __GFP_SENSITIVE && gfpflags_allow_blocking(gfp_mask)) + alloc_flags |= ALLOC_ASI_UNMAP; return alloc_flags; } @@ -4382,7 +4456,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) | - (alloc_flags & ALLOC_KSWAPD); + (alloc_flags & (ALLOC_KSWAPD | ALLOC_ASI_UNMAP)); /* * Reset the nodemask and zonelist iterators if memory policies can be