From patchwork Wed Feb 23 05:21:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junaid Shahid X-Patchwork-Id: 12756368 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C328C43219 for ; Wed, 23 Feb 2022 05:24:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B8ED8D000A; Wed, 23 Feb 2022 00:24:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 169948D0001; Wed, 23 Feb 2022 00:24:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 007EE8D000A; Wed, 23 Feb 2022 00:24:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id E3C1E8D0001 for ; Wed, 23 Feb 2022 00:24:04 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 85B0F181AEF3F for ; Wed, 23 Feb 2022 05:24:04 +0000 (UTC) X-FDA: 79172903208.21.A0B1448 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 10C90120002 for ; Wed, 23 Feb 2022 05:24:03 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-2d07ae11462so163390657b3.8 for ; Tue, 22 Feb 2022 21:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=rf+PEwSCAjx96OEampym4lZ49QLmyoHK2vvPo63tKDU=; b=ClDNkRcR05xwUIt+/3suGkjgzNs7jx8crmYYSBy7YIfyMwXpzxpsuO7L5+YX8vxRfh 1E4PSbB3uEBG+XkPHeMZghVckls3s0XgENlVkTaPWZjkwqbjhFcBMXRc0pNft0u4lgb5 OHkuB86fnETxs+Q/fn9lSwDJhVOhXOzfA1+wbVB6gzJyDg9tXEAFaT4CQ0u+HBjQvZbK xaLThn55HGR54AO8ID3Zi3c8s3jxcmEzd75ic5aZDYq6ErsSaIKBY/8EVr+KDDNXwZv5 bVHGd/T3R6cR1QrhXBnqEawoY6M01XywGeiIoQ1z1l9A03h+cXgMumLSGriK6mCj3H15 pxlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=rf+PEwSCAjx96OEampym4lZ49QLmyoHK2vvPo63tKDU=; b=ocgXIUNhLfLR6S232R30bnlJWQcOQQ618qMegsscJmgIcMjvDtMtUoyEUWGrLhW7xB HAZAAGti5W5TsBysF6x83zmu1BhG9oKcjBV+HE85quU0ZWAEcmNWR6oj4s/ZEtC6LY7j AN8jTz2apitKH1lffZcjaG323R/tK0LfEDKhmZ4EYBU2VFj1nLVwai4CPIOt4dyxnxU9 bMoNhWNzoPiD0zE4GzyycOCiF9+2QO4Lljvp5psteOh76/qNwUyffuF/2Pl/14ytZFf6 80ZqZKVNv9sX0eKunW8so7JLCXcUMfpbNRBdvmKVI7sngjV2zi/V4jlAy63qSV9ZXf6d 1Y3A== X-Gm-Message-State: AOAM532a4IbGgsO4ln/s12N3mebo4x7AW2IJSSvWgzz64gHUsifDp0UL pvieueFVrJyYxyr0iUbez4In3sdMwYEV X-Google-Smtp-Source: ABdhPJxAE8giXqGIDJh7Mviar4JsaaTJZtQQuDBTZF65UTbW8rTNqxt0+PJdM/jBytNMKHkW6V66FJSXIQ6n X-Received: from js-desktop.svl.corp.google.com ([2620:15c:2cd:202:ccbe:5d15:e2e6:322]) (user=junaids job=sendgmr) by 2002:a05:6902:108:b0:621:165e:5c1e with SMTP id o8-20020a056902010800b00621165e5c1emr25436069ybh.204.1645593843385; Tue, 22 Feb 2022 21:24:03 -0800 (PST) Date: Tue, 22 Feb 2022 21:21:44 -0800 In-Reply-To: <20220223052223.1202152-1-junaids@google.com> Message-Id: <20220223052223.1202152-9-junaids@google.com> Mime-Version: 1.0 References: <20220223052223.1202152-1-junaids@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [RFC PATCH 08/47] mm: asi: Add basic infrastructure for global non-sensitive mappings From: Junaid Shahid To: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, pjt@google.com, oweisse@google.com, alexandre.chartre@oracle.com, rppt@linux.ibm.com, dave.hansen@linux.intel.com, peterz@infradead.org, tglx@linutronix.de, luto@kernel.org, linux-mm@kvack.org X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ClDNkRcR; spf=pass (imf29.hostedemail.com: domain of 388QVYgcKCOoVgZMUPeSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--junaids.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=388QVYgcKCOoVgZMUPeSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--junaids.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 10C90120002 X-Stat-Signature: id4hz4r8843gna8zxi9ar5eunzqtwz3m X-HE-Tag: 1645593843-763322 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A pseudo-PGD is added to store global non-sensitive ASI mappings. Actual ASI PGDs copy entries from this pseudo-PGD during asi_init(). Memory can be mapped as globally non-sensitive by calling asi_map() with ASI_GLOBAL_NONSENSITIVE. Page tables allocated for global non-sensitive mappings are never freed. Signed-off-by: Junaid Shahid --- arch/x86/include/asm/asi.h | 12 ++++++++++++ arch/x86/mm/asi.c | 36 +++++++++++++++++++++++++++++++++++- arch/x86/mm/init_64.c | 26 +++++++++++++++++--------- arch/x86/mm/mm_internal.h | 3 +++ include/asm-generic/asi.h | 5 +++++ mm/init-mm.c | 2 ++ 6 files changed, 74 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 521b40d1864b..64c2b4d1dba2 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -15,6 +15,8 @@ #define ASI_MAX_NUM_ORDER 2 #define ASI_MAX_NUM (1 << ASI_MAX_NUM_ORDER) +#define ASI_GLOBAL_NONSENSITIVE (&init_mm.asi[0]) + struct asi_state { struct asi *curr_asi; struct asi *target_asi; @@ -41,6 +43,8 @@ struct asi { DECLARE_PER_CPU_ALIGNED(struct asi_state, asi_cpu_state); +extern pgd_t asi_global_nonsensitive_pgd[]; + void asi_init_mm_state(struct mm_struct *mm); int asi_register_class(const char *name, uint flags, @@ -117,6 +121,14 @@ static inline void asi_intr_exit(void) } } +#define INIT_MM_ASI(init_mm) \ + .asi = { \ + [0] = { \ + .pgd = asi_global_nonsensitive_pgd, \ + .mm = &init_mm \ + } \ + }, + static inline pgd_t *asi_pgd(struct asi *asi) { return asi->pgd; diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 84d220cbdcfc..d381ae573af9 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -1,11 +1,13 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include #include +#include "mm_internal.h" #include "../../../mm/internal.h" #undef pr_fmt @@ -17,6 +19,8 @@ static DEFINE_SPINLOCK(asi_class_lock); DEFINE_PER_CPU_ALIGNED(struct asi_state, asi_cpu_state); EXPORT_PER_CPU_SYMBOL_GPL(asi_cpu_state); +__aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD]; + int asi_register_class(const char *name, uint flags, const struct asi_hooks *ops) { @@ -160,12 +164,17 @@ static void asi_free_pgd_range(struct asi *asi, uint start, uint end) * Free the page tables allocated for the given ASI instance. * The caller must ensure that all the mappings have already been cleared * and appropriate TLB flushes have been issued before calling this function. + * + * For standard non-sensitive ASI classes, the page tables shared with the + * master pseudo-PGD are not freed. */ static void asi_free_pgd(struct asi *asi) { VM_BUG_ON(asi->mm == &init_mm); - asi_free_pgd_range(asi, KERNEL_PGD_BOUNDARY, PTRS_PER_PGD); + if (!(asi->class->flags & ASI_MAP_STANDARD_NONSENSITIVE)) + asi_free_pgd_range(asi, KERNEL_PGD_BOUNDARY, PTRS_PER_PGD); + free_pages((ulong)asi->pgd, PGD_ALLOCATION_ORDER); } @@ -178,6 +187,24 @@ static int __init set_asi_param(char *str) } early_param("asi", set_asi_param); +static int __init asi_global_init(void) +{ + if (!boot_cpu_has(X86_FEATURE_ASI)) + return 0; + + preallocate_toplevel_pgtbls(asi_global_nonsensitive_pgd, + PAGE_OFFSET, + PAGE_OFFSET + PFN_PHYS(max_possible_pfn) - 1, + "ASI Global Non-sensitive direct map"); + + preallocate_toplevel_pgtbls(asi_global_nonsensitive_pgd, + VMALLOC_START, VMALLOC_END, + "ASI Global Non-sensitive vmalloc"); + + return 0; +} +subsys_initcall(asi_global_init) + int asi_init(struct mm_struct *mm, int asi_index) { struct asi *asi = &mm->asi[asi_index]; @@ -202,6 +229,13 @@ int asi_init(struct mm_struct *mm, int asi_index) asi->class = &asi_class[asi_index]; asi->mm = mm; + if (asi->class->flags & ASI_MAP_STANDARD_NONSENSITIVE) { + uint i; + + for (i = KERNEL_PGD_BOUNDARY; i < PTRS_PER_PGD; i++) + set_pgd(asi->pgd + i, asi_global_nonsensitive_pgd[i]); + } + return 0; } EXPORT_SYMBOL_GPL(asi_init); diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 36098226a957..ebd512c64ed0 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1277,18 +1277,15 @@ static void __init register_page_bootmem_info(void) #endif } -/* - * Pre-allocates page-table pages for the vmalloc area in the kernel page-table. - * Only the level which needs to be synchronized between all page-tables is - * allocated because the synchronization can be expensive. - */ -static void __init preallocate_vmalloc_pages(void) +void __init preallocate_toplevel_pgtbls(pgd_t *pgd_table, + ulong start, ulong end, + const char *name) { unsigned long addr; const char *lvl; - for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { - pgd_t *pgd = pgd_offset_k(addr); + for (addr = start; addr <= end; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_pgd(pgd_table, addr); p4d_t *p4d; pud_t *pud; @@ -1324,7 +1321,18 @@ static void __init preallocate_vmalloc_pages(void) * The pages have to be there now or they will be missing in * process page-tables later. */ - panic("Failed to pre-allocate %s pages for vmalloc area\n", lvl); + panic("Failed to pre-allocate %s pages for %s area\n", lvl, name); +} + +/* + * Pre-allocates page-table pages for the vmalloc area in the kernel page-table. + * Only the level which needs to be synchronized between all page-tables is + * allocated because the synchronization can be expensive. + */ +static void __init preallocate_vmalloc_pages(void) +{ + preallocate_toplevel_pgtbls(init_mm.pgd, VMALLOC_START, VMALLOC_END, + "vmalloc"); } void __init mem_init(void) diff --git a/arch/x86/mm/mm_internal.h b/arch/x86/mm/mm_internal.h index 3f37b5c80bb3..a1e8c523ab08 100644 --- a/arch/x86/mm/mm_internal.h +++ b/arch/x86/mm/mm_internal.h @@ -19,6 +19,9 @@ unsigned long kernel_physical_mapping_change(unsigned long start, unsigned long page_size_mask); void zone_sizes_init(void); +void preallocate_toplevel_pgtbls(pgd_t *pgd_table, ulong start, ulong end, + const char *name); + extern int after_bootmem; void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache); diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index 7da91cbe075d..012691e29895 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -12,6 +12,8 @@ #define ASI_MAX_NUM_ORDER 0 #define ASI_MAX_NUM 0 +#define ASI_GLOBAL_NONSENSITIVE NULL + #ifndef _ASSEMBLY_ struct asi_hooks {}; @@ -63,8 +65,11 @@ void asi_unmap(struct asi *asi, void *addr, size_t len, bool flush_tlb) { } static inline void asi_flush_tlb_range(struct asi *asi, void *addr, size_t len) { } +#define INIT_MM_ASI(init_mm) + #define static_asi_enabled() false + #endif /* !_ASSEMBLY_ */ #endif /* !CONFIG_ADDRESS_SPACE_ISOLATION */ diff --git a/mm/init-mm.c b/mm/init-mm.c index b4a6f38fb51d..47a6a66610fb 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -11,6 +11,7 @@ #include #include #include +#include #ifndef INIT_MM_CONTEXT #define INIT_MM_CONTEXT(name) @@ -38,6 +39,7 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, + INIT_MM_ASI(init_mm) INIT_MM_CONTEXT(init_mm) };