| Message ID | 20240528095522.509667-19-kirill.shutemov@linux.intel.com (mailing list archive) |
|---|---|
| State | Handled Elsewhere, archived |
| Headers | show |
| Series | x86/tdx: Add kexec support | expand |
On Tue, May 28, 2024 at 12:55:21PM +0300, Kirill A. Shutemov wrote: > MADT Multiprocessor Wakeup structure version 1 brings support of CPU s/of /for / > offlining: BIOS provides a reset vector where the CPU has to jump to > for offlining itself. The new TEST mailbox command can be used to test > whether the CPU offlined itself which means the BIOS has control over > the CPU and can online it again via the ACPI MADT wakeup method. > > Add CPU offling support for the ACPI MADT wakeup method by implementing Unknown word [offling] in commit message. Please introduce a spellchecker into your patch creation workflow. > custom cpu_die(), play_dead() and stop_this_cpu() SMP operations. > > CPU offlining makes is possible to hand over secondary CPUs over kexec, s/is /it / > not limiting the second kernel to a single CPU. ... > +/* > + * Make sure asm_acpi_mp_play_dead() is present in the identity mapping at > + * the same place as in the kernel page tables. asm_acpi_mp_play_dead() switches > + * to the identity mapping and the function has be present at the same spot in > + * the virtual address space before and after switching page tables. > + */ > +static int __init init_transition_pgtable(pgd_t *pgd) This looks like a generic helper which should be in set_memory.c. And looking at that file, there's populate_pgd() which does pretty much the same thing, if I squint real hard. Let's tone down the duplication. > +{ > + pgprot_t prot = PAGE_KERNEL_EXEC_NOENC; > + unsigned long vaddr, paddr; > + p4d_t *p4d; > + pud_t *pud; > + pmd_t *pmd; > + pte_t *pte; > + > + vaddr = (unsigned long)asm_acpi_mp_play_dead; > + pgd += pgd_index(vaddr); > + if (!pgd_present(*pgd)) { > + p4d = (p4d_t *)alloc_pgt_page(NULL); > + if (!p4d) > + return -ENOMEM; > + set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE)); > + } > + p4d = p4d_offset(pgd, vaddr); > + if (!p4d_present(*p4d)) { > + pud = (pud_t *)alloc_pgt_page(NULL); > + if (!pud) > + return -ENOMEM; > + set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE)); > + } > + pud = pud_offset(p4d, vaddr); > + if (!pud_present(*pud)) { > + pmd = (pmd_t *)alloc_pgt_page(NULL); > + if (!pmd) > + return -ENOMEM; > + set_pud(pud, __pud(__pa(pmd) | _KERNPG_TABLE)); > + } > + pmd = pmd_offset(pud, vaddr); > + if (!pmd_present(*pmd)) { > + pte = (pte_t *)alloc_pgt_page(NULL); > + if (!pte) > + return -ENOMEM; > + set_pmd(pmd, __pmd(__pa(pte) | _KERNPG_TABLE)); > + } > + pte = pte_offset_kernel(pmd, vaddr); > + > + paddr = __pa(vaddr); > + set_pte(pte, pfn_pte(paddr >> PAGE_SHIFT, prot)); > + > + return 0; > +}
On Mon, Jun 03, 2024 at 10:39:30AM +0200, Borislav Petkov wrote: > > +/* > > + * Make sure asm_acpi_mp_play_dead() is present in the identity mapping at > > + * the same place as in the kernel page tables. asm_acpi_mp_play_dead() switches > > + * to the identity mapping and the function has be present at the same spot in > > + * the virtual address space before and after switching page tables. > > + */ > > +static int __init init_transition_pgtable(pgd_t *pgd) > > This looks like a generic helper which should be in set_memory.c. And > looking at that file, there's populate_pgd() which does pretty much the > same thing, if I squint real hard. > > Let's tone down the duplication. Okay, there is a function called kernel_map_pages_in_pgd() in set_memory.c that does what we need here. I tried to use it, but encountered a few issues: - The code in set_memory.c allocates memory using the buddy allocator, which is not yet ready. We can work around this limitation by delaying the initialization of offlining until later, using a separate early_initcall(); - I noticed a complaint that the allocation is being done from an atomic context: a spinlock called cpa_lock is taken when populate_pgd() allocates memory. I am not sure why this was not noticed before. kernel_map_pages_in_pgd() has only been used in EFI mapping initialization so far, so maybe it is somehow special, I don't know. I was able to address this issue by switching cpa_lock to a mutex. However, this solution will only work if the callers for set_memory interfaces are not called from an atomic context. I need to verify if this is the case. - The function __flush_tlb_all() in kernel_(un)map_pages_in_pgd() must be called with preemption disabled. Once again, I am unsure why this has not caused issues in the EFI case. - I discovered a bug in kernel_ident_mapping_free() when it is used on a machine with 5-level paging. I will submit a proper patch to fix this issue. The fixup is below. Any comments? diff --git a/arch/x86/kernel/acpi/madt_wakeup.c b/arch/x86/kernel/acpi/madt_wakeup.c index 6cfe762be28b..fbbfe78f7f27 100644 --- a/arch/x86/kernel/acpi/madt_wakeup.c +++ b/arch/x86/kernel/acpi/madt_wakeup.c @@ -59,82 +59,55 @@ static void acpi_mp_cpu_die(unsigned int cpu) pr_err("Failed to hand over CPU %d to BIOS\n", cpu); } +static void acpi_mp_disable_offlining(struct acpi_madt_multiproc_wakeup *mp_wake) +{ + cpu_hotplug_disable_offlining(); + + /* + * ACPI MADT doesn't allow to offline a CPU after it was onlined. This + * limits kexec: the second kernel won't be able to use more than one CPU. + * + * To prevent a kexec kernel from onlining secondary CPUs invalidate the + * mailbox address in the ACPI MADT wakeup structure which prevents a + * kexec kernel to use it. + * + * This is safe as the booting kernel has the mailbox address cached + * already and acpi_wakeup_cpu() uses the cached value to bring up the + * secondary CPUs. + * + * Note: This is a Linux specific convention and not covered by the + * ACPI specification. + */ + mp_wake->mailbox_address = 0; +} + /* The argument is required to match type of x86_mapping_info::alloc_pgt_page */ static void __init *alloc_pgt_page(void *dummy) { - return memblock_alloc(PAGE_SIZE, PAGE_SIZE); + return (void *)get_zeroed_page(GFP_KERNEL); } static void __init free_pgt_page(void *pgt, void *dummy) { - return memblock_free(pgt, PAGE_SIZE); + return free_page((unsigned long)pgt); } -/* - * Make sure asm_acpi_mp_play_dead() is present in the identity mapping at - * the same place as in the kernel page tables. asm_acpi_mp_play_dead() switches - * to the identity mapping and the function has be present at the same spot in - * the virtual address space before and after switching page tables. - */ -static int __init init_transition_pgtable(pgd_t *pgd) -{ - pgprot_t prot = PAGE_KERNEL_EXEC_NOENC; - unsigned long vaddr, paddr; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - vaddr = (unsigned long)asm_acpi_mp_play_dead; - pgd += pgd_index(vaddr); - if (!pgd_present(*pgd)) { - p4d = (p4d_t *)alloc_pgt_page(NULL); - if (!p4d) - return -ENOMEM; - set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE)); - } - p4d = p4d_offset(pgd, vaddr); - if (!p4d_present(*p4d)) { - pud = (pud_t *)alloc_pgt_page(NULL); - if (!pud) - return -ENOMEM; - set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE)); - } - pud = pud_offset(p4d, vaddr); - if (!pud_present(*pud)) { - pmd = (pmd_t *)alloc_pgt_page(NULL); - if (!pmd) - return -ENOMEM; - set_pud(pud, __pud(__pa(pmd) | _KERNPG_TABLE)); - } - pmd = pmd_offset(pud, vaddr); - if (!pmd_present(*pmd)) { - pte = (pte_t *)alloc_pgt_page(NULL); - if (!pte) - return -ENOMEM; - set_pmd(pmd, __pmd(__pa(pte) | _KERNPG_TABLE)); - } - pte = pte_offset_kernel(pmd, vaddr); - - paddr = __pa(vaddr); - set_pte(pte, pfn_pte(paddr >> PAGE_SHIFT, prot)); - - return 0; -} - -static int __init acpi_mp_setup_reset(u64 reset_vector) +static int __init acpi_mp_setup_reset(union acpi_subtable_headers *header, + const unsigned long end) { + struct acpi_madt_multiproc_wakeup *mp_wake; struct x86_mapping_info info = { .alloc_pgt_page = alloc_pgt_page, .free_pgt_page = free_pgt_page, .page_flag = __PAGE_KERNEL_LARGE_EXEC, - .kernpg_flag = _KERNPG_TABLE_NOENC, + .kernpg_flag = _KERNPG_TABLE, }; + unsigned long vaddr, pfn; pgd_t *pgd; pgd = alloc_pgt_page(NULL); if (!pgd) - return -ENOMEM; + goto err; for (int i = 0; i < nr_pfn_mapped; i++) { unsigned long mstart, mend; @@ -143,30 +116,45 @@ static int __init acpi_mp_setup_reset(u64 reset_vector) mend = pfn_mapped[i].end << PAGE_SHIFT; if (kernel_ident_mapping_init(&info, pgd, mstart, mend)) { kernel_ident_mapping_free(&info, pgd); - return -ENOMEM; + goto err; } } if (kernel_ident_mapping_init(&info, pgd, - PAGE_ALIGN_DOWN(reset_vector), - PAGE_ALIGN(reset_vector + 1))) { + PAGE_ALIGN_DOWN(acpi_mp_reset_vector_paddr), + PAGE_ALIGN(acpi_mp_reset_vector_paddr + 1))) { kernel_ident_mapping_free(&info, pgd); - return -ENOMEM; + goto err; } - if (init_transition_pgtable(pgd)) { + /* + * Make sure asm_acpi_mp_play_dead() is present in the identity mapping + * at the same place as in the kernel page tables. + * + * asm_acpi_mp_play_dead() switches to the identity mapping and the + * function has be present at the same spot in the virtual address space + * before and after switching page tables. + */ + vaddr = (unsigned long)asm_acpi_mp_play_dead; + pfn = __pa(vaddr) >> PAGE_SHIFT; + if (kernel_map_pages_in_pgd(pgd, pfn, vaddr, 1, _KERNPG_TABLE)) { kernel_ident_mapping_free(&info, pgd); - return -ENOMEM; + goto err; } smp_ops.play_dead = acpi_mp_play_dead; smp_ops.stop_this_cpu = acpi_mp_stop_this_cpu; smp_ops.cpu_die = acpi_mp_cpu_die; - acpi_mp_reset_vector_paddr = reset_vector; acpi_mp_pgd = __pa(pgd); return 0; +err: + pr_warn("Failed to setup MADT reset vector\n"); + mp_wake = (struct acpi_madt_multiproc_wakeup *)header; + acpi_mp_disable_offlining(mp_wake); + return -ENOMEM; + } static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip) @@ -226,28 +214,6 @@ static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip) return 0; } -static void acpi_mp_disable_offlining(struct acpi_madt_multiproc_wakeup *mp_wake) -{ - cpu_hotplug_disable_offlining(); - - /* - * ACPI MADT doesn't allow to offline a CPU after it was onlined. This - * limits kexec: the second kernel won't be able to use more than one CPU. - * - * To prevent a kexec kernel from onlining secondary CPUs invalidate the - * mailbox address in the ACPI MADT wakeup structure which prevents a - * kexec kernel to use it. - * - * This is safe as the booting kernel has the mailbox address cached - * already and acpi_wakeup_cpu() uses the cached value to bring up the - * secondary CPUs. - * - * Note: This is a Linux specific convention and not covered by the - * ACPI specification. - */ - mp_wake->mailbox_address = 0; -} - int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, const unsigned long end) { @@ -274,10 +240,7 @@ int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, if (mp_wake->version >= ACPI_MADT_MP_WAKEUP_VERSION_V1 && mp_wake->header.length >= ACPI_MADT_MP_WAKEUP_SIZE_V1) { - if (acpi_mp_setup_reset(mp_wake->reset_vector)) { - pr_warn("Failed to setup MADT reset vector\n"); - acpi_mp_disable_offlining(mp_wake); - } + acpi_mp_reset_vector_paddr = mp_wake->reset_vector; } else { /* * CPU offlining requires version 1 of the ACPI MADT wakeup @@ -290,3 +253,13 @@ int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, return 0; } + +static int __init acpi_mp_offline_init(void) +{ + if (!acpi_mp_reset_vector_paddr) + return 0; + + return acpi_table_parse_madt(ACPI_MADT_TYPE_MULTIPROC_WAKEUP, + acpi_mp_setup_reset, 1); +} +early_initcall(acpi_mp_offline_init); diff --git a/arch/x86/mm/ident_map.c b/arch/x86/mm/ident_map.c index 3996af7b4abf..c45127265f2f 100644 --- a/arch/x86/mm/ident_map.c +++ b/arch/x86/mm/ident_map.c @@ -60,7 +60,7 @@ static void free_p4d(struct x86_mapping_info *info, pgd_t *pgd) } if (pgtable_l5_enabled()) - info->free_pgt_page(pgd, info->context); + info->free_pgt_page(p4d, info->context); } void kernel_ident_mapping_free(struct x86_mapping_info *info, pgd_t *pgd) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 443a97e515c0..72715674f492 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -69,7 +69,7 @@ static const int cpa_warn_level = CPA_PROTECT; * entries change the page attribute in parallel to some other cpu * splitting a large page entry along with changing the attribute. */ -static DEFINE_SPINLOCK(cpa_lock); +static DEFINE_MUTEX(cpa_lock); #define CPA_FLUSHTLB 1 #define CPA_ARRAY 2 @@ -1186,10 +1186,10 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, struct page *base; if (!debug_pagealloc_enabled()) - spin_unlock(&cpa_lock); + mutex_unlock(&cpa_lock); base = alloc_pages(GFP_KERNEL, 0); if (!debug_pagealloc_enabled()) - spin_lock(&cpa_lock); + mutex_lock(&cpa_lock); if (!base) return -ENOMEM; @@ -1804,10 +1804,10 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary) cpa->numpages = 1; if (!debug_pagealloc_enabled()) - spin_lock(&cpa_lock); + mutex_lock(&cpa_lock); ret = __change_page_attr(cpa, primary); if (!debug_pagealloc_enabled()) - spin_unlock(&cpa_lock); + mutex_unlock(&cpa_lock); if (ret) goto out; @@ -2516,7 +2516,9 @@ int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, cpa.mask_set = __pgprot(_PAGE_PRESENT | page_flags); retval = __change_page_attr_set_clr(&cpa, 1); + preempt_disable(); __flush_tlb_all(); + preempt_enable(); out: return retval; @@ -2551,7 +2553,9 @@ int __init kernel_unmap_pages_in_pgd(pgd_t *pgd, unsigned long address, WARN_ONCE(num_online_cpus() > 1, "Don't call after initializing SMP"); retval = __change_page_attr_set_clr(&cpa, 1); + preempt_disable(); __flush_tlb_all(); + preempt_enable(); return retval; }
On Fri, Jun 07, 2024 at 06:14:28PM +0300, Kirill A. Shutemov wrote: > I was able to address this issue by switching cpa_lock to a mutex. > However, this solution will only work if the callers for set_memory > interfaces are not called from an atomic context. I need to verify if > this is the case. Dunno, I'd be nervous about this. Althouth from looking at ad5ca55f6bdb ("x86, cpa: srlz cpa(), global flush tlb after splitting big page and before doing cpa") I don't see how "So that we don't allow any other cpu" can't be done with a mutex. Perhaps the set_memory* interfaces should be usable in as many contexts as possible. Have you run this with lockdep enabled? > - The function __flush_tlb_all() in kernel_(un)map_pages_in_pgd() must be > called with preemption disabled. Once again, I am unsure why this has > not caused issues in the EFI case. It could be because EFI does all that setup on the BSP only before the others have arrived but I don't remember anymore... It is more than a decade ago when I did this... Thx.
On Mon, Jun 10, 2024 at 03:40:20PM +0200, Borislav Petkov wrote: > On Fri, Jun 07, 2024 at 06:14:28PM +0300, Kirill A. Shutemov wrote: > > I was able to address this issue by switching cpa_lock to a mutex. > > However, this solution will only work if the callers for set_memory > > interfaces are not called from an atomic context. I need to verify if > > this is the case. > > Dunno, I'd be nervous about this. Althouth from looking at > > ad5ca55f6bdb ("x86, cpa: srlz cpa(), global flush tlb after splitting big page and before doing cpa") > > I don't see how "So that we don't allow any other cpu" can't be done > with a mutex. Perhaps the set_memory* interfaces should be usable in as > many contexts as possible. > > Have you run this with lockdep enabled? Yes, it booted to the shell just fine. However, that doesn't prove anything. The set_memory_* function has many obscured cases. > > - The function __flush_tlb_all() in kernel_(un)map_pages_in_pgd() must be > > called with preemption disabled. Once again, I am unsure why this has > > not caused issues in the EFI case. > > It could be because EFI does all that setup on the BSP only before the > others have arrived but I don't remember anymore... It is more than > a decade ago when I did this... Are you okay with this? Disabling preemption looks strange, but I don't see a better option.
On Mon, Jun 10, 2024 at 05:01:55PM +0300, Kirill A. Shutemov wrote: > On Mon, Jun 10, 2024 at 03:40:20PM +0200, Borislav Petkov wrote: > > On Fri, Jun 07, 2024 at 06:14:28PM +0300, Kirill A. Shutemov wrote: > > > I was able to address this issue by switching cpa_lock to a mutex. > > > However, this solution will only work if the callers for set_memory > > > interfaces are not called from an atomic context. I need to verify if > > > this is the case. > > > > Dunno, I'd be nervous about this. Althouth from looking at > > > > ad5ca55f6bdb ("x86, cpa: srlz cpa(), global flush tlb after splitting big page and before doing cpa") > > > > I don't see how "So that we don't allow any other cpu" can't be done > > with a mutex. Perhaps the set_memory* interfaces should be usable in as > > many contexts as possible. > > > > Have you run this with lockdep enabled? > > Yes, it booted to the shell just fine. However, that doesn't prove > anything. The set_memory_* function has many obscured cases. > > > > - The function __flush_tlb_all() in kernel_(un)map_pages_in_pgd() must be > > > called with preemption disabled. Once again, I am unsure why this has > > > not caused issues in the EFI case. > > > > It could be because EFI does all that setup on the BSP only before the > > others have arrived but I don't remember anymore... It is more than > > a decade ago when I did this... > > Are you okay with this? Disabling preemption looks strange, but I don't > see a better option. Borislav, given this code deduplication effort is not trivial, maybe we can do it as a separate patchset on top of this one? I also wounder if it makes sense to combine ident_map.c and set_memory.c. There's some overlap between the two.
On Tue, Jun 11, 2024 at 06:47:05PM +0300, Kirill A. Shutemov wrote: > Borislav, given this code deduplication effort is not trivial, maybe we > can do it as a separate patchset on top of this one? Sure, as long as it gets done and doesn't get delayed indefinitely by new and more important features enablement. Usually, we do unifications and cleanups first - then new features but this kexec pile has been long in the making already... > I also wounder if it makes sense to combine ident_map.c and > set_memory.c. There's some overlap between the two. Yeah, we have a bunch of different pagetable manipulating things, all with their peculiarities and unifying them and having a good set of APIs which everything else uses, is always a good thing. And since we're talking cleanups, there's another thing I've been looking at critically: CONFIG_X86_5LEVEL. Maybe it is time to get rid of it and make the 5level stuff unconditional. And get rid of a bunch of code since both vendors support 5level now... Thx.
On Tue, Jun 11, 2024 at 09:46:53PM +0200, Borislav Petkov wrote: > On Tue, Jun 11, 2024 at 06:47:05PM +0300, Kirill A. Shutemov wrote: > > Borislav, given this code deduplication effort is not trivial, maybe we > > can do it as a separate patchset on top of this one? > > Sure, as long as it gets done and doesn't get delayed indefinitely by > new and more important features enablement. I will try to deliver it in timely manner. > Usually, we do unifications and cleanups first - then new features but > this kexec pile has been long in the making already... > > > I also wounder if it makes sense to combine ident_map.c and > > set_memory.c. There's some overlap between the two. > > Yeah, we have a bunch of different pagetable manipulating things, all > with their peculiarities and unifying them and having a good set of APIs > which everything else uses, is always a good thing. Will give it a try. > And since we're talking cleanups, there's another thing I've been > looking at critically: CONFIG_X86_5LEVEL. Maybe it is time to get rid of > it and make the 5level stuff unconditional. And get rid of a bunch of > code since both vendors support 5level now... Can do.
On Wed, Jun 12, 2024 at 12:24:30PM +0300, Kirill A. Shutemov wrote: > I will try to deliver it in timely manner. :-P > > Yeah, we have a bunch of different pagetable manipulating things, all > > with their peculiarities and unifying them and having a good set of APIs > > which everything else uses, is always a good thing. > > Will give it a try. > > > And since we're talking cleanups, there's another thing I've been > > looking at critically: CONFIG_X86_5LEVEL. Maybe it is time to get rid of > > it and make the 5level stuff unconditional. And get rid of a bunch of > > code since both vendors support 5level now... > > Can do. Much appreciated, thanks!
On Wed, Jun 12, 2024 at 11:29:43AM +0200, Borislav Petkov wrote: > > > And since we're talking cleanups, there's another thing I've been > > > looking at critically: CONFIG_X86_5LEVEL. Maybe it is time to get rid of > > > it and make the 5level stuff unconditional. And get rid of a bunch of > > > code since both vendors support 5level now... > > > > Can do. > > Much appreciated, thanks! It is easy enough to do. See the patch below. But I am not sure if I can justify it properly. If someone doesn't really need 5-level paging, disabling it at compile-time would save ~34K of kernel code with the configuration. Is it worth saving ~100 lines of code? Documentation/arch/x86/cpuinfo.rst | 8 +++----- Documentation/arch/x86/x86_64/5level-paging.rst | 9 --------- arch/x86/Kconfig | 24 +----------------------- arch/x86/boot/compressed/pgtable_64.c | 10 +++------- arch/x86/boot/header.S | 4 ---- arch/x86/include/asm/disabled-features.h | 9 +-------- arch/x86/include/asm/page_64.h | 2 -- arch/x86/include/asm/page_64_types.h | 7 ------- arch/x86/include/asm/pgtable_64_types.h | 18 ------------------ arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/head64.c | 5 ----- arch/x86/kernel/head_64.S | 2 -- arch/x86/mm/init.c | 4 ---- arch/x86/mm/pgtable.c | 2 -- drivers/firmware/efi/libstub/x86-5lvl.c | 2 +- tools/arch/x86/include/asm/disabled-features.h | 9 +-------- 16 files changed, 11 insertions(+), 106 deletions(-) diff --git a/Documentation/arch/x86/cpuinfo.rst b/Documentation/arch/x86/cpuinfo.rst index 8895784d4784..0ea70924c89e 100644 --- a/Documentation/arch/x86/cpuinfo.rst +++ b/Documentation/arch/x86/cpuinfo.rst @@ -171,10 +171,10 @@ For example, when an old kernel is running on new hardware. c: The kernel disabled support for it at compile-time. ------------------------------------------------------ -For example, if 5-level-paging is not enabled when building (i.e., -CONFIG_X86_5LEVEL is not selected) the flag "la57" will not show up [#f1]_. +For example, if Linear Address Masking (LAM) is not enabled when building (i.e., +CONFIG_ADDRESS_MASKING is not selected) the flag "lam" will not show up. Even though the feature will still be detected via CPUID, the kernel disables -it by clearing via setup_clear_cpu_cap(X86_FEATURE_LA57). +it by clearing via setup_clear_cpu_cap(X86_FEATURE_LAM). d: The feature is disabled at boot-time. ---------------------------------------- @@ -197,5 +197,3 @@ missing at runtime. For example, AVX flags will not show up if XSAVE feature is disabled since they depend on XSAVE feature. Another example would be broken CPUs and them missing microcode patches. Due to that, the kernel decides not to enable a feature. - -.. [#f1] 5-level paging uses linear address of 57 bits. diff --git a/Documentation/arch/x86/x86_64/5level-paging.rst b/Documentation/arch/x86/x86_64/5level-paging.rst index 71f882f4a173..ad7ddc13f79d 100644 --- a/Documentation/arch/x86/x86_64/5level-paging.rst +++ b/Documentation/arch/x86/x86_64/5level-paging.rst @@ -22,15 +22,6 @@ QEMU 2.9 and later support 5-level paging. Virtual memory layout for 5-level paging is described in Documentation/arch/x86/x86_64/mm.rst - -Enabling 5-level paging -======================= -CONFIG_X86_5LEVEL=y enables the feature. - -Kernel with CONFIG_X86_5LEVEL=y still able to boot on 4-level hardware. -In this case additional page table level -- p4d -- will be folded at -runtime. - User-space and large virtual address space ========================================== On x86, 5-level paging enables 56-bit userspace virtual address space. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index e8837116704c..c62827c2ecea 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -408,8 +408,7 @@ config DYNAMIC_PHYSICAL_MASK config PGTABLE_LEVELS int - default 5 if X86_5LEVEL - default 4 if X86_64 + default 5 if X86_64 default 3 if X86_PAE default 2 @@ -1491,27 +1490,6 @@ config X86_PAE has the cost of more pagetable lookup overhead, and also consumes more pagetable space per process. -config X86_5LEVEL - bool "Enable 5-level page tables support" - default y - select DYNAMIC_MEMORY_LAYOUT - select SPARSEMEM_VMEMMAP - depends on X86_64 - help - 5-level paging enables access to larger address space: - up to 128 PiB of virtual address space and 4 PiB of - physical address space. - - It will be supported by future Intel CPUs. - - A kernel with the option enabled can be booted on machines that - support 4- or 5-level paging. - - See Documentation/arch/x86/x86_64/5level-paging.rst for more - information. - - Say N if unsure. - config X86_DIRECT_GBPAGES def_bool y depends on X86_64 diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index c882e1f67af0..f9b77b66c792 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -10,12 +10,10 @@ #define BIOS_START_MIN 0x20000U /* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U /* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL /* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ unsigned int __section(".data") __pgtable_l5_enabled; unsigned int __section(".data") pgdir_shift = 39; unsigned int __section(".data") ptrs_per_p4d = 1; -#endif /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -113,7 +111,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) * Check if LA57 is desired and supported. * * There are several parts to the check: - * - if the kernel supports 5-level paging: CONFIG_X86_5LEVEL=y * - if user asked to disable 5-level paging: no5lvl in cmdline * - if the machine supports 5-level paging: * + CPUID leaf 7 is supported @@ -121,10 +118,9 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) * * That's substitute for boot_cpu_has() in early boot code. */ - if (IS_ENABLED(CONFIG_X86_5LEVEL) && - !cmdline_find_option_bool("no5lvl") && - native_cpuid_eax(0) >= 7 && - (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { + if (!cmdline_find_option_bool("no5lvl") && + native_cpuid_eax(0) >= 7 && + (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; /* Initialize variables for 5-level paging */ diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S index b5c79f43359b..32361cef909e 100644 --- a/arch/x86/boot/header.S +++ b/arch/x86/boot/header.S @@ -361,12 +361,8 @@ xloadflags: #endif #ifdef CONFIG_X86_64 -#ifdef CONFIG_X86_5LEVEL #define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED) #else -#define XLF56 XLF_5LEVEL -#endif -#else #define XLF56 0 #endif diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c492bdc97b05..19cf1678fcaa 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -38,12 +38,6 @@ # define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31)) #endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ -#ifdef CONFIG_X86_5LEVEL -# define DISABLE_LA57 0 -#else -# define DISABLE_LA57 (1<<(X86_FEATURE_LA57 & 31)) -#endif - #ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION # define DISABLE_PTI 0 #else @@ -149,8 +143,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_UMIP|DISABLE_ENQCMD) #define DISABLED_MASK17 0 #define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK19 (DISABLE_SEV_SNP) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index cc6b8e087192..3b8cb6a8b122 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -60,7 +60,6 @@ static inline void clear_page(void *page) void copy_page(void *to, void *from); -#ifdef CONFIG_X86_5LEVEL /* * User space process size. This is the first address outside the user range. * There are a few constraints that determine this: @@ -91,7 +90,6 @@ static __always_inline unsigned long task_size_max(void) return ret; } -#endif /* CONFIG_X86_5LEVEL */ #endif /* !__ASSEMBLY__ */ diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h index 06ef25411d62..714e88a72c9f 100644 --- a/arch/x86/include/asm/page_64_types.h +++ b/arch/x86/include/asm/page_64_types.h @@ -52,14 +52,7 @@ /* See Documentation/arch/x86/x86_64/mm.rst for a description of the memory map. */ #define __PHYSICAL_MASK_SHIFT 52 - -#ifdef CONFIG_X86_5LEVEL #define __VIRTUAL_MASK_SHIFT (pgtable_l5_enabled() ? 56 : 47) -/* See task_size_max() in <asm/page_64.h> */ -#else -#define __VIRTUAL_MASK_SHIFT 47 -#define task_size_max() ((_AC(1,UL) << __VIRTUAL_MASK_SHIFT) - PAGE_SIZE) -#endif #define TASK_SIZE_MAX task_size_max() #define DEFAULT_MAP_WINDOW ((1UL << 47) - PAGE_SIZE) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 9053dfe9fa03..576aea58b0c0 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -23,7 +23,6 @@ typedef struct { pmdval_t pmd; } pmd_t; extern unsigned int __pgtable_l5_enabled; -#ifdef CONFIG_X86_5LEVEL #ifdef USE_EARLY_PGTABLE_L5 /* * cpu_feature_enabled() is not available in early boot code. @@ -37,10 +36,6 @@ static inline bool pgtable_l5_enabled(void) #define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) #endif /* USE_EARLY_PGTABLE_L5 */ -#else -#define pgtable_l5_enabled() 0 -#endif /* CONFIG_X86_5LEVEL */ - extern unsigned int pgdir_shift; extern unsigned int ptrs_per_p4d; @@ -48,8 +43,6 @@ extern unsigned int ptrs_per_p4d; #define SHARED_KERNEL_PMD 0 -#ifdef CONFIG_X86_5LEVEL - /* * PGDIR_SHIFT determines what a top-level page table entry can map */ @@ -67,17 +60,6 @@ extern unsigned int ptrs_per_p4d; #define MAX_POSSIBLE_PHYSMEM_BITS 52 -#else /* CONFIG_X86_5LEVEL */ - -/* - * PGDIR_SHIFT determines what a top-level page table entry can map - */ -#define PGDIR_SHIFT 39 -#define PTRS_PER_PGD 512 -#define MAX_PTRS_PER_P4D 1 - -#endif /* CONFIG_X86_5LEVEL */ - /* * 3rd level page */ diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 37596a417094..f1c519abb925 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -457,7 +457,7 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start, DPRINTK(ALT, "alt table %px, -> %px", start, end); /* - * In the case CONFIG_X86_5LEVEL=y, KASAN_SHADOW_START is defined using + * KASAN_SHADOW_START is defined using * cpu_feature_enabled(X86_FEATURE_LA57) and is therefore patched here. * During the process, KASAN becomes confused seeing partial LA57 * conversion and triggers a false-positive out-of-bound report. diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index a817ed0724d1..df19bdea1c86 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -52,13 +52,11 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL unsigned int __pgtable_l5_enabled __ro_after_init; unsigned int pgdir_shift __ro_after_init = 39; EXPORT_SYMBOL(pgdir_shift); unsigned int ptrs_per_p4d __ro_after_init = 1; EXPORT_SYMBOL(ptrs_per_p4d); -#endif #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init = __PAGE_OFFSET_BASE_L4; @@ -71,9 +69,6 @@ EXPORT_SYMBOL(vmemmap_base); static inline bool check_la57_support(void) { - if (!IS_ENABLED(CONFIG_X86_5LEVEL)) - return false; - /* * 5-level paging is detected and enabled at kernel decompression * stage. Only check if it has been enabled there. diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 330922b328bf..4b2b2138c163 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -659,12 +659,10 @@ SYM_DATA_START_PTI_ALIGNED(init_top_pgt) SYM_DATA_END(init_top_pgt) #endif -#ifdef CONFIG_X86_5LEVEL SYM_DATA_START_PAGE_ALIGNED(level4_kernel_pgt) .fill 511,8,0 .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC SYM_DATA_END(level4_kernel_pgt) -#endif SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt) .fill L3_START_KERNEL,8,0 diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index eb503f53c319..5a980a452f4c 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -173,11 +173,7 @@ __ref void *alloc_low_pages(unsigned int num) * randomization is enabled. */ -#ifndef CONFIG_X86_5LEVEL -#define INIT_PGD_PAGE_TABLES 3 -#else #define INIT_PGD_PAGE_TABLES 4 -#endif #ifndef CONFIG_RANDOMIZE_MEMORY #define INIT_PGD_PAGE_COUNT (2 * INIT_PGD_PAGE_TABLES) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 93e54ba91fbf..982775ef8b34 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -691,7 +691,6 @@ void native_set_fixmap(unsigned /* enum fixed_addresses */ idx, } #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -#ifdef CONFIG_X86_5LEVEL /** * p4d_set_huge - setup kernel P4D mapping * @@ -710,7 +709,6 @@ int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot) void p4d_clear_huge(p4d_t *p4d) { } -#endif /** * pud_set_huge - setup kernel PUD mapping diff --git a/drivers/firmware/efi/libstub/x86-5lvl.c b/drivers/firmware/efi/libstub/x86-5lvl.c index 77359e802181..f1c5fb45d5f7 100644 --- a/drivers/firmware/efi/libstub/x86-5lvl.c +++ b/drivers/firmware/efi/libstub/x86-5lvl.c @@ -62,7 +62,7 @@ efi_status_t efi_setup_5level_paging(void) void efi_5level_switch(void) { - bool want_la57 = IS_ENABLED(CONFIG_X86_5LEVEL) && !efi_no5lvl; + bool want_la57 = !efi_no5lvl; bool have_la57 = native_read_cr4() & X86_CR4_LA57; bool need_toggle = want_la57 ^ have_la57; u64 *pgt = (void *)la57_toggle + PAGE_SIZE; diff --git a/tools/arch/x86/include/asm/disabled-features.h b/tools/arch/x86/include/asm/disabled-features.h index c492bdc97b05..19cf1678fcaa 100644 --- a/tools/arch/x86/include/asm/disabled-features.h +++ b/tools/arch/x86/include/asm/disabled-features.h @@ -38,12 +38,6 @@ # define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31)) #endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ -#ifdef CONFIG_X86_5LEVEL -# define DISABLE_LA57 0 -#else -# define DISABLE_LA57 (1<<(X86_FEATURE_LA57 & 31)) -#endif - #ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION # define DISABLE_PTI 0 #else @@ -149,8 +143,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_UMIP|DISABLE_ENQCMD) #define DISABLED_MASK17 0 #define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK19 (DISABLE_SEV_SNP)
On Thu, Jun 13, 2024 at 04:41:00PM +0300, Kirill A. Shutemov wrote: > It is easy enough to do. See the patch below. Thanks, will have a look. > But I am not sure if I can justify it properly. If someone doesn't really > need 5-level paging, disabling it at compile-time would save ~34K of > kernel code with the configuration. > > Is it worth saving ~100 lines of code? Well, it goes both ways: is it worth saving ~34K kernel text and for that make the code a lot less conditional, more readable, contain less ugly ifdeffery, ...?
On 6/13/24 09:56, Borislav Petkov wrote: > On Thu, Jun 13, 2024 at 04:41:00PM +0300, Kirill A. Shutemov wrote: >> It is easy enough to do. See the patch below. > > Thanks, will have a look. > >> But I am not sure if I can justify it properly. If someone doesn't really >> need 5-level paging, disabling it at compile-time would save ~34K of >> kernel code with the configuration. >> >> Is it worth saving ~100 lines of code? > > Well, it goes both ways: is it worth saving ~34K kernel text and for that make > the code a lot less conditional, more readable, contain less ugly ifdeffery, Won't getting rid of the config option cause 5-level to be used by default on all platforms that support it? The no5lvl command line option would have to be used to get 4-level paging at that point. Thanks, Tom > ...? >
On Fri, Jun 14, 2024 at 09:06:30AM -0500, Tom Lendacky wrote: > On 6/13/24 09:56, Borislav Petkov wrote: > > On Thu, Jun 13, 2024 at 04:41:00PM +0300, Kirill A. Shutemov wrote: > > > It is easy enough to do. See the patch below. > > > > Thanks, will have a look. > > > > > But I am not sure if I can justify it properly. If someone doesn't really > > > need 5-level paging, disabling it at compile-time would save ~34K of > > > kernel code with the configuration. > > > > > > Is it worth saving ~100 lines of code? > > > > Well, it goes both ways: is it worth saving ~34K kernel text and for that make > > the code a lot less conditional, more readable, contain less ugly ifdeffery, > > Won't getting rid of the config option cause 5-level to be used by default > on all platforms that support it? The no5lvl command line option would have > to be used to get 4-level paging at that point. Yes, there won't be compile-time option to disable 5-level paging. Is it a problem? We benchmarked it back when 5-level paging got introduced and were not able to see a measurable difference between 4- and 5-level paging on the same machine. There's some memory overhead on more page table, but it shouldn't be a show stopper. I would prefer to get 5-level paging enabled if the machine supports it. "no5lvl" cmdline option can be useful for debug or if your workload is somehow special.
On Thu, Jun 13, 2024 at 04:41:00PM +0300, Kirill A. Shutemov wrote: > Documentation/arch/x86/cpuinfo.rst | 8 +++----- > Documentation/arch/x86/x86_64/5level-paging.rst | 9 --------- > arch/x86/Kconfig | 24 +----------------------- > arch/x86/boot/compressed/pgtable_64.c | 10 +++------- > arch/x86/boot/header.S | 4 ---- > arch/x86/include/asm/disabled-features.h | 9 +-------- > arch/x86/include/asm/page_64.h | 2 -- > arch/x86/include/asm/page_64_types.h | 7 ------- > arch/x86/include/asm/pgtable_64_types.h | 18 ------------------ > arch/x86/kernel/alternative.c | 2 +- > arch/x86/kernel/head64.c | 5 ----- > arch/x86/kernel/head_64.S | 2 -- > arch/x86/mm/init.c | 4 ---- > arch/x86/mm/pgtable.c | 2 -- > drivers/firmware/efi/libstub/x86-5lvl.c | 2 +- > tools/arch/x86/include/asm/disabled-features.h | 9 +-------- > 16 files changed, 11 insertions(+), 106 deletions(-) This causes ld: vmlinux.o: in function `rip_rel_ptr': /home/boris/kernel/5th/linux/./arch/x86/include/asm/asm.h:120:(.head.text+0xb96): undefined reference to `page_offset_base' ld: /home/boris/kernel/5th/linux/./arch/x86/include/asm/asm.h:120:(.head.text+0xbaa): undefined reference to `vmalloc_base' ld: /home/boris/kernel/5th/linux/./arch/x86/include/asm/asm.h:120:(.head.text+0xbb4): undefined reference to `vmemmap_base' make[2]: *** [scripts/Makefile.vmlinux:34: vmlinux] Error 1 make[1]: *** [/mnt/kernel/kernel/5th/linux/Makefile:1171: vmlinux] Error 2 make[1]: *** Waiting for unfinished jobs.... make: *** [Makefile:240: __sub-make] Error 2 with my .config. Attached. Also: diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index f9b77b66c792..25559a788aad 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -115,12 +115,10 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) * - if the machine supports 5-level paging: * + CPUID leaf 7 is supported * + the leaf has the feature bit set - * - * That's substitute for boot_cpu_has() in early boot code. */ if (!cmdline_find_option_bool("no5lvl") && native_cpuid_eax(0) >= 7 && - (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { + (native_cpuid_ecx(7) & BIT_UL(16))) { l5_required = true; /* Initialize variables for 5-level paging */ We can simply check CPUID and be done with it, that early. Other than that, I like it. Let's do it. Less ifdeffery, less conditionals. A win-win thing. Thx.
diff --git a/arch/x86/include/asm/acpi.h b/arch/x86/include/asm/acpi.h index ceacac2b335d..21bc53f5ed0c 100644 --- a/arch/x86/include/asm/acpi.h +++ b/arch/x86/include/asm/acpi.h @@ -83,6 +83,8 @@ union acpi_subtable_headers; int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, const unsigned long end); +void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa); + /* * Check if the CPU can handle C2 and deeper */ diff --git a/arch/x86/kernel/acpi/Makefile b/arch/x86/kernel/acpi/Makefile index 2feba7257665..842a5f449404 100644 --- a/arch/x86/kernel/acpi/Makefile +++ b/arch/x86/kernel/acpi/Makefile @@ -4,7 +4,7 @@ obj-$(CONFIG_ACPI) += boot.o obj-$(CONFIG_ACPI_SLEEP) += sleep.o wakeup_$(BITS).o obj-$(CONFIG_ACPI_APEI) += apei.o obj-$(CONFIG_ACPI_CPPC_LIB) += cppc.o -obj-$(CONFIG_ACPI_MADT_WAKEUP) += madt_wakeup.o +obj-$(CONFIG_ACPI_MADT_WAKEUP) += madt_wakeup.o madt_playdead.o ifneq ($(CONFIG_ACPI_PROCESSOR),) obj-y += cstate.o diff --git a/arch/x86/kernel/acpi/madt_playdead.S b/arch/x86/kernel/acpi/madt_playdead.S new file mode 100644 index 000000000000..4e498d28cdc8 --- /dev/null +++ b/arch/x86/kernel/acpi/madt_playdead.S @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/linkage.h> +#include <asm/nospec-branch.h> +#include <asm/page_types.h> +#include <asm/processor-flags.h> + + .text + .align PAGE_SIZE + +/* + * asm_acpi_mp_play_dead() - Hand over control of the CPU to the BIOS + * + * rdi: Address of the ACPI MADT MPWK ResetVector + * rsi: PGD of the identity mapping + */ +SYM_FUNC_START(asm_acpi_mp_play_dead) + /* Turn off global entries. Following CR3 write will flush them. */ + movq %cr4, %rdx + andq $~(X86_CR4_PGE), %rdx + movq %rdx, %cr4 + + /* Switch to identity mapping */ + movq %rsi, %cr3 + + /* Jump to reset vector */ + ANNOTATE_RETPOLINE_SAFE + jmp *%rdi +SYM_FUNC_END(asm_acpi_mp_play_dead) diff --git a/arch/x86/kernel/acpi/madt_wakeup.c b/arch/x86/kernel/acpi/madt_wakeup.c index 30820f9de5af..6cfe762be28b 100644 --- a/arch/x86/kernel/acpi/madt_wakeup.c +++ b/arch/x86/kernel/acpi/madt_wakeup.c @@ -1,10 +1,19 @@ // SPDX-License-Identifier: GPL-2.0-or-later #include <linux/acpi.h> #include <linux/cpu.h> +#include <linux/delay.h> #include <linux/io.h> +#include <linux/kexec.h> +#include <linux/memblock.h> +#include <linux/pgtable.h> +#include <linux/sched/hotplug.h> #include <asm/apic.h> #include <asm/barrier.h> +#include <asm/init.h> +#include <asm/intel_pt.h> +#include <asm/nmi.h> #include <asm/processor.h> +#include <asm/reboot.h> /* Physical address of the Multiprocessor Wakeup Structure mailbox */ static u64 acpi_mp_wake_mailbox_paddr __ro_after_init; @@ -12,6 +21,154 @@ static u64 acpi_mp_wake_mailbox_paddr __ro_after_init; /* Virtual address of the Multiprocessor Wakeup Structure mailbox */ static struct acpi_madt_multiproc_wakeup_mailbox *acpi_mp_wake_mailbox __ro_after_init; +static u64 acpi_mp_pgd __ro_after_init; +static u64 acpi_mp_reset_vector_paddr __ro_after_init; + +static void acpi_mp_stop_this_cpu(void) +{ + asm_acpi_mp_play_dead(acpi_mp_reset_vector_paddr, acpi_mp_pgd); +} + +static void acpi_mp_play_dead(void) +{ + play_dead_common(); + asm_acpi_mp_play_dead(acpi_mp_reset_vector_paddr, acpi_mp_pgd); +} + +static void acpi_mp_cpu_die(unsigned int cpu) +{ + u32 apicid = per_cpu(x86_cpu_to_apicid, cpu); + unsigned long timeout; + + /* + * Use TEST mailbox command to prove that BIOS got control over + * the CPU before declaring it dead. + * + * BIOS has to clear 'command' field of the mailbox. + */ + acpi_mp_wake_mailbox->apic_id = apicid; + smp_store_release(&acpi_mp_wake_mailbox->command, + ACPI_MP_WAKE_COMMAND_TEST); + + /* Don't wait longer than a second. */ + timeout = USEC_PER_SEC; + while (READ_ONCE(acpi_mp_wake_mailbox->command) && --timeout) + udelay(1); + + if (!timeout) + pr_err("Failed to hand over CPU %d to BIOS\n", cpu); +} + +/* The argument is required to match type of x86_mapping_info::alloc_pgt_page */ +static void __init *alloc_pgt_page(void *dummy) +{ + return memblock_alloc(PAGE_SIZE, PAGE_SIZE); +} + +static void __init free_pgt_page(void *pgt, void *dummy) +{ + return memblock_free(pgt, PAGE_SIZE); +} + +/* + * Make sure asm_acpi_mp_play_dead() is present in the identity mapping at + * the same place as in the kernel page tables. asm_acpi_mp_play_dead() switches + * to the identity mapping and the function has be present at the same spot in + * the virtual address space before and after switching page tables. + */ +static int __init init_transition_pgtable(pgd_t *pgd) +{ + pgprot_t prot = PAGE_KERNEL_EXEC_NOENC; + unsigned long vaddr, paddr; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + vaddr = (unsigned long)asm_acpi_mp_play_dead; + pgd += pgd_index(vaddr); + if (!pgd_present(*pgd)) { + p4d = (p4d_t *)alloc_pgt_page(NULL); + if (!p4d) + return -ENOMEM; + set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE)); + } + p4d = p4d_offset(pgd, vaddr); + if (!p4d_present(*p4d)) { + pud = (pud_t *)alloc_pgt_page(NULL); + if (!pud) + return -ENOMEM; + set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE)); + } + pud = pud_offset(p4d, vaddr); + if (!pud_present(*pud)) { + pmd = (pmd_t *)alloc_pgt_page(NULL); + if (!pmd) + return -ENOMEM; + set_pud(pud, __pud(__pa(pmd) | _KERNPG_TABLE)); + } + pmd = pmd_offset(pud, vaddr); + if (!pmd_present(*pmd)) { + pte = (pte_t *)alloc_pgt_page(NULL); + if (!pte) + return -ENOMEM; + set_pmd(pmd, __pmd(__pa(pte) | _KERNPG_TABLE)); + } + pte = pte_offset_kernel(pmd, vaddr); + + paddr = __pa(vaddr); + set_pte(pte, pfn_pte(paddr >> PAGE_SHIFT, prot)); + + return 0; +} + +static int __init acpi_mp_setup_reset(u64 reset_vector) +{ + struct x86_mapping_info info = { + .alloc_pgt_page = alloc_pgt_page, + .free_pgt_page = free_pgt_page, + .page_flag = __PAGE_KERNEL_LARGE_EXEC, + .kernpg_flag = _KERNPG_TABLE_NOENC, + }; + pgd_t *pgd; + + pgd = alloc_pgt_page(NULL); + if (!pgd) + return -ENOMEM; + + for (int i = 0; i < nr_pfn_mapped; i++) { + unsigned long mstart, mend; + + mstart = pfn_mapped[i].start << PAGE_SHIFT; + mend = pfn_mapped[i].end << PAGE_SHIFT; + if (kernel_ident_mapping_init(&info, pgd, mstart, mend)) { + kernel_ident_mapping_free(&info, pgd); + return -ENOMEM; + } + } + + if (kernel_ident_mapping_init(&info, pgd, + PAGE_ALIGN_DOWN(reset_vector), + PAGE_ALIGN(reset_vector + 1))) { + kernel_ident_mapping_free(&info, pgd); + return -ENOMEM; + } + + if (init_transition_pgtable(pgd)) { + kernel_ident_mapping_free(&info, pgd); + return -ENOMEM; + } + + smp_ops.play_dead = acpi_mp_play_dead; + smp_ops.stop_this_cpu = acpi_mp_stop_this_cpu; + smp_ops.cpu_die = acpi_mp_cpu_die; + + acpi_mp_reset_vector_paddr = reset_vector; + acpi_mp_pgd = __pa(pgd); + + return 0; +} + static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip) { if (!acpi_mp_wake_mailbox_paddr) { @@ -97,14 +254,37 @@ int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, struct acpi_madt_multiproc_wakeup *mp_wake; mp_wake = (struct acpi_madt_multiproc_wakeup *)header; - if (BAD_MADT_ENTRY(mp_wake, end)) + + /* + * Cannot use the standard BAD_MADT_ENTRY() to sanity check the @mp_wake + * entry. 'sizeof (struct acpi_madt_multiproc_wakeup)' can be larger + * than the actual size of the MP wakeup entry in ACPI table because the + * 'reset_vector' is only available in the V1 MP wakeup structure. + */ + if (!mp_wake) + return -EINVAL; + if (end - (unsigned long)mp_wake < ACPI_MADT_MP_WAKEUP_SIZE_V0) + return -EINVAL; + if (mp_wake->header.length < ACPI_MADT_MP_WAKEUP_SIZE_V0) return -EINVAL; acpi_table_print_madt_entry(&header->common); acpi_mp_wake_mailbox_paddr = mp_wake->mailbox_address; - acpi_mp_disable_offlining(mp_wake); + if (mp_wake->version >= ACPI_MADT_MP_WAKEUP_VERSION_V1 && + mp_wake->header.length >= ACPI_MADT_MP_WAKEUP_SIZE_V1) { + if (acpi_mp_setup_reset(mp_wake->reset_vector)) { + pr_warn("Failed to setup MADT reset vector\n"); + acpi_mp_disable_offlining(mp_wake); + } + } else { + /* + * CPU offlining requires version 1 of the ACPI MADT wakeup + * structure. + */ + acpi_mp_disable_offlining(mp_wake); + } apic_update_callback(wakeup_secondary_cpu_64, acpi_wakeup_cpu); diff --git a/include/acpi/actbl2.h b/include/acpi/actbl2.h index fa63362469aa..e27958ef8264 100644 --- a/include/acpi/actbl2.h +++ b/include/acpi/actbl2.h @@ -1197,8 +1197,20 @@ struct acpi_madt_multiproc_wakeup { u16 version; u32 reserved; /* reserved - must be zero */ u64 mailbox_address; + u64 reset_vector; }; +/* Values for Version field above */ + +enum acpi_madt_multiproc_wakeup_version { + ACPI_MADT_MP_WAKEUP_VERSION_NONE = 0, + ACPI_MADT_MP_WAKEUP_VERSION_V1 = 1, + ACPI_MADT_MP_WAKEUP_VERSION_RESERVED = 2, /* 2 and greater are reserved */ +}; + +#define ACPI_MADT_MP_WAKEUP_SIZE_V0 16 +#define ACPI_MADT_MP_WAKEUP_SIZE_V1 24 + #define ACPI_MULTIPROC_WAKEUP_MB_OS_SIZE 2032 #define ACPI_MULTIPROC_WAKEUP_MB_FIRMWARE_SIZE 2048 @@ -1211,7 +1223,8 @@ struct acpi_madt_multiproc_wakeup_mailbox { u8 reserved_firmware[ACPI_MULTIPROC_WAKEUP_MB_FIRMWARE_SIZE]; /* reserved for firmware use */ }; -#define ACPI_MP_WAKE_COMMAND_WAKEUP 1 +#define ACPI_MP_WAKE_COMMAND_WAKEUP 1 +#define ACPI_MP_WAKE_COMMAND_TEST 2 /* 17: CPU Core Interrupt Controller (ACPI 6.5) */